With all the hype and progress with different models, namely ChatGPT, AlphaFold, and the coming GPT-4, we’re getting closer to AGI. I genuinely cannot believe the amount of papers that come out everyday- this tweet is a good depiction of what I mean. Over time, predictions on AGI's timeline have been decreasing.
To get to AGI, many people believe scaling models (in terms of compute, dataset size, and parameters) will help get us there. However, there are still 3 fundamental gaps we need to fill with models: common sense, factuality, and causality. In my Substack post of ‘propagating forward’, I go deeper into these gaps.
I've been reading some interesting papers about networks that can help push the goal post forward in these areas, like:
- MIND'S EYE, which shows how you can represent a physical reasoning question → simulate the possible outcomes through MuJuCo → use the simulation results as part of the input in the language model to perform reasoning
- GFlowNets, which is an architecture Yoshua Bengio is really excited about. It basically represents the way humans reason sequentially, adding a new piece of relevant information at each step.