In a recent pre-print paper submitted to the open-access repository ArXiv, Google researchers have highlighted limitations in the ability of transformers, the technology behind large language models (LLMs) like ChatGPT, to generalize tasks. The researchers found that transformers perform well on tasks related to their training data but struggle when presented with out-of-domain tasks or extrapolation tasks. This poses a challenge for the development of artificial general intelligence (AGI), as AGI aims to achieve AI that can perform a wide range of tasks in a manner similar to humans. The researchers' findings suggest that AGI may not be as imminent as some have speculated.
Pedro Domingos, a professor of computer science and engineering, echoed this sentiment, cautioning against overhyping the potential of AGI at this stage. He emphasized that neural networks, like transformers, are opaque and have been trained on vast amounts of data, leading to misconceptions about their capabilities. Domingos believes that many people have mistakenly attributed miraculous abilities to AI systems and that transformers are not the path to human-level intelligence.
While the research conducted by Google researchers focused on the limitations of transformers, it is worth noting that more advanced forms of AI may be better at generalizing. Sharon Zhou, CEO of Lamini AI, expressed her optimism about the future of AI, stating that her company focuses on training models to learn new things rather than solely querying them. Zhou believes that transformers can still be useful and can be steered and aligned to overcome their limitations.
Overall, the research serves as a reality check for those who anticipate AGI in the near future. While transformers have proven effective in specific tasks, their ability to generalize remains a challenge. It is essential to recognize the limitations of AI systems and not overstate their capabilities. Continued research and development will be necessary to overcome these obstacles and achieve the ultimate goal of AGI.