The Embodied AI

Recent advances in AI have been driven by hardware capable of meeting the memory and computational demands of large models trained on vast amounts of data. Since the introduction of the Transformer in 2017, the core technology hasn’t changed much, but scaling model size and data has enabled significant breakthroughs, like human-like language skills. This underscores the idea that increasing computation and memory could eventually lead to even smarter algorithms, possibly up to AGI.

In 1965, Gordon Moore, Intel co-founder, noted that the number of transistors on a microchip doubles approximately every two years, while the cost of computing is halved. This exponential trend, nowadays known ad Moore’s law, has held true for several decades, leading to exponential growth in computing power, making devices faster, smaller, and more energy-efficient. Moore’s Law is not a physical law but a guiding principle for the semiconductor industry. It has driven rapid technological advancements in electronics, from personal computers to smartphones.

The effect of increased and cheaper computations in AI

Moore’s Law has had a profound impact on the development and growth of artificial intelligence, driving many of the current trends we see today. The exponential increase in computational power, fueled by the ability to shrink transistors and fit more of them onto microchips, has enabled AI models to grow in size and complexity. This has allowed for the development of larger neural networks, like GPT-4 and other transformer-based architectures, which require immense computational resources for both training and inference. With more transistors packed into specialized hardware such as GPUs, TPUs, and NPUs, AI models can now process vast amounts of data in parallel, significantly speeding up training times and enabling real-time decision-making in applications like autonomous vehicles and smart devices.

In addition, the falling cost of computation, driven by advances in chip technology, has made AI more accessible and affordable. Cloud platforms now offer powerful AI services on demand, allowing developers, startups, and researchers to build advanced AI systems without needing to invest in expensive hardware. This democratization of AI is accelerating innovation, pushing AI into new areas such as edge computing, where devices like smartphones and IoT sensors can run AI models locally, thanks to energy-efficient chips. The ever-growing need to process and store massive datasets has also benefited from Moore’s Law, as memory and storage chips have become more powerful and less expensive, making it easier for AI systems to handle data-intensive tasks.

Current AI models display impressive abilities in speech and text comprehension, reasoning, and image and video generation, yet remain distant from achieving General Artificial Intelligence (AGI). The clear link between increased computational power and AI performance suggests that future advances in hardware could unlock even more remarkable capabilities. Scaling up model size and computational complexity has already driven significant progress, and as computing power continues to grow, AI could edge closer to general intelligence, though obstacles still persist.

Conclusions

However, while Moore’s Law has fueled these advances, we are now approaching physical limits in chip manufacturing, as transistors near the size of individual atoms. This has led to concerns about the sustainability of this exponential growth in computational power. As AI models grow larger and more energy-hungry, energy efficiency has become a critical issue, with researchers exploring greener approaches to AI development. Despite these challenges, the effects of Moore’s Law have been transformative, and as the industry pushes the boundaries of traditional computing, new technologies like quantum computing and neuromorphic chips may shape the next era of AI. For now, the advancements driven by Moore’s Law continue to enable breakthroughs in AI, from faster training times to the widespread adoption of AI-powered technologies in everyday life.