The hype surrounding “AI agents” – the promise of generative AI robots automating tasks and reshaping work – has outpaced reality. While 2025 was touted as the breakthrough year, the true outcome has been an indefinite postponement of this transformational moment. A growing body of research suggests that fundamental limitations within current AI architectures may prevent truly reliable, fully automated systems from ever emerging.
Mathematical Barriers to Artificial Intelligence
A recent, under-reported study, “Hallucination Stations: On Some Basic Limitations of Transformer-Based Language Models,” argues that Large Language Models (LLMs) have inherent mathematical constraints that prevent them from reliably performing complex tasks. According to Vishal Sikka, a former SAP CTO and AI entrepreneur, these models “cannot be reliable,” even with advanced reasoning capabilities. This means critical applications like nuclear power plant control remain firmly outside the reach of current AI technology.
Coding as a False Dawn?
Despite these limitations, the AI industry emphasizes progress in areas like coding, where AI agents have shown some success. Google’s Demis Hassabis at Davos reported breakthroughs in reducing “hallucinations” – AI-generated errors – and startups like Harmonic are pushing the agent narrative. Harmonic, co-founded by Robinhood CEO Vlad Tenev, claims its AI coding tool, Aristotle, uses formal mathematical verification to ensure reliability. However, this approach is currently limited to verifiable domains like code, excluding subjective or creative tasks like writing essays.
Hallucinations: An Inherent Flaw?
The reality is that AI hallucinations remain pervasive. OpenAI’s own research demonstrates that even state-of-the-art models like ChatGPT consistently fabricate information, misreporting facts with 100% accuracy impossible. This unreliability discourages widespread adoption in corporate environments, where errors can disrupt workflows and negate potential value.
A Pragmatic Path Forward
Industry leaders acknowledge the persistence of hallucinations but believe they can be mitigated through guardrails and filtering mechanisms. Sikka suggests building components around LLMs to overcome their limitations, while Harmonic’s Achim argues that hallucinations may even be necessary for pushing AI beyond human intelligence. The industry consensus is that the gap between guardrails and hallucinations will narrow over time, leading to incremental improvements rather than a sudden revolution.
The Bigger Picture: Automation Inevitable, Reliability a Constant Trade-Off
Despite the mathematical and practical hurdles, the momentum behind AI agents is undeniable. The financial incentives and competitive pressures ensure continued investment and development. As AI systems evolve, they will inevitably perform more cognitive tasks, even if reliability remains imperfect. The ultimate outcome isn’t whether AI agents will exist, but how much risk society is willing to tolerate for the sake of speed and cost savings.
Ultimately, the question isn’t about “good” or “bad” AI, but what the technology is doing to reshape human activity, as computer pioneer Alan Kay suggests. We may be entering an era of massive cognitive automation, the consequences of which will likely be impossible to verify mathematically.
