The Frictions That Make AI Forecasting Hard
To better forecast the impact of AI on any industry, you must understand its overall impact on an integrated ecosystem. It may streamline a piece of the workflow but how will this play out as the other steps digest this acceleration? Where will the bottlenecks be created and can they be cleared? If not, the overall throughput will not increase.
AI forecasting is about predicting what AI systems will be capable of, as well as how those capabilities will impact industries, institutions, and everyday life. That second part is much harder.
“It’s not something you can get from first principles,” Abi Olvera told me when we spoke last week about AI forecasts. I first met Abi after I reached out because of one of her recent articles in which she talks about AI forecasting from the lens of biosecurity.
Abi spent more than seven years as a State Department diplomat working across crisis preparedness, national security, China, international coordination, and emerging technology risks. She served in Dakar and Cairo, where she focused on Egypt’s $12 billion IMF program. She also worked on cybersecurity and law enforcement issues on the China Desk. Across her career, she’s had to think about poverty, national security, crisis preparedness, energy, biosecurity, cybersecurity, and the institutions that have to respond when technology changes faster than policy can. Today, Abi writes the Positive Sum Substack and is launching an organization focused on state capacity and bottlenecks in the economy. She is also a Special Advisor at Golden Gate Institute, and an affiliate at the Institute for AI Policy and Strategy.
In our conversation, Abi and I discussed where she sees the most risk around forecasting gaps, the drivers of those gaps, and what can be done to help fix them.
Forecasting Gaps
Forecasting gaps happen when people assume having access to information is the same as having the ability to act on it. AI makes a domain easier to understand, but doesn’t necessarily make the work easier to execute.
Take radiology for example. In 2016, Geoffrey Hinton told a Toronto machine learning conference that “people should stop training radiologists,” because it was “completely obvious” deep learning would outperform them within five years. And while AI has become very valuable in radiology (the FDA has cleared hundreds of AI-enabled medical imaging tools), the labor-market forecast was very wrong. Radiology did not disappear; in fact demand grew. At the Mayo Clinic, one of the more aggressive AI adopters, radiology headcount reportedly grew 55% since 2016 while the department built a 40-person AI team and used more than 250 AI models.
“There’s been amazing progress, but these AI tools for the most part look for one thing,” said Dr. Charles E. Kahn Jr., a professor of radiology at the University of Pennsylvania’s Perelman School of Medicine and editor of the journal Radiology: Artificial Intelligence. As The New York Times wrote, “Radiologists do far more than study images. They advise other doctors and surgeons, talk to patients, write reports and analyze medical records. After identifying a suspect cluster of tissue in an organ, they interpret what it might mean for an individual patient with a particular medical history, tapping years of experience.”
AI improved the task of interpreting medical images, but radiologists still integrate clinical context, communicate with physicians, guide treatment decisions, perform procedures, and carry accountability. The capability arrived, but the forecast failed because interpreting images was only the tip of the iceberg of the “radiology job” - and as AI became more proficient in this task at the top of the workflow, more radiologists were needed to keep up with the increase in related work.
Abi believes forecasts focus too much on what technology makes possible and not enough on the realities of domains and social incentives governing whether, where, and why people will actually use it. We need to better understand both sides: where does AI change the workflow, and how does the world respond.
Biases and Blind Spots
Part of what makes AI forecasting so hard is that the people closest to the technology are often farthest from the domains where the forecasts will play out. As Abi put it, “Because AI forecasting has tended to be from the more technical, Bay Area-based community, it systematically falls down more in any domain that has to do with social behavior, adoption, or any large real-world or physical world component.” In San Francisco, for example, people tend to start from capability. Software engineers often use AI for tasks where the model can complete much of the workflow end to end. In other domains, however, the workflow extends beyond text into physical systems or interactions with outside actors, which makes AI’s impact both harder to see and harder to forecast.
“People in DC are less likely to be using Claude Cowork or Codex,” she said. “If people use it [in DC], a lot of times they might be using the chatbot version, which is great, but that doesn’t really unleash the parts of AI that are crazy surprising.”
On the other hand, people in San Francisco often underestimate frictions around AI adoption because they are focused on the vision more so than the nitty gritty work that translates vision to reality in the real world.
In truth, neither perspective is complete. San Francisco may be closer to the frontier of technological capabilities, but further from the systems where those capabilities are deployed. DC (or other centers of non-technical power), meanwhile, is closer to those systems where the rubber meets the road, but further from the true possibility of AI technology. Unfortunately, public discourse rarely closes this gap. The people most motivated to speak usually have a stake in the outcome (e.g. a company trying to sell a product or an institution trying to defend its role). Advocacy creates the resources and incentives to shape the conversation, which is valuable, but can also mean the loudest arguments are not always the most representative view of practitioners. And because the content people consume is usually the content that already fits their worldview, public discourse increasingly serves as echo chambers rather than bridges.
Cross-Framework Research
This is why AI forecasting needs more contact with practitioners. They are the ones with the most grounded view of where AI can change the work vs where it can’t, and which parts of the job outsiders are likely to overlook from their AI ivory towers.
Abi believes we need more research that brings together both sides – the technical and the practical. She calls this cross-framework research.
One of her favorite examples of this kind of research is a study that tested whether AI actually helped novices perform molecular biology tasks in a lab. The study compared people with internet access to people with internet plus frontier AI models, then measured whether they could complete hands-on wet-lab tasks over eight weeks. The result was mixed: AI seemed to help on some intermediate steps, but it didn’t significantly increase the number of people who could complete the full lab workflow from start to finish. The study tried to evaluate the delta between simply having better instructions versus being able to execute a difficult physical workflow.
For many industries, a key question with AI is whether it can replace enough tacit knowledge, troubleshooting ability, and hands-on competence to change what a novice can do.
What the World Lets AI Do
This has implications for policy too. If governments don’t know yet exactly how a technology will evolve, rushing to write rigid and detailed rules isn’t helpful. Abi believes more focus needs to go toward building the capacity to respond quickly as evidence gets clearer.
Abi has argued that governments should focus on building ecosystem capacity: better reporting, better evaluations, and more technical expertise inside institutions. This doesn’t require policymakers to perfectly forecast every future use case, but rather gives them the ability to notice what’s happening, test assumptions, and act effectively as they get better information and clearer directions emerge.
AI will transform science, security, industry, and government. But the path from capability to transformation runs through the parts of the world that are hardest to model from the outside. As AI accelerates parts of a workflow, more attention needs to shift to whatever is still slow, physical, tacit, or institutionally constrained. As Abi put it: “Bottlenecks become more important when everything else gets automated.”
The question, then, is not just what AI can do. It’s whether that capability changes the bottleneck. That’s the difference between what AI can technically do and what it can actually do.
Author’s note: An LLM was used for light copy editing only (spelling, grammar, and clarity). Content, meaning, tone, and structure remain unchanged.