Simon Ernst-Sunne, CEO at ScraperAPI: “The fundamental change is that AI agents consume web data directly and act on it, often without a human checking it first”

The growing use of AI is changing what businesses expect from the data they collect. Public web information, once the preserve of developers building individual scraping projects, is increasingly feeding AI models, automated workflows and decisions that happen with little human intervention. That puts a different emphasis on the quality, freshness and reliability of the information being collected.

For ScraperAPI, the challenge is no longer simply getting data off websites. It is making sure that information is accurate, structured and ready for use. A stale product price, an outdated listing or a failed page request might be obvious to a human analyst, but an AI agent can absorb the same information and act on it without recognising that anything is wrong.

The acquisition of Traject Data has helped ScraperAPI broaden its offering, bringing together the infrastructure for collecting public web data with the expertise to deliver finished, structured datasets. The aim is to support customers across the process, whether they want to build and manage their own data pipelines or receive information ready for their business applications.

The company is also turning its attention to AI-generated search results, with monitoring capabilities covering platforms including ChatGPT, Perplexity, Gemini and Google AI Overviews. As these services become another way for consumers and businesses to discover information, organisations are beginning to ask how they are represented in the answers they produce.

In this interview, ScraperAPI’s CEO Simon Ernst-Sunne discusses the changing demands of web data collection, the implications of feeding live information into autonomous AI agents, and why reliable public web data could become a core part of business infrastructure.

You describe a shift from web scraping towards “decision-ready” data. What has changed in the way businesses are using public web data that made this broader offering necessary, and why is simply being able to access the data no longer enough?

Public web data has moved from developer side projects into the core of how businesses run. It now feeds AI systems, automated decisions, and day-to-day operations, and demand reaches well beyond technical teams. 

That shift raises the cost of getting it wrong. An inaccurate price or outdated listing can mislead a decision-making analyst. When the same error feeds into an AI system or automated workflow, it can be repeated across thousands of decisions before anyone notices. So, the data also has to be accurate, structured, and current.

The acquisition of Traject Data last year reflects this change in demand. ScraperAPI has traditionally served developers who build their own pipelines, while Traject Data serves organisations that need finished, reliable data without managing the infrastructure themselves. Bringing the two together makes us end-to-end web data infrastructure, supporting the full spectrum from teams that want to build their own workflows to organisations that need data delivered ready to use.

The release points to data preparation consuming 50–70% of the time and effort in a data project. How much of that burden can ScraperAPI realistically remove, and what are the hardest parts of turning constantly changing public web data into something an AI system can reliably use?

We take away the collection side of data preparation: getting past bot protection, rendering pages, parsing them, and normalising the output into consistent, structured formats delivered straight into a warehouse or an AI agent. That is the part of preparation most teams working with public web data never chose to own. The work specific to each business stays with the customer, such as joining web data with internal data and deciding what a good signal looks like for their model.

The harder problem is that the web keeps changing underneath that data. Anti-bot defences escalate, more pages depend on JavaScript, and a layout change can break a parser overnight. We maintain the schemas on our side, so when a site changes, the customer’s pipeline keeps receiving the same fields. Keeping that stable takes serious engineering, and close to half of our engineering and customer success capacity goes into keeping success rates high.

There is also a less discussed problem specific to AI. An AI model cannot tell a block page, an empty result, or a week-old price from real, current data, but whatever one it receives, it reasons over and then uses for its outputs with the same confidence. Data freshness and failure handling therefore matter as much as access. When these considerations are applied to tools, data collection requests fetch live, uncached pages, outputs arrive as clean markdown or structured JSON, which is a data format the AI model can use directly, and when a page cannot be reached, the system returns a clear error rather than an empty result.

ScraperAPI and Traject Data bring together web data collection with finished-data capabilities. What does the acquisition change for customers in practical terms, and why was bringing those capabilities together preferable to continuing to build the two sides independently?

In practical terms, customers no longer have to choose between building their own scraping setup and finding a separate structured data provider. Developers and data teams can keep using flexible infrastructure to build their own workflows, while organisations that want a more managed approach can get ready-to-use data from the same provider, at a price point that fits each.

It also makes it easier to adapt as needs change. Many teams start by building their own scrapers, then reach a point where they want the data delivered and maintained for them. With both capabilities under one roof, they can make that transition without switching vendors, contracts or integrations.

Bringing the two businesses together made more sense than building each capability independently because each already has production-ready expertise that would take the other time to develop. Traject Data brings years of experience turning public web data into finished solutions at scale, while ScraperAPI brings deep expertise in collecting data from across the public web. Together, we can serve more sources and more complex use cases than either could have managed alone. Traject Data’s products and services continue to support existing customers while its capabilities are progressively integrated into ScraperAPI, with the long-term aim of providing a single access point for all of it.

AI agents increasingly need access to live information rather than data frozen at the point a model was trained. What new problems does that create around data quality, reliability and accuracy, particularly when an agent is using web data to make decisions or take actions autonomously?

The fundamental change is that AI agents consume web data directly and act on it, often without a human checking it first. A person may notice an incorrect price or an outdated listing. An agent will act on it and carry that error into decisions and outputs downstream.

That raises the bar significantly compared to traditional scraping use cases. Accuracy and freshness matter far more when data is feeding autonomous decisions rather than informing a human analyst who can apply their judgement. The tolerance for error is much lower, and the consequences of failure are less visible until they compound and lead to greater damage.

The reliability challenge is also compounded by the nature of the public web itself. As mentioned, web pages restructure, anti-bot defences evolve, and data that was accurate an hour ago might’ve already changed. A static snapshot of the web, however comprehensive, degrades quickly when agents are making decisions in real time. That is why live, uncached access and robust failure handling are not optional features for AI use cases – they are the baseline.

You’re also launching AI search monitoring across platforms including ChatGPT, Perplexity, Gemini and Google AI Overviews. What are businesses currently unable to see about how they are represented in AI-generated answers, and what do you expect them to do differently once they have that visibility?

Most businesses currently have no reliable way of knowing how they appear in AI-generated answers. Unlike traditional search engines, which rank URLs on public results pages, large language models dynamically synthesise conversational responses. Those responses can vary from prompt to prompt, meaning standard rank-tracking tools and web analytics dashboards cannot reliably measure a brand’s presence or citation frequency across AI platforms.

Building a reliable picture requires measurement at scale: running a defined set of prompts repeatedly across ChatGPT, Perplexity, Gemini and Google AI Overviews, then capturing the results as structured data so teams can track trends in mentions, citations and competitor presence over time. Standards for measuring this are still forming, making a consistent, repeatable method all the more important. This is what our AI search monitoring capabilities are designed to support. They run on the same infrastructure that powers our data collection, so teams get that consistency without building it themselves.

With that visibility, businesses can start treating AI answers as part of their existing search and market intelligence work. They can see where competitors appear and where they do not, identify and correct inaccurate descriptions at the source, and focus on the pages and third-party sources that AI assistants actually cite, rather than optimising for signals that no longer determine visibility.

You describe the longer-term ambition as becoming the “data intelligence layer businesses run on”. What would that actually look like in practice, and how far do you think the industry is from treating live public web data as core business infrastructure rather than something collected for individual projects? 

In practice, it means live public web data arriving like other critical business data: continuously, from multiple sources, on a defined schedule and to a defined quality standard, straight into the warehouses, BI tools and AI systems a business already runs. Teams can then spend their time acting on the data rather than collecting and cleaning it. Cloud computing made a similar transition, from a resource provisioned project by project to infrastructure companies rely on without having to think about it. That is the direction web data is heading.

The industry is still early in that transition. Most organisations still collect web data project by project, while preparation consumes 50-70% of a typical data project’s time and effort, according to IBM. AI is accelerating the shift because AI systems increasingly need current external data at scale and on an ongoing basis. That creates pressure to move from ad hoc collection towards something closer to core infrastructure. We already process billions of requests every month for more than 40,000 brands and developers across over 150 countries, giving us a clear view of where that demand is coming from and how it is growing.

For web data to become core infrastructure, it also needs to meet the standards businesses expect from other critical systems. Predictable pricing, reliable delivery and data sourced in a way customers can defend to their own compliance teams. Those are the conditions under which businesses stop treating web data as a project resource and start treating it as something they run on.

More great interviews

• Michael Evans, VP & GM, ANZ & India, Denodo: “Having the right architecture doesn’t automatically mean an agent is ready to operate autonomously”
• Walid Abu-Hadba, CEO of Precisely: “Data quality is the hard problem, and it is hard because it is not a technology challenge”
• Fiona Dear, Co-Director at the Restart Project: “In the next 10 to 15 years, we could lose all repair services that aren’t phones or laptops”