Adam Arellano, Field CTO at Harness: “The biggest misconception that enterprise customers have about adopting AI within business-critical software is that it is going to behave as they expect it to”

There’s nothing quite like going big. Harness says that it is “on a mission to enable 37 million software developers and billions of coding agents to deliver code quickly, reliably, efficiently, and securely”. Easy! And one key person behind this mission is Adam Arellano, the Field Chief Technology Officer (CTO) at Harness, who we interview here as part of our Conversations on AI series.

We expected bold answers to our questions merely from Adam’s bold hairstyle, and we weren’t disappointed. He starts with a stark divide: those “power users” who embrace AI will “be enhanced in ways that are just amazing… there’s a small group that’s going to turn out to be 500% more productive”. But he doesn’t believe that most people are going to see a “dynamic enhancement” of their skills. Turns out you have to put the work in.

He also points out that implementing AI into your company’s workflow isn’t necessarily going to be an instant hit. At Harness itself it took some time to prepare the groundwork – “to set up our engineering pipeline properly” – but now that this foundation has been laid it’s acceleration all the way.

Adam saves his longest answer to the question “What is the biggest misconception enterprise customers still have about adopting AI within business-critical software?” His first response: that he could write a book on the subject. Fortunately he kept his answer here rather more concise: that they expect “that it is going to behave as they expect it to”. We’ll let you enjoy his fuller answer below!

In fact, let’s not hold you back any longer. There’s much to digest here, and we think all of Adam’s feedback is worthy of your full attention. Starting with his thoughts on the impact of AI on the tech industry.

Everyone says AI is transforming software, but where do you believe the industry is still overestimating its impact and where is it underestimating it?

The industry is overestimating AI, thinking that it’s going to be a rising tide that will raise all boats. I think that that is partially true, but I don’t think that most people are going to see the dynamic enhancement of their skill set and their productivity as much as the industry thinks they will. That also leads into where it is underestimating AI.

I think that people underestimate how AI can help particular individuals who work quite well with it or are going to be enhanced in ways that are just amazing. It’s almost like there’s a soft middle that will benefit some. There is a large group that won’t benefit much at all, and then there’s also a small group that’s going to turn out to be 500% more productive because of their use of AI. Those power users that really figure out how to tune their work with AI and how to enhance themselves extremely well are going to be the ones that benefit from it most.

How has AI changed your company’s product roadmap over the past 18 months? Were there projects you accelerated, delayed or abandoned altogether?

We have accelerated our roadmap beyond what we thought we were going to be able to do. We basically took a little while to set up our engineering pipeline properly so that it could handle the machine speed of code being generated. Once we did, we got through our backlog so fast that we had to have more planning sessions than we anticipated and talk more about what our roadmap needs to be, because it’s gone so much faster.

We shipped so many new features, and every single customer request that we got, we were able to ship and integrate so much faster than we were expecting. Planning and designing have actually become more of a prominent task than we were anticipating, and that’s great.

Many SaaS vendors now describe themselves as “AI-powered”. What actually separates companies creating real customer value from those simply adding AI features?

I don’t think there’s a reliable way to separate companies creating real customer value from those simply adding AI features. The reason is that everybody calls everything AI-powered now, so it’s not like there are companies who are providing value through AI that aren’t talking about it. They’re talking about it just like everybody else.

What it comes down to is: are you actually helping the human do the job that they need to do, or are you actually helping people create in ways that they wouldn’t have been able to before? A lot of companies are providing auto-correct and auto-complete to their customers and calling it AI. I don’t think that’s super valuable, but more deep, nuanced insights are.

What is the biggest misconception enterprise customers still have about adopting AI within business-critical software?

I could write a book on this question. The biggest misconception that enterprise customers have about adopting AI within business-critical software is that it is going to behave as they expect it to. That, by itself, makes so many things problematic when you’re talking about business-critical software.

For example, if you want to build something like ADP, a payroll system, the reason why ADP is so clunky and weird is because it’s extremely auditable and predictable. It does what it’s supposed to do, and that’s it. It’s like an old Toyota Corolla that just runs. You don’t do anything fancy with it. You can’t go as fast as you could with the Ferrari, but you also can rely on it always going to do the same thing.

Unfortunately, people think that AI is going to behave the same every single time, and it actually won’t. Even if you have really strong agents, there needs to be some kind of governance and guardrails around it that make sure that you’re getting what you expect out of it. You have to audit what the AI does, just the same way you would have to audit people. Not necessarily to catch some kind of nefarious thing, but just to make sure that it’s behaving as it’s supposed to.

I think it would be best if people had a very tight idea, or a very defined idea, of what they want particular agents to do and made a very predictable agent that does that thing. They should only go for the lifelong large language models or the frontier options when it’s absolutely necessary or when the creativity doesn’t lead to damage.

How is AI changing the role of your customers? Are you replacing repetitive work, augmenting decision-making, or fundamentally changing how people do their jobs?

We are replacing repetitive work for sure. We’re also improving decision making and almost getting in front of it, so it’s not really an augmentation. Augmentation sounds like you’re doing it afterwards, when in reality we’re doing it before. We’re giving them suggestions of decisions to make so that they can pick between suggestions, as opposed to informing them in advance.

The other thing is, we’re fundamentally changing how developers actually get code to production. Instead of it being a process by which they spend most of their time with repetitive work, they’re actually getting to spend more time not even writing code but thinking about the code that they want to create. They spend more time being creative and less time doing repetitive things.

What has been the hardest challenge in bringing AI capabilities into your products?

Technology, data quality, customer trust, regulation, pricing or something else?

Our biggest challenge internally in bringing AI capabilities into our products was actually predictable pricing – so predictable resource allocation. One of our agents, every time we would ask it a question, cost us $16 when we first started.

What we realized is that we had a data problem. We actually needed to have a more concise and predictable search, or we needed to have the AI use a more predictable search that allowed it to spend fewer tokens finding answers and coming up with responses.What we spent a lot of time on was actually building out a knowledge graph, categorizing the data that we had, and creating automated ways of injecting new data. That way, when our customers ask our tool a question, it doesn’t have to go all the way back from the beginning and look at every single thing that’s there. It doesn’t have to spend $16 worth of tokens to finally come up with an answer.

We’ve got it down to pennies now, which is great, so it’s much more efficient. That wasn’t easy. That took quite a while because the amount of data that we hold about our customers, their environments, and their engineering teams is enormous. Finding efficient ways of making sure that data wasn’t duplicated and that it was structured in a way that the AI could use it efficiently was difficult.

How do you balance innovation with responsible AI? Where do you draw the line between moving quickly and ensuring customers can trust the outputs?

So we don’t draw a line between moving quickly and ensuring customers can trust the outputs. We’re actually moving quickly because we can trust the outputs. This doesn’t make sense on the surface, but if you get really good at the basics of building software, make sure that there are the right guardrails around it. You don’t need to slow down. You don’t actually need to go slower. If you build guardrails well, then you’re actually able to accelerate, and that’s what we’ve been doing.

We’ve been getting really, really good at the basics and making sure that our agents are extremely predictable and reliable so that we can speed up and go faster, knowing that the guardrails we put in place keep us and our customers safe. It’s actually the same thing that we sell to our customers.

Think of it: instead of it being the brakes on a car, it’s the spoiler on the back of an F1 car that provides downforce to the tires. By pushing the car down and creating more downforce, what you’re doing is cheating physics and allowing the car to turn corners faster than it normally would by making it heavier than it should be while moving at speed. That’s what we’ve been helping our customers do.

Beyond productivity gains, what business outcome are your customers most excited about when they invest in AI?

I love this question. Besides productivity gains, one of the things that our customers are most excited about is that it’s going to unlock their ability to do more creative thinking, more creative work, and less toil.

Getting things out of the way, like having to send tickets back and forth for security, just means that people will be able to have more time to think about what they actually want to create. They will be able to add layering or add layers and depth into their products and into the things that they make for their customers.

It’s really like freeing up an artist. Instead of having to do all the work of sculpting the entire Michelangelo’s David, you actually get to do a small model of it and then hand it off to somebody else to make it big. Instead of having to sculpt the David, the artist actually made a small model of it and then coached his apprentices through the process of carving the larger one.