Two years ago, many companies didn’t even have a definition of success when implementing AI into their operations. Instead, “they were driven by this AI FOMO that was happening in the CIO suite,” said Krieger in a interview on the Superhuman AI: Decoding the Future Podcast.
Yet, now companies are taking a closer look at their AI investments and seeking some sort of return with measurable results, because “the best products can be grounded in some kind of success metric or evaluation,” he said.
When it comes to adopting a new product, Krieger said, companies should ask two questions: “Is this a good product now, and is this a product that’s going to set up to succeed and scale?”
Otherwise, “When it gets fuzzy, it’s very hard to then evaluate, did it help?”
In terms of Anthropic’s Claude Code, which launched in May, Krieger told customers they can tell how successful the product is based on how often engineers use it.
“I often ask people just to look at the daily active metrics because those don’t lie,” he said. “People do not use tools over and over again every day if they’re not providing value.”
The jury is out on whether AI actually boosts productivity
Some companies have already touted major productivity boosts brought on by AI tools. Google in June said its AI efforts had made its engineers 10% more productive. Buy-now-pay-later company Klarna, whose CEO Sebastian Siemiatkowski said he wanted the company to be ChatGPT’s “favorite guinea pig” also claimed it was able to slow hiring and reduce its workforce to 3,000 people from 7,400 due to AI productivity gains.
An MIT study published Sunday found when AI is used by highly skilled workers, it can boost productivity by 40%. Still, some have cast doubt on using AI for working faster, especially in coding. A July study by Model Evaluation and Threat Research (METR) found AI coding tools often weren’t able to write code at the level of an experienced programmer, and participants in the study rejected suggestions just under half the time. When they did accept the changes, they had to be extra careful.
While Anthropic CEO Dario Amodei said back in March that in three to six months AI would be writing 90% of code, “and then, in 12 months, we may be in a world where AI is writing essentially all of the code,” the July METR study found using AI coding tools made them take 19% longer on their tasks.
“While I like to believe that my productivity didn’t suffer while using AI for my tasks, it’s not unlikely that it might not have helped me as much as I anticipated or maybe even hampered my efforts,” said one participant in the study.
