Artificial intelligence is quickly becoming part of everyday products. But for those who have spent their careers building traditional software, the shift to building AI-powered features can be uncomfortable. The rules of the game are different. Here are some key things to keep in mind when making the transition.
- From Rules to Data
In traditional software, engineers explicitly define the rules—if X, then do Y. With AI, behavior is learned from data rather than hard-coded. This means builders must think about data quality, coverage, and governance as much as they think about code.From Deterministic to Probabilistic
Software is usually deterministic, however AI is inherently non-deterministic (or probabilistic). What I mean by this is that the same query may yield different responses. This uncertainty requires new design considerations such as confidence scoring and user feedback loops.From Testing to Evaluation
You can unit test a software function, but you can’t write a unit test that proves an AI model is always right. Instead, builders rely on evaluation metrics, commonly called “evals.” Evals are scoring mechanisms that help you assess whether your AI system is working, where it’s falling short, and what needs improvement. Think of them as the equivalent of tests in traditional software. But unlike typical unit or integration tests, where inputs and outputs are fixed and correctness is binary, AI evals deal with ambiguity.The Challenge of Measuring the Unmeasurable
Traditional software testing relies on deterministic outcomes. Either the payment processing function correctly calculates the total or it doesn’t. On the other hand, AI evals venture into inherently subjective territory. How do you score the quality of a generated email? The helpfulness of a chatbot response? This ambiguity is both the biggest challenge, and the most important skill to develop, for an AI builder.
The insight here is that perfect evaluation may be impossible, but useful evaluation is essential. Builders should not try to capture every nuance of quality. Instead, the goal should be to look for reliable signals that correlate with real-world success. Sometimes this means accepting proxies—measuring response length as a crude indicator of thoroughness, or checking for specific keywords as a signal of topic relevance.The Human-in-the-Loop Reality
One of the most practical challenges AI builders face is the cost and complexity of human evaluation. While automated metrics are fast and scalable, they often miss nuances that only humans can catch. Successful teams develop hybrid approaches, such as automated evals for rapid iteration, human evaluation for high-stakes decisions and edge cases, and semi-automated systems where humans oversee AI-generated scores.Agency and Control for AI features
Agency, in the context of building AI features, is the system’s ability to take actions, make decisions, or carry out tasks on behalf of the user (which relates to ‘AI agents’). Think of agents paying your bill, writing code, or handling customer support. Unlike traditional tools, AI systems are built to act with varying levels of autonomy. But here’s the part that AI builders often overlook: Every time an AI agent is given more agency, the human user loses some control. So there’s always an agency-control tradeoff to consider. That tradeoff could have significant consequences. On the one hand, if your AI agent suggests a response, the human in the loop can override it. On the other, if it sends the response autonomously, it better be correct. The mistake most inexperienced AI builders make is jumping to full agency before they’ve sufficiently tested what happens when the system gets it wrong. If you haven’t tested how the system behaves under high control, you’re not ready to give it high agency. And if you hand over too much agency without the system earning it first, you may lose visibility into the system, and the trust of your customers.The Reality of AI Hallucinations
One of the most distinct challenges with AI systems, particularly ones using large language models, is their tendency to hallucinate. A hallucination occurs when an AI generates an output that sounds confident but is factually incorrect or fabricated.
Why does Hallucination Happen?
Hallucinations aren’t bugs in the traditional sense. They emerge from how models are trained, predicting the next most likely word based on patterns in data. When the model lacks sufficient grounding or context, the model may simply “fill in the blanks” with plausible but false information.
The Risks
For end users, hallucinations can range from quirky to factually incorrect or costly errors. In domains like healthcare or finance, the consequences could be dire. This makes hallucination management a core design responsibility.
Strategies for Builders
- Grounding with retrieval: Builders must ensure that outputs are anchored in a verified knowledge base so that they are tied to factual data.Transparency cues: AI responses should expose confidence scores or highlighting sources to users.Continuous monitoring: Track hallucination rates over time to detect drift and fine-tune models when necessary.
Hallucinations should be a reminder that AI is not a source of truth—it’s a reasoning tool that needs oversight. This is important to deliver trustworthy systems
8. From Ship-and-Forget to Continuous Learning
Software releases are often final until the next version. AI systems, by
contrast, require ongoing monitoring and retraining. Models can drift as
user behavior, context, or data changes.
9. From Predictable to Trustworthy
In traditional design, users expect precise, predictable outcomes. With AI,
the experience is about trust: communicating uncertainty, offering
transparency, and giving users control. Builders need to create guardrails
and recovery paths for when AI gets it wrong.
10. From Functionality to Responsibility
AI introduces ethical and compliance responsibilities related to bias,
fairness, copyright, and security. These aren’t edge considerations.
Builders should make sure that they’re core to product success and must
be designed into the system from day one.
The Builder’s Mindset Shift
For builders, the shift is less about learning new tools and more about embracing a new mindset. Traditional software is engineered. AI systems need to be engineered, trained but also governed. Success depends not only on technical skill, but also on thoughtful design, monitoring, and responsible use of data.
As AI becomes a necessity for all of us, builders who adapt their approach will be best positioned to deliver the next generation of impactful products.
Shilpa Shastri is a Principal Product Manager at Apptio (an IBM company), where she owns data strategy and GenAI features. Her work bridges product strategy, cloud economics, and AI innovation—helping enterprises adopt AI responsibly and at scale.
