Generative AI has unquestionably increased raw coding velocity as part of the software development lifecycle (SDLC). Tools can churn out syntactically correct boilerplate, REST endpoints, and data transformation logic in a matter of seconds. This increased velocity puts undefined trust before reliability. We now face a more fundamental problem. It has nothing to do with the speed or syntactic correctness of the code produced by AI; it is about semantic correctness. The SDLC bottleneck now resides in non-functional requirements such as latency, resilience, fault tolerance, and security.3
As producing code approaches near-zero marginal cost, the economic and operational value shifts from a consideration of how much engineering time it takes to develop a new application feature to the notion of trust: confidence that a system will meet its SLO (service-level objective) in chaotic, real-world operation. The scarce resource in the SDLC is no longer engineering hours developing features, but rather the trust that a particular expert must stake to underwrite the reliability of a system. The site reliability engineer (SRE) is positioned at the heart of the AI revolution in the SDLC, not as a utility user of AI-generated outputs, but rather as a designer-agnostic validator of AI-generated reliability.2
Testing
AI-generated tests are likely to migrate well beyond unit tests. They can and will incorporate sophisticated processes, such as generative and structure-aware fuzzing.6 Given protocol definitions (such as Protobufs, or OpenAPI specs), AI can generate valid but unexpected inputs that would surface some deep serialization bugs undetectable with standard testing practices. AI could also orchestrate more complex forms of chaos engineering1 by simulating an entire range of system outages and operational failures that might not result in a complete destructiveness, but make it unreliable like mismatched server clocks, or nodes in distributed systems sending wrong data.
But we are not looking for test coverage; we are looking to mitigate risk. For an SRE, the basic job becomes how to guide the AI system. This means establishing a risk model, perhaps even assisted by AI, that provides weights for testing priority based on criterion around code churn, cyclomatic complexity, and centrality in the service dependency graph. A test putting the authentication service through its paces with a P99 latency SLO of P99(latency) <150ms is infinitely more valuable than a thousand tests on an asynchronous batch job. The AI provides the engine for simulation at massive scale, the SRE provides the objective function and determines where to spend expensive computational resources on validating the finest-grained, most critical tests in the system architecture.
Figure 1 represents an AI-driven testing pipeline, where an LLM generates an exhaustive set of tests that are then filtered and prioritized by a risk-weighting engine. This engine’s model can be trained on no shortage of data from production incident data, dependency graphs, and SLO definitions to focus testing on the code paths that matter.
Migration
While testing addresses reliability at the code and system level, reliability risks also emerge during platform shifts. Migration scenarios such as Redis-to-Valkey4,7 reveal how trust must be revalidated even when APIs remain stable.
The case example of a Redis-to-Valkey migration is a canonical example of reliability risk lurking below the abstraction of API compatibility. The command sets are basically identical, but differences in implementation can have first-order effects on production systems. For example, Valkey’s addition of slot-based multithreading7 fundamentally changes key performance aspects from Redis’ single-threaded-per-shard model, potentially introducing new race conditions or simply changing latency distributions for complex scripts.
AI tooling can provide an intelligent way of de-risking this process. An LLM can perform static analysis to identify not only deprecated commands, but also idioms and patterns that may be sensitive to a change in the threading model (such as, what are the chances that calling KEYS extensively in scripts will not be a problem?). In an even more important sense, AI can perform workload synthesis. If you ingest and analyze the output of the MONITOR command for a Redis instance that was being used in production, you could create a high-fidelity load-testing suite to simulate the production command mix, key distribution, and concurrency. This allows engineers to discover when the “semantic drift” occurs; that is, the API contracts are being satisfied, but performance characteristics are diverging—which can cause problems downstream of the Redis instance in the ecosystem before users are affected. Again, the SRE will take the histograms and HDR reports and validate that the P99.9 latencies and memory allocation patterns remain within the error budget of the system under a synthetic load.
Figure 2 shows a comparative dashboard and memory fragmentation metrics for Redis vs. Valkey, both running an AI-synthesized workload based on production traffic. It highlights not just API compatibility, but the latency and memory fragmentation differences that can redefine downstream system reliability.
Observability
Even with careful testing, systems can change in unexpected ways once they are in use. This is why observability, being able to get useful information from complex systems, is important for improving AI reliability.
In a complex microservices environment, observability is not a data problem; it is a signal extraction problem. AI models are particularly good at identifying correlations across high cardinality metrics, distributed traces,5 and structured logs. If an AI model notices a spike in 5xx errors on the API gateway and correlates it with increased cache misses, as well as a sequence of pod OOM Killed events in the caching tier, it might be possible to counsel an engineer to look further up the service dependency chain. The depicted correlation is merely the beginning and not the end. More than just time and effort are needed for an SRE to provide adequate reasoning chains. In this case, the SRE investigates and finds the recent deployment changed the efficient, paginated, HSCAN operation and replaced it with a full keyspace HGETALL inside a function. This change had the side effect of memory utilization on the cache server, which would no longer have limits, eventually leading to ungraceful termination of the cache container with OOM Killed events. This, in turn, leads to the API’s cache usage creating API errors.
Obviously, the AI can identify the symptoms of the error, architecturally and operationally, across the distributed system; however, the underlying pathology can best be diagnosed by an engineer. Moving forward, although it is the engineer’s responsibility to provide a legitimate contextual rationale, as feedback is ingested within the models over time.
Figure 3 can be used to present an incident workflow. The “Before” state shows an incident and a storm of disparate alerts that all point to the same OOM Killed event(s). The “After” state shows the identical OOM Killed disorders occurring in the same service, however, with the AI correlation engine grouping these alerts together and providing the SRE with a hypothesis as to a root cause. The engineer was able to map this hypothesis onto a particular code-level finding (HGETALL misused). The processed finding is part of the new data that the engineer provides to the model.
Curating the Model
The use of AI systems does not mean that the discipline of reliability engineering has become automated; it has become refactored. The traditional aspects of reliability, creating, and analyzing become curation and validation, or more so, instead of the engineer just creating a system, they also must build and refine the AI models that help operate that service in production. The engineer becomes the curator of a model and is responsible for applying the architectural context, business logic, and first principles of distributed systems to turn the generic logic models made by AI into a legitimate, high-fidelity embodiment of their production environment.
In a world where the principles of resilience engineering are more pertinent than ever, the engineer must remember that AI’s capacity to provide scaled, new capacity for leverage does not abrogate the responsibility of the human engineer to apply their deep competencies, so the systems developed are not just fast to develop, but also fundamentally, verifiably, and sustainably trusted at their core.
References
1. Basiri, A., Behnam, N., de Rooij, R., Hochstein, L., Kosewski, L., Reynolds, J., and Rosenthal, C. (2016). Chaos engineering. IEEE Software, 33(3), 35-41.
2. Beyer, B., Jones, C., Petoff, J., and Murphy, N.R. (2016). Site Reliability Engineering: How Google Runs Production Systems. O’Reilly Media.
3. Forsgren, N., Humble, J., and Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps. IT Revolution Press.
4. Redis Documentation. (2024). Redis Commands and Architecture, https://redis.io/docs/
5. Sigelman, B. H., Barroso, L. A., Burrows, M., Stephenson, P. et al. (2010). Dapper, a large-scale distributed systems tracing infrastructure. Technical Report, Google.
6. Sutton, M., Greene, A., and Amini, P. (2007). Fuzzing: Brute Force Vulnerability Discovery. Addison-Wesley.
7. Valkey Project. (2024). Valkey: An Open Source, High-Performance Data Store, https://valkey.io/
Rahul Chandel is an engineering leader with 15+ years of experience building high-performance systems across fintech, blockchain, and cloud platforms at companies like Coinbase, Twilio, and Citrix. He specializes in scalable, resilient architectures and has presented at AWS re:Invent. See https://www.linkedin.com/in/chandelrahul/.
