Achieving 90% reliability in AI systems is often celebrated as a milestone, but industry experts argue this is just the beginning of the journey toward dependable AI. The concept of the “March of Nines” highlights the escalating effort required to improve AI reliability from “usually works” to enterprise-grade software performance.
The March of Nines Explained
The idea behind the March of Nines is that improving reliability by each ‘nine’—from 90% to 99%, then 99.9%, and beyond—requires exponentially more engineering work. In AI workflows, where multiple steps operate sequentially, each with a probability of success, the overall reliability compounds multiplicatively.
For example, a 10-step AI workflow with 90% per-step success yields just a 34.9% chance of end-to-end success. Increasing per-step reliability to 99.9% boosts end-to-end success to 99%, illustrating how critical very high per-step success rates are for dependable enterprise applications.
Measurable Reliability Through Service Level Objectives (SLOs)
Turning reliability into concrete, measurable objectives is essential for improving AI system performance. Teams do this by defining Service Level Indicators (SLIs) related to workflow completion, tool-call success rates, schema validity, policy compliance, latency, and fallback rates.
Setting targets across different impact tiers and managing error budgets help maintain controlled experimentation and minimize unexpected failures, making reliability quantifiable rather than abstract.
Key Engineering Approaches to Add Reliability Nines
Achieving higher reliability involves nine main strategies, including constraining AI autonomy to bounded workflows, enforcing strict contracts at interfaces, layering validation checks, routing based on risk signals, and engineering tool calls with distributed system principles.
Other practices involve making data retrieval predictable and observable, implementing continuous production evaluation pipelines, investing in detailed observability and operational response, and introducing autonomy controls with deterministic fallbacks. These approaches help mitigate the complex failure modes typical in AI-driven systems.
Production Implementation: Example of a Bounded Step Wrapper
One practical implementation is wrapping each model or tool call in a controlled function that enforces validation, retry policies, timeouts, telemetry, and fallbacks to human intervention as needed. This method converts unpredictable AI behaviors into manageable, policy-driven steps.
Retries with jitter backoff handle transient failures, while schema and semantic validations catch bad outputs before progressing. This structure enables more predictable operational outcomes and faster troubleshooting.
Why Enterprises Demand Higher Reliability
Enterprises face significant business risks if AI systems produce errors, as reflected by surveys reporting widespread negative consequences from AI inaccuracies. Higher reliability reduces operational interruptions, enhances trust, and supports broader adoption of AI technologies at scale.
This demand drives investment in stronger measurement frameworks, guardrails, and comprehensive operational controls to meet rigorous enterprise quality standards.
Steps Toward Closing Reliability Gaps
To move up the reliability scale, organizations should:
- Identify critical workflows and define explicit completion SLOs.
- Implement strict validation contracts for all AI outputs and tool interactions.
- Treat connectors and data retrieval as primary reliability concerns with appropriate safeguards.
- Route high-risk actions through additional verification or human approval layers.
- Incorporate incidents into continuous regression test suites to prevent regressions.
Through disciplined engineering practices—bounded workflows, validated interfaces, resilient dependencies, and fast learning cycles—AI systems can progressively achieve the high reliability levels enterprises require.
