Reliability is a product decision

The first version of an AI workflow encourages a narrow definition of success. Give it a representative input, get a useful answer and the feature appears to work.

Operating the same workflow repeatedly changes the question. A customer does not experience a model response in isolation. They experience whether the job starts, whether it finishes, what happens when a provider fails and whether the product can explain its current state.

That has made reliability a product decision for me. The engineering mechanisms matter because each one expresses a choice about the customer promise.

A successful response is only one state

In a live workflow, work can be queued, running, completed, failed, deferred or waiting for a person. A request can succeed at the API boundary while the useful work remains unfinished. A model can return valid output while the wider workflow writes it to the wrong place or advances the wrong lifecycle state.

I learned to ask a more useful question: what does the product believe has happened, and what evidence supports that belief?

That question leads quickly to canonical job state, explicit eligibility rules and durable records of attempts. It also changes the interface. “We received your request” and “your work is complete” describe different commitments. Products create confusion when they collapse those commitments into one green message.

Provider choice is part of continuity planning

FundRobin uses structured model workflows for matching, generation and related AI tasks. The production design includes provider fallback, batch and asynchronous processing, and recovery for unfinished runs.

Those choices began as technical requirements. Their product meaning is continuity. If one model path is unavailable, the team needs to know which work can move to another provider, which work should wait and which differences in output quality are acceptable.

A fallback that silently changes the meaning of a result is not reliable. A retry that duplicates work is not reliable either. Continuity requires a clear contract around the task, its inputs, its state and the output the rest of the product expects.

This is why model portability is more than swapping an API name. The workflow around the model has to absorb differences in latency, structured output, failure modes and cost without losing the customer’s intent.

Recovery begins with deciding what counts as the same job

Retries sound simple until a job has external effects.

If a matching task is repeated, should it replace the earlier result, create a new version or stop because the work already completed? If a scheduled process stops halfway through a cohort, where should it resume? If the provider returns after the application has moved to a new state, is the output still valid?

These are identity and lifecycle questions. Recovery depends on stable job identifiers, idempotent operations where possible and explicit rules for stale work. Without them, an automatic retry can turn one failure into duplicate or contradictory state.

The product decision is the acceptable recovery behaviour. The implementation should make that decision enforceable.

Observability should answer product questions

Infrastructure monitoring can say that a worker is running. That does not tell me whether organisations are receiving useful results, whether a particular path is repeatedly deferred or whether the cost of recovery is distorting the economics of the feature.

The telemetry I care about connects technical events to the product lifecycle:

• how many jobs entered each state;
• where unfinished work accumulates;
• how often a provider fallback is used;
• what completion and recovery cost;
• where a human overrode or stopped the process.

This is operational evidence for prioritisation. A queue that grows slowly every week may deserve more attention than a visible one-off error. A provider path with a low unit price may be expensive once retries and manual review are included.

Observability closes the gap between the system we designed and the system customers are actually using.

Release evidence belongs to the product owner

AI coding agents now execute a substantial part of FundRobin’s implementation and test loop. That makes it possible to move from a well-scoped idea to working software quickly. It also makes release judgement more important.

I do not treat a merged change, a passing test suite and a deployed service as interchangeable evidence. Each answers a different question. A production conclusion needs the deployed version, the health of the relevant services and the behaviour of the customer-facing journey.

The person accountable for the product should understand that chain. Delegating implementation does not delegate the promise made when the feature is released.

Reliability competes with other product work

Reliability is not an unlimited programme. Every recovery mechanism, provider path and dashboard has a cost. The product job is to decide which failures are worth designing around now.

I use a few practical tests:

1. How consequential is a wrong or duplicated action?
2. Can the customer recover without support?
3. Does the failure remain visible, or can it create silent drift?
4. How often does the path run?
5. What evidence would let us improve the decision later?

The answers determine where deterministic rules, human approval, fallback or simple failure are appropriate. A low-consequence draft can tolerate a different recovery path from a payment, a lifecycle transition or a customer-facing submission.

The customer promise is implemented in the failure path

Product teams naturally spend time on the intended journey. Live systems are shaped just as much by interrupted journeys: the provider timeout, the half-finished batch, the stale result and the release that passed CI but does not behave correctly in production.

Reliability becomes concrete when those conditions are treated as product states with owners and expected behaviour. The result is not a system that never fails. It is a product that knows what failure means, preserves the evidence needed to recover and does not ask the customer to absorb its ambiguity.

Related ideas

• The model is not the product — Why the workflow, rather than model output, is the useful unit of design.
• AI workflow economics — How reliability and recovery affect the cost of an AI workflow.

Sources and further reading

1. Monitoring Distributed Systems — Google SRE
2. Canarying Releases — Google SRE
3. Reliability Pillar, AWS Well-Architected Framework — Amazon Web Services