The model is not the product

A model can produce an excellent answer and still sit inside a poor product. I learned this most clearly when FundRobin moved from something that could impress me in a test to something that had to do useful work repeatedly for real organisations. At that point, the interesting questions stopped being mainly about prompting. They became questions about state, evidence, failure, cost, control and what the user should be able to trust.

The model mattered. It just stopped being the centre of the design.

The demo hides most of the work

The first version of an AI feature is unusually easy to overrate. Give a capable model a good prompt and coherent context and it can generate something that looks finished: a funding recommendation, a rationale, a draft proposal, a summary of an organisation.

That output is visible, so it attracts attention. The less visible work sits around it.

Which organisation profile is current? Which opportunity is actually open? Which requirements are hard rules and which require interpretation? What happens when a model provider is unavailable? If a long-running job stops halfway through, can it recover? Can the user tell whether a result is new, stale or incomplete? If the system is about to take a consequential action, who owns the decision?

These are not edge cases once a product is live. They are the product.

FundRobin now processes AI-assisted work across a much larger operating surface than the original prototypes. The product has organically reached more than 250 customers and processed more than 12,000 grants. Repetition changes what you notice. A failure mode that is harmless in ten tests becomes an operating problem when the same workflow runs thousands of times.

One of the most useful shifts in my own thinking was to stop asking, “How do we get a better answer from the model?” as the default question. I now start with a different one:

What promise are we making to the user, and what has to be true for the whole system to keep it?

Change the unit of design from prompt to workflow

A prompt is an implementation detail. The user experiences a piece of work.

For a grant-matching flow, that work might begin with organisation context, pass through eligibility and semantic judgement, produce a recommendation and rationale, and end with a user deciding whether the opportunity deserves attention. A proposal workflow has a different chain: evidence, requirements, source material, drafting, review and eventually a human-owned submission decision.

Once the unit becomes the workflow, the design questions improve:

1. What information must be correct before this stage begins?
2. Which decisions can be represented as deterministic rules?
3. Where does probabilistic reasoning create genuine value?
4. What state has to persist if the work spans minutes, hours or sessions?
5. How do we know a stage completed successfully?
6. What happens when it does not?
7. Where should a person see, challenge or approve the result?

This separation has been important in FundRobin. Eligibility, status transitions, permissions and lifecycle rules are poor places to ask a language model to improvise. Matching messy organisational context to the intent of a funder is exactly the kind of problem where probabilistic reasoning can be useful. Drafting is useful too, but only when the evidence and requirements around the draft remain explicit.

I want AI concentrated where uncertainty and synthesis are useful, with stronger guarantees around the parts of the experience that should behave predictably.

Functionally correct can still be the wrong product

Automated tests miss a different failure mode: the AI can build exactly what was asked for and the result can still feel wrong to a user.

I see this most clearly in workflow and interface design. An agent can create a functionally valid screen around an unfamiliar process because it has optimised for the specification rather than for how people already understand the work.

The human product job is partly anthropological. How do people do this today? Which parts of the existing process are frustrating, and which parts are familiar for a reason? What terminology do they use? Which document or screen acts as their mental anchor? What can we remove, and what should remain recognisable even if the underlying workflow becomes dramatically more efficient?

That is the difference between automating a task and redesigning a product experience.

I do not want AI to reproduce the old process simply because users recognise it. I also do not want to replace it with a technically elegant workflow that makes sense only to the people who built it. The useful design move is to preserve enough familiarity that the user knows where they are, while changing the machinery underneath so the work becomes materially easier.

Reliability is mostly a systems problem

As FundRobin moved into production, some of the most important AI work looked surprisingly conventional.

We introduced patterns for provider fallback, asynchronous and batch processing, canonical job tracking, unfinished-run recovery, structured outputs, usage and cost telemetry, human review and release validation. An adversarial review of one provider-migration design surfaced thirteen issues. Two of the more useful findings were not glamorous model problems at all: a budget guard could fail open, and usage records could be duplicated.

That review reinforced a principle I now use much more broadly: failure semantics matter more than happy-path fluency.

When I look at an AI workflow now, I want to know what happens when:

• the model provider is unavailable;
• a structured response is malformed or incomplete;
• the database read fails;
• a job is interrupted;
• a budget or policy check cannot be performed;
• the external system changes;
• a human does not approve the next step.

A good answer from the model cannot compensate for a workflow that silently loses state or turns uncertainty into an unreviewed action.

This is one reason I am wary of measuring “AI quality” as a single model score. Product quality lives at several layers. Model reasoning is one. Context quality is another. Validation, recovery, interface clarity and human decision rights are others.

A stronger model can improve one layer while leaving the system unreliable.

Architecture becomes product design

The boundary between architecture and product design is thinner in AI systems than I used to think.

Take something as ordinary as background processing. If a matching job can run asynchronously and recover, the user does not need to keep a browser session open and hope a spinner eventually disappears. That architectural decision changes the experience.

Structured outputs change the product too. They let the interface treat a result as components that can be inspected, validated or edited rather than as one opaque block of text.

Provenance changes the interface again. If a recommendation can show which source or piece of organisational evidence informed it, the user has something to interrogate. A recommendation without provenance asks for trust. A recommendation with evidence supports judgement.

The same relationship works in reverse. A product requirement often implies an architectural requirement.

“The user must be able to leave and return” implies durable state.

“The user needs to understand why this opportunity is relevant” implies retained evidence and explanation.

“Nothing should be submitted without explicit approval” implies a control boundary the model cannot silently bypass.

This is where I find the product conversation most interesting. I do not need to be the engineer who personally writes every implementation detail to make these decisions consequential. My role is to define what the system must guarantee, understand the trade-offs well enough to shape the architecture, direct the implementation, and validate whether the behaviour keeps the product promise.

Cost belongs in the product conversation

Operating real workflows also made inference cost a product question, not only an infrastructure line item.

We analysed FundRobin’s production LLM usage because an all-time average was telling the wrong story. A large historical backfill made matching look far more expensive than the current architecture actually was. When we segmented the data by processing mode and architecture era, the commercial conclusion changed: current onboarding and steady-state matching costs were small enough that trial length was primarily an activation question, not an inference-cost constraint.

The important part of that exercise was not the specific number. It was the method.

AI cost only becomes decision-grade when it is attached to the workflow that produced it. A blended historical average can be technically accurate and commercially misleading.

I now prefer to think about cost per completed useful task, not cost per call. A cheap call that causes retries, poor outputs and human clean-up can be expensive. A more expensive reasoning step can be sensible if it materially improves a high-value decision. Batch processing may be better where immediacy adds little value. Deterministic checks should eliminate impossible work before the model is invoked at all.

The product decision is the allocation of reasoning, latency and cost across the workflow.

Human control is not a temporary defect

There is a common story about AI systems in which human review is scaffolding: models get better, autonomy increases, and eventually the human disappears.

I no longer think that is a useful default.

Some review points exist because the model is imperfect. Those may shrink. Others exist because responsibility belongs with a person.

FundRobin can research, match, prepare and draft. An application, however, contains representations about a real organisation and can have external consequences. I want the product to make that boundary explicit. The system can do more of the preparation without quietly inheriting accountability for the final act.

The story itself matters too. A charity or social enterprise should own the impact it wants to create. AI can help assemble evidence, compare an opportunity to the organisation’s context, suggest improvements and draft language. It should remain an agent in service of that story rather than becoming the author of the organisation’s purpose.

That is also why I do not want FundRobin silently submitting applications on behalf of customers. Submission often carries declarations, funder-specific rules and commitments the organisation needs to read and understand. The system can reduce the burden before that point. The final commitment belongs to the customer.

The right level of autonomy depends on three things I return to repeatedly:

• uncertainty: how confidently can the system know what is right?
• consequence: what happens if it is wrong?
• reversibility: can the action be cheaply undone?

A reversible internal recommendation can tolerate more autonomy than a submission, financial commitment or externally visible claim.

That does not mean inserting a human into every step. It means designing the checkpoint where human judgement actually changes the risk or quality of the outcome.

A badly designed “human in the loop” screen is just theatre. If the reviewer cannot see the evidence, understand what changed or exercise meaningful control, the system has not allocated responsibility well. It has merely moved the final click to a person.

Observability closes the product loop

Once AI becomes a workflow, it produces much richer product signals than “the user clicked Generate.”

Which stage fails most often? Where do retries cluster? How much does a user revise a draft? Which inputs are usually missing? Which provider or model path produced the result? What does the completed job cost? Where do people override the recommendation?

These signals are not only engineering telemetry. They are product evidence.

A high edit rate may indicate a weak model, but it may also mean the wrong context is reaching the model. A slow step may require a different interaction pattern rather than a faster model. Repeated user overrides may reveal a bad product assumption. A recovery path that is constantly exercised may tell you the workflow boundary is wrong.

The operating system around the model becomes strategically useful when it generates evidence about how the work actually behaves, not just answers for the user.

That evidence changes the next decision.

The durable advantage is the learning system

Models will continue to improve. Providers will change. Tasks that feel technically difficult today will become easier and cheaper.

That makes me less interested in building a product identity around access to a particular model and more interested in the system that compounds around it: authoritative context, workflow design, control boundaries, user trust, operating data, evaluation and the ability to turn what happens in production into a better product decision.

The model is a powerful component of that system. Sometimes it is the component that makes the entire product possible.

But the product is the promise made to the user.

The job is to design the combination of models, deterministic software, data, tools and human judgement that can keep that promise when the demo is over.