The demo problem
A pilot succeeds under conditions production will never reproduce: curated documents, a friendly evaluator, no concurrency, no adversarial users and no cost ceiling. When the same system meets a real corpus with inconsistent formatting, stale duplicates and permission boundaries, quality falls off a cliff — and nobody has an evaluation baseline to quantify by how much.
Evaluation is the deliverable
Before writing retrieval code we build a golden dataset: real questions from real users, with correct answers agreed by someone qualified to judge. It takes a week and it is the single highest-leverage artefact in the project. Without it, every subsequent decision — chunk size, reranker, model choice, prompt revision — is guesswork dressed up as engineering.
Governance decisions have owners or they have delays
Who approves what the assistant may discuss? What happens when it is wrong? Who reviews flagged conversations? Which data may leave the tenancy? These are organisational decisions, not technical ones, and they are the most common cause of a finished system sitting in staging for three months.
The thin slice discipline
One workflow, real data, real authentication, real logging, real cost measurement. Narrow enough to ship in eight weeks, complete enough that nothing structural is deferred. Everything we learn about the hard parts arrives while the blast radius is still small.
What we check before we commit
Is there a named business metric with a current baseline? Is there an owner empowered to make governance calls? Can we assemble a golden dataset in two weeks? Is the data reachable with permissions intact? Is there a realistic cost ceiling per interaction? Five yeses and we build. Fewer, and we fix that first.