The Questions That Determine Whether Your Chatbot Survives Its First Six Months
Your chatbot was 90% accurate at launch. Three months later, users are complaining. The answers are stale. The chatbot doesn't know about the pricing change your company made last month. It keeps referring to a product line that was discontinued.
This is not a failure. It is a decay curve that every unmaintained chatbot follows. The model did not get worse. The knowledge drifted. Product documentation changed, support processes evolved, and new edge cases emerged from user traffic. The chatbot stayed static while the world moved.
This article is the final piece in a five-part series on building chatbots that actually work. By the end, you will have a one-page project scoping template with the nine questions that determine whether your chatbot survives its first six months in production.
The Decay Curve
In our deployments, we track accuracy against a fixed evaluation set over time. The typical curve is predictable: 90%+ accuracy at launch, dropping 5-10% per month without knowledge base maintenance.
Teams that treat chatbot deployment as a one-time project see this drop and blame the model. They try a different model, and the new model is equally accurate at launch. Then the same decay starts again. The model was never the problem. The knowledge drifted, and nobody was maintaining it.
This pattern is consistent across deployments and is not unique to us. Multiple production observability platforms (LangSmith, Arize) document similar drift patterns in their published production monitoring data. It is the single most common cause of chatbot failure after the first quarter.
The fix is not a better model. The fix is maintenance architecture — who owns the knowledge base, how often it is reviewed, and what triggers an update.
What Maintenance Actually Requires
Most teams underestimate maintenance costs by a factor of three. They budget for the initial deployment and assume the chatbot will take care of itself. It will not.
Knowledge base maintenance. Documents need periodic review — content becomes outdated, products change, policies are revised. This is not optional. In our deployments, we schedule reviews of knowledge base coverage against support ticket miss patterns at regular intervals. Frontmatter metadata needs the same attention as the documents themselves. An entity relationship that was correct at launch may be incorrect after a product restructuring.
Evaluation dataset maintenance. The dataset used to gate deployments must reflect current user traffic. New types of questions appear from real usage. The dataset should grow: initial 100-200 entries is common, scaling to cover the breadth of actual traffic. Stale evaluation datasets produce false confidence — the chatbot passes eval because the dataset only tests scenarios no longer relevant to users.
Observation and triage. Someone must review production traces. Not every day, but on a regular cadence. Are there new failure patterns? Are users asking questions the knowledge base doesn't cover? The evaluation cycle described in Part 4 of this series is continuous, not a one-time setup.
The Escalation Boundary
Every chatbot needs a defined point where it says "I cannot answer this." The escalation design determines whether that handoff is helpful or frustrating. Most teams treat escalation as an afterthought — a "contact us" link at the bottom of the page. That is not escalation design.
In our deployments, escalation triggers are: confidence below threshold, user-expressed frustration, or repeated failure on the same topic. The handoff includes conversation history, the attempted answer, confidence scores, and the category of the question. The principle: the human receiving the handoff should never have to ask "what were they trying to do?" The context must travel with the escalation.
This requires infrastructure. The chatbot must measure its own confidence, detect user frustration (repeated rephrasing, negative sentiment), and package the context into a handoff format. If your chatbot cannot do this, it cannot escalate gracefully. It can only fail silently.
Build AI Agents That Actually Work
Production AI deployments need the right architecture, evaluation framework, and infrastructure. We design and deploy autonomous AI systems for Caribbean businesses — self-hosted, sovereign, built to last.
Book a Fit CallMulti-Step Wizard Flows: The Scope Trap
Wizard flows — guided multi-step processes like password resets, troubleshooting sequences, or application processes — are frequently scoped too early. Each flow is a distinct product. It requires its own intent definition, step logic, error handling, off-script handling, and evaluation dataset.
In our experience, scoping more than one or two wizard flows before the base chatbot is stable is a predictor of project overrun. Each flow adds complexity that compounds: branching paths, user inputs at each step, unexpected answers, and validation at every transition.
Add flows based on production miss data, not upfront speculation. The question is not "how many flows do we need?" but "which questions that users are asking today cannot be answered without a wizard flow?" If the answer is none, don't build wizards. Build better retrieval first.
Simple flows (2-3 steps, linear) take measurably less effort than complex flows (branching, exceptions, backend integration). Complexity should be the scoping dimension, not count of flows.
Working Artifact: The Nine Questions
Based on our operational experience, these are the questions that must be answered before writing a line of code. Every deployment we have done that skipped one of these questions had to go back and answer it later — usually at higher cost.
| # | Question | Why it matters |
|---|---|---|
| 1 | What specific questions must the chatbot answer on day one? | Defines scope. Without this, every question is in scope, and the project never ships. |
| 2 | What documents contain the answers? | Identifies knowledge source. Without this, the chatbot retrieves from wherever documents happen to live. |
| 3 | How are those documents organised and who maintains them? | Governance. Without this, the knowledge base decays from day one. |
| 4 | How will we know if the chatbot answers correctly? | Evaluation method. Without this, you cannot measure improvement or regression. |
| 5 | What does "good enough" look like measured numerically? | Criteria. Without this, the team ships when tired, not when ready. |
| 6 | What happens when the chatbot cannot answer? | Escalation. Without this, the chatbot guesses or goes silent. |
| 7 | Who updates the knowledge base when things change? | Maintenance owner. Without this, nobody is responsible for accuracy six months in. |
| 8 | How often will the evaluation dataset be refreshed? | Maintenance cadence. Without this, the eval set becomes stale and produces false confidence. |
| 9 | What is explicitly out of scope for this deployment? | Boundary. Without this, scope creep is guaranteed. |
Print this table. Fill it in with your team before you write a line of code. The answers will surface disagreements early, when they cost nothing to resolve, rather than late, when they cost weeks of rework.
FAQ
How often should I review the knowledge base?
Monthly for most deployments. Weekly for high-change environments (pricing, compliance, product launches). The review cadence should match the rate of change in the business, not a calendar.
Who should own chatbot maintenance?
The same person or team that owns the source documents. If the product team owns product documentation, they should own the chatbot's product knowledge. The chatbot is a distribution channel for their content, not a separate system.
What is the minimum viable maintenance budget?
One person, half a day per week for knowledge base review, evaluation dataset refresh, and trace triage. Less than that and the decay curve will outpace the maintenance effort.
How do I know when my eval dataset is stale?
Two signals: (1) the chatbot passes all eval checks but users are reporting wrong answers, or (2) the eval dataset has not been updated in three months. Either signal means it is time to refresh.
The Series
This is the final piece in a five-part series on building chatbots that actually work.
- Part 1: Why Your Chatbot Doesn't Work — The demo-to-production gap and why naive RAG fails at scale.
- Part 2: Structured Knowledge — Why document dumps don't work and what to do instead.
- Part 3: Start With a Skateboard — The vertical-slice method for chatbot delivery.
- Part 4: The Evaluation Environment — How to know whether your chatbot is working.
- Part 5: The Questions That Determine Survival — You are here.
We build Automata — autonomous AI agents that handle invoicing, scheduling, procurement, and customer resolution. Self-hosted on Jamaican infrastructure. Every deployment starts with a conversation about your architecture.