Start With a Skateboard: The Vertical-Slice Method

The most dangerous phase of a chatbot project is "Phase 0" — because there is no Phase 0.

Most teams start by building infrastructure. They set up a vector database, configure an LLM provider, build an embedding pipeline, design a retrieval system, and write evaluation scripts. Six weeks later, they have a beautifully architected infrastructure that doesn't answer a single customer question. They have built a factory but no product.

The alternative is the skateboard method: every phase produces a working chatbot. Not a prototype. Not a spec. A deployed, useful thing that gets better. The skateboard is not a prototype of a car — it is a vehicle that moves a person from A to B, just with less range than the car will eventually have.

This article is the third in a five-part series. By the end, you'll have a phase framework you can use to scope your own chatbot project.

---

The Principle: Deployable at Every Step

Building a chatbot that works is not a single delivery event. It is a sequence of working milestones, each one fully functional at its scope. The principle: every phase must produce a deployed, verifiable artifact. No speculative infrastructure. No "we'll wire it together in Phase 3." Each phase's output stands alone.

This mirrors the capability progression that applies to AI agent deployments more broadly — what Part 1 of this series (Beyond the Chatbot) describes as autonomy levels. The difference is that here, we are applying the progression to retrieval capability, not autonomy.

---

Phase 0: The Skateboard — Direct Q&A Matching

Phase 0 starts with the narrowest possible useful scope: a handful of question-answer pairs, hardcoded with direct matching. No LLM. No vector database. No embeddings. It answers the top questions your support team handles every day with 100% accuracy.

This feels wrong to most teams. "That's not AI," they say. Correct. It is not AI. It is a working chatbot that answers real customer questions while you build the infrastructure for Phase 1. It teaches you what questions actually get asked, which ones are most important, and how users phrase their queries. That data is invaluable for designing the structured knowledge layer.

The skateboard ships in days, not weeks. It has no hallucination risk because it has no LLM. Every answer is a known string. Evaluation is binary: the question is in the list or it isn't.

---

Phase 1: The Scooter — Structured Knowledge Retrieval

Phase 1 introduces structured retrieval. Documents are organised in a knowledge hierarchy with frontmatter metadata — the approach described in Part 2 of this series. The chatbot can now answer questions that require connecting multiple entities. It cites sources.

This phase adds an LLM, but the LLM's job is constrained: given a set of retrieved structured documents, synthesise an answer from them. It does not search. It does not guess. It answers from what it was given, and the retrieval system is deterministic enough that the same question reliably retrieves the same documents.

Evaluation at this phase uses a golden dataset — a set of question-answer pairs covering the knowledge base. Spot-checks catch obvious failures. The chatbot is measurably smarter than Phase 0, but it still cannot answer questions about content that hasn't been categorised yet.

---

---

Phase 2: The Bike — Vector Fallback

Phase 2 adds fallback vector search for questions the structured knowledge cannot reach. The retrieval order is: try structured retrieval first — if confidence is below threshold — fall back to vector search — if still below threshold — escalate.

This is where labelled evaluation becomes critical. The chatbot can now answer questions it has never seen before, but the quality of vector search answers varies. A labelled evaluation dataset is introduced. The chatbot can now be measured against a known standard.

LLM-as-Judge evaluation starts here. The evaluation dataset runs automatically, scoring each response for groundedness, relevance, and completeness. Human reviewers spot-check the edge cases. The feedback loop is operational but manual.

---

Phase 3: The Car — Production Hardening

Phase 3 is production hardening. The evaluation pipeline is automated end-to-end: every response scored, every trace queryable, every regression caught before it reaches a user. This is where observability infrastructure (Arize Phoenix, Langfuse, or equivalent) becomes non-negotiable.

The chatbot meets the criteria defined in advance as "good enough for production" — not infinity, just sufficient. The criteria are numerical and explicit: groundedness above 0.9, answer relevance above 0.85, latency below 3 seconds at P95.

No phase ships without passing its predecessor's evaluation. The gate is the eval dataset, not a calendar date.

---

Working Artifact: The Phase Progression Table

Use this table as a project scoping reference. Each phase is independently deployable. Each phase's evaluation method reflects what is possible at that level of infrastructure maturity.

The table works as a diagnostic tool: find where your chatbot is today, and the next phase is defined for you.

---

FAQ

How long does each phase take?

Phase 0: 2-5 days. Phase 1: 2-4 weeks. Phase 2: 4-8 weeks. Phase 3: 4-8 weeks depending on observability maturity. These are ranges, not guarantees — knowledge base quality is the dominant variable.

Can I skip Phase 0?

Yes, but you lose the most valuable data you will ever have about your chatbot: what questions real users actually ask. Phase 0 is as much a research phase as a delivery phase. Skipping it means designing your retrieval architecture against assumptions, not data.

What if my team only has budget for Phase 0?

Phase 0 is useful on its own. A hardcoded Q&A chatbot that reliably answers the top 50 support questions is a legitimate product. It has zero hallucination risk, zero LLM cost, and instant response times. That is not a failure. It is a Phase 0 win.

---

Next in This Series

This is part 3 of a five-part series on building chatbots that actually work.

• Part 1: Why Your Chatbot Doesn't Work — The demo-to-production gap.
• Part 2: Structured Knowledge — Why document dumps don't work.
• Part 3: Start With a Skateboard — You are here.
• Part 4: The Evaluation Environment — How to know if your chatbot is working.

---