when was he born?

a chatbot that answers questions about your own documents is mostly three pieces: a vector database that finds the parts of your documents that matter, a model that writes the answer, and a memory of what's been said. I built one in 2023 with LangChain, Pinecone and Upstash Redis, in a Next.js app streaming with the AI SDK. every one of those has changed its API since, so this is about the ideas, which haven't.

#search, then answer

the usual setup is called retrieval-augmented generation: turn the question into an embedding, find the stored chunks of your documents with the closest embeddings, and hand those chunks to the model along with the question. the model answers from them instead of from whatever it remembers.

that works for the first question. the second one is where it breaks.

#who's he?

search for "when was he born?" and you get chunks about anyone who was ever born. the search has no idea who he is. the question only makes sense with the conversation attached, and embeddings don't do pronouns.

so before searching, a model rewrites the question into one that stands on its own, using the history: "when was he born?" becomes "when was Leo Tolstoy born?". that's the question that gets embedded and searched. LangChain's ConversationalRetrievalQAChain, which I used back then, did exactly this in two model calls: one to rephrase, one to answer.

#two models

the two calls don't need the same model. rephrasing a question is a small job, and nobody reads its output: it's time spent before the first word of the answer shows up. use a small, fast model to rephrase, and save the big one for the answer.

and stream only the answer. the rephrased question is an internal step, so it shouldn't appear in the chat.

#where the history lives

the memory has to come from somewhere on every request. the chat UI already has every message, so it's tempting to send them all with each question. but those messages live in the browser, and anyone can edit what lives in the browser: they could hand your model a conversation that never happened.

keep the history on the server, keyed by session. I used Redis on Upstash: every message gets appended under the session's key, and the rephrase step reads it from there.

#when the answer is in the conversation

one more problem showed up. when the answer was something said earlier in the chat, not something in the documents, the bot couldn't find it. the rephrase step kept only the question, so the search went looking in the documents for something that only existed in the chat.

the fix went in the rephrase prompt: ask for the relevant part of the history along with the standalone question, and pass both on.

given the conversation and a follow-up question, return the part of the
conversation that's relevant to the question, if there is one, and
rephrase the follow-up as a standalone question.

chat history:
{chat_history}

follow-up: {question}

answer in this format:
relevant history: <the excerpt, or N/A>
question: <the standalone question>

with that, the model writing the answer sees both the documents the search found and the bit of conversation that matters.