A field officer finishes a visit, stands at the edge of a chilli field in Chevella, and says into her phone: "Ramesh garu polam lo leaf blight malli vachindi, Blitox cheppanu, rate ekkuva ani annaru." Thirty seconds, Telugu and English mixed, one bar of signal.
That note is the most valuable data our agent will ever get, and it's also the easiest to lose. My teammate wrote about how KhetSmriti reads memory to brief an officer before a visit. This post is about the other half: the write path, which turns a messy voice note into something Hindsight agent memory can actually learn from.
My opinion after building it: the write path decides how smart your agent gets. Recall can only find what retain stored well.
What KhetSmriti is
KhetSmriti ("field memory") is an agent for field officers at agri-input distributors (seeds, fertilisers, crop protection) in Telangana. Before a visit it writes a pre-visit brief from everything it remembers about the farmer. After the visit, the officer logs a note, and weeks later logs the crop outcome. Over time it learns which advice works for each farmer, village and crop.
It runs on Next.js, Groq for the LLM and Whisper, zod for every boundary, and Hindsight as the memory layer. Every Hindsight call goes through one file, src/lib/memory.ts, and shows up live in a Memory panel in the UI.
Step 1: Speak, then let the officer check the transcript
The phone records up to two minutes of audio. The server checks size and type before anything expensive happens:
// src/app/api/transcribe/route.ts
if (audio.size === 0) return fail("empty_audio", "The recording is empty. Please record again.", 400);
if (audio.size > MAX_BYTES) return fail("too_large", "Recording is too long. Keep voice notes under 2 minutes.", 413);Groq's whisper-large-v3 does the transcription, with the language set to Telugu, English or auto. Code-mixed speech is hard, and a single misheard product name becomes a wrong fact in memory forever. So the transcript is always editable before anything else happens. I'd rather have the officer spend five seconds fixing "Blue tax" to "Blitox" than have the agent confidently remember a product that doesn't exist.
Step 2: Structure, with the farmer's own crops as the schema
Free text is hard to filter and compare, so an LLM turns the note into a structured visit: crop, stage, issue, products, objection, reaction. The part I like most is that the zod schema is built per farmer:
// src/lib/visitStructuring.ts
function llmVisitSchema(crops: readonly string[]) {
return z.object({
crop: z.enum(crops as [string, ...string[]]),
issue: z.object({
type: z.enum(["pest", "disease", "soil", "price", "irrigation"]),
name: z.string().min(1).transform((s) => s.trim().toLowerCase()),
}),
advice: z.object({ productIds: z.array(z.string()), note: z.string().min(1) }),
objection: z.string().min(1).nullable(),
farmerReaction: z.enum(["accepted", "hesitant", "rejected"]),
unknownProductMentions: z.array(z.string()).default([]),
// ...
});
}If Ramesh grows chilli and cotton, the model cannot return "paddy". Issue names are lower-cased so "Leaf Blight" and "leaf blight" end up as the same thing in memory. If the output fails validation, the LLM layer retries once with the error appended, then falls back to a second model. It never crashes the page.
The prompt has one rule that matters more than the rest: if the note mentions a product that is not in the catalogue, list it in unknownProductMentions and do not invent an id. After the call, every product id is checked against the catalogue again, and anything unknown is shown to the officer as a warning and never saved as advice. When you're recommending pesticides, a hallucinated product in memory is a bug that keeps coming back.
Step 3: Retain prose, not JSON
This was the least obvious decision. I had a clean JSON object and my instinct was to store it. Instead, KhetSmriti converts it back into plain English before calling retain:
// src/lib/narrative.ts
return [
`Field visit on ${visit.date} to ${farmer.name} (${farmer.id}) in ${village.name} village, ${village.mandal} mandal.`,
`Crop: ${visit.crop}, stage: ${visit.cropStage}.`,
`Issue (${visit.issue.type}): ${visit.issue.name}.`,
describeProducts(visit.advice.productIds, catalogue),
objection,
`Farmer reaction to the advice: ${visit.farmerReaction}.`,
`Farmer profile: ${farmer.landAcres} acres, ${farmer.irrigation} irrigation, ${farmer.priceSensitivity} price sensitivity.`,
].join(" ");The reason is that Hindsight extracts facts and builds observations from text, so the text should read like something a person would write. describeProducts expands an id like P006 into its name, price tier, pack price and dose, so the memory says "premium tier, ₹1,480 per 500 ml" rather than just P006. That's what lets reflect later connect "premium" with "rejected on price".
The bank has a retain mission that tells Hindsight what to pull out of every document: "Extract who was visited, where, which crop and stage, the issue, which products were advised, the farmer's objection and reaction, and whether the advice worked. Keep dates, prices and product ids." It costs one line of config and keeps extraction focused on what the business cares about.
The structured fields still go along for the ride, as tags (farmer:, village:, crop:, officer:, kind:visit) and metadata. So we get both: prose for understanding, and tags for exact filtering.
Step 4: Outcomes are separate memories
Advice without a result is just an opinion. Weeks after a visit, the officer records what happened: applied or not, controlled, partial, failed. That is retained as its own document, outcome:V037, with kind:outcome, next to visit:V037:
const RESULT_TEXT: Record<OutcomeResult, string> = {
controlled: "The advice worked: the issue was controlled.",
partial: "The advice partly worked: the issue was only partially controlled.",
failed: "The advice failed: the issue was not controlled.",
not_applied: "The farmer did not apply the advice.",
};Keeping them separate means the visit memory is never rewritten, the outcome carries its own date, and "what did we advise?" can be told apart from "what actually worked?". Those four sentences are deliberately blunt, because "the advice failed" is a much easier fact for memory to learn from than result: "failed".
Step 5: Reflect, "what works in this village?"
Once enough visits and outcomes are stored, the Insights screen asks Hindsight's reflect three fixed questions per village and crop: what works for the most common problem, how to handle price objections, and what to watch for next month. They're scoped with the village tag:
client().reflect(bankId(), question, { budget: "mid", tags, tagsMatch: "any_strict", includeFacts: true, signal })The answers come back with the memories they were based on, so an officer can tap through to the actual visits. Reflect is the slowest call in the app, so answers are cached for five minutes, and only when all three succeed. A half-failed page is retried on the next request instead of being frozen for five minutes.
This is where the write-path work pays off. On our seed history, reflect found that pink bollworm returns to Moinabad cotton every August, and that Chevella farmers say yes when advice comes with a neighbour's yield result. Neither rule is written anywhere in the code or the prompts. Hindsight found them because every visit and outcome was stored as clear, dated, well-tagged prose.
What changed with memory on
With memory switched off, KhetSmriti's brief for Ramesh says "No history with this farmer yet" and suggests four generic products. With memory on, it recommends the copper fungicide that worked on his own field, cites the visit dates, and warns the officer not to open with the premium product he refused in April. The model is the same. The difference is what the write path put into memory.
Lessons
1. Put the human check before retain, not after. An editable transcript and a review form cost the officer a few seconds. A wrong fact in memory costs you every future brief that recalls it.
2. Build the output schema from the entity you're writing about. A per-farmer z.enum of crops removes a whole class of errors that no prompt can reliably prevent.
3. Retain readable prose, and put the structure in tags. Hindsight learns from text. Filtering needs tags. You don't have to choose.
4. Store outcomes as their own memories. "Advised X" and "X worked" are different facts with different dates. Keep them apart and the agent can learn the difference.
5. Unknown things are warnings, never data. Anything the catalogue doesn't recognise gets shown to a person and kept out of memory.
The code is on GitHub. If you're designing the write side of an agent, the Hindsight docs on retain, missions and observations are worth reading closely, and Vectorize's piece on what agent memory is explains why stateless agents keep starting from zero.