Dispatches
Field notes on putting AI to work · evaluations, agent workflows, shipping without a dev team · by Kalpak Kale
3 dispatches, newest first. Subscribe via RSS, or start with the story.
The 21-case test that decides whether a RAG chatbot ships
A document chatbot for a regulated financial client went from a proof of concept to client beta. What got it there was not the model. It was a weighted evaluation of 21 cases, nine rounds of structured QA, and a daily accuracy report the client ran themselves.
Giving 19 developers AI access through one proxy, and what actually changed
Rolling AI coding tools out to a whole team is not a licensing decision. It is a gateway, a spending policy, an evaluation habit and a set of working rules. Here is how a sixty-person services firm did it, including the 38 tool evaluations that mostly ended in "no".
A brief goes in, software comes out. Shipping client products without a dev team assigned.
An operations manager with an agentic coding tool can take a statement of work to a production system, but only with a fixed process. The one I use, with the payment-first design and security defaults that a travel platform forced me to learn.