← All work

Applied AI · UX · systems integration

Voice-to-knowledge pipeline

Speak a thought on a phone and have it arrive as a structured, linked note. The interesting part is not the pipeline that works — it is the one before it, which could never have worked, and took a read of the platform's own rules to diagnose.

The first design was broken by architecture, not code. Finding that out was worth more than patching it would have been.

The problem

Capture is where most note systems die. Not retrieval, not organisation — capture. If recording a thought takes more than a few seconds, the thought does not get recorded, and the system quietly stops existing.

So the constraint was absolute: the act of recording had to be the same act as having it filed. Say it, and it is organised. No inbox to clear, no triage pass, no discipline required later — because discipline required later is a system with a failure date on it.

The one that works

Dictation on the phone, a local service on the tailnet, an agent turn in the middle, and a structured note out the other end.

The working pipeline: dictated speech becomes a structured, linked note without any manual filing step The phone dictates a thought into a shortcut, which posts the transcript over the tailnet to a small local service. The service runs an agent turn and returns the structured result, which the pipeline files into the knowledge base as a note with frontmatter, a summary, key points and links. Capture — the act of speaking is the act of filing Dictate Text one spoken thought, no app to open POST over the tailnet private network only, nothing exposed publicly A small local service runs one agent turn per note A structured note frontmatter + sectioned body What lands in the knowledge base date · time · type · tags · status source: iphone-dictation Summary · Key Points Action Items · Links Why the shape matters The sections are not decoration. They are what makes the note findable and linkable later, by a person or by an agent. An unstructured transcript is a note that only its author will ever read again.
The filing is not a second step. A template with enforced sections turns dictation into something the rest of the knowledge base can actually use.

The one that could never work — and why that was the useful part

The first version of this routed the whole thing through a chat bot. The phone would send the question as a message, the agent would answer into the chat, and the phone would read the reply back. It looked reasonable. It was impossible, and no amount of debugging would have fixed it, because it was wrong about how the platform works.

Why the first design could never work: a bot never receives its own messages, and a shortcut cannot read a chat as the user The first design sent the phone's question as a message authored by the bot, so the agent never received it. It then tried to read the reply by polling for updates, but the gateway already consumes those updates. Both paths are dead. The working design instead returns the answer inside the same HTTP response to the original request. The first design — two independent faults, either one fatal 1 · send the question the shortcut posted it as the bot so the agent never received it 2 · read the reply the shortcut polled for updates the gateway already consumes those The measurement that settled it Updates waiting to be read by the shortcut: 0 the queue is drained before anything else can read it Two faults, in opposite directions. Fixing either one still leaves the other, and no amount of code changes the rule underneath: a bot never sees its own outgoing messages. What replaced it The answer comes back inside the same request. The shortcut holds the connection open while the agent thinks, so there is nothing to poll and nothing to race.
Both faults are structural. One sends the message as the wrong author; the other reads a queue that is already drained. The design that works stops trying to observe the conversation and instead makes the reply a return value.

The lesson generalises: when a design is impossible, the impossibility is usually about ownership — who authors a thing, who consumes it, who is allowed to see it. Reading the platform's own rules and finding the two places where the plan contradicted them took less time than another round of debugging would have, and it produced a design that is simpler than the one it replaced.

Two pipelines, two jobs

The system deliberately keeps recording and answering apart, because they are different promises. One is a note filed for later; the other is an answer you hear out loud.

Recording and answering are separate paths with separate endpoints The note path files dictated text or audio into an inbox folder for a scheduled processor, producing a structured note. The ask path runs a live agent turn and speaks the answer back to the phone without storing anything. Recording for later "Remember this." dictated text or audio → an inbox folder → a scheduled processor structures it → a linked note in the knowledge base Nothing to wait for. Filing is the whole job. Answering now "Tell me something." a dictated question → a live agent turn → the reply returns in the response → the phone speaks it aloud Nothing is stored. This one is a conversation.
Keeping them apart is why neither is compromised: the note path never makes you wait, and the answer path never files something you only wanted to hear.

Facts, in a form you can check

CaptureA phone dictation shortcut — nine actions, no app to open
TransportHTTP over a private tailnet; the service is not exposed publicly
Output shapeFrontmatter plus four enforced sections: summary, key points, action items, links
Typical latencyAround 6 seconds for a direct answer, around 13 when the agent needs a tool call
Ceiling180 seconds, after which the request is abandoned rather than left hanging
Failure handlingA failed agent run returns an error rather than an empty answer
SchedulingNote processing runs on a timer, so a backlog drains without being watched

Outcome.

What I would change next

Three things, and none of them are cosmetic. The service currently listens on every interface rather than the private one, which is fine on this machine and wrong the moment it joins a network I do not trust. There is no shared secret on the answering endpoint, so any device on the network can spend model credits. And each question is a fresh turn with no memory of the last one — a session handle would fix that, and is the first thing I would add if the thing turns conversational.