Moss moved the search index into the app, and the pause disappeared
Everyone who has talked to a voice agent knows the pause.
You ask a question. The agent says "let me check that for you," or worse, says nothing, and for a long beat you're listening to silence and wondering if the call dropped. Then the answer arrives, and it's usually fine. But the conversation is already damaged. Humans expect a reply to start within a few hundred milliseconds; anything longer reads as confusion, evasion, or a bad connection. A voice agent that answers correctly after two seconds still feels broken.
The strange part is where that pause comes from. It's usually not the language model. Speech recognition, LLM inference, and speech synthesis have all been ruthlessly optimized because everyone can see them on the latency dashboard. The step that quietly eats the budget is retrieval: the moment the agent has to go look something up. The knowledge base lives in a vector database on the other side of the internet, the query text has to be embedded by yet another API before it can even be searched, and by the time the relevant passage comes back, 400 to 600 milliseconds are gone. That's the pause.
Moss attacks that step directly. The company, part of Y Combinator's Fall 2025 batch, builds a real-time semantic search runtime for conversational AI. Instead of hosting your search index behind an API and making every query cross the network, Moss packages the index and ships it into your application, where queries run against local memory. On the company's published benchmark, retrieval comes back in about 3 milliseconds, embedding included. The whole lookup finishes before a round trip to a conventional vector database would have gotten out the door.
Retrieval was built for a different kind of software
The retrieval stack most AI teams use today was designed for workloads where latency was a soft preference, not a hard constraint.
A recommendation engine can take half a second to fetch candidates. A document search page can show a spinner. So the standard architecture put the vector index in a managed cloud service, added an embedding API in front of it, and let every query pay for two network hops plus server-side processing. For the software of 2022, that was a reasonable trade: someone else runs the infrastructure, and nobody notices a few hundred milliseconds.
Conversational AI broke the trade. A voice agent is a pipeline: transcribe the user's speech, reason over it, look things up, generate a reply, synthesize audio. All of it has to fit inside the window of a natural conversational turn, and every stage's latency adds to the total. When one stage in that pipeline routinely costs 400 milliseconds, it doesn't degrade the experience at the margin. It consumes most of the budget the entire pipeline was supposed to share.
Teams building voice agents have responded by caching aggressively, trimming their knowledge bases, or skipping retrieval altogether and stuffing everything into the prompt. Each workaround trades away the thing retrieval was supposed to provide: an agent that actually knows the current contents of the docs, the product catalog, the account history. Moss's founding observation is that the problem was never retrieval itself. It was the network sitting in the middle of it.
A runtime you embed, not a service you call
Moss's architecture has three parts, and the division of labor between them is the whole idea.
Moss Cloud handles the slow work: ingesting documents, embedding them, storing them, and packaging the result into a versioned index. The index is the distributable artifact, documents and vectors bundled together. The runtime is a small engine, written in Rust, that embeds inside your application, pulls the index down over HTTPS, and executes queries against it in local memory.
That last part is what changes the latency math. Once the index has loaded, a query never leaves the process. There is no embedding API call, because the runtime embeds the query itself with built-in models. There is no network hop to a search cluster, because the search happens in the same memory space as the agent asking the question. The developer-facing workflow stays boring in the good sense: create an index with your documents, load it, query it. Three calls in the SDK.
The runtime is more than a bare vector index. Queries run hybrid retrieval, combining semantic similarity with keyword matching, which matters in practice because users say exact things like part numbers and plan names that pure embeddings are bad at. Metadata filtering with operators like $eq, $in, and $near narrows results by structured fields. Connectors pull source data from SQLite, MongoDB, MySQL, and Supabase, and a CLI manages indexes from the terminal.
Moss ships SDKs for Python, TypeScript, Elixir, and C, plus a WebAssembly build that runs the same retrieval inside a browser. The SDKs, examples, and integrations are open source under a BSD 2-Clause license, and there are prebuilt integrations for the frameworks where conversational agents actually get assembled: LiveKit, Pipecat, Vapi, and ElevenLabs on the voice side, LangChain, LlamaIndex, DSPy, CrewAI, and AutoGen for everything else. LiveKit publishes an official starter template pairing its voice agents with Moss, which tells you the voice infrastructure companies see the same bottleneck.
What 3.1 milliseconds means in context
Moss publishes a benchmark in its repository, and the numbers are worth reading carefully rather than rounding to "fast."
The test runs 750 queries against an index of 100,000 documents, retrieving the top 5 results, on a MacBook Pro with an M4 Pro chip. Moss returns a median of 3.1 milliseconds, with a 99th percentile of 5.4. The same workload measured against hosted vector databases lands between 350 and 600 milliseconds at the median: Pinecone at 432, Qdrant at 597, ChromaDB at 351. And the comparison actually understates the gap, because Moss's number includes embedding the query, while the hosted services in the test rely on an external embedding call.
Two caveats belong next to that table. It is a vendor benchmark, run by Moss on its own harness. And local hardware is doing work that a hosted service does elsewhere, so the architectures aren't measuring identical jobs. But that second point is precisely the argument. Moss is claiming that for conversational workloads, doing the work locally is the correct architecture, and the two orders of magnitude between 3 milliseconds and 400 are what the claim looks like when you measure it.
The p99 figure matters more than the median for voice. A conversation doesn't experience your average latency; it experiences the worst lookup in the middle of the most important answer. Tail latencies in the single digits mean retrieval effectively vanishes from the turn budget, and the pipeline can spend its milliseconds on the model instead.
Search that follows the app off the server
Embedding the index in the application has a second consequence that has little to do with speed.
The WebAssembly build means semantic search can run entirely inside a browser tab, and the C library means it can run on a phone or an embedded device. In those deployments, the user's queries never leave the device at all. For products handling medical notes, legal documents, or anything else a customer would rather not stream to a third-party search API, on-device retrieval converts a compliance conversation into a non-issue. The same property makes offline operation possible, which hosted retrieval cannot offer at any latency.
This is the sense in which Moss is positioning itself as a runtime rather than a database. Databases have a location, and your application travels to it. A runtime goes where the application goes: server, edge node, browser, handset. The retrieval code is identical across all of them, with Moss Cloud handling the indexing and syncing so each runtime instance has a current copy. As more AI products push inference toward the client, the retrieval layer either follows or becomes the one piece still tethered to a data center.
Founders who spent their careers on this exact problem
Moss was founded by Harsha Nalluru and Sri Raghu Malireddi, and their backgrounds read like a job description for this company.
Nalluru was a tech lead at Microsoft, where he architected the core stack of the Azure SDK, the client libraries that sit in front of more than 400 cloud services and get downloaded over 100 million times a week on npm. That is years of experience in exactly the artifact Moss sells: a small piece of software embedded in other people's applications, which has to be fast, reliable, and pleasant to integrate, at enormous scale.
Malireddi led machine learning work at Microsoft and Grammarly, shipping LLMs and personalization systems across Office, Bing, and Grammarly to tens of millions of daily users. Grammarly's keyboard is a case study in running serious ML under brutal constraints: on a phone, inside someone else's typing loop, where every millisecond of lag is felt in the fingertips. His personalization work there drove a 300% improvement in retention, which is what happens when latency-sensitive ML actually works.
One founder builds SDKs the world embeds; the other makes models fast on constrained devices. Moss is the intersection of those two careers, and it is a small company: five people in San Francisco, founded in 2024, launched publicly in November 2025.
The early numbers
For a product that launched months ago, the usage Moss reports is unusual in its shape as well as its size.
The company says its runtime is deployed across more than 100 countries, has passed 390,000 package installs, and has served over 10 million minutes of real-time voice, with enterprise customers using it to serve thousands of their own end customers. Around launch it described usage and revenue roughly doubling week over week. Vendor-reported numbers deserve the usual discount, but package installs and voice minutes are consumption metrics, not vanity ones. Nobody accumulates 10 million voice minutes through a runtime that adds lag; the product sits on the hot path, and staying there is the test.
The timing is the tailwind. Voice AI went from demo to deployment over the past two years, and every one of those deployments eventually collides with the retrieval pause. The fixes available until now were all versions of "retrieve less." Moss is a bet that the category instead fixes the architecture, moves the index into the app, and stops treating a lookup as a network event.
Conversation was always the interface where latency shows. Moss's wager is that the winning retrieval layer for conversational AI is the one you never hear.
