RAG is the right tool for procedures and the wrong tool for a live schedule. Most deployments never draw the line, and the customer finds out before the plant does.
By Nishkam Batta
There is a moment in almost every manufacturing AI demo where somebody in the room asks the obvious question.
They have just watched the assistant pull a changeover procedure out of a 300-page work-instruction library in four seconds. Genuinely useful. So they lean forward and ask it something operational: when will 88213 actually ship. The assistant answers. Confidently, in a full sentence, with a date. The room nods. Somebody says this changes everything. Two days later a customer calls asking where their order is, and the date the assistant gave was never real to begin with. Most conversations about RAG implementation services never get to this distinction, and it is the one that decides whether the system survives its first quarter.
The failure here is not a bug, and the vendor is probably not being dishonest. It is a category error, and it is the single most common reason manufacturing AI pilots quietly get switched off around month four.
Retrieval Is a Snapshot, and a Ship Date Is Not
Retrieval-augmented generation works by converting a body of text into vectors, storing them, and finding the passages closest in meaning to whatever the user asked. It then hands those passages to a language model and asks it to answer using them. When the underlying material is a work instruction, a routing sheet, a customer spec, or a supplier agreement, this is close to ideal. Those documents are written once, revised rarely, and their meaning does not change between Tuesday and Wednesday.
A promised ship date is the opposite kind of fact. It is not a document — it is the live output of a scheduling engine that gets recalculated every time a machine goes down, a rush order jumps the queue, a material shipment arrives late, or an operator calls in sick. It holds one value at 9:14 a.m. and a different value at 9:16 a.m., and neither one is written down anywhere permanent.
When a retrieval layer is pointed at scheduling data, what it actually stores is whatever date was true at the moment of indexing. Ask the system the same question three days later and it will return that same indexed date, delivered in exactly the same confident register it uses for the changeover procedure. Nothing in the output distinguishes a firm commitment from a snapshot that has already expired.
This is a recognized and well-mapped failure mode. Barnett and colleagues catalogued seven distinct RAG failure points across three real-world case studies, and stale or misaligned retrieval sits near the top of the list. In legal applications, where the underlying sources are far more stable than a live scheduling board, studies have measured RAG hallucination rates between 17 and 33 percent. A survey of enterprise deployments found that around 80 percent of RAG projects hit critical failures, with roughly one in five ever reaching sustained production use.
The problem is not that these systems are unreliable in general. It is that they are being asked a class of question they were never designed to answer, and they were never given a mechanism for declining.
What RAG Is Genuinely Very Good At
It is worth being specific about the wins here, because the category error runs in both directions — a plant that concludes RAG is overhyped will leave real value sitting on the table.
Mid-market manufacturers sit on enormous unstructured archives of exactly the knowledge that makes a schedule executable in the first place: routing sheets going back decades, setup and changeover instructions from equipment vendors who no longer exist, customer specifications with clause-level tolerances, supplier quality agreements, PPAP files, engineering change notices. This material is genuinely knowledge, it is genuinely hard to search, and the scheduler who knew where all of it lived retired last year.
We worked with a contract manufacturer whose planners spent up to six hours a week just hunting down the correct revision of a setup sheet before they could confirm a job was even schedulable. A retrieval layer over that archive brought typical lookups down to under ten minutes. Nothing about that result required the system to know a live queue position or promise a date to anyone. Every question it answered was a question about a document, which is precisely the shape of problem retrieval was built to solve.
The same is true for onboarding new planners, answering "have we run this part configuration before," and pulling together compliance evidence on demand. These are high-value, document-shaped questions, and a well-built retrieval layer handles them with citations straight back to the source page, which is what makes the output auditable.
Why "Just Connect It to the MES" Does Not Fix It
The instinct at this point is to give the model direct access to the scheduling and MES tables and let it write its own queries so it can answer date questions live. That is a better instinct than indexing a schedule snapshot, and it is still much harder than it sounds.
The BIRD benchmark was built specifically to test how well language models translate natural language into SQL against real, messy databases rather than tidy academic ones. On its test set, GPT-4 reached about 55 percent execution accuracy when handed curated domain hints explaining what the columns meant. Without those hints it dropped to roughly 35 percent. Human performance on the same task sits near 93 percent.
Now consider the gap between that benchmark and a real plant. BIRD's databases average around seven tables. A production scheduling and MES schema, by contrast, often runs to hundreds or thousands of tables, with column names inherited from a decade-old implementation and business logic encoded in conventions nobody wrote down. We see the same sprawl one layer over, in ERP inventory management tables that were built for a transactional system to write to, not for a language model to freely query and reason over. If a model gets it only half right on seven clean tables, treating it as a free-form query engine against a live scheduling schema is not an integration decision — it is a coin flip with a professional tone of voice, and the number it returns is a date your sales team may repeat to a customer.
And there is a second problem underneath the first: a perfectly correct query against a stale plan still produces a wrong date. Scheduling systems are only as current as their last re-plan, and on a busy floor that re-plan can lag reality by hours. Machine downtime, expedite requests, and material shortages are frequently logged after the fact, not in real time. So the schedule the model reads may already be out of date the moment the query runs, no matter how well the SQL was written.
So the honest version of the answer to "when will it ship" is frequently something like: the system says Thursday, the schedule was last re-run two hours ago, and line 3 has missed its plan on the last four jobs of this size. That is a more useful answer than Thursday on its own. It is also an answer no retrieval layer will ever produce by itself, because producing it requires the architecture to know what it does not know.
The Architecture That Actually Holds Up
The systems that survive contact with a plant floor separate the question types before answering, rather than routing everything through one pipeline. In practice that means four things.
A router that classifies the question before answering it. Document questions go to retrieval. Ship-date and status questions go to a live query against the scheduling or MES system. Questions that are actually requests for action — bump a job, hold a work order, expedite a shipment — go to a workflow with an approval step. This classification layer is unglamorous, and it is the difference between a system that is trusted and one that is not.
Parameterized reads instead of free-form SQL. A defined set of read-only queries against agreed views of the schedule and MES data, with the model selecting and filling parameters rather than composing arbitrary SQL. This gives up some flexibility and buys back determinism, which is the correct trade for anything a customer service rep is going to repeat to a customer.
Provenance and a timestamp on every answer. A document answer cites the document and its revision. A ship-date answer states the source system, the time the plan was last recalculated, and, where the plant tracks it, how reliably that line has hit its plan recently. A date with no timestamp attached is not an answer — it is a promise nobody actually signed off on.
A human approval gate on anything that writes. Reading a schedule is low risk. Re-sequencing a queue, expediting a job, or committing a new ship date to a customer is not. The person who owns the outcome should own the click, with the reasoning and the source data in front of them.
None of this is exotic architecture. It is mostly the discipline to admit that two different questions need two different mechanisms, right at the point in a project where it is tempting to promise one interface that answers everything.
Four Questions Worth Asking a Vendor
If you are evaluating a manufacturing AI proposal, these four will tell you most of what you need to know within twenty minutes.
How does the system distinguish a document question from a live ship-date question, and what happens at that boundary? If there is no boundary, everything downstream is guesswork.
When I ask for a ship date, what exactly gets read, and how old is the plan behind it? Vague answers here are the whole ballgame.
What does the system do when it cannot answer reliably? A system with no refusal behavior will hallucinate a date instead of declining, because declining was simply never built.
Show me an answer with its sources and timestamp attached. Not in a slide. In the working system.
The Test to Run This Week
If you already have an assistant deployed, run one experiment. Ask it for the ship date on an order you can physically track on the floor. Then go check the line. Then ask it the same question again a few hours later and see whether the answer changed after conditions did.
That takes about forty minutes, and it tells you more about your AI investment than any dashboard will. A system that gets the document questions right and honestly flags the ship-date questions as time-sensitive is working correctly. A system that answers everything with equal confidence is not more capable — it just has fewer ways of telling you it is wrong.
