How to build a document intelligence pipeline

What matters here

• Score confidence per field, not per document. A 96% document hides the one field that is wrong.
• Validate against a system that already knows. A total that matches the purchase order is worth more than any model score.
• Send the uncertain fields to a person, not the whole document. Re-reading a correct page is how review queues get abandoned.
• Keep the source document linked to every extracted value, so a disputed number can be checked in seconds.
• Set thresholds per field. A date being wrong costs less than an amount being wrong.
• Track corrections by field. A field people always fix is a threshold that is wrong or a template that changed.

Who this is for

You run operations or an AI platform. Documents arrive as PDFs, a model reads them well most of the time, and nobody is comfortable letting it write to a system of record.

How it works in practice

What has to happen between a document arriving and its values being trusted.

1. 1

   The document is classified

   Type and template, because an invoice and a contract need different extraction entirely.

2. 2

   Fields are extracted with a score each

   Per field, because that is the level at which a value is right or wrong.

3. 3

   Values are validated against a system of record

   The purchase order, the contract, the vendor master.

4. 4

   Uncertain fields go to a person

   Just those fields, with the document open at the right place.

5. 5

   Confirmed values are written, with the document linked

   So the number and its evidence stay together.

6. 6

   Corrections are tracked by field

   Because a field that is always corrected is telling you something specific.

Paste these into Claude Code or Codex with the Tray Headless plugin installed. Each stage runs on its own. The systems named in them are the worked example rather than a requirement, and every prompt says so.

Once per project, run /tray-workflows:set-workspace to pick the workspace these build in. Point it at a sandbox first.

1. 1

   Set up and classify before extracting

   An invoice and a contract need different extraction entirely.

   Headless skills build-workflow

   Copy

   Use build-workflow. The systems in play are Google Drive, NetSuite and
   the systems these documents relate to, or whatever we run in those
   seats.
   
   Classify each document before extracting: what type it is, and where
   possible which template or supplier layout it follows.
   
   Extraction accuracy depends heavily on knowing what you are reading. A
   generic pass over an unknown document type produces the confident wrong
   answers this whole pipeline exists to avoid.
   
   Route anything unclassifiable to a person rather than guessing. A
   document nobody can identify is not a document to extract optimistically.
   
   Record the classification and its confidence alongside the document, so
   a bad extraction can be traced back to a misclassification rather than
   blamed on the extractor.

2. 2

   Extract per field, and score each one

   A document-level score is an average, and averages hide outliers.

   Headless skills build-workflow tray-patterns

   Copy

   Use build-workflow. For each field, produce the value, a confidence
   score for that field, and where on the page it came from.
   
   The location matters. A reviewer who can be shown the exact region a
   value came from checks it in seconds; one given a value and a forty page
   PDF does not check it at all.
   
   Set a threshold per field rather than one for the document. Getting a
   date wrong by a day is a nuisance; getting an amount wrong by a decimal
   place is money out of the door, and those two do not deserve the same
   bar.
   
   Never emit a single document-level confidence and gate on it. A page at
   96% is usually one field at 40% and nineteen at 99%, and the 40% one is
   the reason you built this.

   Capture the page region for every value. It is what makes review fast, and review speed is what decides whether the queue gets worked at all.

3. 3

   Validate against something that already knows the answer

   External agreement beats internal confidence.

   Copy

   Validate extracted values against systems that hold the truth:
   
     A total against the purchase order it references
     A vendor against the vendor master, matched on identifiers rather than
     on the name printed at the top
     A date against the contract term it should fall inside
     Line items against what was actually received
     Arithmetic: do the lines sum to the total, does tax compute
   
   A value that agrees with an independent source is trustworthy regardless
   of what the model scored it. A value that disagrees is suspect regardless
   of a high score, and that is the more important half.
   
   Where validation is impossible because there is no reference, say so
   explicitly and treat the value as unvalidated rather than as validated by
   default.

4. 4

   Send the uncertain fields to a person, not the document

   Re-reading a correct page is how review queues get abandoned.

   Headless skills tray-gotchas

   Copy

   Use tray-gotchas, then build review around the reviewer:
   
     Show only the fields below threshold or failing validation
     Show the extracted value, the page region it came from, and what it was
     validated against
     Let them correct the field and move on, without touching the rest
     Give the queue an ageing view, because a review queue with no age is a
     queue nobody prioritises
   
   Route by what the field is about: an amount mismatch goes to whoever owns
   the money, a vendor mismatch to procurement.
   
   Then write only confirmed values, and link the document to every one of
   them. When somebody disputes a number in six months, the answer should be
   one click rather than a search through a shared drive.

5. 5

   Track corrections by field, not by rate

   A field people always fix is telling you something specific.

   Copy

   Report by field rather than in aggregate:
   
     Correction rate per field, which finds the threshold that is set wrong
     Fields corrected despite scoring above threshold, which is the
     dangerous category
     Validation failure rate per source system
     Review queue depth and age
     Documents that could not be classified
   
   Fields corrected despite a high score is the number that matters. Each
   one is a value the pipeline would have written without asking, and a
   cluster of them in one field usually means a supplier changed their
   template.
   
   A single overall accuracy figure hides all of this. It goes up as volume
   shifts towards easy documents and tells you nothing about the hard
   ones.

6. 6

   Validate it, then hand the thresholds to operations

   Because templates change and the cost of an error varies by field.

   Copy

   Run the per-step schema checks and the whole-workflow audit before this
   touches production. Run it against a set of documents with known correct
   values, including a few deliberately awkward ones, and check the
   per-field scores against what was actually right.
   
   Then open the same workflow in Tray Build so operations can set the
   per-field thresholds, add validation sources and change the review
   routing in the visual canvas.

What it connects to

Documents come in, values are checked against systems that already know, and the document stays attached to what it produced.

Named systems are the ones most teams run, not the only ones that work. Each is an authentication in your Tray workspace, referenced by name, so the workflow never holds a credential. Where we have a connector page, the name links to it.

Questions people ask

Related guides