From messy PDFs and scanned documents to validated, traceable, production-ready structured data
Businesses have no shortage of documents.
Purchase orders, invoices, inspection reports, employment applications, medical records, contracts, internal reports, spreadsheets, technical drawings, and operational forms continue to arrive in different formats and layouts.
The problem is not storing these documents.
The problem is turning the information inside them into reliable structured data.
A finance system does not want a PDF. An ERP does not want a scanned purchase order. A database cannot do much with a handwritten form sitting inside a shared folder.
What those systems need are things like:
{
"purchase_order_number": "PO-10452",
"supplier": "ABC Engineering",
"order_date": "2026-08-28",
"total_amount": 12500.00
}That sounds simple until you work with real enterprise documents.
A document may have multiple columns, inconsistent labels, tables split across pages, handwritten values, low-quality scans, diagrams, charts, or information whose meaning depends entirely on its position on the page.
And that leads to an important distinction:
A structured output is not necessarily a correct output.
An LLM can return perfectly valid JSON containing a completely wrong value.
This article explains how to design a document extraction pipeline that addresses that problem from end to end: parsing, schema design, extraction, validation, evaluation, human review, deployment, and operational monitoring.
I also use Unstract as a practical example to show how such a pipeline can be built and deployed. The original experiments covered both the open-source edition and Unstract Cloud, including its agentic workflow for schema and prompt generation.
TL – DR
A reliable AI document extraction system is not simply:
Document → LLM → JSON
A production system is closer to:
Document → Parsing → Structure Preservation → Schema → Extraction → Validation → Evaluation → Review → Deployment → Monitoring
The quality of the final result depends on every stage.
If parsing destroys the relationship between a label and its value, the LLM may confidently extract the wrong information.
If the schema describes the wrong fields, the model may produce technically valid but operationally useless data.
If evaluation is performed only against documents used during development, the reported accuracy may look much better than real-world performance.
And if the system cannot show where an extracted value came from, humans may struggle to verify questionable results.
These are not minor implementation details. They are the foundation of a trustworthy document intelligence system.
Why Structured Data Extraction Is Harder Than It Looks
Organizations are interested in document extraction for a straightforward reason: structured data can participate in downstream business processes.
Once information is converted into a consistent format, it can be:
- inserted into databases,
- analyzed,
- pushed into ERP systems,
- used by applications,
- passed into automation workflows,
- validated against business rules,
- and combined with information from other systems.
The challenge is that enterprise documents rarely behave like clean datasets.
Consider a purchase order.
One supplier might put the purchase order number in the top-right corner. Another may place it below the company address. A third might call it PO Number, while another uses Order Reference.
The line items can be presented as a standard table, merged cells, multiple sections, or even split over several pages.
Some documents are scanned images.
Some contain handwriting.
Others rely on charts, diagrams, stamps, checkboxes, or visual relationships that disappear when the document is converted into plain text.
The source article demonstrates this problem with a purchase order containing a messy layout.
The hidden problem: parsing happens before extraction
One of the most important ideas in document AI is that the LLM does not see the original document unless you deliberately preserve that information.
Most systems first convert the document into a machine-readable representation such as text, Markdown, HTML, or another intermediate structure.
That parser can introduce errors before the LLM even begins its work.
For example:
Original:
Item | Description | Qty | Price
---- | ------------------------ | --- | -----
101 | Industrial Bearing | 10 | 125.00
Parsed incorrectly:
Item
Description
Qty
Price
101
Industrial Bearing
10
125.00The values still exist.
But the relationship between them may be weakened or lost.
The model can produce valid JSON and still assign 125.00 to the wrong field.
This is why document extraction should be treated as a pipeline problem, not simply an LLM prompting problem.
How Document Extraction Has Evolved
Document extraction has gone through several major generations.
The key change has been a shift from telling software where information is located toward explaining what the information means.
1. Rule-Based Extraction
Traditional extraction systems combine technologies such as:
- OCR,
- fixed templates,
- coordinates,
- keywords,
- regular expressions,
- and hard-coded rules.
For a stable document, this approach can work very well.
For example:
Find "Invoice Number"
Read the value 150 pixels to the right.The weakness becomes obvious when the layout changes.
A supplier modifies its form.
A field is renamed.
A new document variant is introduced.
A table moves to another page.
Suddenly another rule is required. Over time, the extraction system turns into a large collection of special cases.
Rule-based extraction is therefore not inherently bad. It is simply most effective when the input is highly predictable.
2. Machine Learning and Document Understanding Models
The next generation used machine learning to understand document structure.
Instead of relying exclusively on fixed coordinates, models could recognize:
- text regions,
- tables,
- key-value pairs,
- sections,
- and other layout elements.
This was a major improvement.
But there was another cost: training data.
Organization-specific document understanding often requires annotated examples. Creating these datasets can become expensive and time-consuming, particularly when an organization has only a small number of examples for certain document types.
3. LLM-Based Extraction
Large language models changed the workflow again.
Instead of training a specialized model for every document family, engineers can describe the required fields and expected output.
For example:
Extract:
- supplier_name
- purchase_order_number
- order_date
- currency
- total_amount
Return the result using the following schema...This is dramatically more flexible.
The model can often handle previously unseen document layouts without requiring a new training cycle.
But this does not mean the problem is solved.
LLMs introduced flexibility.
They did not eliminate the engineering required for reliable extraction.
Three Practical Approaches to LLM Document Extraction
There is no single correct way to feed documents into an LLM.
Three approaches are especially useful.
Approach 1: Parser First, Then LLM
The document is first converted into a machine-readable representation:
PDF
↓
Parser
↓
Text / Markdown / HTML
↓
LLM
↓
Structured JSONThis architecture is relatively easy to implement and can be efficient.
However, the parser becomes critical.
If table structure, spatial relationships, headings, or cross-page context are lost, the downstream model receives incomplete or misleading information.
Approach 2: Vision LLM
Another approach is to send document pages directly to a vision-capable model.
This preserves much more visual context and can be particularly useful for:
- scanned forms,
- complicated tables,
- charts,
- technical layouts,
- diagrams,
- and other visually structured documents.
The trade-off is cost and operational complexity.
Vision-based processing can be:
- slower,
- more expensive,
- harder to trace,
- and more difficult to scale for large document volumes.
Approach 3: Hybrid Parsing
A practical compromise is to use traditional parsers for information they handle well and vision models for content that depends on visual interpretation.
For example:
Text → conventional parser
Tables → structure-aware parser
Charts → vision model
Diagrams → vision model
↓
Combined document
↓
LLMThis hybrid architecture is often attractive because it avoids paying the cost of vision processing for every character while still handling layout-dependent information.
The source article also describes using conventional parsing for text and tables while using vision models for charts, figures, and diagrams.
Where Agentic Workflows Enter the Picture
LLM-based extraction does not necessarily have to be one giant prompt.
An agentic workflow can divide the problem into smaller specialized tasks.
For example:
Document
↓
Parser Agent
↓
Field Discovery Agent
↓
Schema Agent
↓
Prompt Generation Agent
↓
Extraction Agent
↓
Validation Agent
↓
Human ReviewEach component has a narrower responsibility.
The agents can inspect example documents, suggest fields, reconcile similar fields, generate schemas, construct extraction instructions, evaluate outputs, and refine the configuration.
That changes where engineering effort goes.
With old systems, teams spent their time maintaining templates and extraction rules.
With modern AI systems, they spend more time managing:
- parsers,
- schemas,
- prompts,
- validation rules,
- evaluation datasets,
- confidence mechanisms,
- traceability,
- and deployment workflows.
Building a Reliable LLM-Based Extraction Pipeline
A production-grade pipeline should have several explicit stages.
Stage 1: Parse the Document
The first goal is not extraction.
The first goal is to preserve as much meaning and structure as possible.
The parser should ideally retain:
- page boundaries,
- tables,
- section hierarchy,
- labels,
- values,
- layout relationships,
- and references to the original document location.
A weak parser can poison every downstream stage.
The LLM cannot reliably reconstruct information that was discarded before it received the document.
Stage 2: Define a Schema
The schema tells the system exactly what it is trying to extract.
For example:
{
"purchase_order_number": "string",
"supplier_name": "string",
"order_date": "date",
"currency": "string",
"total_amount": "number"
}A schema can be manually written or inferred from a task description and example documents.
Once approved, it should be stored and versioned rather than regenerated unpredictably for every document.
The reason is simple: schema changes are business changes.
If downstream systems expect supplier_name, changing the schema silently can break those integrations.
The source workflow treats schema creation as a controlled operation: generate, review, approve, version, and reuse until extraction requirements change.
Stage 3: Extract the Data
The parsed document, schema, and extraction instructions are then sent to the LLM.
Conceptually:
Parsed Document
+
Approved Schema
+
Extraction Instructions
↓
LLM
↓
Structured OutputThe expected result should be machine-readable and constrained by the schema.
This is where structured output becomes important.
But schema compliance alone is not enough.
Validation Is Necessary — But It Is Not Sufficient
Suppose the system expects:
{
"total_amount": 12500.00
}and the model returns:
{
"total_amount": 12500.00
}The JSON is valid.
The data type is correct.
The field exists.
The schema passes.
But what if the actual document says:
Total Amount: 12,500.00
Tax: 1,250.00
Grand Total: 13,750.00The model could have selected the wrong amount.
Schema validation would not catch that because 12500.00 is still a perfectly valid number.
This creates one of the most important principles in document AI:
Schema-valid does not mean source-correct.
The original article makes this distinction explicitly and argues that accuracy evaluation is required in addition to structural validation.
Stage 4: Evaluate Extraction Accuracy
The next question is:
Did the model extract the right value?
A useful evaluation framework compares extracted values against human-verified reference data.
For example:
| Field | Expected | Extracted | Correct |
|---|---|---|---|
| PO Number | PO-10452 | PO-10452 | ✅ |
| Supplier | ABC Engineering | ABC Engineering | ✅ |
| Date | 2026-08-28 | 2026-08-28 | ✅ |
| Total | 13,750.00 | 12,500.00 | ❌ |
This is far more informative than merely checking whether the JSON parses.
However, production systems have an important complication:
There is usually no human reference answer available for every live document.
That is why additional mechanisms such as:
- source citations,
- value localization,
- confidence estimates,
- and human review queues
become important.
A Reliable Extraction Pipeline Needs a Feedback Loop
A useful production architecture looks more like this:
┌──────────────┐
│ Document │
└──────┬───────┘
↓
┌──────────────┐
│ Parser │
└──────┬───────┘
↓
┌──────────────┐
│ Schema │
└──────┬───────┘
↓
┌──────────────┐
│ LLM │
└──────┬───────┘
↓
┌──────────────┐
│ Validation │
└──────┬───────┘
↓
┌─────────┴─────────┐
↓ ↓
High confidence Low confidence
↓ ↓
Automation Human Review
↓ ↓
Database Correct / Approve
↓
Feedback Loop
The critical idea is that AI should not automatically become the final authority for every field.
Some documents are straightforward.
Others require human verification.
A mature system knows the difference.
Putting the Pipeline Into Practice with Unstract
Unstract provides a practical environment for building schema-guided document extraction workflows.
The platform supports document parsing, field-specific extraction, configurable prompts, selectable LLMs, and deployment through APIs or ETL workflows.
The original implementation tested Unstract against a messy purchase order and later against a hand-filled employment application and construction blueprint.
Setting Up Unstract Open Source
The source workflow begins by cloning the Unstract repository and starting the platform:
git clone https://github.com/Zipstack/unstract.git
cd unstract
./run-platform.shThe original article then accessed the local frontend at:
http://frontend.unstract.localhostwith the default demonstration credentials provided in that environment.
For a real production environment, credentials should of course be changed and infrastructure hardened before exposing the system to business data.
Creating a Prompt Studio Project
Unstract’s Prompt Studio provides a no-code environment for configuring document extraction.
The general workflow is:
- Create a project.
- Configure an LLM.
- Configure the parser.
- Upload sample documents.
- Define extraction fields.
- Specify output types.
- Test the results.
The original setup used a purchase order with a deliberately difficult layout.
Each extraction requirement was defined separately, with output types such as:
- text,
- number,
- date,
- JSON.
This field-level configuration is important because different data types may require different validation and interpretation rules.
Choosing the Parser
The parser can have as much impact on accuracy as the LLM itself.
In the experiment, LLMWhisperer was selected as the parser, while Unstructured was also mentioned as another option.
The goal was not merely to turn the PDF into text.
The goal was to preserve enough of the original structure for the model to understand which values belonged to which labels and table rows.
The resulting parsed representation retained enough row structure and cross-page relationships for the sample extraction to work correctly.
Moving to Agentic Extraction
Unstract Cloud adds another layer of automation.
At the time of the original experiment, its Agentic Prompt Studio was described as a beta feature that used multiple agents for:
- field discovery,
- schema generation,
- prompt creation,
- and extraction.
This is particularly interesting because it moves part of the configuration process from manual engineering to automated reasoning.
Instead of manually deciding every field up front, the system can inspect the documents and suggest a structured representation.
But the human remains part of the loop.
That distinction matters.
Agentic automation should reduce configuration effort, not remove accountability.
Testing an Employment Application
The original experiment used a sample hand-filled employment application.
The pipeline selected an LLM for both extraction and agent tasks and used LLMWhisperer for parsing.
The workflow then proceeded through multiple agent stages.
Agent 1: Field Discovery
The first stage identified potential fields and proposed information such as:
- field names,
- descriptions,
- data types,
- example values.
Agent 2: Field Reconciliation
Different documents may describe the same concept using different labels.
For example:
Phone
Telephone
Mobile Number
Contact NumberA reconciliation stage can recognize these as potentially equivalent concepts.
Agent 3: Schema Finalization
The proposed fields are consolidated into a JSON Schema that becomes the contract for downstream extraction.
This multi-agent process is described in the source article as part of the Agentic Prompt Studio workflow.
I then tested the data extraction from a construction blueprint (see below).
Human Approval Still Matters
The interesting part of the experiment was not simply that agents generated the schema.
It was that the user could review and edit the generated configuration before approving it.
The workflow included:
Document
↓
Field suggestions
↓
Human review
↓
Reference data
↓
Schema generation
↓
Human approval
↓
Prompt generation
↓
Human approval
↓
ExtractionThat makes the process much safer than blindly accepting whatever an agent produces.
In the sample employment application, the generated values matched the approved reference data.
However, the author correctly noted that this should not be interpreted as a fully independent accuracy benchmark because both reference and extracted results came from the same document.
That qualification is extremely important.
Testing a More Difficult Case: Construction Blueprints
Structured forms are one challenge.
Technical drawings are another.
The next experiment involved a construction blueprint.
The system was able to identify fields from the left and right sections but initially failed to identify information contained inside diagrams.
Instead of treating that failure as evidence that the platform was useless, the schema was manually improved.
A dimension object was added for the top diagram.
The extraction prompt was regenerated.
The system was then run again.
The defined fields were successfully extracted.
This experiment demonstrates a principle that is often overlooked:
The schema is part of the intelligence of the extraction system.
A strong model with a weak schema can still produce poor results.
Human review of schema design can therefore be more valuable than simply switching to a more expensive model.
Deploying the Extraction System
Once the extraction workflow has been tested, it needs to become part of the surrounding business infrastructure.
The source workflow supports two important deployment models:
API Deployment
Applications submit documents and receive structured results.
Conceptually:
Application
↓
Document Upload
↓
Extraction API
↓
Structured JSON
↓
Application / ERP / DatabaseThis is useful when extraction needs to happen on demand.
For example, an internal application might upload a purchase order and immediately receive:
{
"po_number": "PO-10452",
"supplier": "ABC Engineering",
"total": 13750.00
}The source experiment used a workflow-based API deployment, allowing the same extraction tool to be reused within the deployment workflow.
Testing the API with Postman
The deployed API can be tested without immediately writing a custom application.
The experiment used a predefined Postman collection:
API Deployments
↓
Download Collection
↓
Import into Postman
↓
Process Document
↓
Upload file
↓
Send
↓
Receive JSONThis is a useful development pattern because it separates extraction testing from application development.
Before integrating the API into an ERP or custom application, engineers can verify:
- input handling,
- response structure,
- extraction quality,
- latency,
- error handling,
- and authentication behavior.
ETL Deployment
An API is not always the best architecture.
Suppose an organization receives thousands of purchase orders every month in a shared Google Drive folder.
Nobody wants an employee to upload every file manually.
An ETL pipeline is more appropriate:
Google Drive
↓
New Document
↓
Parser
↓
Extraction
↓
Validation
↓
PostgreSQL
↓
ERP / Analytics / ApplicationsThe source experiment configured Google Drive as the input source and PostgreSQL as the destination.
The demonstrated workflow ran periodically, processed new purchase orders, and stored the structured output in PostgreSQL.
This is where document extraction becomes operational automation rather than a demonstration.
The following screenshot shows an example of an extracted record in the Postgres database.
API vs ETL: Which Should You Use?
The choice depends on the business process.
| Requirement | Better Fit |
|---|---|
| Extract one document immediately | API |
| User uploads documents interactively | API |
| Process files continuously from a folder | ETL |
| Process thousands of documents on a schedule | ETL |
| Integrate with an application | API |
| Feed a database automatically | ETL |
Many enterprises will eventually use both.
For example:
Real-time documents → API
Batch documents → ETLBoth can use the same underlying extraction configuration.
That reduces duplication and makes governance easier.
The Most Important Lesson: Reliability Is a Pipeline Property
One of the strongest conclusions from the experiments is that extraction accuracy cannot be attributed solely to the model.
Consider this chain:
Parser
↓
Schema
↓
Prompt
↓
LLM
↓
Validation
↓
Evaluation
↓
ReviewA failure at any layer can result in incorrect business data.
A model cannot recover a table relationship destroyed by the parser.
A validator cannot detect every semantically wrong number.
A perfect extraction prompt cannot compensate for a missing schema field.
And a high benchmark score does not guarantee performance on unseen documents.
Therefore:
Document extraction reliability is a system-level property.
The Problem With Naive Accuracy Testing
This is one of the most important limitations discussed in the original article.
Imagine the workflow does this:
Document
↓
Generate schema
↓
Generate reference data
↓
Extract same document
↓
CompareThe resulting score can look excellent.
But the model has effectively been tested on material that was already part of the development process.
That is not a strong measure of generalization.
A better evaluation design is:
Development Set
↓
Schema + Prompt Development
↓
Frozen Configuration
↓
Held-Out Test Set
↓
Independent EvaluationThe test documents should ideally contain previously unseen layouts and realistic variations.
The article explicitly recommends separating documents used for schema and prompt development from a held-out test set.
This is the same basic principle used throughout machine learning:
Never confuse development performance with generalization performance.
One Accuracy Number Is Not Enough
Suppose an extraction system reports:
Accuracy: 94%
That sounds impressive.
But what exactly is 94%?
Was it:
- field-level accuracy?
- document-level accuracy?
- character accuracy?
- exact-match accuracy?
- value-level F1?
- accuracy after excluding difficult documents?
A single aggregate number can hide the real failure modes.
The article references a more granular evaluation strategy using order-insensitive unified value F1 for value accuracy, together with word-level and page-level grounding F1 for source traceability.
This is an important direction for serious document AI evaluation.
A production dashboard should ideally distinguish at least:
Extraction correctness
Schema correctness
Grounding / provenance
Coverage
Human-review rate
Processing cost
LatencyThat gives engineering teams much more useful information than one headline score.
Multi-Document Extraction Is a Different Problem
Not every extraction task is about one document.
Consider a purchase order where additional information about each item is stored in a separate bill of materials.
The documents may be connected through:
Material NumberNow the task becomes:
Purchase Order
+
Bill of Materials
↓
Match common identifiers
↓
Combine information
↓
Generate structured resultThis is no longer just document extraction.
It becomes a cross-document reasoning problem.
The original experiment found that the agentic workflow could not correctly construct the required schema for such a multi-document case.
That limitation is important because real enterprise workflows frequently depend on relationships across multiple files.
Traceability Is a First-Class Requirement
Imagine the system tells you:
Total Amount = $13,750Now the finance team asks:
Where did that number come from?
A trustworthy extraction system should be able to answer.
Ideally, every important value should include something like:
{
"value": 13750,
"source": {
"page": 3,
"location": "Grand Total"
}
}Or the interface should allow the user to click the field and see its source location.
This turns extraction from:
“The AI says this is the answer.”
into:
“The AI says this is the answer, and here is the evidence.”
That difference becomes extremely important when documents affect financial, legal, operational, or compliance decisions.
The source article highlights traceability as a valuable capability but notes that the tested beta implementation did not always point users to the exact correct location.
Human-in-the-Loop Should Be Designed, Not Added Later
It is tempting to think that a fully automated system is the ultimate goal.
In many real-world environments, that is the wrong target.
A better objective is:
Automate what is safe to automate and route uncertainty to humans.
For example:
Confidence ≥ 99%
↓
Automatic processing
Confidence 90–99%
↓
Optional review
Confidence < 90%
↓
Mandatory reviewThe exact thresholds should be determined using real evaluation data.
The human reviewer should be given:
- extracted value,
- source location,
- confidence,
- document preview,
- expected schema,
- and the ability to correct the result.
This makes human review fast rather than forcing someone to manually inspect the entire document.
An Important Missing Capability: Requirement-Driven Schema Generation
Another limitation highlighted in the article is that agentic schema generation can start from the document rather than from the actual business requirement.
That distinction matters.
Imagine a 300-page technical document contains hundreds of possible fields.
The system may identify many of them.
But your business may only care about 12.
So instead of:
Document
↓
Find everything
↓
Human deletes unnecessary fieldsa better system should support:
Business Requirement
↓
Required Fields
↓
Schema
↓
Document ExtractionFor example:
“Extract supplier information, PO number, delivery date, line-item quantities, unit prices, tax, and grand total.”
This requirement should drive schema creation.
The source article points to a generative AI toolkit called GAIK that uses this requirement-oriented approach for schema extraction and structured outputs.
Why Long Documents Make Schema Design Harder
Long documents introduce another scaling problem.
A document with hundreds or thousands of potential fields can produce an enormous candidate schema.
That means the user may spend more time removing irrelevant fields than defining the fields they actually need.
For long enterprise documents, schema generation should therefore be:
requirement-aware, hierarchical, and selective.
Instead of asking:
“What information exists in this document?”
ask:
“What information do I need from this document for this business process?”
That small change can dramatically reduce unnecessary extraction work.
A Production-Grade Architecture
Putting everything together, a robust document intelligence platform could look like this:
┌────────────────────┐
│ Document Sources │
│ │
│ Drive / Email / API│
│ SharePoint / S3 │
└─────────┬──────────┘
↓
┌────────────────────┐
│ Document Ingestion │
└─────────┬──────────┘
↓
┌────────────────────┐
│ Parsing / OCR / │
│ Vision Processing │
└─────────┬──────────┘
↓
┌────────────────────┐
│ Structure-aware │
│ Document Model │
└─────────┬──────────┘
↓
┌───────────────┴────────────────┐
↓ ↓
Requirement / Schema Document Context
Definition │
↓ │
┌───────────────┐ │
│ JSON Schema │◄───────────────────────┘
└───────┬───────┘
↓
┌───────────────┐
│ Extraction │
│ LLM / Agent │
└───────┬───────┘
↓
┌───────────────┐
│ Validation │
│ Rules + Types │
└───────┬───────┘
↓
┌───────────────┐
│ Grounding + │
│ Confidence │
└───────┬───────┘
↓
┌──────┴───────┐
↓ ↓
Trusted Uncertain
Output Output
↓ ↓
Database / Human Review
ERP / API ↓
↓ Corrections
└───────────────┘
↓
Evaluation Data
↓
Continuous Improvement
This architecture separates responsibilities and creates clear points where quality can be measured.
Practical Design Recommendations
When building your own extraction platform, several principles are worth keeping from the beginning.
Treat parsing as a core AI component
Do not choose a parser merely because it extracts text.
Measure whether it preserves the relationships your extraction task actually needs.
Version your schemas
A schema should have an explicit version.
For example:
purchase_order_v1
purchase_order_v2
purchase_order_v3This makes production changes auditable.
Keep prompts versioned too
A schema change without a corresponding prompt version can make results difficult to reproduce.
Store source evidence
Do not save only:
{
"amount": 13750
}Prefer:
{
"amount": 13750,
"page": 3,
"source_text": "Grand Total: 13,750.00"
}where your architecture permits it.
Build a human-review queue
Not every result should be automatically trusted.
Evaluate unseen documents
A model that performs well on development documents may still fail badly on new layouts.
Measure cost
A vision LLM may improve accuracy while dramatically increasing inference cost.
The correct architecture is therefore an optimization problem involving:
accuracy + latency + cost + traceability + review workload.
Monitor document drift
Suppliers change templates.
Business forms evolve.
Scanning quality changes.
New document variants appear.
An extraction system should therefore be monitored continuously rather than treated as a one-time deployment.
What Unstract Demonstrates Well
Based on the experiments described in the source article, Unstract demonstrates a useful end-to-end concept:
Document
→ Parser
→ Schema
→ Prompt
→ LLM
→ Structured Data
→ API / ETLThe agentic workflow adds another layer:
Document
→ Field Discovery
→ Field Reconciliation
→ Schema Generation
→ Prompt Generation
→ ExtractionThe experiments also show why human approval remains valuable, especially when dealing with diagrams, unusual layouts, and complex business requirements.
But Don’t Confuse a Successful Demo With Production Accuracy
This is perhaps the most important editorial conclusion from the entire experiment.
A system can successfully extract:
- a purchase order,
- an employment application,
- or a construction blueprint
and still not be production-ready.
Before deploying at scale, you need to know:
How does it perform on unseen documents?
What happens when the parser fails?
What happens when the document contains handwriting?
What happens when fields are missing?
What happens when the schema changes?
What happens when two documents need to be joined?
Can users verify the source of every important value?
How many documents require human review?
How much does every processed document cost?
What happens when the LLM or parser is upgraded?
Those questions matter more than a successful screenshot.
Final Takeaways
Reliable AI document extraction is no longer about writing one clever prompt.
It is about engineering a complete system.
The core pipeline is:
Parse
↓
Preserve Structure
↓
Define Requirements
↓
Generate / Approve Schema
↓
Extract
↓
Validate
↓
Evaluate
↓
Ground to Source
↓
Human Review Where Necessary
↓
Deploy
↓
MonitorThe lessons from the experiments are especially clear.
First, parsing matters.
Poor document representation can cause extraction failures before the LLM even sees the information.
Second, schemas matter.
A model cannot reliably extract fields that have not been clearly defined.
Third, validation is not the same as correctness.
Valid JSON can contain invalid business information.
Fourth, evaluation must use independent data.
Testing on documents used to generate the schema or reference data can create misleadingly optimistic results.
Fifth, source traceability matters.
People need to know not only what value the AI extracted, but where that value came from.
Sixth, multi-document workflows need separate consideration.
Linking information across purchase orders, bills of materials, contracts, invoices, and other records is a more complex problem than extracting fields from one document.
Finally, agentic systems can reduce configuration effort, but they do not eliminate the need for human oversight.
The long-term opportunity is not simply to build an AI that can read documents.
It is to build an AI system that can read documents, understand business requirements, produce structured information, prove where the information came from, recognize uncertainty, and integrate safely with business systems.
That is the difference between an impressive AI demo and a document extraction platform that an organization can actually trust.
Suggested Hero Image
Concept: A realistic enterprise document-processing pipeline showing messy PDFs, invoices, purchase orders, scanned forms, and blueprints entering an AI document extraction system, then flowing through parsing, schema generation, LLM extraction, validation, human review, and finally into a clean database/ERP. Modern enterprise AI architecture, professional technical illustration, no readable brand logos, 16:9.
Suggested Technical Diagram
MESSY DOCUMENTS
↓
OCR / PARSER / VISION
↓
STRUCTURE PRESERVATION
↓
BUSINESS REQUIREMENTS
↓
JSON SCHEMA
↓
LLM / AI AGENTS
↓
STRUCTURED OUTPUT
↓
VALIDATION
↓
GROUNDING + CONFIDENCE
↓
┌───────────────────┐
│ HIGH CONFIDENCE │ → AUTOMATION
├───────────────────┤
│ LOW CONFIDENCE │ → HUMAN REVIEW
└───────────────────┘
↓
DATABASE / ERP / APIEditorial note: I deliberately did not reproduce the source article’s wording, author bio, comment section, Medium promotional material, or recommendation feed. The technical substance has been retained and reorganized into an original article suitable for a personal/technical blog. The uploaded source contains the complete article text and the embedded screenshots/visuals referenced throughout the original publication.
@credit:
Umair Ali Khan, Ph.D
