Introducing OpenOCR Stratify

Why document structure matters

The page below comes from the USGS report Status of Water-Quality Conditions in the United States, 2010–20. It has two text columns, a section heading, and a table with a caption and abbreviation notes. The document is real: every image here is rendered from the published PDF. We have drawn the region boundaries manually to explain the proposed workflow.

Read across both columns as though they were one paragraph and unrelated sentences become neighbors. Detach the table from its notes and abbreviations lose their definitions. Flatten its indented rows and a relationship between a category and its entries can disappear. A list of correctly recognized words does not preserve these distinctions. Select the highlighted regions below to inspect the original pixels.

These are practical problems for applications built on extracted documents. Search needs passages with coherent boundaries. A financial workflow needs to distinguish a total from a nearby subtotal. An assistant answering questions about a paper needs the qualifications in a caption as well as the values in a chart. A clean-looking transcription can conceal a structural error until someone relies on it.

The same issue becomes more difficult across a long document. A report may start as a digital PDF, contain several scanned appendices, and end with photographs of signed forms. Processing every page in the same way is convenient, but it does not necessarily match the material. Stratify is intended to preserve enough information about each page and region to make a more deliberate choice possible.

How Stratify would process a document

The workflow starts with the source page and ends with a document someone can inspect. Each part has a concrete purpose. A provider may already cover several of them in one request, so these are not four mandatory model calls.

Find the regions. Inspect the document’s pages, orientation, image quality, and usable embedded text. Locate the paragraphs, columns, tables, captions, and other regions, retaining their positions and relationships. That gives the rest of the workflow a map of what needs to be read.

Choose the readers. Use the customer’s configured engines for the identified regions. A customer might pair a layout detector with a vision model, or keep an existing provider’s paragraph extraction and choose a different reader for the tables. The processing plan needs to make those choices and their limits explicit.

Extract the content. Prepare each selected reader’s input at a suitable resolution, including neighboring context where it matters. Run the recognition step, retain useful existing extraction, and record which source region and engine each result came from.

Reassemble the document. Restore reading order, reconcile overlapping crops, and reconnect headings, captions, notes, and tables. Deliver usable text or structured data with source references and visible gaps where processing is incomplete.

If a provider already returns a useful table with cell relationships and page coordinates, Stratify should retain that work. Repeating a successful extraction adds cost without a clear purpose. The development documentation describes the proposed behavior and integration boundaries in more detail.

Reading in pieces without losing the whole

In the annotated USGS page, P1 and P2 identify the main text columns. H1 is a heading, P3 its paragraph, C1 the table caption, N1 the explanatory notes, and T1 the table itself. These are editorial identifiers for this example. In a working pipeline, a layout detector would supply region locations and likely types, giving later steps a stable way to refer to each part.

Preparing T1 requires a decision about context. Its column headers and cells fit inside one rectangle, but the definitions of MCL, HBSL, and other abbreviations sit above it in N1. The caption in C1 explains what the table describes. Including all three regions gives a reader more context without sending the entire page. Toggle the example above to compare both crops; the underlying document pixels stay unchanged.

The system also needs to know how a crop relates to the original. Rendering can change resolution; pages can be rotated; providers can report positions using different coordinate systems. A result associated only with the final crop image is not enough for a user who wants to highlight the corresponding area of the original PDF. That transformation needs to remain traceable.

Overlapping crops create another assembly problem. Including surrounding text can help a reader interpret a table, but that text should not appear twice in the final document. Context sent to a model and content selected for output are related, distinct decisions. The region representation needs to preserve that distinction.

Some failures cannot be fixed by preparing a different input. Enlarging a poor scan does not restore a missing digit. A page cut off at the edge of a photograph may remain incomplete. In those cases, the useful result is an honest indication of uncertainty and a link to the source, rather than a confidently completed value.

Combining a segmenter with an OCR or vision model

A segmenter answers a question about layout: where are the parts, and what roles do they appear to have? A recognition engine answers a question about their content. Some providers handle both. Others offer a dedicated layout detector or a model that is particularly useful for a certain kind of input.

Stratify would support explicit combinations. A customer could select one detector and one recognition model for all regions, or keep a provider’s paragraph extraction and send selected tables or annotations to another reader. The blank IRS form below shows why the input matters: address fields, checkbox choices, a dependents table, and numbered income rows are different structures on the same page.

A table reader might preserve the dependents column headings and entry cells together. A form reader would need to retain the relationship between a checkbox and its label. A text reader could handle the margin instructions separately. This form is blank; a faithful result would preserve empty fields rather than inventing values. Asking an LLM to explain a form is a different operation from transcribing its contents, and the workflow should make that distinction explicit.

The coordination is the part we want OpenOCR to supply: preparing the inputs, sending each to the configured engine, recording what completed, and keeping the results connected to their source regions. A customer should not have to build a separate cropping and reconstruction service just to test a segmenter with a preferred reader.

This does not assume that specialized combinations will always win. A document model that receives the whole page may understand context that a crop leaves out. A general vision model may already read a particular table accurately. We need to evaluate the actual combination on representative inputs, including the work required to prepare and assemble them.

Returning a document you can inspect

The results need to come back as a document, not an unordered collection of model responses. In the USGS example, P1 precedes P2, H1 belongs with P3, and the table retains links to its caption and abbreviation notes. The identifiers carry through so a user can compare any part of the extraction with the original page.

That representation is also the foundation for selective retries. Suppose the paragraphs are acceptable but a table cell is wrong. A user could select T1, choose another engine, and request another pass. Its replacement would need to be reconciled with the accepted document while leaving the accepted paragraphs, caption, and notes in place. A retry should not silently rewrite unrelated content.

Keeping a history of that decision would help explain the result. Which engine produced the first version? What input did the second engine receive? Did the replacement include a different crop or more context? These are useful questions when comparing alternatives, investigating an error, or deciding whether a new result is trustworthy enough for the next step.

A failed region should also remain visible. If a provider times out while reading T1, the system may still have usable paragraph results. Returning those as a complete document would hide the missing work. The planned result needs per-region completion information so the application can distinguish a partial extraction from a finished one and decide whether to retry or ask for review.

The exact response schema is still being designed. What matters at this stage is preserving the information required for these behaviors. Flattening everything into a single text string too early would make selective replacement and inspection much harder to introduce later.

Building on provider capabilities

OpenOCR already supports provider-native PDF processing for compatible engines and page rendering for engines that accept images. Some providers also return their own layout or document analysis. Stratify would extend those foundations with a shared representation for regions and relationships across engines.

There are useful capabilities to build on. Mistral OCR documents structured document output, including layout information. AWS Textract offers analysis of tables, forms, and layout. PaddleOCR provides a dedicated layout detector that is a candidate for pairing with a separate recognition model.

These responses are not interchangeable without translation. A provider’s “block” may represent a line, a paragraph, or a table. Coordinates may be normalized or expressed in another unit. A table can carry cell positions, merged cells, and relationships that would disappear if we kept only its text. The common format needs to retain those distinctions.

That is why the first planned step is preserving useful provider structure. Before introducing another recognition pass, we need to establish what the existing result contains and whether it can support region selection and assembly. Supporting a provider in OpenOCR today should not be taken to mean that every advanced operation from that provider is already exposed.

Adapters will need to be checked against the relevant operation and model version. A list of text chunks is not the same thing as a set of regions with coordinates. The integration has to establish what can be traced to the page, what can be reused, and what would require another detector or recognition pass.

How we plan to evaluate it

A longer pipeline introduces more places for an error to occur. A detector can miss a footnote, a crop can remove a qualification, and assembly can duplicate a sentence. The evaluation needs to consider the document that comes out of the entire workflow, not just the accuracy of a model on isolated regions.

We plan to compare proposed Stratify combinations with individual providers on the same inputs. The set needs documents that expose different problems: multi-column papers, dense tables, small captions, mixed digital and scanned pages, equations, and annotations. A simple letter should also be represented, since extra processing needs to justify itself on ordinary documents as well as difficult ones.

Text accuracy is one measure. Reading order, missing content, duplicate regions, and table relationships matter too. For the USGS table, recognizing a value while attaching it to the wrong constituent is still an incorrect result. For the blank form, turning an empty checkbox into a selected answer would be another kind of failure. A score that overlooks those relationships would give an incomplete picture of whether the pipeline helped.

Elapsed time and total cost belong in the same evaluation. A crop may be small but still involve rendering, detection, another provider request, and reconstruction. A selective retry is useful only if it produces an acceptable result without repeating unnecessary work. The plan should also account for retries that fail, rather than reporting only the cost of a successful final call.

We have not published Stratify benchmark results, pricing, or claims of performance gains. Confidence information needs its own evaluation: scores returned by different providers may describe different things and cannot simply be averaged. Until those signals are understood, customer-selected processing and visible uncertainty are a more defensible starting point than automatic routing based on a single score.