Document Inventory

Text extraction from a PDF is an old, solved, boring problem. This is the other half of it, and it stayed unsolved for much longer: the illustrations. A manual has a photograph of a control panel on page eight; a lecture deck has a Rembrandt on slide twelve; a textbook has a fibre-classification table on page thirty. Extract the text and all three vanish. What comes back is prose with holes in it, and the holes are frequently where the meaning was.

An inventory is what you get instead. The document comes back as units — pages, or slides — and each unit carries its text plus every visual thing on it as a separate, addressable element: an id like page.7.img.1, a bounding box in points, the extracted bytes, a hash of those bytes, and the text that sits nearest it on the page. The document stops being a file and becomes a structure you can hold a conversation about.

The identifier is the product. Everything else in the inventory is recoverable by running the parse again; the id is the thing an application stores, in a database, for months, and expects to still mean the same picture when it comes back. That single requirement shapes almost every decision underneath.

The obvious way to number the pictures in a PDF is to list the page's image resources and count them. It is also wrong. That list is unordered by specification, and it de-duplicates: an image painted five times in five places appears once. A caller who says 'the second figure' means the second one they can see, not the second entry in a dictionary they have never seen.

So elements are paint-order occurrences instead: the parser walks the page's drawing instructions the way a renderer would, and each act of painting gets its own id. An image drawn five times is five elements, because five is how many a reader sees. Nested drawing containers are walked too, with their transformation matrices composed, so a figure reachable only through two levels of indirection still lands at the coordinates it actually occupies.

That gives the same file the same identifiers on every parse. What it cannot give is a promise across versions — no PDF library guarantees identical enumeration after an upgrade. So instead of pretending, the inventory records which library produced it, at what version, and a hash of every extracted asset. Drift you can detect is honest; drift you cannot is the failure this exists to prevent. Adding a capability later — captions, chrome detection, region finding — is always additive: fields appear on elements that already exist, and nothing is ever renumbered.

There is a category of document that defeats all of this quietly. A design-led technical manual does not embed photographs of its product; it draws them, with tens of thousands of vector strokes. Parse one and you get an inventory with no figures at all — which is true, since there are no embedded images, and completely misleading, since the document is nothing but figures. One 135-page manual in our reference set returned zero elements while carrying nearly four hundred thousand drawing operations.

Two things follow. The first is a detector that clusters the drawing operations into figures — and the filters matter more than the clustering, because most vector paths in a document are decoration: a page-covering background rectangle, a table's ruling lines, an underline. The same manual now yields nearly eight hundred addressable figures. The second, and more important, is that the parser counts the drawing operations for free while it is already walking the page, so an inventory reporting no figures can also report that the page is covered in drawing, and name the fix. An empty answer that explains itself is worth more than a clever one that does not.

A slide deck's every page carries the master's background panel, a footer stamp and a logo. Describing the same background thirty-six times is money spent to learn nothing. So the inventory marks what recurs: the same bytes, or the same box, appearing across three or more pages is furniture rather than content. On a fifty-six-slide lecture deck that is ninety-three of two hundred and twenty-nine occurrences.

Three pages, not two — and the number was measured, not chosen. On an art-history deck, twenty images appear on exactly two pages, and seventeen of them are real artworks, because the deck reveals a slide and then repeats it with an annotation. A rule of 'appears more than once' would have filed Lidell Sawyer's In the Twilight as page furniture and called it a forty-two per cent saving. The residual is honest in the right direction: chrome appearing on only two pages is missed, and a missed logo costs one model call while a suppressed artwork costs the reader the thing they came for.

Nothing is ever deleted on the strength of that verdict. A furniture element keeps its id, its box, its bytes and its caption, and gains a record of why it was classed and by which rule. Removing it would renumber everything after it, and silent removal would make 'filtered' impossible to tell apart from 'absent'.

A photograph of a shoe is visibly a shoe. That it is Untitled (Shoe), a gelatin silver print from 1974, is not recoverable from the pixels at any resolution — it exists only in the credit line beside it. So the inventory keeps two fields and never merges them: what the document asserts about a figure, read out of the nearest credit block and traced to the exact words it came from; and what a model saw when it looked at the picture. A wrong identity traces back to the text that claimed it. A wrong description traces back to the model that read it. A blended string traces back to neither, which is why it is not offered.

The same discipline runs through the whole structure. Pixels we rendered ourselves are labelled as rendered on every asset, stated rather than left to be inferred from a missing field. A figure the parser deliberately skipped carries the reason on itself, so a forty per cent drop in described elements reads as a cost saving rather than a hole. The consistent rule: a reduction in coverage that is not visible in the response is a defect, even when the code is working exactly as written.

A PDF stores text as runs placed at coordinates — which is the output of the layout algorithm, not its input. For a left-to-right document the two happen to coincide and nobody notices. For an Arabic one they do not: read the runs in the order they sit on the page and you get the sentence backwards. Recovering the reading order is the inverse of one rule in the Unicode bidirectional algorithm, and it is worth being precise that it is the inverse — the algorithm itself runs the other way, because it is what a renderer needs and the renderer has already run.

Then there is a subtler one. Arabic letters frequently arrive as presentation forms — the contextual shapes a font draws, stored as separate codepoints. They render identically to the letters anyone would type and they are not the same characters, so searching an extracted document for a word plainly visible on the page finds nothing. Over half the characters in one report were affected. Folding them back to base letters costs nothing and is the difference between text and an image of text. The temptation is to reach for blanket Unicode normalisation, which also rewrites a trademark sign into two letters and a fraction into three characters — so the fold is restricted to the blocks that exist to hold glyph shapes, and every character an author actually chose is left as written.

A table drawn with ruling lines is found exactly, with its grid and its cell text. A table laid out with nothing but whitespace is invisible to line-based detection, and rather than guess at one, the inventory counts the gap and reports it. A map that covers its whole page is a single path covering a whole page, which is geometrically indistinguishable from the background rectangle the filters exist to remove — so it is filtered, and the response says which rule fired. The grammar that reads credit lines knows English conventions only; on an Arabic art book it finds nothing and asserts nothing, which is correct behaviour and an open gap at the same time.

None of that is a snag list. Every one of those limits was found the same way — by pointing the parser at a document nobody had tried before — and that is now the expectation rather than the surprise. Four reference documents produced four distinct failures; a thirty-document library produced two more. So the tooling includes a triage pass that reads a folder and flags the documents it handles badly, and the promise made is not that documents work. It is that a document which does not work says so.

In Gateway, a document becomes an inventory in one call:

POST /v1/media/process
{"source": {"key": "…/manual.pdf"},
 "ops": [{"op": "regions"},                    // figures the document DRAWS
         {"op": "render", "dpi": 150},         // …made visible
         {"op": "describe", "targets": "regions"}]}

# every element comes back with an id, a box, a hash and a URL —
# and page.7.img.1 still means the same picture next year

The parse step is free and local — no model, no network, no GPU. Detection and rendering are free too, and slow enough on a drawing-dense document that they are never run inside a request. Only the describing costs anything, and only for the elements you point it at.