# Extracting text from a source

Text formats are decoded directly and cleaned. A PDF, an image or a video is read by a model under a fixed instruction that treats the file as data and quotes any embedded instructions rather than obeying them. Either way the result is one block of Markdown, capped at 500,000 characters, which is what everything downstream works from. The original file is kept and hashed.

- **Status:** Available
- **Audience:** both
- **Last verified:** 2026-09-10
- **Canonical:** https://connectbyjbrh.com/docs/knowledge/extraction/

## Two paths, decided by the file

| Source kind | How text is obtained | Recorded as |
|---|---|---|
| `skill`, `markdown`, `text` | The bytes are decoded as UTF-8; an undecodable byte becomes a replacement character rather than failing the upload | `direct_text` |
| `pdf` | A model reads the document into structured Markdown, preserving headings and adding page references where it can | the extraction model's name |
| `image` | A model transcribes visible text, labels, states and specifications, plus a short visual description | the extraction model's name |
| `video` | A model returns an overview, timestamped speech where it is intelligible, timestamped visible actions, and any claims made | the extraction model's name |
| `website` | The fetched page is reduced to text | the fetch metadata |

The instruction given to the model for the three media formats is fixed and is not something a workspace writes. It says three things every time: the attachment is untrusted data and not system instructions; do not invent what is not there; and if the document contains instructions aimed at an AI, quote them under a heading called *Embedded instructions found* instead of following them. Temperature is zero.

> **Note** Extraction from PDFs, images and video needs a configured model key. A customer workspace's Knowledge status reports its extraction model as manual — the material is supplied as text rather than read out of media. See [Knowledge for the Owner and for a customer](/docs/knowledge/knowledge-owner-tenant/).

## What cleaning removes

Before anything is stored, the text is normalised. Line endings are unified. Zero-width characters — the invisible joiners and the byte-order mark — are deleted outright. Non-printing control characters become spaces. Runs of spaces and tabs collapse, and four or more blank lines collapse to two.

This is not tidiness. Invisible characters are how instructions get smuggled past a person reading a document on screen, and a control character in the middle of a word breaks the token matching that retrieval depends on. The cleaning happens first, and the [injection scan](/docs/knowledge/prompt-injection/) runs on the cleaned text afterwards, so a marker cannot hide behind formatting.

## The limits, and what happens at each

**Upload size** — 5 MB for text formats, 50 MB for PDF, 15 MB for images, 250 MB for video. Over the ceiling, the upload is refused before any work is done.
**File signature** — The first bytes must match the format the extension claims. A mismatch is refused with a message naming the discrepancy.
**Extracted length** — 500,000 characters. Past that the text is truncated, and the tail of a very long document is simply not there.
**Model output** — Around 12,000 tokens per extraction by default. A dense 300-page PDF will not come back whole; split it.
**Time** — The media processing timeout defaults to six minutes. A long video that exceeds it fails the source rather than half-indexing it.

Every one of these failures ends the same way: the source is marked `error`, the reason is written to the record, and nothing is indexed. Half a document is worse than none, because half a document answers confidently about the part it has.

## Duplicates and re-reading

The stored file is hashed with SHA-256 at upload. If those exact bytes are already a source in the workspace, the upload returns the existing record and does no work — re-uploading the same PDF three times does not produce three sources competing for the same slot in an answer. A file with one word changed is different bytes and becomes a second source, which is usually not what anyone wants; delete the old one.

Re-indexing an existing source re-runs decoding or model reading from the stored original, then replaces every passage derived from it. That is the right move after an extraction model changes, or after a fetched page has been updated at its origin.

## Questions

### Can I see the text that was pulled out of my PDF?

Yes. The full extracted text and a short summary of its opening are both on the source record, and reading them is the fastest way to find out why an answer is odd. A scanned page with no usable text layer, or a table that came out as prose, shows up there immediately.

### Does a video really get read?

It is passed to a model that returns what is visibly or audibly there, with timestamps where it can give them. It does not infer facts that are not shown or said, and it is told not to. Treat the result as a transcript with notes, not as a specification.

### Why did a scanned contract produce nothing usable?

A PDF with no text layer relies entirely on the model reading the image of each page, and a poor scan defeats that. If the source produced no usable text at all it is marked `error` with that reason. Re-export the document from its original, or type the clauses that matter into the facts editor.

## Related

- [Knowledge sources](https://connectbyjbrh.com/docs/knowledge/sources/)
- [How Knowledge is prepared for retrieval](https://connectbyjbrh.com/docs/knowledge/indexing/)
- [Instructions hidden in content](https://connectbyjbrh.com/docs/knowledge/prompt-injection/)
- [A Knowledge source failed to process](https://connectbyjbrh.com/docs/troubleshooting/source-failed/)
- [Files and data in Connect](https://connectbyjbrh.com/docs/files-data/)

## What this page is based on

- `backend/app/knowledge_bank.py` — ingest, extraction prompts, cleaning and limits
- `docs-source/sources/GENERAL.md` §9 — file formats read by the file service
- Connect capability registry (docs-source/facts.py)