# Adding a Knowledge source, end to end

Adding a source runs through hashing, extraction, cleaning, scanning, contradiction checking and passage building before anything can be answered from it. Most sources arrive `active` in under a minute. A source that trips the scanner or contradicts approved material lands in `needs_review` instead, and waits for a person rather than quietly changing what Connect says.

- **Status:** Available
- **Audience:** both
- **In the app:** #/knowledge-sources-advanced
- **Last verified:** 2026-09-10
- **Canonical:** https://connectbyjbrh.com/docs/workflows/add-knowledge/

## The stages

1. Trigger — somebody chooses a file, or gives a public URL, on the Knowledge screen.
2. User event — the upload carries the bytes with a title, an authority, a category and the scopes it should serve.
3. Authentication and workspace resolution — the session fixes the workspace before anything is written; an operator session and a customer session reach the same service by different routes.
4. Ingest — the declared type is checked against the file's first bytes, the size ceiling for that type is applied, and the bytes are written to a temporary file and hashed with SHA-256.
5. Canonical record — a source row is created with status `processing`. If those exact bytes are already a source here, the existing record is returned and nothing further runs.
6. Reasoning and extraction — text formats are decoded directly; PDF, image and video are read by a model under a fixed instruction that treats the file as data and quotes any embedded instructions rather than obeying them.
7. Knowledge, memory and rules — the text is cleaned of invisible and control characters, scanned for instruction patterns, and, for fact, instruction and playbook authority, compared against approved material for contradictions.
8. Autonomy and approval — warnings or conflicts move the source to `needs_review`. Warnings can be approved by a person with a note; a conflict has to be resolved in the documents themselves.
9. Action — passages are written: heading-aware, about 1,800 characters with 180 characters of overlap, at most 240 of them, each with an optional embedding.
10. Result — the source becomes `active`, carrying a passage count, an indexed timestamp and a summary of its opening. Anything unresolved leaves it in `needs_review` and it answers nothing.
11. Relationship and memory — nothing is written to memory. Knowledge is supplied truth and does not become something Connect learned.
12. Audit and usage — the ingest is logged as a success or a warning with the source, its type, its authority and the counts of warnings and conflicts; model-based extraction is metered against the workspace's AI budget.

## What you see, and what can fail

| Stage | What you see | What changes | What can fail |
|---|---|---|---|
| Ingest | The upload progressing | A temporary file and a hash | Over the size ceiling; the file signature contradicts the extension; an unsupported type |
| Canonical record | The source appears, marked processing | A row with authority, category and scopes | Nothing — but an identical file silently returns the existing source |
| Extraction | Processing, for anything from seconds to minutes | The extracted text on the record | No usable text; the media timeout; an exhausted or missing model key |
| Scanning | Warnings listed on the source | Warnings and conflicts stored | Nothing fails here; findings park the source instead |
| Passage building | A passage count appears | Passage rows, replacing any previous set | Text longer than the extraction cap is truncated before this point |
| Result | `active`, or `needs_review` with a reason | The source becomes readable by retrieval, or not | Activation is refused while a conflict stands |

> **Careful** Every failure at extraction leaves the source `error` with the reason recorded, and indexes nothing. That is deliberate: half a document answers confidently about the half it has.

## Verifying it actually worked

1. Check the status and the passage count on the source.
   - Result: `active` with a plausible count is the minimum. A twenty-page PDF that produced two passages did not extract properly.
2. Read the extracted text, or at least the summary.
   - Result: This catches the failure nothing else catches — a document that parsed cleanly into the wrong content, such as a scan read as layout rather than prose.
3. Search for a question a customer would ask, in the scope that suits the work.
   - Result: The new source should appear with a section name you recognise. If it does not, the usual causes are scope and authority rather than the text.
4. For anything commercial, ask the question in a scope where a commitment would be needed.
   - Result: Presence is not sufficiency. A source can rank first and still sit below the bar for stating a price or a warranty.

Doing all four takes a couple of minutes and catches the two failures that otherwise surface as a customer complaint: material that is present but unreachable, and material that is reachable but not allowed to prove anything.

## Idempotency and repeating the work

Adding the same bytes twice is safe: the hash matches and the existing source comes back untouched. Adding a slightly edited file is not the same operation — it produces a second source that competes with the first, which is the most common way a workspace ends up with two answers to one question. Re-index the original, or delete it, rather than uploading a revision alongside it.

Re-indexing replaces every passage derived from a source and re-runs the scans. It is the right response to a changed extraction model, a fetched page that has been updated at its origin, or a source whose passages look wrong. It does not re-fetch a URL by itself.

## Questions

### How long should this take?

A text or Markdown file is effectively instant. A PDF, an image or a video depends on the model reading it, and the media timeout defaults to six minutes — a source still processing after that will end as `error` rather than hanging indefinitely.

### Can I add a source and set its authority later?

Yes, but changing authority re-runs the trust checks rather than simply saving a value, so a source that was quietly fine as `reference` can land in `needs_review` when promoted. Doing it deliberately, with time to read the result, beats doing it on the way to a meeting.

### What if extraction produced text but it reads badly?

Fix it at the source. Re-export the document with a real text layer, or supply the part that matters as Markdown. Passages are built from what extraction returned, so a poor read is baked into every answer afterwards.

## Related

- [Knowledge sources](https://connectbyjbrh.com/docs/knowledge/sources/)
- [Extracting text from a source](https://connectbyjbrh.com/docs/knowledge/extraction/)
- [How Knowledge is prepared for retrieval](https://connectbyjbrh.com/docs/knowledge/indexing/)
- [Ingesting a Knowledge source, end to end](https://connectbyjbrh.com/docs/workflows/knowledge-ingest/)
- [A Knowledge source failed to process](https://connectbyjbrh.com/docs/troubleshooting/source-failed/)
- [When two sources disagree](https://connectbyjbrh.com/docs/knowledge/source-conflict/)

## What this page is based on

- `backend/app/knowledge_bank.py` — ingest, extraction, scanning, chunking and status
- `backend/app/api.py` — the upload and URL endpoints
- `docs-source/sources/GENERAL.md` §9 — the shared file service and refused constructs
- Connect capability registry (docs-source/facts.py)