Adding a Knowledge source, end to end
Adding a source runs through hashing, extraction, cleaning, scanning, contradiction checking and passage building before anything can be answered from it. Most sources arrive active in under a minute. A source that trips the scanner or contradicts approved material lands in needs_review instead, and waits for a person rather than quietly changing what Connect says.
The stages#
- Trigger — somebody chooses a file, or gives a public URL, on the Knowledge screen.
- User event — the upload carries the bytes with a title, an authority, a category and the scopes it should serve.
- Authentication and workspace resolution — the session fixes the workspace before anything is written; an operator session and a customer session reach the same service by different routes.
- Ingest — the declared type is checked against the file's first bytes, the size ceiling for that type is applied, and the bytes are written to a temporary file and hashed with SHA-256.
- Canonical record — a source row is created with status
processing. If those exact bytes are already a source here, the existing record is returned and nothing further runs. - Reasoning and extraction — text formats are decoded directly; PDF, image and video are read by a model under a fixed instruction that treats the file as data and quotes any embedded instructions rather than obeying them.
- Knowledge, memory and rules — the text is cleaned of invisible and control characters, scanned for instruction patterns, and, for fact, instruction and playbook authority, compared against approved material for contradictions.
- Autonomy and approval — warnings or conflicts move the source to
needs_review. Warnings can be approved by a person with a note; a conflict has to be resolved in the documents themselves. - Action — passages are written: heading-aware, about 1,800 characters with 180 characters of overlap, at most 240 of them, each with an optional embedding.
- Result — the source becomes
active, carrying a passage count, an indexed timestamp and a summary of its opening. Anything unresolved leaves it inneeds_reviewand it answers nothing. - Relationship and memory — nothing is written to memory. Knowledge is supplied truth and does not become something Connect learned.
- Audit and usage — the ingest is logged as a success or a warning with the source, its type, its authority and the counts of warnings and conflicts; model-based extraction is metered against the workspace's AI budget.
What you see, and what can fail#
| Stage | What you see | What changes | What can fail |
|---|---|---|---|
| Ingest | The upload progressing | A temporary file and a hash | Over the size ceiling; the file signature contradicts the extension; an unsupported type |
| Canonical record | The source appears, marked processing | A row with authority, category and scopes | Nothing — but an identical file silently returns the existing source |
| Extraction | Processing, for anything from seconds to minutes | The extracted text on the record | No usable text; the media timeout; an exhausted or missing model key |
| Scanning | Warnings listed on the source | Warnings and conflicts stored | Nothing fails here; findings park the source instead |
| Passage building | A passage count appears | Passage rows, replacing any previous set | Text longer than the extraction cap is truncated before this point |
| Result | active, or needs_review with a reason | The source becomes readable by retrieval, or not | Activation is refused while a conflict stands |
Verifying it actually worked#
Check the status and the passage count on the source.
Result
activewith a plausible count is the minimum. A twenty-page PDF that produced two passages did not extract properly.Read the extracted text, or at least the summary.
Result This catches the failure nothing else catches — a document that parsed cleanly into the wrong content, such as a scan read as layout rather than prose.
Search for a question a customer would ask, in the scope that suits the work.
Result The new source should appear with a section name you recognise. If it does not, the usual causes are scope and authority rather than the text.
For anything commercial, ask the question in a scope where a commitment would be needed.
Result Presence is not sufficiency. A source can rank first and still sit below the bar for stating a price or a warranty.
Doing all four takes a couple of minutes and catches the two failures that otherwise surface as a customer complaint: material that is present but unreachable, and material that is reachable but not allowed to prove anything.
Idempotency and repeating the work#
Adding the same bytes twice is safe: the hash matches and the existing source comes back untouched. Adding a slightly edited file is not the same operation — it produces a second source that competes with the first, which is the most common way a workspace ends up with two answers to one question. Re-index the original, or delete it, rather than uploading a revision alongside it.
Re-indexing replaces every passage derived from a source and re-runs the scans. It is the right response to a changed extraction model, a fetched page that has been updated at its origin, or a source whose passages look wrong. It does not re-fetch a URL by itself.
Questions#
How long should this take?
A text or Markdown file is effectively instant. A PDF, an image or a video depends on the model reading it, and the media timeout defaults to six minutes — a source still processing after that will end as error rather than hanging indefinitely.
Can I add a source and set its authority later?
Yes, but changing authority re-runs the trust checks rather than simply saving a value, so a source that was quietly fine as reference can land in needs_review when promoted. Doing it deliberately, with time to read the result, beats doing it on the way to a meeting.
What if extraction produced text but it reads badly?
Fix it at the source. Re-export the document with a real text layer, or supply the part that matters as Markdown. Passages are built from what extraction returned, so a poor read is baked into every answer afterwards.