Connect by JBRH Open Connect

IMAP

IMAP is the protocol for reading a mailbox that stays on the server. The client browses folders, fetches whole messages or single parts, and sets flags such as \Seen. Unlike POP3 it does not download and delete, which means the client has to keep an accurate idea of where it had got to — and that cursor is where mail is lost.

Status
Reference What this means
Audience
both, developer
Last verified
Product version
6.3.2

The identifiers, in the order they trip people up#

Folder (mailbox)
A named container. Hierarchy uses a server-chosen delimiter, often / or ., which the client must ask for rather than assume.
Sequence number
A message's position in the folder *right now*. It changes whenever anything is expunged. Never store one.
UID
A number that is stable for the lifetime of the folder and always increases. This is the one to store.
UIDVALIDITY
A folder-level token. If it changes, every UID you cached is meaningless and the client must resynchronise from scratch.
UIDNEXT
The UID the next arriving message will probably get. Useful as a cheap 'has anything arrived' check.
MODSEQ
With CONDSTORE, a modification counter that lets a client ask what changed since a point — including flag changes, which UIDs alone cannot express.

The UIDVALIDITY rule is the one that causes silent data problems. A folder recreated with the same name is a different folder as far as the protocol is concerned; a client that keeps using its cached UIDs will read the wrong messages or none at all, and nothing about the connection looks broken while it happens.

What a client must remember, and where that goes wrong#

A correct IMAP client keeps state on disk: the folder list, UIDVALIDITY per folder, the highest UID it has processed, and the flag state it believes each message has. That set is the cursor, and the cursor is a liability as much as an asset.

FailureSymptomWhy it is hard to spot
Cursor advanced before processing succeededMessages skipped permanentlyNothing errors. The mailbox looks quiet
Cursor advanced on a refusalA block of mail is never readThe refusal was handled; the loss was the side effect
UIDVALIDITY change ignoredWrong messages processed, or noneThe connection is healthy and the folder exists
Flags cached and not revalidatedMessages reprocessed, or missed as already seenAnother client changed them; nothing told you
Cursor stored per connectionEverything reprocessed after a restartCorrectness is preserved and cost is not

Does Connect use IMAP?#

Used, as one of three mail transports. A mailbox can be connected as Gmail through the Gmail API, as Microsoft Graph, or as any IMAP/SMTP server. providers.build() reads only the mailbox's transport and config, so the choice is per-mailbox and invisible downstream.

Two design decisions in Connect are direct answers to the failures above.

  • The cursor does not advance on a refusal. When the workspace's daily allowance is spent, mail is held rather than dropped and the read position deliberately stays where it is. The refusal is shown in Needs You, so the held work is visible rather than silent, and the messages are read once the allowance permits.
  • Provider data and canonical data are separate. What the adapter fetched is stored in tenant_* tables in the provider's own shape; the canonical threads, messages and contacts are what the engine and every screen read, with connect_core.bridge_* copying between them. A bug in one adapter therefore cannot corrupt a workspace's conversation history, and adding a transport changes nothing downstream.

Health is judged separately from connectivity. A mailbox that authenticates and returns nothing is recorded as a quiet mailbox — a health signal rather than an error — because "connected" and "working" are different claims, and an IMAP session can be perfectly healthy while pointed at a folder nothing arrives in.

Efficiency, and the extensions worth asking for#

  1. IDLE — the server pushes a notification when something changes, instead of the client polling. Cheaper for both sides; needs a connection kept open and re-established on drop.
  2. CONDSTORE and QRESYNC — ask for changes since a MODSEQ, including flag changes and expunges. This is what makes resynchronisation cheap rather than a full folder scan.
  3. Partial FETCH — retrieve BODYSTRUCTURE first, then only the parts you need. Fetching headers and a text part beats fetching a message with a 20 MB attachment you will not read.
  4. FETCH in batches — one command for a UID range rather than one command per message. The round trips dominate on a folder of any size.

Not every server offers these, and a client must read the CAPABILITY response rather than assume. A shared-hosting IMAP server with no CONDSTORE forces full scans, and that is a property of the mailbox worth knowing before blaming the client for being slow.

Questions#

Why must a client store UIDs rather than sequence numbers?

Sequence numbers describe positions in the folder as it is at this instant, and they shift whenever a message is expunged. A stored sequence number points at a different message after any deletion. UIDs are stable for as long as UIDVALIDITY is unchanged, which is precisely the guarantee a cursor needs.

What should happen when UIDVALIDITY changes?

Discard every cached UID for that folder and resynchronise. It is not a failure — a folder can be recreated for entirely ordinary reasons — but treating the old UIDs as valid produces wrong results rather than errors, which is much more expensive to discover.

Is IMAP slower than a provider API?

For incremental synchronisation, usually yes, because a provider API can return a change list directly while IMAP without CONDSTORE must compare state. What IMAP gives in exchange is that it works with any server, including one no vendor supports, and it is the reason a mail product can connect a mailbox it has never heard of.