Unicode, scripts and transliteration
A language is what somebody speaks; a script is the set of characters it is written in. One language can be written in several scripts and one script carries many languages, so a transcript of the same conversation can come back in different characters without the language having changed. Text comparison has to account for that, and naive comparison does not.
Four layers people flatten into one#
- Bytes
- The encoded form. UTF-8 is the sane default; its length in bytes tells you nothing about its length to a reader
- Code points
- The characters Unicode defines. Still not what a person calls a character
- Grapheme clusters
- What a reader perceives as one character — a base letter with its marks, an emoji with a modifier. This is the unit for cursor movement and truncation
- Script
- The writing system the characters belong to: Latin, Devanagari, Arabic, Tamil, Han and dozens more
Truncating a string by bytes splits a character. Truncating by code points splits a grapheme cluster and can change what the text means or leave a stray combining mark. Truncating by grapheme cluster is what a person expects, and it is the only one of the three that needs a library rather than a slice.
Normalisation, folding and sorting#
The same visible text can have more than one encoding: a letter with an accent may be one code point or a base letter followed by a combining mark. They look identical and compare unequal. Normalisation converts to a canonical form — composed or decomposed — and comparing text without normalising it first is a bug that appears only for some inputs, which is the worst kind.
Case folding is not universal either. Whether a capital letter maps to the lower-case form you expect depends on the language, and the Turkish dotted and dotless *i* are the standard example of a case operation that is wrong unless it knows what language it is folding.
Sorting is a third, separate decision. Alphabetical order is defined by the locale, not by the character encoding, and code-point order is not alphabetical order in any language including English.
Transliteration is not translation#
Transliteration rewrites text into a different script while keeping the same language: Hindi written in Latin letters is still Hindi. Translation changes the language. Confusing the two produces confident nonsense in both directions — a transliterated line treated as English, or a translation offered where a reader wanted the original.
Romanisation is also lossy and unstandardised in practice. The same spoken word reaches Latin script several different ways depending on who or what wrote it down, and converting back is guesswork. A name, therefore, is a poor key: two records for one person are entirely possible with no character in common.
Does Connect use Unicode and script handling?#
It has to — the product speaks and writes to people across languages, and a transcript is the place the layers above become visible. Three things follow, and they are worth knowing before you go looking for a fault that is not one.
- A transcript may return a language in a script you did not expect. Speech recognition and the model decide how what was said is written down; the same conversation can appear romanised or in its own script. The language has not changed and neither has the meaning.
- Searching a transcript is string matching. A name typed in Latin letters does not match the same name written in another script, because nothing in a plain search knows they are the same word.
- Identity does not rest on names. A person in Connect is resolved by Identity — one address on one channel, keyed by that channel and value — rather than by matching display names across scripts. That is why cross-channel identity works at all, and why duplicates are proposed for a person to decide rather than merged automatically.
The regional speaking-style layer on the voice side is about how a reply sounds rather than how text is encoded, and the two are separate concerns: a style choice does not change what script a transcript is written in.
Practical rules#
- Store and transmit UTF-8, everywhere, without exception.
- Normalise before comparing, and store what you normalised so the comparison is reproducible.
- Fold case with a language in hand, or not at all.
- Sort with a locale-aware collation; never assume the encoded order.
- Count and truncate by grapheme cluster when a limit is meant for a reader.
- Key records on identifiers, not on names — the identifier is the same in every script.
Questions#
Why is my call transcript written in Latin letters when the call was not in English?
Because transcription chooses a script as well as a language, and a romanised rendering of the same words is a legitimate output. It is a transliteration rather than a translation — the language is unchanged. If you search that transcript, search using the script it was written in.
Two records exist for the same person with differently spelled names. Why?
Because a name is not a reliable key across scripts and romanisations. Connect resolves a person by identities — an address on a channel — so a person reached on two channels that never shared an address can legitimately arrive as two records. Duplicate detection proposes the join and a person confirms it, because merging is not cleanly reversible.
Does searching handle characters like % and _ correctly?
Yes. Those are wildcard characters in SQL pattern matching, and typing one into a search box used to widen the search rather than narrow it. They are escaped now and matched as ordinary characters, which is what somebody typing a literal underscore intends.