001Notes

Email Before AI: MIME, Canonical Text, Multilingual Processing, and Privacy

On this page23

A beginner-first tutorial following email from MIME bytes through canonical text, language detection, translation, Unicode offsets, PII detection, redaction, and safe model input.

3 Sept 2026Project Lessons11 min readhigh confidence

Article details

Status: Building Publicly
Subcategory: Thunderbird AI
Last reviewed: 3 Sept 2026
Prerequisites: No email-format or machine-learning knowledge required

23 sections

An email looks like a page of text in Thunderbird. On disk or on the network, it is closer to a small package containing headers, nested documents, encodings, attachments, and instructions for how a mail reader should assemble them.

That difference matters before machine learning begins. A classifier trained on duplicated HTML and plain-text alternatives learns the wrong word frequencies. A summarizer that mistakes an old quoted reply for the newest message can reverse who decided what. An extractor that silently rewrites an invoice number during translation may return a fluent but unusable answer.

This tutorial follows one fictional email from raw source to model input. No personal mailbox data is used.

1. The five objects people casually call “the email”

Keep these representations separate:

Representation	What it contains	What it is good for
Wire source	Original headers, boundaries and encoded bodies	Authentication and forensic evidence
MIME tree	The parser’s structural interpretation	Choosing bodies and attachments
Decoded source text	Human-readable characters	Exact quotations and citations
Analysis text	Relevant body with segments labelled or excluded	Classification, extraction and retrieval
Policy-safe text	Redacted, bounded text permitted for a model	Endpoint inference

The model normally needs the last representation. A trustworthy product must retain a route back to the decoded source.

2. Start with bytes, not meaning

A message following RFC 5322 is divided into headers and a body by one blank line:

From: Aster Bank <alerts@asterbank.test>
Subject: Book Nook transaction
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

INR 805.00 was debited at Book Nook.

The blank line is structural. Header continuation lines, line endings, character sets and transfer encodings all affect parsing. UTF-8 bytes E2 82 B9 represent the rupee sign ₹; interpreting them as three unrelated Latin-1 characters corrupts the source before any tokenizer sees it.

Base64 and quoted-printable are not encryption. They are transport-safe encodings. Decoding them is analogous to opening a shipping box: it reveals the payload but does not yet tell us which payload is authoritative.

3. MIME is a tree

Consider this simplified structure:

multipart/mixed
├── multipart/alternative
│   ├── text/plain
│   └── text/html
└── application/pdf; name="receipt.pdf"

multipart/alternative usually means “several versions of the same content.” Concatenating both children duplicates the message. multipart/mixed usually means distinct parts, such as a body plus attachments. A nested message/rfc822 may contain an entire forwarded email with its own headers and MIME tree.

The parser therefore performs a recursive operation:

Recursive MIME interpretation

parse(part)=decode leaforcombine(parse(children), subtype policy)

The multipart subtype determines whether children are alternatives, independent parts, signed content, or related resources.

The important beginner intuition is that MIME parsing is not “remove HTML tags.” Structure is decided before cleaning prose.

4. HTML is presentation plus text

An HTML alternative can contain the visible sentence, navigation, tracking pixels, hidden preview text, quoted blocks and styles. A naïve tag-stripper turns:

<p>Paid <strong>INR 805</strong> at Book Nook.</p>
<style>.amount { color: green }</style>

into a mixture of content and CSS tokens. A renderer-aware extraction path should ignore non-visible elements, insert sensible boundaries around blocks, decode entities, and preserve links separately.

Even then, the visible HTML body is not automatically more authoritative than the plain alternative. Selection is a policy decision. Thunderbird’s rendered-message path and parser sidecar let analysis consume the same locally available message while retaining how the body was chosen.

5. Quoted replies are real source text with a different role

Suppose the newest message says:

The release is Friday.

On Monday, Daniel wrote:
> The release is Thursday.

Both dates occur in the email. A bag-of-words model sees Friday and Thursday; a summarizer may choose either. Segmenting quoted history gives the downstream system a better representation:

body      → The release is Friday.
quoted    → The release is Thursday.

This is not deletion. The decoded source still contains both spans. The analysis view says which segment is the current author’s contribution.

Signatures, unsubscribe footers and forwarded-message separators create similar problems. Their vocabulary can dominate short messages while carrying little intent.

6. Source maps prevent cleaned text from becoming invented text

If analysis removes characters 28–90, offsets in the cleaned string no longer point to the same positions in the source. A source map records where every retained range came from.

decoded source:  [0.....................96]
analysis text:   [0......22]
source range:    [0......22]

For a retained analysis character at position j, store a mapping back to source position i:

Offset mapping

sourceOffset=M[analysisOffset]

The mapping lets an extracted amount highlight the original decoded characters instead of trusting a rewritten copy.

This distinction is foundational for citations, entity spans and human review.

7. Canonical text and model text have different jobs

Canonical source text should be stable, exact and reconstructible. Model text may be shortened, translated, redacted or reorganized for a task.

decoded source
   ├── canonical source + offsets       authoritative
   ├── analysis body                    derived
   ├── English translation              derived aid
   ├── redacted endpoint text           policy-safe derivative
   └── embedding / summary              model artifact

If a summary says “Maya approved Atlas,” that sentence is not promoted into the source because it is convenient. The source must independently contain a supporting span.

8. Translation must protect exact tokens

General translation models may change punctuation, reorder phrases, localize numbers, or split identifiers. Before translation, ThunderbirdAI can protect URLs, amounts and reference-shaped tokens:

Aprobar INV-2026-42 por INR 805
          ↓ protect
Aprobar ⟦P0⟧ por ⟦P1⟧
          ↓ translate
Approve ⟦P0⟧ for ⟦P1⟧
          ↓ restore
Approve INV-2026-42 for INR 805

The translation helps retrieval or a model understand the prose. Citations and deterministic claims still point to the decoded original.

9. Complete dry run

Input message:

Hello team,
The launch is Friday. Reference INV-2026-42.
--
Maya
On Monday Daniel wrote:
> The launch was Thursday.

Normalize line endings without changing the decoded characters.
Identify the signature delimiter and quoted-reply transition.
Produce three labelled segments: body, signature and quote.
Retain the full original text and range metadata.
Use only the body for the default analysis view.
Protect INV-2026-42 if translation is requested.
Apply PII policy before any permitted endpoint request.

The resulting model text is shorter, but the source has not been overwritten.

10. The actual synthetic run

The accompanying experiment executed the segmentation procedure over the message above.

Decoded source116 characters

Analysis body56 characters

Segments retained3 labelled ranges

Personal mail usednone

The shorter analysis text is not evidence that it is “better” in every task. A question about what Daniel previously proposed may require the quoted segment. Segmentation enables task-specific inclusion; it should not erase history.

11. Experiments worth trying

Duplicate the plain and HTML alternatives, then measure how term counts and classifier confidence change.
Include ten replies beneath a one-line answer and compare a full-body summary with a current-segment summary.
Translate a sentence containing URLs, currency and IDs with and without protection.
Delete the source map and attempt to highlight an extracted value after whitespace normalization.
Feed malformed boundaries and unsupported charsets into the parser and verify that failure is explicit rather than silently producing empty evidence.

That establishes the source-text foundation:

A model never receives “the email.” It receives one engineered representation of the email. Correctness begins by naming that representation and keeping its route back to source.

The next two parts extend that rule. Translation creates another derivative of the source, while privacy policy decides which derivative a destination is permitted to receive.

12. Multilingual processing without losing the evidence

An inbox may mix English, Hindi, Spanish, Japanese and code-switched sentences. Exact identifiers, currencies and URLs often remain unchanged while the grammar around them changes.

Aprobar INV-2026-42 por INR 805.
Please approve INV-2026-42 for INR 805.
कृपया INR 805 वाला INV-2026-42 स्वीकृत करें।

A multilingual system must understand the request without rewriting the source evidence. Two common architectures make different tradeoffs.

12.1 Translate into one pivot language

source language → local translation → English classifier/embedder/model

This reuses strong English components and one downstream vocabulary. It adds translation latency, can lose information or corrupt identifiers, and makes source alignment harder.

12.2 Use multilingual models directly

source language → multilingual classifier/embedder/model

This removes one transformation and often preserves source alignment. Quality may vary sharply between high-resource and low-resource languages, models may be larger, and code-switched text remains challenging. Neither architecture wins universally; compare them on the actual languages and tasks.

13. Language detection is a prediction too

Short messages are ambiguous:

OK
Gracias
Invoice INV-42 ready

A sender can change languages within one sentence. Treat detected language as metadata with a score and possibly span-level labels, not as a perfect mailbox fact. When confidence is weak, a multilingual fallback may be safer than selecting the wrong translation model.

Record states such as disabled, unavailable, failed, partial, and translated. Empty translated text must not be mistaken for a successful translation of an empty source.

14. Protect exact tokens before translation

Suppose the Spanish source says:

Aprobar INV-2026-42 por INR 805.

An English derivative may say “Approve INV-2026-42 for INR 805.” That helps an English classifier, but the Spanish source remains authoritative. Before translation, replace exact-value spans with collision-resistant markers and restore them afterward.

The fictional run protects an invoice ID, amount and URL, then restores all three exactly. Markers must not collide with source text, be translated, or become reordered ambiguously. Extracting exact identifiers before translation is often safer than recovering them from translated prose.

15. Unicode has several coordinate systems

Humans count visible graphemes. Unicode defines code points. JavaScript strings index UTF-16 code units. Some characters occupy two code units.

For A🙂B, a reader sees three characters and Unicode has three code points, but JavaScript indexes four UTF-16 units: A, the emoji’s high surrogate, its low surrogate, and B.

In Hi 🙂 Maya approved Atlas, a Python model may report the code-point start for Maya as 5. JavaScript’s UTF-16 start is 6 because the emoji occupies two code units. Convert the coordinate system and validate the returned span:

Cross-runtime span contract

JS.slice(convert(start), convert(end))=returned entity text

Exact equality catches offset drift before a model hint is stored or highlighted.

Combining marks add another distinction: é can be one precomposed code point or e plus a combining accent. Normalization changes offsets, so apply it only as a versioned transformation with an explicit source map.

16. Translation boundaries and code switching

Languages reorder clauses, so a translated entity may not map monotonically to one source range. Word alignment models estimate mappings but remain probabilistic. Safer interfaces cite the original message, store translation spans as derived, and show original plus translated snippets during review.

Code switching challenges one-language pipelines:

Kal meeting hai, please review INV-42 before 5 PM.

A multilingual encoder may handle the mixed sentence directly. A translation pipeline may translate only one part or misidentify the language. Evaluation therefore needs realistic mixed-language fixtures, not one polished paragraph per language.

Retrieval and generation also require separate tests. A multilingual embedder may retrieve Hindi and English messages into a shared space while the answer model responds poorly in Hindi. Translation-first retrieval may find the right message but lose an exact phrase. Measure cross-language recall, exact-token preservation, classification by language, span validity, answer-language adherence and citation fidelity separately.

If a 30,000-character body permits only 12,000 translated characters, expose partial status and coverage:

Translation coverage

coverage=processed source characterseligible source characters

Coverage does not measure translation quality; it prevents partial output being presented as complete.

17. PII policy and redaction

Before an assistant sends text to any endpoint, it should answer three different questions:

What potentially sensitive spans are present?
How sensitive is each span in this context?
What may this destination receive?

Those are detection, contextual classification and policy. Collapsing them into one “PII filter” makes failures difficult to inspect.

PII can include contact identifiers, phone numbers, government-linked identifiers, payment cards, credentials and contextual person names. Shape alone does not determine risk: a public support address differs from a private address tied to a medical appointment.

18. Pattern recognizers, validation and overlap

Deterministic recognizers find known shapes quickly and return exact spans. They struggle with formatting variation, contextual names, overlapping numeric shapes, and adversarial Unicode or spacing.

Validation removes some false candidates. Payment-card shapes can use the Luhn checksum: starting at the right, double every second digit, subtract nine from doubled values above nine, and add all adjusted digits.

Luhn decision

valid⇔Σ adjustedDigits mod 10 = 0

A valid checksum means plausible shape, not a real active card or proof of ownership.

Overlapping recognizers can disagree. A 16-digit card-shaped value contains 12-digit substrings that a broad government-ID rule may match. Prefer validated, more-specific spans or retain the conflict for review. Replacing spans in discovery order can leak suffixes or create nested redactions.

19. Run the policy, including the failure

Input

Policy-safe output

Five of six fictional expectations pass. The invalid card-shaped number is the deliberate failure: a 12-digit substring activates the broad Aadhaar-shaped rule. A privacy detector needs a labelled false-positive corpus, not only successful regex examples.

A contextual NER model such as a GLiNER-style detector can distinguish candidates using surrounding language, but it may hallucinate boundaries. Combine flexible candidate detection with exact source-span validation. The model proposes sensitive spans; a separate policy engine decides network permission.

20. Severity, destination and provenance

Detection might produce type=API key, severity=critical, and span=81…113. Destination policy then maps severity and trust to allow, redact, or block. A local deterministic stage, private loopback model and explicit cloud endpoint may receive different representations.

Redaction changes length and offsets:

source:   Contact maya@example.test about INV-42
redacted: Contact <EMAIL> about INV-42

Keep canonical local text, redacted text, final endpoint-safe text, matches and policy decision as separate fields. Never overwrite the source with a derivative. Do not let metadata-only debug traces retain raw credentials either.

Redaction can damage the task. Replacing two addresses with one <EMAIL> token prevents answering “Which address received the receipt?” Options include local execution, stable aliases such as <EMAIL_1>, blocking the endpoint request, or sending only non-sensitive context. Measure privacy recall and task success together.

Incoming analysis and outgoing data-loss prevention also differ. Outgoing systems need recipient context and should normally warn before sending rather than silently rewrite a draft.

21. Evaluate the complete input boundary

Measure sensitive-span precision and recall by type and severity, policy-decision accuracy, characters leaked after redaction, downstream task success, false positives per ordinary message, Unicode evasions, and worst-case runtime.

Then test the combined pipeline:

malformed MIME and unsupported charsets must fail visibly;
quoted history and signatures must retain source ranges;
translation must preserve protected identifiers;
emoji and combining marks must not shift highlights;
partial processing must remain visible;
every destination must receive only its permitted representation;
diagnostics must not reintroduce removed content.

The consolidated lesson is that safe model input is not cleaned prose. It is a versioned, source-linked derivative produced by parsing, language processing and destination policy—each with its own uncertainty and failure modes.

Primary references