Email Classification and Personal AI: Rules, Naive Bayes, Transformers, Specialists, and Evaluation

Let us make the hidden steps explicit. A practical classifier does not jump directly from a MIME email to a trustworthy category.

decoded fields → representation → scores → confidence → precedence → routing label

The representation is the form of the message the algorithm can operate on. A regular expression uses characters. Naive Bayes uses word counts. A transformer uses contextual token vectors. These are three different views of the same source.

The candidate score is also algorithm-specific. A rule may produce a Boolean match. Naive Bayes produces relative log scores. The ModernBERT adapter produces a normalized top-label score. None of these numbers has meaning without its scoring procedure.

The precedence policy is ordinary program logic that decides whose result to trust. If a user-approved template maps one verified statement format to finance, should a weaker statistical model override it? Usually not. If an exact rule does not match and the local learned model is unsure, should a contextual model get a chance? Perhaps. If every method is weak, should the system guess anyway? Often it should fall back or ask for review.

This separation turns “the AI classified it” into inspectable questions: Which fields were read? Which representation was built? Which candidates were produced? What did each score mean? Which branch selected the output? Would a stricter threshold have abstained?

A learned classifier replaces some hand-written phrase choices with statistics estimated from examples. Each training record needs an input and the output a human wants.

{
  label: "finance",
  subject: "Card transaction",
  author: "alerts@example.test",
  text: "card transaction merchant debited"
}

The label is called the target, class, or ground truth. The subject, author, and body are features in raw form. Thunderbird’s local sparse learner joins those fields before tokenization. Sender-like text can therefore influence the result alongside the subject and body.

Training examples silently define the problem. If every finance example comes from bank@example.test, the model may learn the sender token rather than financial language. It can score beautifully on more mail from that address and fail on a receipt from a new merchant. This is shortcut learning: a feature correlates with the label in training but does not capture the intended concept.

Good dataset design asks whether examples represent later messages, whether near-duplicate threads leak across train and test, whether senders are tied to labels, whether ambiguous messages are labelled consistently, whether rare classes have enough examples, and whether “none of these” is possible.

For this tutorial the training set is intentionally tiny and fictional. That makes every count inspectable. It also makes the resulting accuracy unsuitable as a claim about real inboxes.

Collect every unique training token into an ordered vocabulary:

[invoice, payment, receipt, interview, recruiter, vacancy]

Represent a document by counting how many times each vocabulary word appears. For “invoice payment receipt”:

invoice payment receipt interview recruiter vacancy
   1       1       1        0         0        0

This row is a vector: an ordered list in which each position has a fixed meaning. It is sparse because a realistic vocabulary may have tens of thousands of positions while one email uses only a small fraction.

Two consequences matter. Order disappears. “invoice payment receipt” and “payment invoice receipt” have identical vectors. “Dog bites person” and “person bites dog” also match even though their meanings differ.

Repetition remains. “invoice invoice payment” places 2 in the invoice coordinate. Multinomial Naive Bayes treats that as two observations. A repeated footer can overpower a short original body.

We can preserve short phrases with n-grams such as password_reset, but the vocabulary grows rapidly. We can reduce common-word influence with TF-IDF, but ordinary Multinomial Naive Bayes has a clean probabilistic story with non-negative counts. Representation and algorithm must agree.

Use the smallest dataset that teaches the mechanism:

finance → "invoice payment receipt"
jobs    → "interview recruiter vacancy"

Classify invoice payment. There are two examples, one per class, so both priors are one half. There are six unique vocabulary words. Each class has three token occurrences.

Without smoothing, invoice occurs zero times in jobs. Because the document score is a product, one zero makes the entire jobs score zero. With a large vocabulary, almost every new message contains something unseen. The classifier becomes brittle.

Laplace smoothing adds one imaginary count to every vocabulary word:

For finance, invoice and payment each have probability (1+1)/(3+6)=2/9. For jobs, each is unseen, so each has probability (0+1)/(3+6)=1/9.

Finance is four times larger. Normalizing the two scores gives 80% and 20% for display.

The 80% display share does not mean the model is correct 80% of the time. It came from two toy messages under assumptions that are badly violated. It says only how these model scores divide after normalization.

The implementation used for this page follows the source behavior rather than a library shortcut:

for (const example of trainingExamples) {
  const tokens = tokenize(subject + "\n" + author + "\n" + body);
  classExampleCount[label] += 1;
  for (const token of tokens) {
    vocabulary.add(token);
    tokenCount[label][token] += 1;
    classTokenTotal[label] += 1;
  }
}

At inference:

score[label] = log(classExampleCount[label] / allExampleCount);
for (const token of tokenize(newMessage)) {
  const numerator = (tokenCount[label][token] ?? 0) + 1;
  const denominator = classTokenTotal[label] + vocabulary.size;
  score[label] += log(numerator / denominator);
}

The artifact trains on twelve fictional records—four each for finance, jobs, and security—and evaluates twelve different held-out messages. Held out means the model does not see those rows while learning counts.

This is an implementation proof. You can inspect surviving tokens, relative scores, predicted label, and exact gap mapping. It proves that code can learn and separate these simple examples. It is not a quality proof for real email: the dataset is balanced, tiny, synthetic, and lexically friendly.

Compare:

The bank approved my card dispute.
We sat on the river bank after lunch.

A bag of words sees bank in both. A contextual transformer builds a different representation for that token because neighboring words participate in the computation.

Its tokenizer usually uses subwords, not our whitespace tokens. Each token ID retrieves a learned vector; positional information preserves order. In self-attention, every token creates query, key, and value vectors. Query–key dot products determine how much one position reads from another, and weighted values form a new contextual representation. After many layers, bank beside river need not resemble bank beside card.

A conventional fine-tuned classifier adds a learned output head. Thunderbird’s private adapter here is different: it loads a natural-language inference model for zero-shot classification. It does not train a new fifteen-way head on this fictional mail.

NLI compares a premise and hypothesis:

premise:    A recruiter scheduled your engineering interview.
hypothesis: This email is about jobs.

The adapter repeats that hypothesis for every candidate label, scores entailment, and chooses the strongest.

“Uses ModernBERT” does not fully describe the algorithm. The encoder, NLI training, hypothesis template, candidate labels, and normalization procedure jointly define the classifier.

The recorded artifact uses:

tasksource/ModernBERT-base-nli
revision de4ab7e77845098b7fab7f6ab9d370ddff27b19c
role: modernbert-intent
hypothesis: "This email is about {}."
execution: local CPU runner

The benchmark contains only fictional one-sentence emails, with direct, paraphrase, and confounder examples for every intended label.

We ran two tiers. The teaching tier contains nine messages intended as finance, jobs, or security, with only those labels competing. The contract tier contains 45 messages—three per Thunderbird label—with all fifteen competing. Each request was repeated three times after five warm-ups; the fixture records median latency and prediction stability.

The constrained run achieved 100% accuracy on 9 messages. The full run achieved 48.9% on 45 messages. Warm median latency grew from roughly 73 ms with three candidates to 353 ms with fifteen. The cold request took about 2.0 seconds, including lazy model loading.

The model did not suddenly become less intelligent. We asked a harder question. job-hunt competed with jobs; calendar with action-required; shipping looked like travel; support looked legal; promotional editorial copy resembled newsletters. Because zero-shot NLI evaluates hypotheses, more candidates also require more paired inference work.

The experiment proves only that the pinned local adapter ran, separated the narrow teaching set, and exposed substantial ambiguity under the current full vocabulary. It does not establish production quality, universal model ranking, GPU latency, calibration, multilingual performance, or adversarial safety. Forty-five synthetic rows are a diagnostic probe.

The recorder requires an explicit --allow-live and accepts only a loopback endpoint. Normal builds consume JSON; they do not start a model, download weights, or send content anywhere.

Rules and models have complementary strengths. Thunderbird resolves them in a visible order:

1. user-approved template override
2. exact deterministic rule
3. sufficiently confident user-trained sparse model
4. configured transformer hint
5. safe fallback

This is not score averaging. The scales differ, and higher-trust decisions may deserve precedence regardless of lower-trust scores.

If the sparse model is only medium confidence and a high-confidence transformer disagrees, the transformer may resolve it. A high sparse result is not casually displaced. If neither learned result reaches threshold, the system falls back.

Retain category, confidence, source, reason, and every candidate. Diagnostics let a developer distinguish bad representation, weak training, label ambiguity, threshold error, and wrong precedence. Without them, every failure becomes “AI was wrong.”

Benchmarks need targeted counterexamples. Step through five sparse-model failures.

Negation: “not a job alert” retains strong jobs tokens. Unigrams cannot represent the scope of not.

Polysemy: bank may mean finance or river land. Contextual encoders help because neighbors change its representation.

Boilerplate leakage: an unsubscribe footer can mention security and jobs below a personal body. Parse current authored text separately and evaluate parser plus classifier.

Out of distribution: a closed classifier ranks known labels even for pottery. Consider other/unknown, calibrated thresholds, dedicated OOD examples, and fallback.

Tokenization blind spots: CV OK; HR to call becomes only call. Changing minimum length may recover abbreviations while adding noisy words, so measure the tradeoff.

Email classification is not a contest where the newest model defeats every older method. It is a stack of contracts.

The label ontology defines distinctions. The parser defines visible content. The tokenizer defines symbols that become evidence. The representation defines relationships that survive. The learner estimates associations. Calibration interprets scores. The cascade chooses whose judgment wins. Evaluation determines which failures become visible. The product action determines the cost of being wrong.

Rules handle precise stable phrases. Multinomial Naive Bayes turns small local datasets into an inspectable model whose arithmetic we can reproduce. Contextual transformers handle paraphrase and word sense more naturally, but their behavior depends on training task, label wording, candidate competition, and runtime cost. Confidence becomes useful only after measurement and decision policy. A fallback keeps forced guesses from becoming invisible automation.

A classifier does not discover the one true category hidden inside an email. We define a question, construct a representation, learn or encode a decision rule, and choose how much uncertainty the product may act on.

Once those choices are explicit, classification stops looking mysterious. It becomes a system we can calculate, animate, test, criticize, and improve. The next parts ask how that system becomes personal, when other model roles should enter, and what evidence justifies promoting a new configuration.

Suppose a newsletter is predicted as promotion and the user changes it to developer update. The system could add an exact sender rule, add one labelled sparse example, change a template family, or update a global model. Each spreads the correction differently.

An exact rule has narrow blast radius. A sparse example generalizes to shared words. A template generalizes to repeated structure. Fine-tuning can change unrelated behaviour and is the hardest to explain or reverse. Before learning, define the target: category, reply style, priority, summary format and folder placement are different tasks with different data.

Thunderbird’s experimental training records remain named and versioned. An example retains label, source identity, text, inclusion state and holdout status. A feedback click such as Wrong is not sufficient training data because it may mean wrong category, wrong retrieval, unsupported answer, poor wording or privacy concern. Store structured issue type and trace reference; make accidental feedback reversible.

If every finance example comes from one bank, the model may memorize the sender. Test unseen senders, unseen template families, paraphrases, near-miss labels, balanced classes and the deployment distribution. Sender memorization may still be useful, but name it rather than calling it general language understanding.

Active learning asks a person to label cases expected to teach more than random examples. Prediction entropy is one signal:

Disagreement between a rule, sparse model and transformer is another useful review queue. A candidate version should first run in shadow mode: the current decision remains visible, the challenger records its output, disagreements are measured, and promotion occurs only after explicit gates pass.

Fine-tuning becomes reasonable only when contextual errors persist, enough representative consented examples exist, a frozen evaluation shows a meaningful improvement, budgets permit the artifact, and rollback is solved. LoRA or adapters reduce trainable parameters; they do not remove data-quality, consent or evaluation requirements.

Run cheap, precise stages first and call flexible models only when necessary.

The fictional workload resolves three of five requests deterministically and sends only two to learned specialists. That counts routing decisions; it is not a universal latency claim.

GLiNER-style NER returns text, label, start, end and confidence; accept only exact spans inside bounded input. ModernBERT returns one intent from the declared vocabulary. Both are hints behind explicit opt-in and a verified loopback manifest. Installing weights does not authorize their outputs or convert them into source evidence.

Specialists also fail differently: classifiers choose wrong labels, NER shifts boundaries, embedders lose neighborhoods, rerankers demote good candidates, summarizers omit or invent, and generators produce unsupported language. Evaluate each interface before an end-to-end number hides the repair location.

A monolith can still be reasonable for a prototype, rare operation or tiny workload where orchestration dominates. Compare total system cost rather than following an architectural slogan.

Quantization represents weights with fewer bits, commonly four or eight plus scale metadata.

An 8B model at four bits suggests about 4 GB of raw weights before overhead. The approximation can change exact-number or formatting behaviour, so measure the task.

Completion budgets can reverse rankings. A reasoning model may consume a short cap before emitting the required visible citation. Empty or truncated output is a failed product result, not hidden success.

An answer-model evaluation should separate exact identifiers, numbers and dates; no-answer behaviour; citation presence and validity; claim support; format; prompt-injection resistance; and completion under budget. Privacy leakage and exact-ID substitution are often hard failures rather than small deductions in a weighted average.

A candidate is Pareto-dominated when another is no worse on every relevant dimension and better on at least one. A fast, slightly weaker model and a slower, stronger model may both remain valid operating points. Compare quality with p95 latency, memory, model size, energy and privacy placement.

Use role-specific metrics: classifier macro F1/calibration, NER span F1, embedder Recall@k/MRR, reranker nDCG gain, summarizer factual coverage, and generator task/citation pass rate. Synthetic fixtures isolate failures and are publishable; redacted real-shaped data adds messiness; consented local held-out data estimates deployment value. Do not turn six fictional cases into a general accuracy claim.