GAGodwin Etim AkpanGeospatial & Data Engineer← All projects

Machine learning / Text representation

NLP & Document Clustering
Compare representations. Measure what survives.

A reproducible comparison of lexical and learned text features: headline classification across 42 categories, held-out topic clustering, and a linked investigation of rare-word probability estimation.

Research project · Repeated experiments & held-out evaluation
206,959Retained headlines
42Headline categories
0.410Mean TF-IDF macro-F1
18Clustering configurations
Explore the engineering ↓View GitHub project ↗
Project in focus

Which representation supports prediction?

Which representation supports prediction? — Mean and individual split scores for a majority baseline, TF-IDF, and averaged Word2Vec. Macro-F1 weights the 42 categories equally.
Mean and individual split scores for a majority baseline, TF-IDF, and averaged Word2Vec. Macro-F1 weights the 42 categories equally.View full-size figure ↗

01 / The question

Which text representations preserve the signal needed for classification and clustering?

Useful text systems need more than plausible semantic neighbours. This study tests whether representations support category prediction and whether unsupervised groups recover known topics. The experiments separate feature learning, model selection, and held-out evaluation.

My contributionBuilt the text-cleaning and evaluation workflow, trained lexical and embedding baselines, evaluated held-out document clusters, and produced reproducible result exports.

02 / System walkthrough

From input to evidence.

The main stages of the implemented analytical workflow.

  1. 01

    Clean and partition

    Remove empty headlines, normalized duplicates, and conflicting-label groups. Run three stratified 80/20 splits with identical partitions for each classifier.

  2. 02

    Fit training-only representations

    Compare unigram/bigram TF-IDF with 200-dimensional CBOW Word2Vec averages. Fit embedding scaling on training rows and use the same Logistic Regression regularization.

  3. 03

    Evaluate every category

    Compare against the majority-class baseline; retain macro-F1, accuracy, precision, recall, support, and full confusion counts for all 42 categories.

  4. 04

    Test document clusters

    Strip headers, footers, and quotations from 20 Newsgroups. Fit TF-IDF and normalized 100-component LSA on the earlier partition, then evaluate K-means on the later partition.

03 / Engineering decisions

Why the implementation matters.

Make leakage checks part of preparation

Deduplicating normalized headlines before splitting prevents identical text from appearing on both sides. Document clustering also removes exact normalized train/test overlap.

Separate model selection from evaluation

For each clustering representation, choose k using mean training silhouette across three seeds. Report held-out ARI and NMI for every candidate; known topics do not train the clusters.

Keep sparse text sparse

TF-IDF stays sparse. Truncated SVD provides a bounded 100-dimensional semantic representation; no dense document-by-vocabulary conversion is required.

Show all categories

Aggregate scores are paired with a 42-category performance chart and full CSV exports, so frequent categories do not hide weaker parts of the task.

04 / Results & interpretation

What the experiment showed.

Headline scores are means across three overlapping stratified splits. Cluster scores use the official later 20 Newsgroups partition. All transformations are fitted on training text.

Headline classification / three splits

0.410 macro-F1

TF-IDF + LR averages 57.2% accuracy. Averaged Word2Vec + LR records 0.257 macro-F1 under the declared settings.

Topic clustering / held-out partition

0.185 mean ARI

Normalized LSA-100 at training-selected k=20 produces mean NMI 0.372. Topic recovery remains partial; cluster separation and semantic agreement are different questions.

Reproducible evidence

9 + 18 runs

Nine classifier runs and eighteen clustering configurations, with source checksums, fixed settings, full metric tables, and prediction reconciliation.

Figure / 02

Look beyond the headline score

Look beyond the headline score — All 42 categories, ordered by mean TF-IDF F1. Full per-class precision, recall, and support are available in the result tables.
All 42 categories, ordered by mean TF-IDF F1. Full per-class precision, recall, and support are available in the result tables.View full-size figure ↗
Figure / 03

Measure topic recovery on unseen documents

Measure topic recovery on unseen documents — Held-out ARI and NMI for both representations and all candidate cluster counts. Whiskers show the observed seed range, not confidence intervals.
Held-out ARI and NMI for both representations and all candidate cluster counts. Whiskers show the observed seed range, not confidence intervals.View full-size figure ↗

Further investigation / Statistical NLP

How much probability belongs to words not yet observed?

Text classification asks which words distinguish a category. Probability estimation asks a complementary question: how should a model account for words absent from a small sample? I implemented and evaluated MLE, Simple Good–Turing, and absolute discounting to examine that trade-off.

At 1k tokens / 50k vocabulary

0.179 vs 0.285 JS

Absolute discounting reduced mean Jensen–Shannon divergence relative to MLE; Simple Good–Turing achieved 0.182. Lower is better.

Unseen probability mass / same setting

1.88 vs 45.63 pp

Mean absolute error, in percentage points, for Simple Good–Turing versus MLE. Reserving probability for unseen words makes a substantial difference at small sample sizes.

At 1M tokens

Vocabulary matters

Simple Good–Turing led mean JS for the 50k vocabulary; MLE was slightly lower for the full vocabulary. More data changes the ranking rather than creating one universal winner.

120 estimator runs: four exact sample sizes from 1k to 1M tokens, five seeded draws, and two training-derived vocabulary settings. A disjoint 3.40-million-token Text8 reference evaluates the estimates. Charts show means and sample standard deviations across draws.

Figure / 01

Sample size changes the value of smoothing

Mean Jensen–Shannon divergence by sample size for three unigram estimators, comparing capped and full training vocabularies.
Repeated experiment on canonical Text8. Both smoothing methods improve mean JS over MLE at 1k tokens; the ranking changes with sample size and vocabulary. Shaded bands are ±1 sample SD across five draws, not confidence intervals.View full-size figure ↗
Figure / 02

Look beyond overall divergence

Absolute error in unseen-word probability mass by sample size; smoothed estimators improve on the zero unseen mass assigned by MLE.
Unseen-mass error examines the part of the distribution that a sample misses. It complements JS: a model can estimate total unseen mass well without assigning accurate probabilities to individual rare words.View full-size figure ↗
Explore the probability-estimation experiment ↗

05 / Scope & limitations

Scope & limitations.

  • GeneralizationHeadline results use repeated random splits from one publisher corpus. The splits overlap and do not establish future-period or cross-publisher performance. Historical category conventions and related labels remain part of the task.
  • Representation scopeWord2Vec is a fixed CBOW average with no pretrained contextual model. Equal regularization and training data do not make the two feature families equally expressive or independently optimized.
  • Cluster interpretationHeaders, signatures, and quotations are removed with heuristics. Exact normalized overlaps are removed; paraphrases and thread-level similarities can remain. Silhouette uses a fixed sample and different representation geometries, so its absolute values should not rank feature families.
  • Runtime and reproducibilityThese are single-machine Python experiments. No distributed speedup or production-serving claim is made. Fixed seeds and one Word2Vec worker improve repeatability; numerical libraries and versions can still change exact results.
  • Probability-estimation experimentOne ordered Text8 split and five draws measure sampling variability, not cross-domain generalization. Vocabulary is fixed from the training pool; unseen mass is allocated uniformly. This is a unigram study, not an evaluated contextual language model. MLE KL is infinite where reference events have zero estimated probability; the outputs preserve that fact.

06 / Next iteration

Where I would take it next.

The next technical priorities for this project.

  1. Evaluate temporal transfer

    Reserve a future news period or a second publisher and keep all tuning inside the training period.

  2. Test modern semantic encoders

    Compare sentence representations and supervised tuning under the same deduplication and hold-out rules.

  3. Inspect useful cluster labels

    Add human assessment of cluster themes and document retrieval quality before proposing operational categories.

07 / Code & reproducibility

Explore the implementation.

Explore the implementation, selected results, and technical documentation on GitHub.

  • Nine freshly executed classifier runs, full per-class reports, and confusion counts.
  • Eighteen clustering runs with held-out ARI/NMI, sampled silhouette, and cluster/topic tables.
  • Input and code hashes, pinned versions, eight preparation/resume tests, and reconciled predictions.
  • The separate rare-word probability study retains its verified 120 estimator runs.

Figures are generated from the executed result tables. Seed ranges describe observed variation, not confidence intervals.

Download classification runs ↓Download clustering runs ↓Download probability-estimation results ↓
Explore the GitHub project ↗Return to selected work →