AI & MLPython

Classifying Text with BERT Features and Cosine Similarity

Text classification is often introduced as a supervised problem: collect labeled examples, train a model, and predict a new document's label. A BERT notebook in my learning collection explores another starting point—compare representations and assign labels through similarity.

The retained example includes BERT feature extraction and cosine-similarity classification. It uses a pretrained language model as a source of features rather than as a chatbot.

From text to a vector

Once documents have numerical representations, cosine similarity compares their directions. A labeling rule can use those comparisons to assign a category.

The representation of a label matters too. Comparing a paragraph with a short category name is different from comparing it with several examples that describe the category.

This exposes several choices that a high-level classification function can hide: which model output becomes the document vector, how longer text is handled, and how the categories are represented.

Similarity needs a test set

Avoiding a new training run does not remove the need to evaluate assignments. Two comments about the same event can express opposing positions. A model may recognize their shared subject while missing the distinction the task requires.

For a useful comparison, I would keep the labels and test documents fixed, then inspect how feature and pooling choices change the assignments. Ambiguous cases help expose what a representation captures.

Where this fits

The 2018 BERT paper is a useful historical anchor for contextual language features. The notebook shows their appeal as reusable inputs to a smaller downstream task.

It remains a retained learning example, without a verified personal accuracy result. A public companion would need clear source attribution, a permitted sample, and cleared outputs.