Natural Language Processing (NLP) is probably the hottest topic in Artificial Intelligence (AI) right now. After the breakthrough of GPT-3 with its ability to write essays, code and also create images from text, Google announced its new trillion-parameter AI language model that’s almost 6 times bigger than GPT-3. These are massive advances in the discipline that keep pushing the boundaries to new limits.

How is this possible? How can machines interact with human language? There are dozens of subfields in NLP, but we must start with the basics. In another post I went through some tips on how to begin the NLP journey. Now it’s time to talk about normalizing text.

Why do we need text normalization?

When we normalize text, we attempt to reduce its randomness, bringing it closer to a predefined “standard”. This helps us to reduce the amount of different information that the computer has to deal with, and therefore improves efficiency. The goal of normalization techniques like stemming and lemmatization is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form.

An example

Jaron Lanier said:

“It would be unfair to demand that people cease pirating files when those same people aren’t paid for their participation in very lucrative network schemes. Ordinary people are relentlessly spied on, and not compensated for information taken from them. While I’d like to see everyone eventually pay for music and the like, I’d not ask for it until there’s reciprocity.”

Let’s start by saving the phrase as a variable called “sentence”:

In another post I went through some techniques to perform Exporatory Data Analysis over text, so now I’ll focus on different types or methods.

Expanding contractions

In our sentence, we have the words “we will” contracted as “we’ll”, which should be managed before further normalization.

Contractions are words or combinations of words that are shortened by dropping letters and replacing them by an apostrophe, and removing them contributes to text standardization

There are different ways to expand contractions, but one of the most straight forward one is to create a dictionary of contractions with their corresponding expansions:

Then, we can use regular expressions to update the text:

Tokenize

Tokenization is the process of segmenting running text into sentences and words. In essence, it’s the task of cutting a text into pieces called tokens.

Next, we should remove punctuations.

Remove punctuations

Now we can perform stemming and lemmatization over the sentence.

Stemming

Stemming is the process of reducing the words to their word stem or root form. The objective of stemming is to reduce related words to the same stem even if the stem is not a dictionary word. For example, connection, connected, connecting word reduce to a common word “connect”.

The most common algorithm for stemming English, and one that has repeatedly been shown to be empirically very effective, is Porter’s algorithm.

What happened here? Words have been reduced, but some of them not to a real English word. Stemming refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time. The result? Stemming a word or sentence may result in words that are not actual words.

This happens because there are mainly two errors in stemming:

• Over-stemming: where a much larger part of a word is chopped off than what is required, which in turn leads to words being reduced to the same root word or stem incorrectly when they should have been reduced to more stem words. For example, the words “university” and “universe” that get reduced to “univers”.
• Under-stemming: occurs when two or more words could be wrongly reduced to more than one root word when they actually should be reduced to the same root word. For example, the words “data” and “datum” that get reduced to “dat” and “datu” respectively (instead of the same stem “dat”).

An improvement to the Porter Stemmer is the Snowball Stemmer, which stems words to a more accurate stem.

Still not good, right? Let’s follow a different approach.

Lemmatization

Unlike stemming, lemmatization reduces words to their base word, reducing the inflected words properly and ensuring that the root word belongs to the language.

It’s usually more sophisticated than stemming, since stemmers works on an individual word without knowledge of the context. In lemmatization, a root word is called lemma. A lemma is the canonical form, dictionary form, or citation form of a set of words.

Just like for stemming, there are different lemmatizers. For this example, we’ll use WordNet lemmatizer.

It’s possible to improve performance over lemmatization even further if you provide the context in which you want to lemmatize, which you can do through parts-of-speech (POS) tagging.

POS tagging is the task of assigning each word in a sentence the part of speech that it assumes in that sentence. The primary target of POS tagging is to identify the grammatical group of a given word: whether it is a noun, pronoun, adjective, verb, adverbs, etc. based on the context.

POS tagging improves accuracy

You can use POS as a speech parameter, which in Python is noun by default. For example, the word ‘leaves’ without a POS tag would get lemmatized to the word ‘leaf’, but with a verb tag, its lemma would become ‘leave’.

To get the best results, you’ll have to feed the POS tags to the lemmatizer, or otherwise it might not reduce all the words to the lemmas you desire.

Conclusion

While lemmatization helps a lot for some queries, it equally hurts performance. On the other hand, stemming increases recall while harming precision. Getting better value from text normalization depends more on pragmatic issues of word use than on formal issues of linguistic morphology.

Even though text normalization is considered pretty solved for modern languages such as English, there exist many (e.g. historic) languages for which the problem is harder to solve, due to a lack of resources and unstable orthography.

So, should we always normalize?

It depends on the nature of the problem. Topic modeling, for example, relies on the distribution of content words, the identification of which is dependent on a string match between words, which is achieved by lemmatizing their forms so that all variants are consistent across documents.

Lemmatization is also important for training word vectors, since accurate counts within the window of a word would be disrupted by an irrelevant inflection like a simple plural or present tense infleciton. On the other hand, some sentiment analysis methods (e.g. VADER), have different ratings depending on the form of the word and therefore the input should not be stemmed or lemmatized.

All preprocessing does not require normalization for the eventual model or application to be effective, and it may actually impede the success or accuracy of the model or application. We need to ask ourselves: is important information being lost by normalizing? Or is irrelevant information being removed?