NLP

NLP stands for natural language processing, and it is associated with language. These can be solutions such as making sentences, finding the sentiment of a sentence or question, and answering. In our data science class, we also learned how to do text preprocessing and make language models.

Text Preprocessing

Machine learning models can only understand numbers. That means we cannot directly input text into a model. Therefore, we first needed to do some preprocessing to the text.

Examples of text preprocessing:

Remove HTML tags

import re
def remove_html_tags(text):
    re_html = re.compile('<.*?>')
    return re_html.sub(r'', text)

Remove URLs

text = 'My profile link: https://www.github.com/danielkim'
def remove_url(text):
    re_url = re.compile('https?://\S+|www\.\S+')
    return re_url.sub('', text)

Remove Punctuations

import string
exclude = string.punctuation

def remove_punc(text):
    return text.translate(str.maketrans('', '', exclude))

Remove stopwords

Remove common words that are not important

from nltk.corpus import stopwords
stopwords_english = stopwords.words('english')

def remove_stopwords(text):
    new_text = []
    for word in text.split():
        if word in stopwords_english:
            continue
        else:
            new_text.append(word)
    return ' '.join(new_text)

Tokenization

Change the text into an array

from nltk.tokenize import word_tokenize, sent_tokenize
sent1 = 'Life is a matter of choices and every choice makes you!'
print(word_tokenize(sent1))

Vectorization

Changing text into numbers