Computer scienceData scienceNLPText processingText normalization

Tokenization implementation

NLTK modules

Report a typo

You have a text that you want to tokenize. Match each line of the code with its result.

text = "I have got a cat. He's 3 years old."

Match the items from left and right columns

print(word_tokenize(text))

print(sent_tokenize(text))

print(regexp_tokenize((text), "[A-z]+"))

['I', 'have', 'got', 'a', 'cat', '.', 'He', "'s", '3', 'years', 'old', '.']

['I', 'have', 'got', 'a', 'cat', 'He', 's', 'years', 'old']

['I have got a cat.', "He's 3 years old."]

___

Create a free account to access the full topic