Computer scienceData scienceNLPText processingText normalization

Tokenization implementation

Theory

Word tokens

Report a typo

There is a sentence in the input. Print the tokens using the method that returns word and punctuation tokens.

Tip: Do not forget to import the module you need

Sample Input 1: 

If you want to be happy, learn programming!

Sample Output 1: 

['If', 'you', 'want', 'to', 'be', 'happy', ',', 'learn', 'programming', '!']
Write a program in Python 3





___

Create a free account to access the full topic