Computer scienceData scienceNLPText processingText normalization

Tokenization

What is a token?

Report a typo

Choose the right definition of the word token.

Select one option from the list

It's an individual characters or a symbol in a text

A complete word or sentence in a text corpus

It's a unit which can be obtained during tokenization

It's a unique identifier assigned to each word or phrase in a vocabulary

___

Create a free account to access the full topic