2024-03-11

Encodings

Text Input

$$\text{{text}} = \text{“Tokens in NLP.”}$$

Tokenised Letters

$$[\text{T}, \text{o},\text{k},\text{e},\text{n},\text{s},\text{ },\text{i},\text{n}, \text{ }, \text{N},\text{L}, \text{P}, \text{.}]$$

Index Encoding Scheme

$$[\text{’ ‘}: 0, \text{.}: 1, \text{L}: 2, \text{N}: 3, \text{P}: 4, \text{T}: 5,\text{e}: 6, \text{i}: 7, \text{k}: 8, \text{n}: 9, \text{o}: 10, \text{s}: 11]$$

Encoded Text

$$[5, 10, 8, 6, 9, 11, 0, 7, 9, 0, 3, 2, 4, 1]$$

One Hot Encoding