Sun, 13 Apr 2025 17:53:43 +0000

The Tokenizer is arguably the most important module.

The Tokenizer takes the list generated from the Optimizer and iterates through each word. When it finds a keyword in the list, it goes through the following steps:
1. Finds the end of the declaration the keyword is indicating
2. Sends this full declaration to a function that'll parse it into a token

The final result is arranged as an array of tokens that emulates the steps the final executable must go through.