Lexical Analysis

Hello Dear Readers,

Let’s talk about something different today. I am currently in my 7th semester, and one of my core subjects is Compiler Construction. If you’re familiar with compilers, you would know they consist of different phases. The first phase of Compiler Construction is lexical analysis.

Lexical analysis involves tokenizing your code. Let me break this down: when you write code in a high-level language, the compiler’s first task is to break it into tokens. The purpose of tokenization is to convert everything you’ve written in your code into basic elements, such as keywords, operators, identifiers, literals, and punctuation. In the lexical analysis phase, the lexical analyzer reads the source code character by character and converts it into meaningful sequences based on the syntax of the programming language. This lexical analyzer is also called a scanner.

Here’s an example.

Let’s tokenize this basic C++ code.

Remember, tokenization doesn’t specifically start from the main function, and the header files are not excluded. Before lexical analysis begins, the preprocessor handles header files (i.e., #include statements). The preprocessor essentially copies the contents of the included header files into the source file before tokenization. Therefore, from the compiler’s perspective, these files are part of the source code and are tokenized along with everything else. They are not excluded from the tokenization phase.

Get Mahrukhrizvi’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

This is the tokenization of the above written Hello World Program.

1. int
2. main
3. (
4. )
5. {
6. cout
7. <<
8. “Hello, World!”
9. <<
10. endl
11. ;
12. return
13. 0
14. ;
15. }

Purpose of Lexical Analysis:

• Simplifies the parsing process: By converting source code into tokens, the syntax analysis (parsing) phase can more easily process the structure of the code.

Error handling: The lexer can detect errors such as invalid characters or incorrect identifiers during tokenization.