BareASM 2: Developing a preprocessor

In this article we will design a preprocessor for our assembler. This preprocessor will only handle the macros in the specification, which we covered in our first article.

You can follow our project on our Substack or Medium, and you can find its full code on its GitHub repository.

Specification

Here we will make some additions to and clarify our specification.

Tabs will be used as indentation for labels and sublabels but may also be used at the end of lines, such as before comments.

Macros must exist on their own line; they can be preceded by whitespace. In this context, whitespace refers only to ‘ ‘ and ‘\t‘ characters. If, for example, the macro produces more code, the preceding whitespace will also precede all lines of code generated, i.e., ‘ #repeat 3 “db 0x00“‘ will produce ‘ db 0x00‘ on 3 separate lines.

Symbols in the preprocessor will be broken down into tokens in a slightly simpler way, such that strings will be considered a valid token regardless of what is between the quotes. Obviously still account for escape sequences, but beyond this, all non-symbol characters will be treated as whitespace. This would mean that code such as the following is valid: ‘#define+SYMBOL,”value”‘, ‘#include()”filename”’, etc. Code like this is still discouraged but would be accepted by the preprocessor.

Design

In this section we will cover how our preprocessor will work and how it fits into the broader assembler’s design.

After the ‘Assemble‘ function instantiates a preprocessor object, the constructor will load the entire file into memory as a vector of strings where each line is a separate string. This vector will be stored as a public member variable of the preprocessor object so the assembler can easily access it once it has been processed.

The preprocessor’s ‘Preprocess‘ function will iterate through each line in the file, and it will check for any symbols in the line and resolve them in the ‘ResolveSymbolsInLine‘ function; if no symbols have been defined, it will skip this stage. Then it will check if the first non-whitespace character in the line is a ‘#‘ and if so, it will process the line as a macro in the ‘ProcessMacro‘ function or skip to the next line if not.

The ‘ResolveSymbolInLine‘ function will iterate through each character in the line and will hold a temporary symbol string variable and a result string variable. All characters that are valid in a symbol will be concatenated to the symbol variable, and all non-valid characters for a symbol will cause the accumulated symbol characters to be checked against a map for any defined values; if so, then either the assigned value or the accumulated symbol characters will be concatenated to the result, the symbol variable will be reset to an empty string, and all non-valid symbol characters will also be concatenated to the result. This function will also have mechanisms to handle strings and allow symbols to be redefined.

The ‘ProcessMacro‘ function will split the line up into tokens using a basic lexical analyser. We will create a more developed one in our next article, but this will simply serve to tokenise macros, symbols, integers and strings and remove any comments or whitespace. Then it will take the first token, which would be the macro name, and use it to transfer execution to a function that can handle the appropriate macro, i.e., the ‘ProcessIncludeMacro, ‘ProcessDefineMacro’ or ‘ProcessRepeatMacro’ functions.

The ‘ProcessIncludeMacro‘, ‘ProcessDefineMacro‘ and ‘ProcessRepeatMacro‘ functions will perform the appropriate transformations to the source code and be able to handle any and all potential errors that may arise.

The following diagram depicts the flow of control for each macro between functions:

This design is intended to allow easy addition of more macros if needed and keeps the overall design of our system simple and easy to understand.

Implementation

We will include snippets of code for our preprocessor here. The full code can be found on GitHub.

Get The Routinely Interrupts’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

First we will add the following directories and files to our project:

- src/
  - inc/
    - preprocessor.hpp
  - src/
    - preprocessor.cpp
  - tests/
    - define.asm
    - repeat.asm
    - include.asm

We will also add a public function called ‘Assemble‘ to our assembler class which will return a bool and not take any arguments; this will include the following code:

bool Assembler::Assemble()
{
    Preprocessor preprocessor = Preprocessor(this->inputFilename);
    if (!preprocessor.success) {
        std::cerr << 
        return false;
    }
    
    if (!preprocessor.Preprocess()) {
        return false;
    }
    return true;
}

This code instantiates an object of the preprocessor class and passes in the source code file name and checks whether the file could be found by the constructor; then it will return a relevant error message or continue and begin preprocessing the source code and check whether that was successful.

The following code is from preprocessor.hpp:

#pragma once

#include <map>
#include <vector>
#include <string>

class Preprocessor {
private:
    std::map<std::string, std::string> definitions;

    static std::vector<std::string> SplitString(std::string str);
    static bool IsInteger(std::string str);
    static std::string ProcessString(std::string str, bool &success);
    static bool VerifySymbolSyntax(std::string symbol);

    std::string ResolveSymbolsInLine(std::string line);

    bool ProcessMacro(int line, int character);
    bool ProcessIncludeMacro(int line, std::string prefix, std::vector<std::string> tokens);
    bool ProcessDefineMacro(int line, std::vector<std::string> tokens);
    bool ProcessRepeatMacro(int line, std::string prefix, std::vector<std::string> tokens);
public:
    bool success;
    std::vector<std::string> data;

    Preprocessor(std::string filename);

    bool Preprocess();
};

Here we have declared a class and a set of member variables and functions as well as static helper functions. We will go through each of these functions in the order that they would be executed with code to illustrate how they achieve their purpose.

First the constructor, as invoked by the ‘Assemble‘ function, will ensure that the file is accessible through the following mechanism:

std::ifstream file(filename);
if (!file.good()) {
    success = false;
    return;
}

The publicly accessible member variable ‘success‘ is a bool which is initialised to true. After the constructor verifies the file is accessible, it will collect the data line by line into the data member variable of type ‘std::vector<std::string>’, where each line is a separate string.

The ‘Preprocess’ function is then also invoked by the ‘Assemble‘ function and will iterate through each line of the file. The following code shows this process:

for (int line = 0; line < this->data.size(); line++) {
    if (this->data[line]== "") { continue; }

    if (this->definitions.size() != 0) {
        this->data[line] = this->ResolveSymbolsInLine(this->data[line]);
    }

    for (int character = 0; character < this->data[line].length(); character++) 
    {
        if (this->data[line][character] != ' ' && this->data[line][character] != '\t' && this->data[line][character] != '#') {
            break;
        } else if (this->data[line][character] == '#') {
            if (!this->ProcessMacro(line, character)) {
                return false;
            }
            line--;
            break;
        }
    }
}

The code above will first check if the line is empty, and if so, it will skip it; otherwise, if any symbols have been defined, it will then try and resolve any symbols in the line. Then it will iterate through each character in the line from the beginning, checking if the first non-whitespace character found is a ‘#‘, in which case it will process the line as a macro by passing it into the ‘ProcessMacro’ function. Then it will check if the line was successfully processed and will decrease the line variable by one, as the macro will be removed, so the current line must be reprocessed.

The ‘ProcessMacro’ function will find the prefix to the macro, call a function to split the string into tokens (lexical analysis), and refer the execution to the appropriate function for the specific macro. It does this through the following code:

std::string prefix = this->data[line].substr(0, character);
std::vector<std::string> tokens = SplitString(this->data[line]);
this->data.erase(this->data.begin()+line);

std::transform(tokens[0].begin(), tokens[0].end(), tokens[0].begin(), ::toupper);
if (tokens[0] == "#INCLUDE") {
    return this->ProcessIncludeMacro(line, prefix, tokens);
} else if (tokens[0] == "#DEFINE") {
    return this->ProcessDefineMacro(line, tokens);
} else if (tokens[0] == "#REPEAT") {
    return this->ProcessRepeatMacro(line, prefix, tokens);
} else {
    std::cerr << "ERROR: Unknown macro, " << tokens[0] << ", on line: " << line+1 << std::endl; 
    return false;
}
return true;

The code shown above is responsible for finding the prefix to the macro, splitting the string, erasing the macro from the source code and finally setting the macro to uppercase, as the macro name is case-insensitive, and then it will transfer execution to the appropriate function and return an error if no function can be found to handle the macro.

Each of the three specific macro processing functions will first perform checks such as ensuring the correct number of arguments and that each of the arguments is in the correct format, returning error messages if necessary. Then they will perform the appropriate transformations to the source code.

For now, once preprocessing has finished, the preprocessor will print out the final source code ready for assembling to the user. We have also created 3 assembly code files for testing our preprocessor; these can be found in the tests directory on our GitHub.

This implementation should allow for easy addition of more macros and makes the source code easier to understand.

Conclusion

In this article we designed and implemented a basic preprocessor for our assembler. This article is up to date for commit 8cf1dff.

In the next article we will begin designing and implementing a lexical analyser to separate our source code up into relevant tokens.