Skip to content

Commit 6f56412

Browse files
tomlogicdpgeorge
authored andcommitted
py/lexer: Process CR earlier to allow newlines checks on chr1.
Resolves an issue where lexer failed to accept CR after line continuation character. It also simplifies the code.
1 parent 5feeba8 commit 6f56412

File tree

1 file changed

+7
-12
lines changed

1 file changed

+7
-12
lines changed

py/lexer.c

Lines changed: 7 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -137,23 +137,18 @@ STATIC void next_char(mp_lexer_t *lex) {
137137
lex->chr1 = lex->chr2;
138138
lex->chr2 = lex->reader.readbyte(lex->reader.data);
139139

140-
if (lex->chr0 == '\r') {
140+
if (lex->chr1 == '\r') {
141141
// CR is a new line, converted to LF
142-
lex->chr0 = '\n';
143-
if (lex->chr1 == '\n') {
144-
// CR LF is a single new line
145-
lex->chr1 = lex->chr2;
142+
lex->chr1 = '\n';
143+
if (lex->chr2 == '\n') {
144+
// CR LF is a single new line, throw out the extra LF
146145
lex->chr2 = lex->reader.readbyte(lex->reader.data);
147146
}
148147
}
149148

150-
if (lex->chr2 == MP_LEXER_EOF) {
151-
// EOF, check if we need to insert a newline at end of file
152-
if (lex->chr1 != MP_LEXER_EOF && lex->chr1 != '\n') {
153-
// if lex->chr1 == '\r' then this makes a CR LF which will be converted to LF above
154-
// otherwise it just inserts a LF
155-
lex->chr2 = '\n';
156-
}
149+
// check if we need to insert a newline at end of file
150+
if (lex->chr2 == MP_LEXER_EOF && lex->chr1 != MP_LEXER_EOF && lex->chr1 != '\n') {
151+
lex->chr2 = '\n';
157152
}
158153
}
159154

0 commit comments

Comments
 (0)