On 21 July 2024 19:33:11 BST, Juliette Reinders Folmer <[email protected]>
wrote:
>The crux - to me - is that it is an undocumented breaking change, which by definition is a bug.
There are two parts of this which are bugs, in my opinion:
- That it wasn't documented, e.g. with a line in UPGRADING listing the affected tokens.
- That the tokenisation consumes the comment as part of the token, rather than just performing a
lookahead.
One is easily fixed; the other is more subtle, but maybe fixable.
>As I've said before, I'm not against changing the tokenization, what I'm speaking
>up about is that it was done in an inconsistent, semi-random and undocumented way.
As others have said, there is nothing unusual in the process that was followed here. A minor change
was proposed via Pull Request, discussed with multiple core contributors, and wasn't deemed
significant enough for a wider discussion or RFC.
The documentation probably *should* have been caught during that review, because it's a common
checklist item. The behaviour of the token stream could have been, but we got unlucky and nobody
thought of it. There's no guarantee that a different process would have done better - there
have been changes which went through a whole RFC process, then a year later someone points out a
flaw that could have been avoided; that's life.
Now that we have spotted it, we need to decide what to do.
Regards,
Rowan Tommins
[IMSoP]