Hi
On 7/20/24 18:40, Juliette Reinders Folmer wrote:
Tim, you're making my point for me. This is *exactly* why the current
change should be reverted.
I am not sure how you read "PHP users have no idea what a token is" as an argument in favor of reverting the change, because reverting the change means that completely reasonable code suddenly stops working with a parser error in a patch version and PHP users will rightfully come to PHP's issue tracker to complain.
And not, like it is now, an undocumented, random change creating an
inconsistency in the Tokenizer.
The tokenizer is doing the right thing: It tokenizes the PHP source code. It is absolutely normal that PHP first and second-digit updates make changes to the token stream. New tokens are added, old tokens are removed, tokens may appear in places where they previously could not appear for well-formed PHP programs. Tools working on the token stream need to adapt and this change is no different from any other change to PHP's syntax in that regard (except that documenting the change was forgotten).
Best regards
Tim Düsterhus