On 20.07.2024 at 18:51, Tim Düsterhus wrote:
>
>> And not, like it is now, an undocumented, random change creating an
>> inconsistency in the Tokenizer.
>
> The tokenizer is doing the right thing: It tokenizes the PHP source
> code. It is absolutely normal that PHP first and second-digit updates
> make changes to the token stream. New tokens are added, old tokens are
> removed, tokens may appear in places where they previously could not
> appear for well-formed PHP programs. Tools working on the token stream
> need to adapt and this change is no different from any other change to
> PHP's syntax in that regard (except that documenting the change was
> forgotten).
If the tokenizer would tokenize a whole file as a single token, would
that also be correct? Of course, I'm exaggerating, but
<https://3v4l.org/qIf2c> doesn't look correct to me
– "yield /* comment
*/ from" shouldn't be a single token.
Cheers,
Christoph