Re: Request for opinions: bug vs feature - changeintokenization of yield from

From: Date: Sun, 21 Jul 2024 11:28:57 +0000
Subject: Re: Request for opinions: bug vs feature - changeintokenization of yield from
References: 1 2 3 4 5 6 7 8 9 10 11  Groups: php.internals 
Request: Send a blank email to [email protected] to get a copy of this message 
Hi

On 7/20/24 20:31, Christoph M. Becker wrote:

If the tokenizer would tokenize a whole file as a single token, would
that also be correct?  Of course, I'm exaggerating, but


The function is documented as:


token_get_all() parses the given code string into PHP language tokens using the Zend engine's lexical scanner.


So if Zend engine's lexical scanner would tokenize a complete PHP file as a single token, then the output would be correct, yes.


<https://3v4l.org/qIf2c> doesn't look correct to me – "yield /* comment
*/ from" shouldn't be a single token.



Looking at this from a user expectation perspective and not a technical perspective, neither should yield from be a single token in:


    var_dump(token_get_all('<?php yield from $foo;'));


Best regards
Tim Düsterhus

Thread (32 messages)

« previous php.internals (#124533) next »