Skip to content

Parser doesn't properly handle non-BMP characters #498

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
robbie01 opened this issue May 2, 2025 · 1 comment
Closed

Parser doesn't properly handle non-BMP characters #498

robbie01 opened this issue May 2, 2025 · 1 comment
Assignees

Comments

@robbie01
Copy link

robbie01 commented May 2, 2025

It seems like the parser isn't properly handling a UTF-16 surrogate pair and tries to create an illegal character.

This precludes the use of the hy package (which implements the Hy language). It includes some exotic characters to test if a specific behavior of ast.parse occurs.

Traceback (most recent call last):
  File "/home/robbie/bookshelf/main.py", line 1, in <module>
    import hy, hy_main # type: ignore
  File "/home/robbie/bookshelf/.venv/lib/python3.11/site-packages/hy/__init__.py", line 11, in <module>
    import hy.importer  # NOQA
  File "/home/robbie/bookshelf/.venv/lib/python3.11/site-packages/hy/importer.py", line 13, in <module>
    from hy.compiler import hy_compile
  File "/home/robbie/bookshelf/.venv/lib/python3.11/site-packages/hy/compiler.py", line 14, in <module>
    from hy.errors import HyCompileError, HyLanguageError, HySyntaxError
  File "/home/robbie/bookshelf/.venv/lib/python3.11/site-packages/hy/errors.py", line 9, in <module>
    from hy.compat import PYPY, PY3_13
  File "/home/robbie/bookshelf/.venv/lib/python3.11/site-packages/hy/compat.py", line 22, in <module>
    if "def" in ast.unparse(ast.parse("𝕕𝕖𝕗 = 1")):
  File "/home/robbie/graalpy-24.2.1-linux-amd64/lib/python3.11/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 1
    𝕕𝕖𝕗 = 1
    ^
SyntaxError: invalid character '\udd55' (U+DD55)
@msimacek
Copy link
Contributor

msimacek commented May 2, 2025

I tried to run hy's tests and I made a fix for this and few other things. I got the tests to pass 597 out of 609 tests. The REPL works, although readline support is limited (no autocomplete etc.). PR in #500

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants