Generating Python in 2020?
I’m trying to figure out the right way to generate Python code programatically these days. I’m at best a very amateur Pythonista, and this post only covers a morning of web research. I’d love to get comments/advice!
My first assumption was that I’d use the language’s built-in ASTs and there would be a built-in “unparser” to spit out Python syntax. That turns out to only be true as of Python 3.9. For earlier versions there’s a host of options, some of which perhaps still apply to current versions. Here’s the rundown of my morning’s research.
Build Python asts and unparse them
As of Python 3.9, there is an ast.unparse method built into CPython.
For older versions of Python including Python 2, there is a 3rd-party library called astunparse, which is a refactoring of utilities from the Python 2 and 3 source trees, so presumably reliable and version-flexible.
Possible Alternative: build CSTs and unparse them
libCST is a well-supported library from Instagram for generating Concrete Syntax Trees, refactoring, and producing python output. Plenty of recent activity on their github. For parsing it depends on parso, which seems to be a well-supported 3rd-party library for parsing. Requires Python 3.6.
Get Joe Hellerstein’s stories in your inbox
Join Medium for free to get updates from this writer.
Note that there is a new parser as of Python 3.9. that does not generate CSTs, though it does generate backwards-compatible ASTs. There is a discussion in a bug report on this about the future of libCST and parso, and the community seems motivated to maintain them. Still, I’m not sure it makes sense to embrace CSTs in a new development effort.
Other Libraries
typed-astunparse is a version of astunparse that generates typed Python from a typed-ast. Requires Python 3.5.
horast is a wrapper on typed-ast and typed-astunparse to handle Comments and Directives.
redbaron appears to be a standalone parser and code modifier. Not a lot of activity in the repo.
What Do you Know?
I’d love to hear in the comments from others who’ve gone down this path!








