gh-70870: Clarify dual usage of 'free variable'

[ncoghlan]

• Define closure variable and free variable in glossary
• Use free (closure) variable to link to the closure variable term when describing
  API elements that use "free" in the narrower "closure variable" sense rather than
  the broader formally defined sense of any reference to names other than those of
  local variables
• Switch entirely to the closure variable term when describing API elements that
  don't otherwise make any reference to the term "free"

Closes #70870
Closes #98117

• Issue: [doc] Questionable terminology ('free variables') for describing what locals() does #70870

---

📚 Documentation preview 📚: https://cpython-previews--122545.org.readthedocs.build/

[encukou]

Now, whenever you read free variable, you're left wondering about which of the two meanings is meant. Can we avoid that?

Would it make sense to accept the “pragmatic” meaning of free variable?

[edit: Oh, I understood this wrong: the distinction is in globals/builtins, not in variables used in an inner scope, right?]
..., and in the few cases that exclude variables only accessed in inner nested scopes, say that explicitly?

After all, “used” in executionmodel.html's definition is itself not too specific. Does a nonlocal without “actual” use count? Does a def statement that binds a cell from an outer scope count?

[ncoghlan]

Now, whenever you read free variable, you're left wondering about which of the two meanings is meant. Can we avoid that?

Would it make sense to accept the “pragmatic” meaning of free variable?

We can't, as the formal meaning isn't specific to Python.

The heart of the problem is that the use of the closure/free name pairing when cell variables were introduced was formally incorrect, since the named free variables were potentially only a subset of all free variables if a nested function also referenced global or builtin names. However, the docs haven't historically made that clear, they just called them free variables, so folks inferring the meaning of the phrase from the way those parts of the CPython docs used it would understandably conclude that it referred specifically to references to closure variables (e.g. I had to go read the code to determine whether the symtable docs were using "free" in its formal sense or its pragmatic sense, and it turned out to be the latter).

That naming decision was made decades ago though, so the best we can reasonably do is:

1. acknowledge that the phrase "free variable" is now unavoidably ambiguous at least as far as CPython is concerned, and to some degree for Python in general
2. indicate that some contexts (such as the formal language spec) always use one of the two meanings so it doesn't need to be spelled out on each use (this mostly applies to cases where "free variable" is being used correctly in the formal sense, since I didn't want to modify the related wording in the execution model docs as part of this PR)
3. where practical, update the wording of affected documentation to remove the ambiguity (this mostly involved either adding a "closure" qualifier or explicitly mentioning that globals and builtins were excluded when the term is being used in its pragmatic sense)

And yeah, references to variables in outer scopes are included in both the formal and pragmatic uses of the phrase. The difference is just that the pragmatic use omits any references to global and builtin names, since the compiler doesn't actually keep track of those outside the individual lookup opcodes, so the only way to get that list is to scan the full AST or opcode array for the code block.

(While the ambiguity is unfortunate, the situation isn't bad enough to justify making implementation changes solely due to this)

[encukou]

One more idea that might be off the mark: how about using nonlocal rather than closure variable?
The keyword already can't be used for global/built-in variables ( nonlocal scopes are the local scopes of the enclosing functions, so it's not a synonym for “non-local”). Some variables are nonlocal implicitly.

[ncoghlan]

We use nonlocal in the symtable API to specifically mean "marked with the nonlocal keyword", rather than as a general term for all closure variables.

The function object attribute is also called __closure__ rather than __nonlocals__, as is the parameter to exec and eval, so there's no getting away from closure variables being a term worth defining.

[ncoghlan]

(Converted to draft, since I don't know when I'll get back to this and incorporate Petr's feedback)

[JelleZijlstra]

I read the PR and all makes sense to me. Happy to take another look after you incorporate Petr's feedback.

[willingc]

Docs look good to me @ncoghlan. Thanks for the many clarifications.

[ncoghlan]

Thanks for the reviews, folks!

[miss-islington-app]

Thanks @ncoghlan for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

[bedevere-app]

GH-125088 is a backport of this pull request to the 3.13 branch.