[venv] Adding a .gitignore file to virtual environments

[brettcannon]

BPO	39236
Nosy	@brettcannon, @vsajip, @asottile, @tirkarthi

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-01-06.19:45:05.714>
labels = ['type-feature', 'library', '3.9']
title = '[venv] Adding a .gitignore file to virtual environments'
updated_at = <Date 2020-01-07.16:32:50.101>
user = 'https://github.com/brettcannon'

bugs.python.org fields:

activity = <Date 2020-01-07.16:32:50.101>
actor = 'vinay.sajip'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-01-06.19:45:05.714>
creator = 'brett.cannon'
dependencies = []
files = []
hgrepos = []
issue_num = 39236
keywords = []
message_count = 2.0
messages = ['359459', '359517']
nosy_count = 4.0
nosy_names = ['brett.cannon', 'vinay.sajip', 'Anthony Sottile', 'xtreak']
pr_nums = []
priority = 'low'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue39236'
versions = ['Python 3.9']

Linked PRs

• GH-83417: Allow venv to add a .gitignore file to environments via a new scm_ignore_file parameter #108125

[brettcannon]

In a discussion on Twitter, the idea of having venv lay down a .gitignore file in a newly created virtual environment that consisted of nothing but * came up (https://twitter.com/codewithanthony/status/1213680829530099713). The purpose would be to help prevent people from inadvertently committing their venv to git. It seems pytest does something similar for .pytest_cache (got one complaint but have chosen to keep it otherwise).

To me this seems like a good enhancement. Since this would mostly benefit beginners then it should probably be an opt-out if we do it at all. Maybe make --no-ignore-file to opt out?

FYI Mercurial does not support subdirectory hgignore files like git does, so this may be git-specific (for now): https://www.selenic.com/mercurial/hgignore.5.html.

[vsajip]

I'm -0.5 on this. Beginners will probably shoot themselves in the foot multiple ways using git (or any DVCS), in terms of checking things in they didn't mean to (e.g. build artifacts which are not in the venv). As they can easily undo this, I don't think we need to do this.

[ofek]

I think this is a very good idea and not only would assist beginners but has come to be expected for tools that create directories that should not be version controlled.

[import-pandas-as-numpy]

Resurrecting this to add some thoughts.
To introduce myself, I run a third-party anti-malware scanning service against the Python Package Index.
We've observed a fairly long-standing trend of venv inclusion in source distributions.
This is problematic for a few reasons:

• Many of the commands in the virtual environment creation itself cause flags on our signature analysis schema. We've taken to 'whitelisting' virtual environment hashes on a per-file basis as we encounter them, but this is by no means evergreen. Whitespacing issues and minute differences in versioning have lead to a constant whack-a-mole of trying to maintain these hashes.
• This often dramatically increases the package size of smaller files. As I suspect is a 'granted', the individuals most predisposed to unintentional inclusion of their virtual environments in their workflows are also the most likely to have heavily polluted virtual environments in the first place.
• This creates a 'recursive checking' problem, where by we're also now checking every dependency of a particular package by virtue of enumerating the virtual environment. This might sound ideal, but as we scan all new and uploaded packages on the Python Package Index, these packages are "secure" in our eyes by virtue of simply existing in the first place.
• Because we aggregate the signatures that a package might display, including the virtual environment substantially increases the propensity that a package may be verdicted as potentially malicious. To me, it stands to reason that if you include enough code, anything is going to flag as malware without human analysis.

We would like to scan the package uploaded and make a determination on the code therein, less so the dependencies it uses. When we begin adding additional packages/libraries into the codebase, our whitelisting and other false-positive reducing techniques begin to degrade.

We've seen malicious modifications to virtual environments in the past, where the included libraries are used to contain malicious code or libraries that otherwise would be prohibited from the Python Package Index (or, in some cases, were already removed from the Python Package Index for being malicious.)

As PyPI shifts towards a third-party anti-malware solution, I think this would be extremely beneficial to revisit this conversation with that in mind.

[dstufft]

I'm +1 on this change, it's a reasonable mitigation to a common mistake that is trivial to undo if you really do want to check your venv into your git repo.

[shenanigansd]

Another +1, basically what dstufft said, I help teach Python on Discord, and I have had to help many beginners delete their venv(s) from git, but I've never had anyone actually want to upload it.

[hugovk]

+1, I've seen packages on PyPI that have accidentally included the venv and the packager had no idea before I reported it.

For comparison, some other tools' directories that include .gitignore files:

• .idea for PyCharm
• .mypy_cache
• .pytest_cache
• .ruff_cache
• .tox/[env-name]

And some common venv names are part of this Python template .gitigore: https://github.com/github/gitignore/blob/4488915eec0b3a45b5c63ead28f286819c0917de/Python.gitignore#L122-L129

[brettcannon]

@vsajip has your mind changed at all on this topic?

I also noticed that Nox inserts a .gitignore into the virtual environments it creates.

While we wait to here from Vinay, are there any other ignore files we should be considering, i.e., Mercurial, Bazaar, or Subversion (I listed those specifically as they are supported by packaging standards)? Should there be some CLI flag to disable the creation of the ignore file(s)?

[vsajip]

Well, I originally said I was -0.5 on this, and that's still my view (learners/novices uploading to PyPI could make all kinds of mistakes, and I'm not sure we want to try and catch them all), but I don't object strongly to this change if you want to go ahead and do it - please loop me in on the PR.

[adamchainz]

I also noticed that Nox inserts a .gitignore into the virtual environments it creates.

Maybe that is from the virtualenv package, which has been creating such files since v20.0.21 (2020-05-20):

$ cat .venv/.gitignore
# created by virtualenv automatically
*

Adding this to venv would be "upstreaming" the feature and keeping the two virtualenv packages more consistent.

The full list of precedents that I know of: Coverage.py, Mypy, pytest, and virtualenv. (And pending on tox.)

[ofek]

Hatch also does this

[brettcannon]

Apparently, .bzrignore supports the same syntax (if we want to bother with support Bazaar since it seems to no longer be supported).

[vsajip]

I'd say stick with .gitignore for now - I doubt there's enough demand for Bazaar support.

[hugovk]

Agreed, let's start with only Git.

Of the open source projects tracked by Open Hub, Bazaar accounts for ~0% (13,042 repos) compared to Git at 74% (1,079,642), and is the least popular of the five SCMs represented:

[adamchainz]

Better data for what is used now - Stack Overflow survey 2022: https://survey.stackoverflow.co/2022/#section-version-control-version-control-systems . 82% Git, 1% SVN, <1% Mercurial.

[brettcannon]

please loop me in on the PR.

#108125

[bersbersbers]

For anyone wondering why this seems necessary, because they have a .gitignore file in their .venv already: vscode-python has been doing this on creation of an environment from the beginning (late 2022).

[brettcannon]

vscode-python has been doing this on creation of an environment from the beginning (late 2022).

To be specific, we have been adding a .gitignore file since we introduced our opinionated "Python: Create Environment" command. Our experience informed my decision to push for this here as we have had zero issues since introducing it in the VS Code extension.