Skip to content

zipfile regression in 3.11 beta: ignores utf8 flag and removes it from existing entries #95463

Closed
@obfusk

Description

@obfusk

Bug report

apksigcopier CI started failing on 3.11 beta because of this seemingly unnecessary change in #32007 (which added fallback encoding support):

     def _encodeFilenameFlags(self):
         try:
-            return self.filename.encode('ascii'), self.flag_bits
+            return self.filename.encode('ascii'), self.flag_bits & ~_MASK_UTF_FILENAME
         except UnicodeEncodeError:

which results in:

  • the utf8 flag in ZipInfo.flag_bits (whether set manually or from an existing entry that already had it) being completely ignored (i.e. set/unset based on whether the file name can be encoded in ascii or requires utf8, regardless of the current value) when file headers and the central directory are written, whereas before it was already being set when required but not unset if not required (and having ascii filenames with the utf8 flag set is completely fine and common, but now impossible);
  • resultingly, removal of the utf8 flag from existing zip entries (with names that can be encoded in ascii but have it set nonetheless) in the central directory (though presumably not from the individual file headers for unmodified existing entries, making them inconsistent with the central directory), e.g. when appending to an existing zip file.

Which resulted in zip files that were no longer bitwise identical, making the existing lack of support for creating reproducible/deterministic zip files even worse than before.

Please revert this change; I'm happy to make a PR for that.

Your environment

  • CPython versions tested on: 3.11.0-beta.5 (Ubuntu 20.04.4, GitHub actions), 3.11.0-beta.4 (Debian testing/unstable)
  • Operating system and architecture: see above (both amd64)

Metadata

Metadata

Labels

3.11only security fixes3.12only security fixesrelease-blockertype-bugAn unexpected behavior, bug, or error

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions