s3tools-announce Mailing List for s3cmd

Dear s3cmd users and contributors,
I'm glad to announce that

s3cmd v2.4.0 has been released on Github, Sourceforge and PyPi.

This release should not contain any breaking change except a few output
message changes.
With this release, we reach the awesome milestone of exactly 100
contributors to s3cmd.

Thanks again to all the contributors, the testers and the issue reporters.


s3cmd-2.4.0      - 2023-12-12
===============
* Added "setversioning" command for versioning configuration (Kuan-Chun
Wang)
* Added "settagging", "gettagging", and "deltagging" commands for
bucket/object tagging (Kuan-Chun Wang)
* Added "setobjectretention" and "setobjectlegalhold" commands (Etienne
Adam/Withings SAS)
* Added "setownership" and "setblockpublicaccess" commands
* Added "cfinval" command to request Cloudfront to invalidate paths (#1256)
* Added "--keep-dirs" option to have the folder structure preserved on
remote side
* Added --skip-destination-validation option for "setnotification" command
(Kuan-Chun Wang)
* Added "--max-retries" flag and "max_retries" config option (#914)
* Added FIPS support (Michael Roth)
* Added "object ownership" and block public access" values to "info"
command output for buckets
* Added to "ls" command a "DIROBJ" tag for directory objects in S3 remote
* Added server profiles to run-tests.py to skip tests depending on the
server type
* Fixed "TypeError: sequence item 1: expected str instance, bytes found"
error with Python 3.12 (#1343)
* Fixed a missing return for "object_batch_delete" of S3.py  (James Hewitt)
* Fixed "object is not callable" error because of md5 FIPS test (#1005)
* Fixed "compute_content_md5 is not defined" error for "setversioning"
(#1312) (Gavin John)
* Fixed list objects to use NextMarker when only prefixes are returned
(Albin Parou)
* Fixed upload to not retry when an S3 compatible server is full
* Fixed recursive delete of objects named with whitespace (#976)
* Fixed the mime type when uploading directories to be
"application/x-directory"
* Fixed "string indices must be integers" error for sync when in dry-run
mode (#1313) (Shohei Tanaka)
* Fixed SignatureDoesNotMatch error when modifying an object on Cloudflare
R2 (#1332) (Philip C Nilsson)
* Fixed Cloudfront invalidation issue for paths with wildcard or special
characters
* Fixed Cloudfront crash because of error reporting for retries
* Fixed Cloudfront "unable to parse URL" error (#1292)
* Improved the handling of "empty" files on the remote side to sync with
local folders
* Improved "abortmp" command by requiring an object to avoid bad accidents
when using Ceph (Joshua Haas)
* Improved file download by retrying when encountering SlowDown or
TooManyRequests errors (Robin Geiger)
* Improved error messages in case of connection error or host unreachable
* Improved error messages to be more explicit for upload errors after
retries
* Improved remote2local attributes setting code
* Improved remote2local with more explicit error messages when setting
attributes (#1288)
* Improved remote2local output messages by using the "mkdir" prefix instead
of "make dir"
* Improved the SortedDict class
* Improved run-test.py by using "--include" when calling Curl instead of
"-include" (Matthew James Kraai)
* Improved GitHub CI by enabling pip cache in actions/setup-python (Anton
Yakutovich)
* Improved GitHub CI by adding a "codespell" check on push and PRs
(Yaroslav Halchenko)
* Updated GitHub CI tests to use more recent versions of Minio and Python
* Upgraded GitHub actions (Anton Yakutovich)
* Cleanup and update of copyright headers, docs, comments and setup.py
* Cleanup to fix "invalid escape sequence" syntax warnings
* Many other bug fixes and cleanups


Shortlog below.


Thanks,

Florent


Albin Parou (1):
      fix: Use NextMarker if only prefixes are returned

Anton Yakutovich (4):
      Bump GitHub actions
      Enable pip cache in actions/setup-python
      Extract deps from `setup.py` to `requirements.txt`
      Return additional pip install from `setup.py`

Etienne Adam (1):
      Add setobjectretention and setobjectlegalhold commands

Florent Viard (54):
      Change version to development 2.3.0-dev
      Merge pull request #1233 from maroth96/maroth96-add-fips-support
      Fixes #1005 - Fixes object is not callable error because of md5 FIPS
test
      Merge pull request #1277 from jeffwang0516/notification-option
      Merge pull request #1278 from Fumesover/master
      Better error message in case of connection error or host unreachable
      Retry a file download in case of SlowDown or TooManyRequests error
      Fixes #914 - Add "--max-retries" and "max_retries" config option
      Don't retry upload when server is full
      Improve error message to be more explicit for upload errors after
retries
      run-tests.py: Fixes os error when trying to skip first tests
      run-tests.py: Added server profiles to skip tests depending on the
server type
      Update CI test.yml to use last minio version and use python 3.10
instead of 3.10-dev
      Merge pull request #1281 from drakulavich/patch-1
      Merge pull request #1282 from drakulavich/ci/pip_cache
      Cloudfront: Remove wildcard import for exceptions
      Cloudfront: Fixes paths with wildcard or special characters to not be
invalidated
      Cloudfront: Fixes crash because of error reporting for retries
      Fixes #1256 - Add cfinval command to request cloudfront to invalidate
paths
      Merge pull request #1034 from Jamstah/patch-1
      Crypto: Remove old python legacy code
      Move hash functions into Crypto module
      Fixes #976 - Fixes recursive delete of objects named with whitespace
      Add to ls command a "DIROBJ" tag for directory objects in S3 remote
      Add "--keep-dirs" option to have the folder structure preserved on
remote side
      SortedDict: Add support for reversed() and ensure SortedDictIterator
is recognized as an iterable
      remote2sync: children files and folders have to be deleted before
parent folders
      remote2local: Use rmdir to delete folders
      run-tests.py: Add test for keep-dirs feature
      Refactor to treat copy_pairs like remote_list and update_list for
sync commands
      Set correct mime type when uploading directories
      Merge pull request #1287 from fviard/feature_keep_dirs
      Merge pull request #1289 from jahschwa/bug/abortmp-require-object
      Use "mkdir" prefix instead of "make dir" for remote2local output
messages
      Fixes #1288 - More explicit errors when setting attributes for
remote2local
      Cleanup of remote2local attributes setting code
      Don't use os.path for handling s3 paths
      Merge pull request #1284 from jeffwang0516/set-versioning
      Fixes #1292 - cloudfront: Parse all aws possible urls in
httpurl_to_s3uri
      Merge pull request #1314 from xcir/fix/issue1313
      CI: Add python 3.11 and 3.12 to the test matrix
      Fixes #1343 - bytes to be used for repl string in stripNameSpace
      cleanup: fixes "invalid escape sequence" Syntax Warnings
      Merge pull request #1324 from yarikoptic/enh-codespell
      cosmetic: comment typo fixes
      Merge pull request #1333 from PhilipNilsson7/master
      Merge pull request #1296 from kraai/curl-include-option
      Merge pull request #1297 from withings-sas/legalhold_and_retention
      cleanup: update copyright headers and other values
      Fixes #1359 - add setownership and setblockpublicaccess commands +
related entries in info
      cosmetic: typo fix
      Merge pull request #1283 from jeffwang0516/tagging-support
      s3cmd: handle cases of empty key or value for settagging
      Update version to 2.4.0

Gavin John (1):
      Fixes error "'compute_content_md5' is not defined" for setversioning

James Hewitt (1):
      Add missing return

Joshua Haas (2):
      bug: require object in abortmp
      make error message consistent

Matthew James Kraai (1):
      Pass --include to curl

Michael Roth (2):
      Add FIPS support
      Move md5 to BaseUtils

PhilipCNilsson7 (1):
      fix SignatureDoesNotMatch error when modifying an object on
Cloudflare R2

Shohei Tanaka (1):
      Fix type error(#1313)

Yaroslav Halchenko (5):
      Add github action to codespell master on push and PRs
      Add rudimentary codespell config
      ignores for codespell
      Fixed manually some ambigous typos
      [DATALAD RUNCMD] run codespell throughout

jeffwang0516 (5):
      Add option to skip destination validation when applying notification
config
      Update help message
      Support versioning configuration
      Refactoring
      Support bucket/object tagging command

Dear s3cmd users and contributors,

I'm glad to announce that

s3cmd v2.3.0 has been released on github, sourceforge and PyPi.

This release should not contain any breaking change.

Thanks again to all the contributors, the testers and the issue reporters.


s3cmd-2.3.0      - 2022-10-03
===============
* Added "getnotification", "setnotification", and "delnotification"
commands for notification policies (hrchu)
* Added support for AWS_STS_REGIONAL_ENDPOINTS (#1218, #1228) (Johan Lanzrein)
* Added ConnectionRefused [111] exit code to handle connection errors
(Salar Nosrati-Ershad)
* Added support for IMDSv2. Should work automatically on ec2 (Anthony Foiani)
* Added --list-allow-unordered to list objects unordered. Only
supported by Ceph based s3-compatible services (#1269) (Salar
Nosrati-Ershad)
* Fixed --exclude dir behavior for python >= 3.6 (Daniil Tararukhin)
* Fixed Cloudfront invalidate retry issue (Yuan-Hsiang Lee)
* Fixed 0 byte cache files crashing s3cmd (#1234) (Carlos Laviola)
* Fixed --continue behavior for the "get" command (#1009) (Anton Ustyugov)
* Fixed unicode issue with fixbucket (#1259)
* Fixed CannotSendRequest and ConnectionRefusedError errors at startup (#1261)
* Fixed error reporting for object info when the object does not exist
* Fixed "setup.py test" to do nothing to avoid failure that could be
problematic for distribution packaging (#996)
* Improved expire command to use Rule/Filter/Prefix for
LifecycleConfiguration (#1247)
* Improved PASS/CHECK/INCLUDE/EXCLUDE debug log messages
* Improved setup.py with python 3.9 and 3.10 support info(Ori Avtalion)
* Many other bug fixes


Shortlog below.


Thanks,
Florent


Anthony Foiani (2):
      use IMDSv2 to retrieve instance metadata
      make IMDSv2 optional

Carlos Laviola (1):
      treat 0 byte cache files as nonexistent

Daniil Tararukhin (1):
      fix os walk when excluding directories

Florent Viard (13):
      Update CI test.yml to run on PR
      Fixes #1261 - Always read a conn after a request to be able to reuse it
      Fixes #1259 - Fixes unicode issue with fixbucket
      Fixes #1259 - Correct typo when retrieving response from IMDSv2
      Ensure 404 exc content is filled even for object_info
      Fixes #1009 - Fixes --continue behavior for the "get" command
      Fixes #996 - Avoid crashing when setup.py test is called
      Fixes #1247 - Update expire command to use Rule/Filter/Prefix
for LifecycleConfiguration
      Fixes #1261 - Additional fix for CannotSendRequest at startup
      Fixes ConnectionRefusedError not found error for python 2
      Use str instead of repr in FileList debug logs
      Use quotes around files in FileLists for debug logs
      Update version to 2.3.0

Johan Lanzrein (1):
      Take into account the AWS_STS_REGIONAL_ENDPOINTS

Ori Avtalion (1):
      List support of Python 3.9, 3.10

Salar Nosrati-Ershad (3):
      excepting/handling tcp connection refused error
      feat: implement list objects allow-onordered option as a config
      feat: add `--list-allow-unorderd` command-line switch

Yuan-Hsiang Lee (1):
      fixes cloudfront invalidate retry

hrchu (1):
      Add notification commands

Dear s3cmd users and contributors,

I'm glad to announce that
s3cmd v2.2.0 has been released on github, sourceforge and PyPi.

This release is mostly made of bug fixes and should not contain any
breaking change.

Thanks again to all the contributors, the testers and the issue reporters.


s3cmd-2.2.0      - 2021-09-27
===============
* Added support for metadata modification of files bigger than 5 GiB
* Added support for remote copy of files bigger than 5 GiB using
MultiPart copy (Damian Martinez, Florent Viard)
* Added progress info output for multipart copy and current-total info
in output for cp, mv and modify
* Added support for all special/foreign character names in object
names to cp/mv/modify
* Added support for SSL authentication (Aleksandr Chazov)
* Added the http error 429 to the list of retryable errors (#1096)
* Added support for listing and resuming of multipart uploads of more
than 1000 parts (#346)
* Added time based expiration for idle pool connections in order to
avoid random broken pipe errors (#1114)
* Added support for STS webidentity authentication (ie AssumeRole and
AssumeRoleWithWebIdentity) (Samskeyti, Florent Viard)
* Added support for custom headers to the mb command (#1197) (Sébastien Vajda)
* Improved MultiPart copy to preserve acl and metadata of objects
* Improved the server errors catching and reporting for cp/mv/modify commands
* Improved resiliency against servers sending garbage responses
(#1088, #1090, #1093)
* Improved remote copy to have consistent copy of metadata in all
cases: multipart or not, aws or not
* Improved security by revoking public-write acl when private acl is
set (#1151) (ruanzitao)
* Improved speed when running on an EC2 instance (#1117) (Patrick Allain)
* Reduced connection_max_age to 5s to avoid broken pipes as AWS closes
https conns after around 6s (#1114)
* Ensure that KeyboardInterrupt are always properly raised (#1089)
* Changed sized of multipart copy chunks to 1 GiB
* Fixed ValueError when using more than one ":" inside add_header in
config file (#1087)
* Fixed extra label issue when stdin used as source of a MultiPart upload
* Fixed remote copy to allow changing the mime-type (ie content-type)
of the new object
* Fixed remote_copy to ensure that meta-s3cmd-attrs will be set based
on the real source and not on the copy source
* Fixed deprecation warnings due to invalid escape sequences
(Karthikeyan Singaravelan)
* Fixed getbucketinfo that was broken when the bucket lifecycle uses
the filter element (Liu Lan)
* Fixed RestoreRequest XML namespace URL (#1203) (Akete)
* Fixed PARTIAL exit code that was not properly set when needed for
object_get (#1190)
* Fixed a possible inifinite loop when a file is truncated during
hashsum or upload (#1125) (Matthew Krokosz, Florent Viard)
* Fixed report_exception wrong error when LANG env var was not set (#1113)
* Fixed wrong wiki url in error messages (Alec Barrett)
* Py3: Fixed an AttributeError when using the "files-from" option
* Py3: Fixed compatibility issues due to the removal of getchildren()
from ElementTree in python 3.9 (#1146, #1157, #1162, # 1182, #1210)
(Ondřej Budai)
* Py3: Fixed compatibility issues due to the removal of encodestring()
in python 3.9 (#1161, #1174) (Kentaro Kaneki)
* Fixed a crash when the AWS_ACCESS_KEY env var is set but not
AWS_SECRET_KEY (#1201)
* Cleanup of check_md5 (Riccardo Magliocchetti)
* Removed legacy code for dreamhost that should be necessary anymore
* Migrated CI tests to use github actions (Arnaud J)
* Improved README with a link to INSTALL.md (Sia Karamalegos)
* Improved help content (Dmitrii Korostelev, Roland Van Laar)
* Improvements for setup and build configurations
* Many other bug fixes


Shortlog below.


Thanks,
Florent


AJP (1):
      Fix printing error in `report_exception(e)`

Aaron Maxwell (17):
      Branch for Aaron / --follow-symlinks
      Provide --bucket-prefix for run-tests.py
      Test case for following symlinks (simplest case) (currently fails)
      Implementation of --follow-symlinks option (nonrecursive case)
      Test for s3cmd put --follow-symlinks
      Negative test for s3cmd put --follow-symlinks
      Manual page documentation for --follow-symlinks
      More thorough and precise tests for --follow-symlinks
      Guard against common error when using --bucket-prefix
      correct error in test
      use Autotest-3, not autotest-3
      Compartmentalized symlink-safe directory walk, with all tests
passing on python2.5 and python2.6
      Fix leak in recursive-symlink-filtering algorithm
      document fswalk function
      name refactor to indicate _fswalk_follow_symlinks is an internal
helper function
      Update Changelog
      Option to follow local symlinks for sync and put
(--follow-symlinks option) including tests and documentation.  Also
addition to run-tests.py: --bucket-prefix option, to allow different
developers to run tests in their own sandbox

Abhishek Varshney (1):
      fix: handle empty md5 in find_md5_one

Albert Bendicho (4):
      Off-by-one on MAX_CHUNKS verification
      max_chunks now a config option
      MAX_CHUNKS constant removed, now a config option
      Added multipart_max_chunks

Alec Barrett (1):
      Update wiki URL in error message

Aleksandr Chazov (4):
      Allow setting SSL authentication
      Update man pages
      make names more understandable
      allow skip key file config option for embedded keys case

Alex Harford (1):
      Ensure testsuite-out is created before running dd

Alex Offshore (1):
      Return decoded response in get_policy and get_cors

Alexander Allakhverdiyev (1):
      Fix Python 3 bytes string encoding when getting IAM credentials

Alexander Todorov (2):
      Disable hostname checks for SSL connections on RHEL 7.2
      In case check_hostname isn't defined catch the error

Andrew Gaul (7):
      Sort s3cmd ls output by bucket name
      Capitalize command descriptions for consistency
      Horizontally align object policy info
      Emit user metadata in object info
      Support signed URL content disposition type
      Emit storage class in info subcommand
      Handle missing ETag in subcmd_bucket_list

Anton I. Sipos (1):
      Fix crash when cloudfront operations receive 500 errors. Retries
parameter was not being sent to send_request as a keyword arg.

Arnaud J (1):
      Migrate CI tests to use github actions

Arto Jantunen (2):
      Add a close method to the ConnMan api
      Add option to disable connection pooling

Arturas Moskvinas (1):
      Do not close stdout, it might be used when file is downloaded
and destination is "-"

Billy Crook (2):
      Added support for listing StorageClass attribute. Disabled by
default.  Enable with list_storageclass = True in .s3cfg
      Adding -l option for long_listing list_storageclass now provided
by long_listing and -l. Minor whitespace cleanup.

Bobby Impollonia (1):
      Add missing import of errno in S3.py

Brady (2):
      Install instructions for PIP
      Correct typo in instructions

Brandon Galbraith (1):
      Update s3cmd.1

Brendan O'Connor (1):
      Bugfix: use extra headers -- allows Requester Pays for buckets
you don't own. Patch from http://arxiv.org/help/bulk_data_s3

Brian C. Lane (1):
      Fix handling of config tri-state bool values (like acl_public)

Børge Nerdal Kjelsrud (1):
      Add option to ignore error on copy when key no longer exists

Carlo Cabanilla (1):
      fix bug in datetime tuple concatenation

Castedo Ellerman (1):
      content_encoding != charset, bug #153

Charlie Schluting (2):
      s3cmd du can gobble gigs of RAM on a bucket with millions of
objects. Re-worked 'du' to traverse the structure and store only the
sum at each level, allowing python to free memory at each folder. Went
from 4GB consumption (and being killed) on a test bucket with ~50M
objects in thousands of directories, to a max of 80M usage.
      that's not necessary any more.

Chris Lamb (1):
      Support relative expiry times in signurl.

Christian Rodriguez (2):
      Fix #849 by adding requester pays header to HEAD requests
      Merge branch 'bug/849'

Christopher Noyes (3):
      allow adding encoding to content type in s3
      don't add charset if it's already there, make param a comma
delimited str, not multiple
      remove comments, add space after ;

Christopher Porter (1):
      Fixing bug 3091912 - KeyError when copying multiple keys

Cisco Guillaume (1):
      Merge pull request #1 from GuillaumeCisco/gcisco-fix-appdata-directory

Craig Ringer (1):
      Add support for creating signed S3 URLs with the new signurl command

Dale Lovelace (1):
      Allow --acl-grant on AWS groups e.g.

Damian Gerow (2):
      Support proxying SSL connections
      Only support proxied SSL with Python >= 2.7

Damian Martinez (1):
      Adding MultiPartCopy class and enable for object_copy when
remote file is greater than 5gb.

Dan Carroll (1):
      Fixed typo: paramters -> parameters

Daniel Gryniewicz (1):
      Add getlifecycle command

Daniel Harris (1):
      Fix potential divide by zero errors

Dave Cole (1):
      Apply include/exclude to destinations too, via
mdomsch/s3cmd@268e7c3bd37cf60920025d556449fc1b93a1c590 Refs
s3tools/s3cmd#109.

David Eckelkamp (1):
      Let Makefile determine version number from PkgInfo.py file

David Kohen (2):
      Add support for IAM roles and temp tokens
      IAM role typo fixes

David Liu (1):
      Fail silently on SIGPIPE

Devon Jones (3):
      added ability to set home based on environment variable
      added S3CMD_CONFIG documentation to the man page
      Merge remote-tracking branch 'upstream/master'

Dmitry Kamenka (1):
      Copy uri_params in bucket_list

Dongmao Zhang (2):
      add new option --limit-rate to throttle download/upload speed
      only gettimeofday when limitrate is required

Doug Crozier (4):
      Changed "Buckets list" test to use pbucket rather than hardcoded
names. This fixes issue #1066.
      Allow use of Docker to run test cases with multiple Python versions
      Fix for 'dictionary changed size during iteration' error on
Python 3 with --cache-file (Issue #945)
      Test cases added for --cache-file

EC2 Default User (1):
      .gitignore: add s3cmd.egg-info

Emmanuel (1):
      managed files that was not remote copy to upload them directly

EnlytonJ (1):
      Windows fix re / vs os.path.sep

Eric Connell (1):
      added the ability to upload files by reading from stdin

Eric Drechsel (1):
      Fix for ignored --default-mime-type option

Eric Mill (1):
      rewrite of README into markdown

Etienne Michon (1):
      Typo in manual

Eugene Brevdo (4):
      Added --continue-put and --upload-id, allowing skipping of files
or part files during upload when their size and md5sum match the file
being uploaded.
      Minor bugfix to --upload-id check
      Fixed a small bug in MultiPart upload
      Fixed a upload corruption bug in MultiPart continue.

Florent Viard (403):
      BugFix: Fix dateS3toPython to gracefully handle returned date
that are not in GMT to respect iso8601. Ex.: 2014-03-05T18:45:50+00:00
instead of 2014-03-05T18:45:50Z
      Fixes date functions to return timestamps based on UTC time
value. + so simplified formatDateTime().
      Minor changes in README to fix points 1 and 2 in my comment at
https://github.com/s3tools/s3cmd/pull/369.
      Cosmetic: removed one empty space in excess of end of run summary.
      Fix MD5 comparison with s3 compatible servers that are not Amazon S3.
      Small cleanup of code, using "in" instead of "find" for strings
when possible.
      Fix an error in my previous commit:
b5b61abc8eb159f5932b936c7ca47e618615f6a3. (.find() forgotten to be
removed)
      Improve the exit code of s3cmd in the case of an error from S3
to give a clearer status of the reason of the failure. Some existing
exit codes were modified when they were not already in use.
      Disable dns support for buckets when "host_bucket" hostname
doesn't have a %(bucket)s item in it.
      Backported changes from the following pull request:
https://github.com/s3tools/s3cmd/pull/369 regarding aws pricing.
      Updated README.md with latest changes of S3 prices:
https://aws.amazon.com/fr/blogs/aws/aws-data-transfer-price-reduction/?sc_ichannel=EM&sc_icountry=global&sc_icampaigntype=launch&sc_icampaign=EM_128506540&sc_idetail=em_1778261631&ref_=pe_411040_128506540_12
      Improvement of error reporting in the case of an issue when
importing S3 sub-files. Now correctly cover cases like syntax errors
in these files.
      Fixes #423 (https://github.com/s3tools/s3cmd/issues/423).
Autodetected locale default should be set in default config for only
being used when this field is not overriden, neither by command line
or by a config file.
      Fixes #448 : Fixes S3UriS3 regexp to also recognize "s3://" as a
valid S3UriS3 entity with no bucket and no object.
      Renaming some functions for the code to be more understandable.
      Fixes #424 : In python 2.7, parsing invalid xml now raise a
ParserError(that didn't exist in python <2.7) instead of a ExpatError
+ cleanup in Utils.py (see issue #424 for more details)
      Improved handling for errors 501 and 504: Error 501: It means
that a feature is not implemented in the server, so no need for
useless retries. Error 504: Gateway timeout situation, exit code
should indicate a server error.
      Cleanup: Add prologue to files to state that they are UTF-8 encoded.
      Fix run-test script to take into account the config file that is
specified on the command line.
      Fixes encoding/decoding issues and so support for all strange
characters (chinese, japanese, ...) in file names, folder or path
arguments (Should fix bug like #417i) when UTF-8 is not the system
def$ Now, in theory, everything that gets in should be converted to
unicoded and then things would be converted back to string (user
defined encoding) at the outmost parts when dealing with the system.
When talking to S3, convert unicode to strings in utf-8 encoding.
      Remove useless error import
      Pass2 of "fixes encoding/decoding issues": os.walk replaced to
handle unicodise conversion + fixed the removal of most of useless or
invalid unicodise/decodise in the code.
      Fix for issue #432. Read the response from httplib even if there
is a 0 byte to read. So httplib doesn't think that there is a response
waiting to be read when the next request come. Straightforward to fix
thanks to the detailed analysis of Edwintorok.
      Use deunicodise consistentely in object_copy and object_modify
      fix encoding : Added encode_to_s3 and decode_from_s3 function +
switched to "unicode" call instead of "decode" as it is supposed to be
more efficient.
      Fix wrong exception used for EOF from S3 error during recv_file.
(S3Error expect an http response, not a text message)
      For copy and modify, take care to check if there is an error
after the start of the operation (reported by a status 200 but an
"Error" root entry in the xml body) In addition, for the move, It is
good to check for the CopyObject xml reply to see if it is a success.
But, if there is a status of 200 and no "data" in response, don't
fail. Other S3 server implementations use only status code and don't
return any body when there is no error.
      Another fix related to fix encoding issues. (file.name /
stream.name are bytes, so unicodise them also)
      Fix invalid file transfer progress info in case of local2remote
and remote2remote: If there was failed copy, seq number was preserved
but previously uploaded file number was forgotten, thus user could
have faced an invalid indication like: 14/8 when there is 6 upload
following 6 remote copy failures.
      Add a link to the known issues page (on github) within the
header and footer of s3cmd generated error reports. The goal is to
reduce the number of duplicated "bug reports" for known issues.
      Fixes #528 (https://github.com/s3tools/s3cmd/issues/528).
There is a lot of different "python-magic" libraries/versions, all
doing different things, so all cases need to be covered. file-5.11 has
be switched with "filemagic" as it should be more commonly used.
Introspection in each python magic lib has been done to ensure to
handle encoding or not only when it is needed.
      Fixes issue #576 (https://github.com/s3tools/s3cmd/issues/576).
EX_SOMEFAILED was still used in a PR that was merged instead of
EX_PARTIAL.
      Merge pull request #572 from fviard/fix_encoding_magic
      Merge pull request #574 from koenrh/https-by-default
      Don't slurp whole file while computing SHA256 checksum. Rework
of the PR of petr-tichy. When using large --multipart-chunk-size-mb or
--disable-multipart, the whole file was read into memory.
      Merge pull request #580 from Tim-Anderson/error_stop
      Merge pull request #583 from Tim-Anderson/issue577
      Merge pull request #586 from Tim-Anderson/remove_ignore_failed_copy
      Merge pull request #587 from taz77/master
      Merge pull request #589 from jcwilson/master
      Merge pull request #560 from Flipboard/iter
      Merge pull request #593 from jrschumacher/set-cors
      Merge pull request #603 from mdomsch/bug/testsuite
      Following #603 : WARNING OUTPUT FORMAT CHANGE. Changed the
format of output strings to be more consistant. Warning, the format of
output for a lot of actions was changed. Thus, any automatic string
for parsing the output could have to be fixed when using s3cmd after
this checkin. Ex: "File hello/world.txt stored as
s3://mybucket/world.txt" will now be: "upload: 'hello/world.txt' ->
's3://mybucket/world.txt'
      Fixes #602 (and complete #603) Fix run-tests.py to have all
tests passing against HEAD
      If s3error message is none, don't print an useless ": None"
      Return NotFound error with message s3Error: NoSuchBucket when
there is no bucket found to delete
      Raise standard S3Error in S3.py instead of simply exiting. So
the normal error logic can't operate on any unexpected error. Avoid to
have empty errors in output like: "ERROR: S3 error:" (Because of an
400 InvalidBucketName error returned by a non AWS3 server)
      Merge pull request #590 from andrewgaul/content-disposition
      Fix stop_on_error config type
      Attempt to fix the first bug of issue #600 : When a subfolder
"xxx" was selected as source, the sync comparison was done against the
destination base whereas it had to be done based on the "xxx"
subfolder of the destination.
      Fixes run-tests following the fix for
https://github.com/s3tools/s3cmd/issues/600
      Merge pull request #616 from mdomsch/bug/612
      Fix bad help info for the "--no-check-certificate" switch.
Thanks to @psi29a for proposing the fix.
      Merge pull request #618 from mdomsch/bug/futurewarning
      Fixes #592 : Ignore any leading slash before bucket_name in an
S3Uri. Original issue: "Running s3cmd ls s3:///foo lists owned buckets
instead of emitting an error."
      Fix for error reporting in case of upload error. (Following the
fix for #577)
      Fixes #581 regarding the reporting of SSL verification issues.
Other problems of this issues should already be fixed.
      Merge pull request #621 from mdomsch/bug/ssl-checking
      Merge pull request #619 from fviard/second_fix_for_577
      Merge pull request #617 from fviard/fix_subfolder_source
      Merge pull request #623 from mdomsch/bug/testsuite2
      Reworked Upload and Download errors to be more consistent. Add
detection and handling of different OSErrors like "File name too
long."
      Merge pull request #628 from mdomsch/bug/curl
      Merge pull request #627 from fviard/upload_download_errors_rework
      Merge pull request #630 from mdomsch/feature/ia-class
      Merge pull request #611 from bsapiro/master
      Added --server-side-encryption-kms-id config option. Following PR #611.
      Fixes bug in PR #611
      Fixes #636. Also remove the temporary file when encountering an S3Error.
      Merge pull request #599 from shigezobriand/master
      Fix "info" command for servers not supporting the
"request_payer" feature. Based on PR #640 of Qiankunzheng.
      Fixes #644 : Minor fix to not output "u quoted" invalid
parameter in the case of an invalid command.
      Reworked source and destination input of "remote2local" to cover
gracefully more cases.
      Block "-" (stdin / stdout) to be used in parameters for sync as
it has no sense.
      Only warn for the non regular files that are skipped.
      In the case of local2remote, raise an error if a source arg is missing.
      Fix automatic tests for recent changes.
      Merge pull request #650 from fviard/fix-remote2local-input
      Don't fail the copy if a non AWS s3 server doesn't support setting ACLs.
      Have extra labels coherent when progress is enabled or not and
for multipart uploads.
      Added the --stats option that is similar to the one of rsync.
      Added "--stats" info for sync commands (remote2local,
remote2remote, local2remote)
      Merge pull request #652 from s3tools/bug/598
      Merge pull request #655 from fviard/rework_progress
      Fix duplicated code for the error alert message. Based on
@edunham PR: https://github.com/s3tools/s3cmd/pull/609
      Fixes #661 : Don't try to perform a delete if there is nothing
to delete. + Restore warning that was in 1.5.2.
      Fixes #659 : Don't fail if "content-type" field is missing in
info like for eucalyptus walrus.
      Fixes #646: number of "files to update" are not reported in the
summary (there was only the new files to be added)
      Fixes #646 : remote2remote sync -  "Updated" files were not
reported as to be remotely copied when in dry-run mode.
      Merge pull request #667 from kobake/skip-restore-already-in-progress
      When using --no-preserve, still add 'md5' custom attribute if
md5 check is enabled.
      Fixes #674 : When using AWS_CREDENTIAL_FILE, crash in case of
invalid key in .aws/credentials file + updated the code for actual
name of the keys.
      Merge pull request #711 from Shamanoid/patch-2
      Merge pull request #559 from jmif/master
      Merge pull request #701 from makery/master
      Don't ask to set access_token in run_configure as it should not
be used very often.
      Merge pull request #720 from harshavardhana/etag
      Merge pull request #684 from harshavardhana/try-fail
      Fix for local2remote in the case of "." as a source. Fixes #706
      Fixes #709 : Fixes sync command with windows + cleanup
      Fix the error message to avoid confusion in a special cases like
"no space left on device".
      Merge pull request #702 from artms/quiet
      Fixes #744 - Ensure to check non None or empty bucket names
before calling check_bucket_name.
      Merge pull request #755 from andrewgaul/list-missing-etag
      Merge pull request #714 from
alces-software/fix/generalize-wildcard-certificate-forgiveness
      Minor optimization and cosmetic fix in ConnMan.
      Fix Issue #672 - Patch based on PR #729 proposed by MLopez-Ibanez.
      Merge pull request #763 from dang/getlifecycle
      Fixes getBucketFromHostname for when host_bucket config has no
"bucket" element.
      Fix the infamous "UnboundLocalError" for conn exception hidding
meaningful exceptions
      Added an explicit error message when the servers hostname are
wrong or unresolvable.
      Avoid having an error when user uses "https://" prefix to host_base...
      Fixes #741 Issues validating ssl certs when using a custom port
for host_bucket address.
      Real fix for #772. Use urlparse to extract hostnames in connman
      Merge pull request #779 from cybojanek/master
      Follow up on #779, not using a CONNECT "tunnel" for http proxies
as it is not the most common case.
      Fixes #777 : Fix __repr__ returning unicode and sign_v2 not encoded to s3.
      Merge pull request #757 from mozawa/max-ls-keys
      Merge pull request #781 from kwarunek/master
      Fixes recursive delete that was broken by the new "truncated" tag.
      Fixes orphan var in "du" exception handler.
      run-tests.py Update expirations tests dates to the future.
      Merge pull request #784 from mozawa/configure-endpoint
      Merge pull request #785 from shaform/py3k
      Fix failure of copy or rename for s3-like servers that don't
have support for acl.
      Fix error of "info" command when s3-like servers don't implement
"acl" or another feature.
      Merge pull request #789 from shaform/py3k
      Merge pull request #787 from Abhishekvrshny/fix-empty-md5
      Avoid errors with modify command when s3-like servers don't support acl.
      Fix possible bug in previous change for info command and get_lifecycle.
      Fixes #722 : basename fonction was broken for local files for
windows as the os path separator is not the same on linux.
      Aws sometimes uses 301 code for redirects + redirect to aws base host.
      Fix signature v4 when aws redirect to aws base host.
      Fixes #778 : Workaround for aws sending early errors before
requests are completed + support for http expect continue feature.
      Updated maintainer in setup.py.
      Fixes #797 : unicodisation missing for exclude patterns coming from files.
      Fixes #800 (second part): "cfg.include" was not checked like is
cfg.exclude to use the non memory optimal del subcmd.
      Fix for part of #660: fix detection for "sync single-file.txt
s3://mybucket/single-file.txt"
      Fix to supporting python 1.6. Use "List" for http methods and
not a kind of dict.
      Backport more code from httplib for compatibility with python 2.6
      Fix previous patch on custom httpconnection, it is _send_output
and not send_output.
      Attemps to fix last issues of #769 .
      Initial version of a .travis.yml and it's .travis.s3cfg
configuration for CI with travis-ci.
      custom_httplib27 - Missing symbols in python 2.6
      Missing bit in previous commit.
      [travis] Fixes allow_failures not being strings.
      [travis] Added run-test-minio.py + have travis run it.
      Added travis-ci build status of master in the github displayed readme.
      Merge pull request #806 from shaform/py3k
      Merge pull request #808 from robd003/master
      Merge pull request #809 from shaform/py3k
      Fixes #810 : Also handle redirects for path styles syntax.
      Fixes #816 : Fix recv_file 3xx,4xx, handlers + add region
extraction of 3xx redirections.
      Fixes #798 : HEAD request can't parse xml response to get redirection url.
      Fix missing imports in custom_httplib27.
      Move urlencode func into Utils and fix encoding   issues when
having uid groupid with special chars.
      Small cleanup in S3.py
      Py3 support: remove usage of deprecated has_key.
      Py3 support: fix SortedDict (keys() are now views and iterators
need __next__)
      Update README to indicate python versions supported by s3cmd.
      Changed unicodise/deunicodise wrappers to fit with python3 string model.
      Python3 support: Fixed stderrr, stdout unicode output.
      Remove some code that was there for supporting py version < 2.6
      Use the stdout/stderr buffer directly instead of detaching.
      Python3 support: fix error reporting at import time and unicode.
      Python3 support WIP: Added Custom_httplib3x to patch python3
version of httplib as it is done for py2.
      [Py3 WIP] Fix for compatibility.
      PY3 support: fixed S3Uri and Exceptions regarding unicode/str
compatibility.
      Missing change that was needed for custom_httplib functions...
      PY3 support: more unicode related things.
      PY3 support: long was removed in py3. Replace by int
      PY3 support: ensure to use the adequate / sign for division with python3.
      PY3 support: Added a warning at runtime to warn of WIP
      PY3 support: Added silent versions for unicodise/deunicodise funcs
      Small cosmetic changes in Custom_httplib(s)
      Remove legacy code. s3cmd is now only compatible py >2.6
      Py3 support: Reworked request/signature regarding unicode/bytes mix.
      Py3 support: signature v2 - limit list of subresources url that are signed
      Py3 support: ensure that request body are encoded when "unicode" strings
      Signature v2: add "cors" to list of subresources to include in signature
      Improve get_bucket_location to be able to work all the time with aws
      Use pretty print everywhere in debug for request responses
      Getlocation for each type of request when region is not set and
signature v4 used
      PY3 support: more fixes for str/unicode related issues.
      Typo fix in debug log message
      Merge pull request #837 from alexjh/allow-skip-tests
      Merge pull request #851 from burstworks/master
      PY3 support: use getitem instead of getslices for slices
      PY3 support: Fix for regexp/unicode in runtest-minio
      Use io buffers everywhere for files or stdin/stdout + Use custom
stream_name attribute.
      Fixes signurl (v2) for python3.
      run-tests-minio: re-activated a test that is now working.
      Urlencode string only when formatting the url and not earlier.
      Get ride of the "extra" parameter for requests and use a clean
"params" dict for all the query strings.
      Ensure to not have "{}" in functions parameters to be safe.
      In the v4 canonical query string, parameter without values
should still have an equal sign.
      Fixes signature v2 invalid call to format_param_str.
      PY3 support: Fixes acl.py use of repr.
      Remove bad duplicated code.
      Little performance + memory improvement. + Fixes randomless in auto tests.
      Typo fixes.
      Sort headers in requests to always have the same order.
      Fixes failing tests with special chars for PY3
      Small changes on run-tests.py
      Change .travis.yml now all tests are passing with minio.
      Merge pull request #870 from yossi-stratoscale/issue-869
      Merge pull request #858 from chauffer/patch-1
      Documentation: Fixes #872 . Updated following the proposal of @cthorley.
      Merge pull request #857 from guygurari/cp-mv-bug
      Fix error message to be accurate with all cases + Fixed auto
tests for last change.
      Update version to 2.0.0
      Update setup.py classifiers to indicate that Python3 is supported.
      Changed version to 2.0.0+ for github nightly versions
      Fixes #887 - Invalid code inside cf __str__ breaking cf invalidation.
      Fixes #889 - info output cosmetic fix.
      Fixes #885 - Specific error message when getting policy by non owner.
      Small cleanup of legacy checks in Config.py
      Fixes #888 - Fixes cfinvalinfo broken in 2.0.0
      Fixes #886 - Don't force to have a / on last parameter for
"modify" command.
      Fixes #890 - Missing json import after cleanup patch.
      Fixes #892 - Ensure to add "LICENSE" file to release tarballs.
      Fixes #894 - Detect and report error 403 in getpolicy for info command
      Fixes #895 - Refactored py3 source to be byte-compilable with
py2 to fix warnings during "setup.py install"
      Remove python3 support warning.
      [PY3 support] Fixes --dump-config broken because of "verbosity"
      Cosmetic - use "with" instead of an open/close
      Fix encoding issues with config files + PY3 support for --configure
      Use "unicode" for config attributes
      Small rework for pre existing hash cache file detection
      PY3 fix: Open/Save hash cache file in "binary" mode
      Fix for non string cmdline parameters being ignored
      Cleanup: Don't have anymore cfg as a global
      Fixes missing local cfg in copy/move
      Fixes last missing local cfg
      Fixes #905 - Fixes https with proxy when no port defined for server.
      Fixes #908 - Fixes unicode in req headers for py2 and py3
      Merge pull request #917 from thelan/master
      Fixes #921 - fixes gpg_command for python3
      Fixed #922 - getenv return unicode in py3 and bytes in py2
      Merge pull request #923 from harshavardhana/fix-etag
      Fix unicode issue in report in the case of an error
      Fixes #920 Always treat hostnames in lower case. Signature issues.
      Merge pull request #925 from jmaitrehenry/marker
      Fix #922 - Handle libmagic missing on windows
      Fixes getBucketFromHostname not using "lowercase" host_bucket.
      Avoid crashing python when encountering a CertificateError
      Update version to 2.0.1
      Fixes sourceforge mailing list addr in RELEASE_INSTRUCTIONS file.
      Update version to 2.0.1+ for MASTER dev.
      Fixes #929 - strip namespace well even without xml declaration
      Fixes #933 Fix dst copy when the destination file was modified
      Merge pull request #938 from Nossnevs/feature/fix_nr_of_paths
      Fixes #525 - Fixed no invalidate of remotely copied files during sync
      Fixes error reporting for import error and utf8 characters as
cmdline arguments
      Added clarification comment for the socket_timeout configuration
value OS limit
      Partial fix for #941 - Retry for all kind of requests in case of
OS network timeout events
      Fixes #941 - Restore retries for some 400 errors like RequestTimeout
      Partial fix for #913 - Only throttle retries for s3 slowdown errors
      Fixes #913 - Fix limitrate to recover of sudden changes in
bandwidth and to not slow down too much the transfers
      Merge pull request #959 from mouchar/2.0.1
      Little help clarification for --delete-after. Fixes #958
      Merge pull request #947 from alexallah/master
      Addition to #947 - ensure it works with py2. aws requests always
returns utf-8 encoded data
      Merge pull request #940 from bcl/master-acl_public
      Fixes bug created by #940 - cmdline set values are real booleans
      Complete #940 - upload_id, content_disposition and content_type
default value is now "" and not anymore None
      Merge pull request #968 from LocutusOfBorg/avoid-distutils
      Merge pull request #970 from radeksimko/b-exclude-connection-header
      Merge pull request #975 from cybojanek/add_restore_subresource
      Merge pull request #966 from Trane9991/master
      Merge pull request #973 from mcv21/nulldisplay
      Fixes #960 - Do a retry if exc has errno but errno is None.
Python3 support.
      Fixes auto-test following recent aws changes regarding bucket
names. See: #957
      Fixes #967 - In addition with patch for #960, ensure that error
logging with not crash
      Update version to 2.0.2
      Fixes #987 - Fixes using IAM with python3. Config options should
be unicode.
      Fixes #987 - Before py36, py3 json was only able to load unicode str
      Merge pull request #986 from vpresnyak/output_alignment_fix
      Complete the fix #986 - To satisfy py2 and p3, there should be
float before and after the division
      Issue #956 - Fixes size reporting using k instead of K as it a
multiple of 1024
      Fixes #956 - fix size and alignement of du and ls output reporting
      Merge changes of release 2.0.2
      Merge pull request #994 from harshavardhana/no-need
      Merge pull request #1007 from hrchu/patch-1
      Merge pull request #1010 from starrify/fix-minor-help-message-typo
      Convert anything given to getTreeFromXml to utf8 bytes if needed
      Fixes #1006 - fixes multiple python3 encoding issues with
cloudfront functions
      Merge pull request #1021 from offshore/patch-1
      Merge pull request #1033 from junousi/public_url_https
      Fixes #1006 - missing return in previous commit
222af5d8ce90bbb2f7784360c019f5a6af2ef8bd
      Merge pull request #1042 from serprex/master
      Merge pull request #1061 from k0ste/help
      Merge pull request #1063 from timgates42/bugfix/typo_unnecessarily
      Change version to development 2.0.2+
      setup.cfg: set universal bdist_wheel flag
      setup.py: update meta files with support for recent python versions
      Merge pull request #1011 from zmedico/signurl_use_dns_based_on_bucket_host
      travis-ci: Disable tests for python 2.6 as not supported anymore
by travis-ci default ubuntu distribution
      Merge pull request #1071 from nitrocode/patch-1
      Complete the previous change to rename INSTALL file to INSTALL.md
      Merge pull request #1067 from NotTheDr01ds/bug/1066
      Merge pull request #1068 from NotTheDr01ds/feature/run-tests-in-docker
      run-tests.py: Don't fail ACL tests if public_url_use_https is used
      svnignore: update to match latest gitignore content
      Little cleanup. TODO is out of date and obsolete
      Add a .dockerignore because of this stupid docker that does not
know how to exclude files he has no business with
      Update run-tests.docker.README.md with a small display fix
      Create FUNDING.yml
      Factorize the region detection inner request and stop retrying
requests that fail the ssl certificate checks
      Fixes #1059 - Hack for SignatureDoesNotMatch error when host
port 80 or 443 is specified, due to stupid servers
      Merge pull request #1014 from Quiq/master
      Merge pull request #1032 from
vainu-arto/make-connection-pooling-configurable
      run-test-*: Minor fixes
      Small cleanup
      Fixes #377 - Avoid crashing when a file disappeared during cache
generation
      Fixes #1080 - url encode bucket_name for path-style requests.
Fixes an issue with CephRGW and the tenant feature.
      Fixes #1081 - Don't force signature v4 if bucket_name uses special chars
      Fixes #1082 - Allow to use signature v4 only, even for a command
with no target bucket specified
      Fixes #1054 - Relax limitation on special chars for --add-header key names
      Fixes #1020 - Update storage-class cmd help msg with new aws s3 classes
      Fixes open file descriptors leaking by not being closed on time
      Added a comment about how to escape the comment char in exclude files
      Fixes #989 - in aws_credential_file, fixes read_string only
available with py3 + refactor for possible unicode issues
      Fixes #847 - set a few str as unicode to fix UnicodeDecodeError
with python2 when doing a sync remote 2 remote
      Merge pull request #1084 from prabhakar267/master
      Fixes py headers to use system default python and not force
python2 anymore
      setup.py: Update authors and cleanup
      Update version to 2.1.0
      Change version to development 2.1.0+
      Fixes #1087 - fixes ValueError when using more than one ":"
inside add_header in config file
      Fixes #1089 - Ensure that KeyboardInterrupt are always properly raised
      Update RELEASE_INSTRUCTIONS with the new procedure to upload to Pypi
      Rework and cleanup of MultiPart and MultiPartCopy (following the
merge of PR #74)
      Add progress info output for multipart copy and current-total
info in output for cp, mv and modify
      Move s3_quote to Utils module
      Multipart copy - lower multipart copy chunk size to 1 GiB
      Move cmd - take care in that copy s3 xml result can be a
multipart copy result
      Url encode source for copy/mv/modify to support all
special/foreign character names in object names
      Cosmetic cleanup
      MultipartCopy avoid retrieving again object info when we already
know its size
      Ensure to catch and display server errors for cmd cp,mv,modify errors
      Cleanup - no message was set in 2 cases for S3UploadError
      MultiPart - Fixes extra label issue with stdin as source
      MultiPartCopy - Also copy acl like direct copy
      MultipartCopy - Preserve metadata of the object that is copied
      Fixes #1091 - Ugly hack to have modify work with files bigger than 5Gb
      Allows to set mime-type (ie content-type) also for remote copies also
      Allows to set public ACL with "modify" to match "copy"
      Remove legacy code for dreamhost. I guess that they are now supporting acl
      Small cleanup of _build_attr_header
      Fixes typo/failure in multipart cp when progress meter is off
      Refactored object_copy to avoid duplicated code in object_modify
      Fixes #685 - Grammatical correction a debug message
      Metadata for copy should  now be consistent for multipart or not
and for aws or not cases
      Fix remote_copy to ensure that meta-s3cmd-attrs will be set
based on the real source and not on the copy source
      Fixes _build_attr_header for case of wrong old value being use
for wrong attribute
      Simplify the code to use _build_attr_header in remote_copy
      Fixes #1088 - Handle servers indicating (lying) that listObject
result is truncated, when it is not the case
      Fixes #1090 - Be more resilient to a server sending random data
in ?lifecycle response
      Fixes #1093 - Fixes error encountered with Minio send empty
content instead of a valid ?requestPayment response.
      Merge pull request #1051 from chazer/ssl-auth
      Fixes #1094 - Fixes typo bug + mini cleanup
      Fixes #1096 - Add http error 429 to the retryable errors
      Fixes #1101 - "files-from" option was resulting in an
AttributeError with Python3
      Cleanup of a few lines ending with a ";"
      Cleanup - Remove unreachable code and an useless test
      Fixes #1113 - Fixes report_exception wrong error when LANG env
var was not set
      Fixes #346 - Add support for listing and resume of multipart
uploads of more than 1000 parts
      Utils/parseNodes: cleanup - avoid returning empty dicts inside
the result list
      Ref #1114 - Add a time based expiration for idle pool
connections in order to avoid random broken pipe errors
      Fixes #1114 - Reduce connection_max_age to 5s as it looks like
that aws closes https conns after around 6s
      Merge pull request #1118 from Patouche/issues/1117
      Merge pull request #1119 from Patouche/fix/datetime-tz
      Merge pull request #1120 from anbnyc/patch-1
      Merge pull request #1124 from tirkarthi/fix-warnings
      Merge pull request #1137 from ondrejbudai/python39-fixes
      Merge pull request #1141 from onagurna/master
      Merge pull request #1145 from xrmx/cleanup_check_md5
      travis.yml : Add python 3.9 to the ci test matrix
      Merge pull request #1144 from k3ntako/master
      Change import to have the py3 import first
      Merge pull request #1160 from siakaramalegos/patch-1
      Merge pull request #1165 from FFKL/master
      Merge pull request #1163 from Rjerk/wip-fix-empty-lc-prefix
      Merge pull request #1152 from orztt/wip-private-acl
      Change dev version to 2.1.0-dev to respect PEP 440 that is now
enforced by pip
      Fixes #1125 - avoid infinite loops when files are truncated
during hashsum or upload
      Config: Get token expiration from AssumeRoleWithWebIdentityResult response
      Minor fixes in s3cmd
      Create a new BaseUtils module for helper functions that don't
need access to Config
      Ensure that "https" url namespaces are also stripped in xml
      Complete work for AssumeRole And AssumeRoleWithWebIdentity support
      Use explicit raw strings for regexp patterns in Config
      Cleanup: Factor unicodise/deunicodise to be reused inside Config
      Fixes #1190 - Set properly PARTIAL exit code for object_get when needed
      Merge pull request #1199 from arnaudj/dev/migrateTravisToGitHubActions
      CI: Add python 3.10 to test matrix
      Merge pull request #1193 from rvanlaar/patch-1
      Merge pull request #1204 from akete/issue-1203
      Merge pull request #1198 from sebvajda/master
      Fixes #1201 - Raise an explicit error when only env var
AWS_ACCESS_KEY is set
      Update version to 2.2.0

Francois Gaudin (2):
      Fix custom header support for cp
      Modify command equivalent to a self copy to modify metadata

Francois Lebel (1):
      Fix the example for recursive listing.

François Wirion (1):
      Fix missing slash in directory semantics documentation

George Hickman (2):
      Don't include python files in the manifest.
      Add a makefile for releasing

George Melika (1):
      Added a --disable-content-encoding to disable adding the
content-encoding header to the pushed objects

Gianfranco Costamagna (2):
      Fix typo (overriden -> overridden)
      Add fallback code to make it compatible with Python2
implementation (we could simplify by adding whichcraft, but a new
dependency just for one which command seems an overkill)

Gianni Chiappetta (1):
      Always attempt to set the content-type

Giorgos Logiotatidis (1):
      If response['headers'] is empty, then don't update dict.

GuillaumeCisco (3):
      Fix app data directory
      Add .idea directory to gitignore
      Add fallback for old windows versions

Guy Gur-Ari (4):
      run-tests.py: use lowercase bucket name
      don't allow mv/cp of multiple files to single file
      raise exception for missing '/' during mv, cp
      remove unreachable code

Harshavardhana (5):
      s3cmd: Fix s3 path style signature v4 handling.
      Missing etag in HTTP headers, handle gracefully.
      build: Do not throw exceptions if no arguments were passed for setup.py
      Don't need to compute ETag md5sum match if it is not a real md5sum
      Do not convert all regions to lower case

Haruka Iwao (1):
      * S3/Config.py   Get user input aws access key, secret and gpg
passphrase out of debug format string

James Brown (1):
      Respect the $TZ environment variable to show local times if requested

Jan Kasiak (2):
      Change proxy SSL URI to relative
      Add restore subresource to V2 signature

Jason Dalton (8):
      Replaced mktemp with alternative temp file generation based on
secure randoms in order to avoid owner-restricted permission issues
with mktemp files
      Added cleanup of temp file if error or keyboard interrupt occurs
during transfer.
      cleanup
      Revert "cleanup"
      Revert "Replaced mktemp with alternative temp file generation
based on secure randoms in order to avoid owner-restricted permission
issues with mktemp files"
      Set permission on copied file based on umask.
      Updated references to temp file in exception handlers
      Relaxed timeout for http connection during IAM role lookup

Jason Godfrey (1):
      Quick fix for cfcreate and cf modify not working. These commands
would generate a KeyError for 'Origin'. (Bug 3536626)

Jens Braeuer (2):
      Added support for "Static Websites"
      Remove recursion detection for symlinks.

Jerome Leclanche (11):
      Avoid catching SystemExit in a try block
      Add __repr__ for S3Uri class
      Add S3.MultiPart module with basic functionality
      Make use of S3.MultiPart functionality by default
      Rename MultiPartUpload.bucket into MultiPartUpload.s3
      Implement multipart threading
      Use the ThreadPool interface to thread multipart uploads and
return a proper response
      Dynamically increase the chunk size depending on the file size
      Add the --enable-multipart option to s3cmd
      Add multipart_num_threads and multipart_chunk_size to Config
      Properly pass multipart enabling, default enabled on > 100MB

Joe Fiorini (1):
      Added command to set bucket access policy

Joe Mifsud (2):
      Add support for invalidating multiple CF distributions with
--cf-inval when the bucket is the origin for multiple distributions
      Switching to set()

Jose Fernandez (magec) (2):
      Make use of TMP variable to change tempdir
      Document Environment variables

Josep del Rio (4):
      When using default index, invalidate path
      Corrected the new flags for CF index invalidation
      Removed debug messages, make sure default index is defined
      Fixed exception for root document

Josh C. Wilson (1):
      move progress_meter default setting down to Config class

Jukka Nousiainen (2):
      Set public_url() protocol based on use_https
      Parameter for public URL protocol

Julien Maitrehenry (1):
      Change the marker value for NextMarker in bucket_list_streaming

Julien Recurt (1):
      Adding option signurl_use_https for signurl to use https prefix

Ka-Hing Cheung (1):
      enlarge write and read buffers

Karl Matthias (1):
      Add support for putting arbitrary additional headers in s3cfg file.

Karsten Sperling (15):
      Use python-magic for guessing MIME types if available
      Catch the right exception, doh.
      Added example /etc/magic file with better support for web file types
      Add support for setting Content-Encoding header where it can be
guessed, in particular Content-Encoding: gzip is handled.
      Flush stdout after logging messages, in case buffered pipes are involved
      Merge branch 'master' of git://github.com/s3tools/s3cmd
      Added example /etc/magic file with better support for web file types
      Add support for setting Content-Encoding header where it can be
guessed, in particular Content-Encoding: gzip is handled.
      Flush stdout after logging messages, in case buffered pipes are involved
      Fix up code that was merged incorrectly
      Merge branch 'mime-guessing'
      Add -M short option for --guess-mime-type
      Fix ValueError when magic module is not present
      Add support for http://pypi.python.org/pypi/filemagic version of
python-magic
      Merge branch 'mime-guessing'

Karthikeyan Singaravelan (1):
      Fix deprecation warnings due to invalid escape sequences.

Kelly McLaughlin (2):
      Handle empty return bodies when processing S3 errors.
      Reuse a single connection when doing a bucket list

Kentaro (2):
      Update base64.encodestring for Python 3.9 compatibility
encodestring() method was removed from base64 in Python 3.9.
      Update base64 import to support Python 2 and 3

Kevin Daub (5):
      Fixed typo in output messages.
      Added --server-side-encryption as an option and adds the header
to put and put-copy.
      Reverted out manpage update
      Added Server-Side Encryption (SSE) to output of --info
      Fixed formatting

Kiana McNellis (2):
      Fixed bug where the error report will not show if there was an
import error due to dependency on unicodise which was not successfully
imported
      Changed the exit code for import errors to not rely on
dependency of EX_GENERAL being defined when not imported

Koen Rouwhorst (1):
      Use HTTPS by default

Konstantin Shalygin (1):
      s3cmd: python 3.8 compat

Krzysztof Warunek (1):
      support endpoint with custom path

Lance Batson (1):
      fix issue with error display after config modify

Leandro Ostera (1):
      Update s3cmd --configure

Liu Lan (1):
      fix get bucket info error when LC use the filter element

Luke Winslow (1):
      Fixed issue with mixed path separators with s3cmd get
--recursive on Windows.

Marco Jakob (7):
      Get the access token from the environment
      Do not refresh role
      Do not refresh role
      Add option for token-refresh
      Add option for token-refresh
      Add option for token-refresh
      Add option for token-refresh

Mark Titorenko (2):
      Generalize wildcard certificate forgiveness
      Updated based on feedback.

Masashi Ozawa (2):
      add "limit" option to "ls" and "la" commands to return the
specified number of objects instead of returning all objects in the
bucket.
      Enable to configure "host_base" as "S3 Endpoint" in interactive
configuration.

Matt Domsch (485):
      Apply excludes/includes at local os.walk() time
      add --delete-after option for sync
      add more --delete-after to sync variations
      Merge remote-tracking branch 'origin/master' into merge
      Merge branch 'delete-after' into merge
      add Config.delete_after
      Merge branch 'delete-after' into merge
      fix os.walk() exclusions for new upstream code
      Merge branch 'master' into merge
      Merge branch 'master' of git://github.com/s3tools/s3cmd
      add --delay-updates option
      finish merge
      Handle hardlinks and duplicate files
      hardlink/copy fix
      remote_copy() doesn't need to know dst_list anymore
      handle remote->local transfers with local hardlink/copy if possible
      sync: add --add-destination, parallelize uploads to multiple destinations
      add local tree MD5 caching
      fix getting uid
      Apply excludes/includes at local os.walk() time
      add --delete-after option for sync
      add more --delete-after to sync variations
      add Config.delete_after
      fix os.walk() exclusions for new upstream code
      add --delay-updates option
      Handle hardlinks and duplicate files
      hardlink/copy fix
      remote_copy() doesn't need to know dst_list anymore
      handle remote->local transfers with local hardlink/copy if possible
      sync: add --add-destination, parallelize uploads to multiple destinations
      add local tree MD5 caching
      fix getting uid
      HashCache: add missing break during purge
      merge manpage conflict]
      Merge branch 'cache-local-md5' into hardlink-fixes
      sync: refactor parent/child and single process code
      Merge branch 'econnell/master' into econnell-merge
      merge relistan/master
      Merge remote-tracking branch 'brendano/master' into brendano-merge
      Merge remote-tracking branch 'ksperling/master' into ksperling-merge
      Merge remote-tracking branch 'michaeltyson/master' into michaeltyson-merge
      Merge remote-tracking branch 'joseprio/master' into joseprio-merge
      Merge remote-tracking branch 'ghickman/packaging' into ghickman-merge
      Merge remote-tracking branch 'mattswe/patch-1' into mattswe-merge
      Merge remote-tracking branch 'joefiorini/access-policy' into
joefiorini-merge
      Merge remote-tracking branch 'Roguelazer/tz_support' into Roguelazer-merge
      Merge remote-tracking branch 'godfreja/fix_cf_keyerror' into
godfreja-merge
      Merge remote-tracking branch 'manos/master' into manos-merge
      Merge remote-tracking branch 'ghickman/manifest-cleanup' into merge
      fix mime detection for Fedora python-mime
      fix multipart uploads
      fix --cf-invalidate after merge
      cannot use both --cf-invalidate and --add-destination
      note mdomsch merge branch in version string
      Merge remote-tracking branch 'ejnoyes/master' into test-merge
      Merge remote-tracking branch 'spikegronim/fix-hang-bug' into merge
      Merge remote-tracking branch 'tycho/master' into merge
      finish merge
      fix up failed merge of --delete-after-fetch
      update mdomsch-merge version string
      fixup trivial merge conflict
      remove unused and incorrect function hash_file()
      wrap get_md5() with try/except IOError
      remove debug output that mixes unicode and ascii strings poorly
      be verbose when both hardlinking and copying of local files fails
      update_acl(): fix output message
      Handle local hardlink and copy failures by downloading the objects
      add output on remote copy, fixes test cases 27 and 28
      add --files-from=FILE to allow transfer of select files only
      accept --files-from=- to read from stdin
      add --files-from to manpage
      use --files-from only on local source (not remote, not dest)
      fix sync --dry-run traceback, bug #124
      fix single file sync local2remote traceback, bug #128
      sync remote2remote: set target_uri on updated files too, bug 131
      Split md5/hardlink stuff out of SortedDict into FileDict
      fix sync single remote file to existing local file traceback, bug 88
      add --max-delete for del and sync commands, fixes bug #155
      sync: ignore --delete-removed if no source files, bug #155
      provide a better exception / error reporting message, bug #158
      Merge branch 'files-from' into merge, pull request #116
      Merge remote-tracking branch 'borgenk/master' into merge, PR #143
      Merge remote-tracking branch 'yar/master' into merge, PR #147
      Merge remote-tracking branch 'aisipos/master' into merge, PR #151
      Merge remote-tracking branch 'developmentseed/master' into merge, PR #152
      Merge branch 'bug/155' into merge, PR #156
      Merge branch 'bug/153' into merge, PR #157
      Merge remote-tracking branch 'abuchanan-grio/master' into merge, PR #162
      Merge remote-tracking branch 'tjasond/master' into merge, PR #163
      add --xattr option referring to file system extended attribute
containing md5sums
      Merge branch 'xattr' into merge
      fix conditional import xattr module
      Merge branch 'xattr' into merge
      push md5_xattr to Config only if xattr module was imported
      Merge branch 'xattr' into merge
      Don't emit "cowardly" message unless we have source files & delete files
      Merge branch 'bug/155' into merge
      Get md5 from remote attribute header in recursive syncs too
      Merge branch 'bug/remote-md5sum' into merge
      FileLists: get md5 from headers
      Merge branch 'bug/remote-md5sum' into merge
      move x-amz-meta-s3cmd-attrs handling into S3.py
      use metadata-stored md5 in bucket list, put, and info
      Merge branch 'bug/remote-md5sum' into merge
      look at metadata md5 for get as well
      Merge branch 'bug/remote-md5sum' into merge
      s3cmd get to stdout: set file_exists to avoid traceback, bug #175
      don't purge the HashCache if using --files-from
      HashCache: catch KeyError in unmark_for_purge(), bug #194
      HashCache and FileDict: ignore files with device = 0 or inode = 0
      Merge branch 'files-from' into merge
      Merge branch 'bug/194' into merge
      fixup merge
      Merge remote-tracking branch 'ivebeenlinuxed/master' into merge
      Merge remote-tracking branch 'reed-sandberg/master' into merge
      Merge remote-tracking branch 'kuenishi/more_multipart' into merge
      Merge remote-tracking branch 'scharissis/fix_help_typo' into merge
      finish merge of #186
      Merge remote-tracking branch 'orenhe/simplify_wildcard_handling'
into merge
      Merge remote-tracking branch 'robysath/master' into merge
      Merge remote-tracking branch 'spartman/remote-copy-fix' into merge
      remote_copy(): fix value in failed_copy_list
      Merge branch 'no-mime-magic-option' of
git://github.com/radomirml/s3cmd into merge
      Merge branch 'master' of git://github.com/brandongalbraith/s3cmd
into merge
      Merge branch 'listallmybuckets-error' of
git://github.com/leostera/s3cmd into merge
      Fix for ignored --default-mime-type option
      Merge branch 'bug/default-mime-type' into merge
      really merge @edrex https://github.com/s3tools/s3cmd/pull/244
      include python version in error dump
      remove python-2.5-ism when guessing mimetype (bug #218)
      don't hardlink 0-length files
      encode filenames to system encoding before calling shutils.copy2()
      getting the file system encoding is loop invariant
      Merge pull request #250 from mdomsch/bug/python-version-on-error
      Merge pull request #251 from mdomsch/bug/218
      Merge pull request #253 from mdomsch/bug/unicode-stat
      Print a useful error message when MemoryError is raised
      Merge pull request #259 from 36degrees/patch-1
      fixes for Windows not including a pwd module
      fix divide-by-zero when no files were transferred
      handle OSError: No such file or directory in get_local_filelist
      Merge branch 'bug/memoryerror' into merge
      Merge branch 'bug/divide-by-zero' into merge
      Merge branch 'bug/lstat-removed-file' into merge
      Merge branch 'bug/windows-no-pwd' into merge
      Merge git://github.com/robd003/s3cmd into merge
      --no-check-md5 should disable md5sum calculation, including
preserved attributes
      Merge branch 'bug/217' into merge
      fix put - s3://  (upload from stdin, #270)
      Merge branch 'bug/put-from-stdin' into merge
      os.gete[ug]id() doesn't exist on non-Unix
      Merge branch 'bug/put-from-stdin' into merge
      Merge git://github.com/flebel/s3cmd into merge
      add 'make tarball' and 'make rpm' to Makefile
      remove useless import of S3.Exceptions
      reduce "Process files that was not remote copied" to debug()
      add s3cmd.spec to .gitignore
      add a friendly error message on UnicodeEncodeError exception
      report LANG= environment variable on crash
      handle empty directory on S3 sync remote2local
      _download: Exception != OSError
      _download: handle S3Error
      Merge pull request #92 from
ksauzz/support-get-operation-to-multibyte-objectname
      Ignore S3 objects with empty names
      Merge pull request #284 from swiftserve/fix_config_file_verbosity
      set content-type, content-charset, and not content-encoding
      use ~/.s3cfg in run-tests too
      move empty object name detection where it belongs
      delete unused keys_to_remove logic
      s/0o775/0755/ for python 2.4 compatibility
      Merge pull request #286 from swiftserve/fix_verbosity_config
      Merge pull request #291 from Knewton/master
      Merge pull request #293...
 
[truncated message content]

Dear s3cmd users and contributors,

I'm glad to announce that

s3cmd v2.1.0 has been released on github, sourceforge and PyPi.

This release is mostly made of bug fixes and should not contain any
breaking change.

Thanks again to all the contributors, the testers and the issue reporters.


s3cmd-2.1.0      - 2020-04-07
===============
* Changed size reporting using k instead of K as it a multiple of 1024 (#956)
* Added "public_url_use_https" config to generate public url using
https (#551, #666) (Jukka Nousiainen)
* Added option to make connection pooling configurable and
improvements (Arto Jantunen)
* Added support for path-style bucket access to signurl (Zac Medico)
* Added docker configuration and help to run test cases with multiple
Python versions (Doug Crozier)
* Relaxed limitation on special chars for --add-header key names (#1054)
* Fixed all regions that were automatically converted to lower case
(Harshavardhana)
* Fixed size and alignment of DU and LS output reporting (#956)
* Fixes for SignatureDoesNotMatch error when host port 80 or 443 is
specified, due to stupid servers (#1059)
* Fixed the useless retries of requests that fail because of ssl cert checks
* Fixed a possible crash when a file disappears during cache generation (#377)
* Fixed unicode issues with IAM (#987)
* Fixed unicode errors with bucked Policy/CORS requests (#847) (Alex Offshore)
* Fixed unicode issues when loading aws_credential_file (#989)
* Fixed an issue with the tenant feature of CephRGW. Url encode
bucket_name for path-style requests (#1080)
* Fixed signature v2 always used when bucket_name had special chars (#1081)
* Allow to use signature v4 only, even for commands without buckets
specified (#1082)
* Fixed small open file descriptor leaks.
* Py3: Fixed hash-bang in headers to not force using python2 when
setup/s3cmd/run-test scripts are executed directly.
* Py3: Fixed unicode issues with Cloudfront (#1006)
* Py3: Fixed http.client.RemoteDisconnected errors (#1014) (Ryan Huddleston)
* Py3: Fixed 'dictionary changed size during iteration' error when
using a cache-file (#945) (Doug Crozier)
* Py3: Fixed the display of file sizes (Vlad Presnyak)
* Py3: Python 3.8 compatibility fixes (Konstantin Shalygin)
* Py2: Fixed unicode errors sometimes crashing remote2remote sync (#847)
* Added s3cmd.egg-info to .gitignore (Philip Dubé)
* Improved run-test script to not use hard-coded bucket names(#1066)
(Doug Crozier)
* Renamed INSTALL to INSTALL.md and improvements (Nitro, Prabhakar Gupta)
* Improved the restore command help (Hrchu)
* Updated the storage-class command help with the recent aws s3 classes (#1020)
* Fixed typo in the --continue-put help message (Pengyu Chen)
* Fixed typo (#1062) (Tim Gates)
* Improvements for setup and build configurations
* Many other bug fixes


Shortlog below.


Thanks,
Florent


Alex Offshore (1):
      Return decoded response in get_policy and get_cors

Arto Jantunen (2):
      Add a close method to the ConnMan api
      Add option to disable connection pooling

Doug Crozier (4):
      Changed "Buckets list" test to use pbucket rather than hardcoded
names. This fixes issue #1066.
      Allow use of Docker to run test cases with multiple Python versions
      Fix for 'dictionary changed size during iteration' error on
Python 3 with --cache-file (Issue #945)
      Test cases added for --cache-file

EC2 Default User (1):
      .gitignore: add s3cmd.egg-info

Florent Viard (36):
      Fixes #987 - Fixes using IAM with python3. Config options should
be unicode.
      Fixes #987 - Before py36, py3 json was only able to load unicode str
      Complete the fix #986 - To satisfy py2 and p3, there should be
float before and after the division
      Issue #956 - Fixes size reporting using k instead of K as it a
multiple of 1024
      Fixes #956 - fix size and alignement of du and ls output reporting
      Convert anything given to getTreeFromXml to utf8 bytes if needed
      Fixes #1006 - fixes multiple python3 encoding issues with
cloudfront functions
      Fixes #1006 - missing return in previous commit
222af5d8ce90bbb2f7784360c019f5a6af2ef8bd
      Change version to development 2.0.2+
      setup.cfg: set universal bdist_wheel flag
      setup.py: update meta files with support for recent python versions
      travis-ci: Disable tests for python 2.6 as not supported anymore
by travis-ci default ubuntu distribution
      Complete the previous change to rename INSTALL file to INSTALL.md
      run-tests.py: Don't fail ACL tests if public_url_use_https is used
      svnignore: update to match latest gitignore content
      Little cleanup. TODO is out of date and obsolete
      Add a .dockerignore because of this stupid docker that does not
know how to exclude files he has no business with
      Update run-tests.docker.README.md with a small display fix
      Create FUNDING.yml
      Factorize the region detection inner request and stop retrying
requests that fail the ssl certificate checks
      Fixes #1059 - Hack for SignatureDoesNotMatch error when host
port 80 or 443 is specified, due to stupid servers
      run-test-*: Minor fixes
      Small cleanup
      Fixes #377 - Avoid crashing when a file disappeared during cache
generation
      Fixes #1080 - url encode bucket_name for path-style requests.
Fixes an issue with CephRGW and the tenant feature.
      Fixes #1081 - Don't force signature v4 if bucket_name uses special chars
      Fixes #1082 - Allow to use signature v4 only, even for a command
with no target bucket specified
      Fixes #1054 - Relax limitation on special chars for --add-header key names
      Fixes #1020 - Update storage-class cmd help msg with new aws s3 classes
      Fixes open file descriptors leaking by not being closed on time
      Added a comment about how to escape the comment char in exclude files
      Fixes #989 - in aws_credential_file, fixes read_string only
available with py3 + refactor for possible unicode issues
      Fixes #847 - set a few str as unicode to fix UnicodeDecodeError
with python2 when doing a sync remote 2 remote
      Fixes py headers to use system default python and not force
python2 anymore
      setup.py: Update authors and cleanup
      Update version to 2.1.0

Harshavardhana (1):
      Do not convert all regions to lower case

Jukka Nousiainen (2):
      Set public_url() protocol based on use_https
      Parameter for public URL protocol

Konstantin Shalygin (1):
      s3cmd: python 3.8 compat

Pengyu Chen (1):
      Fixed: Minor typo in help message for `--continue-put`

Prabhakar Gupta (1):
      Fix INSTALL.md formatting issues

Ryan Huddleston (1):
      With this commit 4 years ago OSError was caught:
https://github.com/s3tools/s3cmd/commit/889c5a7ba1ef8fc48f0c1be593e7ddf1e518a408

Tim Gates (1):
      Fix simple typo: unecessarily -> unnecessarily

Vlad Presnyak (1):
      Fixing proper handling of file sizes. Will correctly return
floating point or integer as appropriate.

Zac Medico (1):
      Fix signurl to support path-style bucket access via
check_bucket_name_dns_support

hrchu (1):
      Update restore cmd help

matteobar (1):
      Copyright year updated

nitro (1):
      README: brew install and renamed to INSTALL.md

Dear s3cmd users and contributors,

I'm glad to announce that

s3cmd v2.0.2 has been released on github, sourceforge and PyPi.

This release is mostly made of bug fixes.

Thanks again to all the contributors, the testers and the issue reporters.

 s3cmd-2.0.2      - 2018-07-15
===============
* Fixed unexpected timeouts encountered during requests or transfers
due to AWS strange connection short timeouts (#941)
* Fixed a throttle issue slowing down too much transfers in some cases (#913)
* Added support for $AWS_PROFILE (#966) (Taras Postument)
* Added clarification comment for the socket_timeout configuration
value OS limit
* Avoid distutils usage at runtime (Matthias Klose)
* Python 2 compatibility: Fixed import error of which with fallback
code (Gianfranco Costamagna)
* Fixed Python 3 bytes string encoding when getting IAM credentials
(Alexander Allakhverdiyev)
* Fixed handling of config tri-state bool values (like acl_public)
(Brian C. Lane)
* Fixed V2 signature when restore command is used (Jan Kasiak)
* Fixed setting full_control on objects with public read access (Matthew Vernon)
* Fixed a bug when only one path is supplied with Cloudfront. (Mikael Svensson)
* Fixed signature errors with 'modify' requests (Radek Simko)
* Fixes #936 - Fix setacl command exception (Robert Moucha)
* Fixes error reporting if deleting a source object failed after a move (#929)
* Many other bug fixes (#525, #933, #940, #947, #957, #958, #960, #967)

Important info:
AWS S3 doesn't allow anymore uppercases and underscores in bucket
names since march 1, 2018

Shortlog below.


Thanks,
Florent


Alexander Allakhverdiyev (1):
      Fix Python 3 bytes string encoding when getting IAM credentials

Brian C. Lane (1):
      Fix handling of config tri-state bool values (like acl_public)

Florent Viard (18):
      Fixes sourceforge mailing list addr in RELEASE_INSTRUCTIONS file.
      Update version to 2.0.1+ for MASTER dev.
      Fixes #929 - strip namespace well even without xml declaration
      Fixes #933 Fix dst copy when the destination file was modified
      Fixes #525 - Fixed no invalidate of remotely copied files during sync
      Fixes error reporting for import error and utf8 characters as
cmdline arguments
      Added clarification comment for the socket_timeout configuration
value OS limit
      Partial fix for #941 - Retry for all kind of requests in case of
OS network timeout events
      Fixes #941 - Restore retries for some 400 errors like RequestTimeout
      Partial fix for #913 - Only throttle retries for s3 slowdown errors
      Fixes #913 - Fix limitrate to recover of sudden changes in
bandwidth and to not slow down too much the transfers
      Little help clarification for --delete-after. Fixes #958
      Addition to #947 - ensure it works with py2. aws requests always
returns utf-8 encoded data
      Fixes bug created by #940 - cmdline set values are real booleans
      Complete #940 - upload_id, content_disposition and content_type
default value is now "" and not anymore None
      Fixes #960 - Do a retry if exc has errno but errno is None.
Python3 support.
      Fixes auto-test following recent aws changes regarding bucket
names. See: #957
      Fixes #967 - In addition with patch for #960, ensure that error
logging with not crash

Gianfranco Costamagna (1):
      Add fallback code to make it compatible with Python2
implementation     (we could simplify by adding whichcraft, but a new
dependency just for one which command     seems an overkill)

Jan Kasiak (1):
      Add restore subresource to V2 signature

Matthew Vernon (1):
      Allow setting full_control on objects with public read access

Matthias Klose (1):
      Avoid distutils usage at runtime

Mikael Svensson (1):
      Fixes bug when only one path is supplied. Before it displayed
the nr of characters in the string.

Radek Simko (1):
      Avoid passing 'connection' header to 'modify' requests

Robert Moucha (1):
      Fixes #936 - Fix setacl command exception

Taras (1):
      replace `Config()` with `self`

Taras Postument (6):
      Issue #906 add support for $AWS_PROFILE
      make it work with python2
      Completed some PR tasks:- better imports handling- more debug
logging- better error logging- use `Config().update_option()` to set
keys
      pr tasks:     - unicodize env variables as well     - treat each
param from aws  credentials as optional     - print errors from
reading aws credentials file as warnings
      add helper function to read the deprecated credentials format
      add more comments

Dear s3cmd users and contributors,

I'm glad to announce that

s3cmd v2.0.1 has been released on github, sourceforge and PyPi.

This release is mostly made of bug fixes.
A lot of regressions for Python 3 have been fixed and Python 3 support
should now be good enough.

Thanks again to all the contributors, the testers and the issue reporters.

s3cmd-2.0.1      - 2017-10-21
===============
* Support for Python 3 is now stable
* Fixed signature issues due to upper cases in hostname (#920)
* Improved support for Minio Azure gateway (Julien Maitrehenry,
Harshavardhana)
* Added signurl_use_https option to use https prefix for signurl (Julien
Recurt)
* Fixed a lot of remaining issues and regressions for Python 3 (#922, #921,
#908)
* Fixed --configure option with Python 3
* Fixed non string cmdline parameters being ignored
* Windows support fixes (#922)
* Don't force anymore to have a / on last parameter for the "modify"
command (#886)
* Removed the python3 support warning
* Detect and report error 403 in getpolicy for info command (#894)
* Added a specific error message when getting policy by non owner (#885)
* Many other bug fixes (#905, #892, #890, #888, #889, #887)

Shortlog below.


Thanks,
Florent


Florent Viard (35):
      Changed version to 2.0.0+ for github nightly versions
      Fixes #887 - Invalid code inside cf __str__ breaking cf invalidation.
      Fixes #889 - info output cosmetic fix.
      Fixes #885 - Specific error message when getting policy by non owner.
      Small cleanup of legacy checks in Config.py
      Fixes #888 - Fixes cfinvalinfo broken in 2.0.0
      Fixes #886 - Don't force to have a / on last parameter for "modify"
command.
      Fixes #890 - Missing json import after cleanup patch.
      Fixes #892 - Ensure to add "LICENSE" file to release tarballs.
      Fixes #894 - Detect and report error 403 in getpolicy for info command
      Fixes #895 - Refactored py3 source to be byte-compilable with py2 to
fix warnings during "setup.py install"
      Remove python3 support warning.
      [PY3 support] Fixes --dump-config broken because of "verbosity"
      Cosmetic - use "with" instead of an open/close
      Fix encoding issues with config files + PY3 support for --configure
      Use "unicode" for config attributes
      Small rework for pre existing hash cache file detection
      PY3 fix: Open/Save hash cache file in "binary" mode
      Fix for non string cmdline parameters being ignored
      Cleanup: Don't have anymore cfg as a global
      Fixes missing local cfg in copy/move
      Fixes last missing local cfg
      Fixes #905 - Fixes https with proxy when no port defined for server.
      Fixes #908 - Fixes unicode in req headers for py2 and py3
      Merge pull request #917 from thelan/master
      Fixes #921 - fixes gpg_command for python3
      Fixed #922 - getenv return unicode in py3 and bytes in py2
      Merge pull request #923 from harshavardhana/fix-etag
      Fix unicode issue in report in the case of an error
      Fixes #920 Always treat hostnames in lower case. Signature issues.
      Merge pull request #925 from jmaitrehenry/marker
      Fix #922 - Handle libmagic missing on windows
      Fixes getBucketFromHostname not using "lowercase" host_bucket.
      Avoid crashing python when encountering a CertificateError
      Update version to 2.0.1

Harshavardhana (1):
      Don't need to compute ETag md5sum match if it is not a real md5sum

Julien Maitrehenry (1):
      Change the marker value for NextMarker in bucket_list_streaming

Julien Recurt (1):
      Adding option signurl_use_https for signurl to use https prefix

matteobar (1):
      Added note on new Python 3+ support

Dear s3cmd users and contributors,

I'm glad to announce that

s3cmd v2.0.0 is now available on github, sourceforge, and PyPi.

After almost 1 year and half, and the contributions from 23 different
developers,
this release is an important milestone because it adds the support for
Python 3!

This is my first release and so I would like to thanks Matt Domsch for his
support
and all the contributors, the testers and the issue reporters.


s3cmd-2.0.0      - 2017-06-26
===============
* Added support for Python 3 (Shaform, Florent Viard)
* Added getlifecycle command (Daniel Gryniewicz)
* Added --cf-inval for invalidating multiple CF distributions (Joe Mifsud)
* Added --limit to "ls" and "la" commands to return the specified number of
objects (Masashi Ozawa)
* Added --token-refresh and --no-token-refresh and get the access token
from the environment (Marco Jakob)
* Added --restore-priority and --restore-days for S3 Glacier (Robert Palmer)
* Fixed requester pays header with HEAD requests  (Christian Rodriguez)
* Don't allow mv/cp of multiple files to single file (Guy Gur-Ari)
* Generalize wildcard certificate forgiveness (Mark Titorenko)
* Multiple fixes for SSL connections and proxies
* Added support for HTTP 100-CONTINUE
* Fixes for s3-like servers
* Big cleanup and many unicode fixes
* Many other bug fixes


Shortlog below.

Thanks,
Florent


Abhishek Varshney (1):
      fix: handle empty md5 in find_md5_one

Alex Harford (1):
      Ensure testsuite-out is created before running dd

Andrew Gaul (1):
      Handle missing ETag in subcmd_bucket_list

Arturas Moskvinas (1):
      Do not close stdout, it might be used when file is downloaded and
destination is "-"

Christian Rodriguez (1):
      Fix #849 by adding requester pays header to HEAD requests

Daniel Gryniewicz (1):
      Add getlifecycle command

David Eckelkamp (1):
      Let Makefile determine version number from PkgInfo.py file

Florent Viard (96):
      Don't ask to set access_token in run_configure as it should not be
used very often.
      Fix for local2remote in the case of "." as a source. Fixes #706
      Fixes #709 : Fixes sync command with windows + cleanup
      Fix the error message to avoid confusion in a special cases like "no
space left on device".
      Fixes #744 - Ensure to check non None or empty bucket names before
calling check_bucket_name.
      Minor optimization and cosmetic fix in ConnMan.
      Fix Issue #672 - Patch based on PR #729 proposed by MLopez-Ibanez.
      Fixes getBucketFromHostname for when host_bucket config has no
"bucket" element.
      Fix the infamous "UnboundLocalError" for conn exception hidding
meaningful exceptions
      Added an explicit error message when the servers hostname are wrong
or unresolvable.
      Avoid having an error when user uses "https://" prefix to host_base...
      Fixes #741 Issues validating ssl certs when using a custom port for
host_bucket address.
      Real fix for #772. Use urlparse to extract hostnames in connman
      Follow up on #779, not using a CONNECT "tunnel" for http proxies as
it is not the most common case.
      Fixes #777 : Fix __repr__ returning unicode and sign_v2 not encoded
to s3.
      Fixes recursive delete that was broken by the new "truncated" tag.
      Fixes orphan var in "du" exception handler.
      run-tests.py Update expirations tests dates to the future.
      Fix failure of copy or rename for s3-like servers that don't have
support for acl.
      Fix error of "info" command when s3-like servers don't implement
"acl" or another feature.
      Avoid errors with modify command when s3-like servers don't support
acl.
      Fix possible bug in previous change for info command and
get_lifecycle.
      Fixes #722 : basename fonction was broken for local files for windows
as the os path separator is not the same on linux.
      Aws sometimes uses 301 code for redirects + redirect to aws base host.
      Fix signature v4 when aws redirect to aws base host.
      Fixes #778 : Workaround for aws sending early errors before requests
are completed + support for http expect continue feature.
      Updated maintainer in setup.py.
      Fixes #797 : unicodisation missing for exclude patterns coming from
files.
      Fixes #800 (second part): "cfg.include" was not checked like is
cfg.exclude to use the non memory optimal del subcmd.
      Fix for part of #660: fix detection for "sync single-file.txt
s3://mybucket/single-file.txt"
      Fix to supporting python 1.6. Use "List" for http methods and not a
kind of dict.
      Backport more code from httplib for compatibility with python 2.6
      Fix previous patch on custom httpconnection, it is _send_output and
not send_output.
      Attemps to fix last issues of #769 .
      Initial version of a .travis.yml and it's .travis.s3cfg configuration
for CI with travis-ci.
      custom_httplib27 - Missing symbols in python 2.6
      Missing bit in previous commit.
      [travis] Fixes allow_failures not being strings.
      [travis] Added run-test-minio.py + have travis run it.
      Added travis-ci build status of master in the github displayed readme.
      Fixes #810 : Also handle redirects for path styles syntax.
      Fixes #816 : Fix recv_file 3xx,4xx, handlers + add region extraction
of 3xx redirections.
      Fixes #798 : HEAD request can't parse xml response to get redirection
url.
      Fix missing imports in custom_httplib27.
      Move urlencode func into Utils and fix encoding   issues when having
uid groupid with special chars.
      Small cleanup in S3.py
      Py3 support: remove usage of deprecated has_key.
      Py3 support: fix SortedDict (keys() are now views and iterators need
__next__)
      Update README to indicate python versions supported by s3cmd.
      Changed unicodise/deunicodise wrappers to fit with python3 string
model.
      Python3 support: Fixed stderrr, stdout unicode output.
      Remove some code that was there for supporting py version < 2.6
      Use the stdout/stderr buffer directly instead of detaching.
      Python3 support: fix error reporting at import time and unicode.
      Python3 support WIP: Added Custom_httplib3x to patch python3 version
of httplib as it is done for py2.
      [Py3 WIP] Fix for compatibility.
      PY3 support: fixed S3Uri and Exceptions regarding unicode/str
compatibility.
      Missing change that was needed for custom_httplib functions...
      PY3 support: more unicode related things.
      PY3 support: long was removed in py3. Replace by int
      PY3 support: ensure to use the adequate / sign for division with
python3.
      PY3 support: Added a warning at runtime to warn of WIP
      PY3 support: Added silent versions for unicodise/deunicodise funcs
      Small cosmetic changes in Custom_httplib(s)
      Remove legacy code. s3cmd is now only compatible py >2.6
      Py3 support: Reworked request/signature regarding unicode/bytes mix.
      Py3 support: signature v2 - limit list of subresources url that are
signed
      Py3 support: ensure that request body are encoded when "unicode"
strings
      Signature v2: add "cors" to list of subresources to include in
signature
      Improve get_bucket_location to be able to work all the time with aws
      Use pretty print everywhere in debug for request responses
      Getlocation for each type of request when region is not set and
signature v4 used
      PY3 support: more fixes for str/unicode related issues.
      Typo fix in debug log message
      PY3 support: use getitem instead of getslices for slices
      PY3 support: Fix for regexp/unicode in runtest-minio
      Use io buffers everywhere for files or stdin/stdout + Use custom
stream_name attribute.
      Fixes signurl (v2) for python3.
      run-tests-minio: re-activated a test that is now working.
      Urlencode string only when formatting the url and not earlier.
      Get ride of the "extra" parameter for requests and use a clean
"params" dict for all the query strings.
      Ensure to not have "{}" in functions parameters to be safe.
      In the v4 canonical query string, parameter without values should
still have an equal sign.
      Fixes signature v2 invalid call to format_param_str.
      PY3 support: Fixes acl.py use of repr.
      Remove bad duplicated code.
      Little performance + memory improvement. + Fixes randomless in auto
tests.
      Typo fixes.
      Sort headers in requests to always have the same order.
      Fixes failing tests with special chars for PY3
      Small changes on run-tests.py
      Change .travis.yml now all tests are passing with minio.
      Documentation: Fixes #872 . Updated following the proposal of
@cthorley.
      Fix error message to be accurate with all cases + Fixed auto tests
for last change.
      Update setup.py classifiers to indicate that Python3 is supported.

Guy Gur-Ari (4):
      run-tests.py: use lowercase bucket name
      don't allow mv/cp of multiple files to single file
      raise exception for missing '/' during mv, cp
      remove unreachable code

Harshavardhana (2):
      Missing etag in HTTP headers, handle gracefully.
      build: Do not throw exceptions if no arguments were passed for
setup.py

Jan Kasiak (1):
      Change proxy SSL URI to relative

Joe Mifsud (2):
      Add support for invalidating multiple CF distributions with
 --cf-inval     when the bucket is the origin for multiple distributions
      Switching to set()

Krzysztof Warunek (1):
      support endpoint with custom path

Marco Jakob (7):
      Get the access token from the environment
      Do not refresh role
      Do not refresh role
      Add option for token-refresh
      Add option for token-refresh
      Add option for token-refresh
      Add option for token-refresh

Mark Titorenko (2):
      Generalize wildcard certificate forgiveness
      Updated based on feedback.

Masashi Ozawa (2):
      add "limit" option to "ls" and "la" commands to return the specified
    number of objects instead of returning all objects in the bucket.
      Enable to configure "host_base" as "S3 Endpoint" in     interactive
configuration.

Matt Domsch (2):
      Catch 'connection reset by peer' (errno 104) and retry
      pretty print server responses

Robert Palmer (1):
      Added support for S3 Glacier archive retrieval speed options

Shaform (11):
      use print function
      Fix has_key with 2to3
      Fix except with 2to3
      Fix numliterals with 2to3
      update relative import
      actually run python3 for tests
      run python3 for run-tests.py
      fix relative imports
      fix imports
      fix logging
      fix has_key, numliterals, except, imports on s3cmd

Simone Esposito (1):
      S3/Config: `os.environ.get` -> `os.getenv`

Vincent (1):
      FIX: update s3cmd.1 --disable man entry option

Yossi Abraham (1):
      Properly passing parsed hostname and port

marcojakob (3):
      Remove optparser switches
      Make _access_token_refresh private and set it to true if access_token
was already set
      Add CLI option for specifying access_token, see PR #694

s3cmd v1.6.1 is released to github, sourceforge, and PyPi.

This primarily fixes bugs from previous releases, especially SSL library
challenges introduced by the ever-changing python 2.7.x series SSL support.

Thanks to Florent Viard for reviewing a lot of proposed patches. This round
we had contributions from 8 different developers.

Shortlog below.

Thanks,
Matt


Albert Bendicho (4):
      Off-by-one on MAX_CHUNKS verification
      max_chunks now a config option
      MAX_CHUNKS constant removed, now a config option
      Added multipart_max_chunks

Alexander Todorov (2):
      Disable hostname checks for SSL connections on RHEL 7.2
      In case check_hostname isn't defined catch the error

Andrew Gaul (1):
      Emit storage class in info subcommand

Florent Viard (19):
      Fixes #636. Also remove the temporary file when encountering an
S3Error.
      Fix "info" command for servers not supporting the "request_payer"
feature. Based on PR #640 of Qiankunzheng.
      Fixes #644 : Minor fix to not output "u quoted" invalid parameter in
the case of an invalid command.
      Reworked source and destination input of "remote2local" to cover
gracefully more cases.
      Block "-" (stdin / stdout) to be used in parameters for sync as it
has no sense.
      Only warn for the non regular files that are skipped.
      In the case of local2remote, raise an error if a source arg is
missing.
      Fix automatic tests for recent changes.
      Don't fail the copy if a non AWS s3 server doesn't support setting
ACLs.
      Have extra labels coherent when progress is enabled or not and for
multipart uploads.
      Added the --stats option that is similar to the one of rsync.
      Added "--stats" info for sync commands (remote2local, remote2remote,
local2remote)
      Fix duplicated code for the error alert message.     Based on
@edunham PR: https://github.com/s3tools/s3cmd/pull/609
      Fixes #661 : Don't try to perform a delete if there is nothing to
delete. + Restore warning that was in 1.5.2.
      Fixes #659 : Don't fail if "content-type" field is missing in info
like for eucalyptus walrus.
      Fixes #646: number of "files to update" are not reported in the
summary (there was only the new files to be added)
      Fixes #646 : remote2remote sync -  "Updated" files were not reported
as to be remotely copied when in dry-run mode.
      When using --no-preserve, still add 'md5' custom attribute if md5
check is enabled.
      Fixes #674 : When using AWS_CREDENTIAL_FILE, crash in case of invalid
key in .aws/credentials file + updated the code for actual name of the keys.

Harshavardhana (1):
      s3cmd: Fix s3 path style signature v4 handling.

Matt Domsch (14):
      remote copies: copy ACL also (#547)
      only copy the ACL if --acl-public wasn't specified
      Allow --requester-pays for PUT commands too (#585)
      add "+" to PkgVersion so we can tell it's a git checkout
      Include version at the top of --debug output
      Update RELEASE_INSTRUCTIONS
      Add github release instructions
      Explicitly catch and raise CertificateError in send_request()
      Validate SSL certificate hostname against target bucket hostname
(#598)
      Don't unconditionally disable hostname checking
      more hostname fixes, move debugs around
      Add --host and --host-bucket command line options
      Allow acl_public = True/False to be in config file
      Remove newer python-ism from S3/Utils.py

Shigeaki Okazaki (1):
      Warn when skipping non-existing files or symbolic links

Teemu Ikonen (2):
      Fix for #634. Partial get start position was dropped if requested
fell back to signature v2
      Fix start_position dropped on 307 redirect

kobake (1):
      Ignore RestoreAlreadyInProgress exception on restore command

wiof (2):
      Control MAX_CHUNKS
      Define MAX_CHUNKS = 10000

After seven months, and dozens of new features and bug fixes from 24
contributors, s3cmd v1.6.0 is now available.  In this cycle, Florent Viard
has stepped up to co-maintain the tool, and has done an outstanding job
reviewing pull requests, fixing bugs, and doing everything a good
maintainer should. Thank you Florent!

Here's the high-level overview of all the cool new capabilities (including
support for Infrequent Accessed storage announced this morning by AWS!):

s3cmd-1.6.0      - 2015-09-18
===============
* Support signed URL content disposition type
* Added 'ls -l' long listing including storage class
* Added --limit-rate=RATE
* Added --server-side-encryption-kms-id=KEY_ID
* Added --storage-class=CLASS
* Added --requester-pays, [payer] command
* Added --[no-]check-hostname
* Added --stop-on-error, removed --ignore-failed-copy
* Added [setcors], [delcors] commands
* Added support for cn-north-1 region hostname checks
* Output strings may have changed.  Scripts calling s3cmd expecting
  specific text may need to be updated.
* HTTPS is now the default
* Many unicode fixes
* Many other bug fixes

Pick up your copy wherever fine software is distributed:
https://github.com/s3tools/s3cmd/releases/tag/v1.6.0
https://sourceforge.net/projects/s3tools/

Full shortlog since v1.5.2 appended.

Thanks,
Matt



Andrew Gaul (3):
      Horizontally align object policy info
      Emit user metadata in object info
      Support signed URL content disposition type

Billy Crook (2):
      Added support for listing StorageClass attribute.     Disabled by
default.  Enable with list_storageclass = True in .s3cfg
      Adding -l option for long_listing     list_storageclass now provided
by long_listing and -l.     Minor whitespace cleanup.

Brady (2):
      Install instructions for PIP
      Correct typo in instructions

David Liu (1):
      Fail silently on SIGPIPE

Dongmao Zhang (2):
      add new option --limit-rate to throttle download/upload speed
      only gettimeofday when limitrate is required

Florent Viard (53):
      Improved handling for errors 501 and 504:     Error 501: It means
that a feature is not implemented in the server, so no need for useless
retries.     Error 504: Gateway timeout situation, exit code should
indicate a server error.
      Cleanup: Add prologue to files to state that they are UTF-8 encoded.
      Fix run-test script to take into account the config file that is
specified on the command line.
      Fixes encoding/decoding issues and so support for all strange
characters (chinese, japanese, ...) in file names, folder or path arguments
(Should fix bug like #417i) when UTF-8 is not the system def$     Now, in
theory, everything that gets in should be converted to unicoded and then
things would be converted back to string (user defined encoding) at the
outmost parts when dealing with the system.     When talking to S3, convert
unicode to strings in utf-8 encoding.
      Remove useless error import
      Pass2 of "fixes encoding/decoding issues": os.walk replaced to handle
unicodise conversion + fixed the removal of most of useless or invalid
unicodise/decodise in the code.
      Fix for issue #432. Read the response from httplib even if there is a
0 byte to read. So httplib doesn't think that there is a response waiting
to be read when the next request come.     Straightforward to fix thanks to
the detailed analysis of Edwintorok.
      Use deunicodise consistentely in object_copy and object_modify
      fix encoding : Added encode_to_s3 and decode_from_s3 function +
switched to "unicode" call instead of "decode" as it is supposed to be more
efficient.
      Fix wrong exception used for EOF from S3 error during recv_file.
(S3Error expect an http response, not a text message)
      For copy and modify, take care to check if there is an error after
the start of the operation (reported by a status 200 but an "Error" root
entry in the xml body)     In addition, for the move, It is good to check
for the CopyObject xml reply to see if it is a success.     But, if there
is a status of 200 and no "data" in response, don't fail. Other S3 server
implementations use only status code and don't return any body when there
is no error.
      Another fix related to fix encoding issues. (file.name / stream.name are
bytes, so unicodise them also)
      Fix invalid file transfer progress info in case of local2remote and
remote2remote: If there was failed copy, seq number was preserved but
previously uploaded file number was forgotten, thus user could have faced
an invalid indication like: 14/8 when there is 6 upload following 6 remote
copy failures.
      Add a link to the known issues page (on github) within the header and
footer of s3cmd generated error reports.     The goal is to reduce the
number of duplicated "bug reports" for known issues.
      Fixes #528 (https://github.com/s3tools/s3cmd/issues/528).
There is a lot of different "python-magic" libraries/versions, all doing
different things, so all cases need to be covered.     file-5.11 has be
switched with "filemagic" as it should be more commonly used.
Introspection in each python magic lib has been done to ensure to handle
encoding or not only when it is needed.
      Fixes issue #576 (https://github.com/s3tools/s3cmd/issues/576).
EX_SOMEFAILED was still used in a PR that was merged instead of EX_PARTIAL.
      Merge pull request #572 from fviard/fix_encoding_magic
      Merge pull request #574 from koenrh/https-by-default
      Don't slurp whole file while computing SHA256 checksum. Rework of the
PR of petr-tichy.     When using large --multipart-chunk-size-mb or
--disable-multipart, the whole file was read into memory.
      Merge pull request #580 from Tim-Anderson/error_stop
      Merge pull request #583 from Tim-Anderson/issue577
      Merge pull request #586 from Tim-Anderson/remove_ignore_failed_copy
      Merge pull request #587 from taz77/master
      Merge pull request #589 from jcwilson/master
      Merge pull request #560 from Flipboard/iter
      Merge pull request #593 from jrschumacher/set-cors
      Merge pull request #603 from mdomsch/bug/testsuite
      Following #603 : WARNING OUTPUT FORMAT CHANGE. Changed the format of
output strings to be more consistant.     Warning, the format of output for
a lot of actions was changed. Thus, any automatic string for parsing the
output could have to be fixed when using s3cmd after this checkin.     Ex:
    "File hello/world.txt stored as s3://mybucket/world.txt"     will now
be:     "upload: 'hello/world.txt' -> 's3://mybucket/world.txt'
      Fixes #602 (and complete #603) Fix run-tests.py to have all tests
passing against HEAD
      If s3error message is none, don't print an useless ": None"
      Return NotFound error with message s3Error: NoSuchBucket when there
is no bucket found to delete
      Raise standard S3Error in S3.py instead of simply exiting. So the
normal error logic can't operate on any unexpected error.     Avoid to have
empty errors in output like:     "ERROR: S3 error:"     (Because of an 400
InvalidBucketName error returned by a non AWS3 server)
      Merge pull request #590 from andrewgaul/content-disposition
      Fix stop_on_error config type
      Attempt to fix the first bug of issue #600 : When a subfolder "xxx"
was selected as source, the sync comparison was done against the
destination base whereas it had to be done based on the "xxx" subfolder of
the destination.
      Fixes run-tests following the fix for
https://github.com/s3tools/s3cmd/issues/600
      Merge pull request #616 from mdomsch/bug/612
      Fix bad help info for the "--no-check-certificate" switch. Thanks to
@psi29a for proposing the fix.
      Merge pull request #618 from mdomsch/bug/futurewarning
      Fixes #592 : Ignore any leading slash before bucket_name in an S3Uri.
    Original issue:     "Running s3cmd ls s3:///foo lists owned buckets
instead of emitting an error."
      Fix for error reporting in case of upload error. (Following the fix
for #577)
      Fixes #581 regarding the reporting of SSL verification issues. Other
problems of this issues should already be fixed.
      Merge pull request #621 from mdomsch/bug/ssl-checking
      Merge pull request #619 from fviard/second_fix_for_577
      Merge pull request #617 from fviard/fix_subfolder_source
      Merge pull request #623 from mdomsch/bug/testsuite2
      Reworked Upload and Download errors to be more consistent. Add
detection and handling of different OSErrors like "File name too long."
      Merge pull request #628 from mdomsch/bug/curl
      Merge pull request #627 from fviard/upload_download_errors_rework
      Merge pull request #630 from mdomsch/feature/ia-class
      Merge pull request #611 from bsapiro/master
      Added --server-side-encryption-kms-id config option. Following PR
#611.
      Fixes bug in PR #611

François Wirion (1):
      Fix missing slash in directory semantics documentation

Gianfranco Costamagna (1):
      Fix typo (overriden -> overridden)

Giorgos Logiotatidis (1):
      If response['headers'] is empty, then don't update dict.

Josh C. Wilson (1):
      move progress_meter default setting down to Config class

Ka-Hing Cheung (1):
      enlarge write and read buffers

Koen Rouwhorst (1):
      Use HTTPS by default

Matt Domsch (74):
      signurl should honor Config.use_https
      Merge branch 'master' of github.com:s3tools/s3cmd into upstream-master
      fix signurl test
      retry recv_file() on IOError(EPIPE)
      Revert "signurl should honor Config.use_https"
      close http(s) connection after IOError(EPIPE) or other unknown error
      report_exception() on uncaught IOError before exit
      accept 'us-east-1' as valid Config.bucket_location in [mb] command
      Merge pull request #483 from mdomsch/bug/broken-pipe
      Merge pull request #478 from petrus-jvrensburg/patch-1
      Merge pull request #459 from thesues/throttle
      s3cmd help: --limit-rate works for uploads too
      Merge pull request #480 from fviard/improve-errors-501-504
      Merge pull request #468 from ppe/support-display_name-in-acl-revoke
      Sign x-emc-* headers (#486)
      Merge pull request #488 from glogiotatidis/master
      Ensure content-type header is treated as a string and long int
      Merge pull request #494 from andrewgaul/emit-user-metadata
      Merge pull request #493 from andrewgaul/align-policy
      Merge pull request #496 from fviard/fix_for_432
      Merge pull request #495 from fviard/batch_encoding_fixes
      Merge pull request #504 from LocutusOfBorg/fix-typo
      Merge pull request #507 from LocutusOfBorg/debian-patches
      escape - and 'Cache-Control' items in manpage
      Merge branch 'master' of github.com:s3tools/s3cmd into upstream-master
      replace /tmp/ with ~/ in manpage examples
      remove references to donations
      update manpage for formatting
      Remove useless imports and unused variables
      Merge pull request #499 from fviard/fix_copy_modify_check_errors
      Merge pull request #508 from fviard/fix_encoding_multipart_filename
      more IOError EPIPE catches
      Merge pull request #512 from fviard/fix_invalid_progress_info
      add --requester-pays (#511)
      add Payer to [info] command output
      move setting requester-pays into S3Request() init
      Merge pull request #515 from mdomsch/feature/requestor-pays
      add [payer] command
      add requester-pays tests
      Merge branch 'BCrookAtRA-master' into upstream-master
      Merge pull request #532 from ddzialak/master
      [info] use last-modified value, not date
      note s3:GetBucketLocation permission is needed too
      Makefile: file is README.md now
      Merge pull request #537 from flisky/master
      Merge pull request #538 from kahing/bigger_chunk_size
      Merge pull request #569 from fviard/common_issues_link
      Merge pull request #568 from blindroot/master
      Try using Config.bucket_location if we haven't received a redirect
      Merge branch 'master' of github.com:s3tools/s3cmd into upstream-master
      Merge pull request #550 from iceboundflame/10513-sigpipe
      Merge pull request #571 from wirion/master
      Downloads: print destination file name not <fdopen>
      Fix testsuite for recent changes
      introduce class ExitScoreboard
      Fix Rename (NoSuchKey) test string tested for
      fix md5 checks on download of multipart-uploaded files  (#604)
      Merge pull request #605 from mdomsch/bug/604
      Fix use of scoreboard
      Don't check hostnames in SSL certs when --no-check-certificate
      Fix FutureWarnings from S3/AccessLog.py
      add --[no-]check-hostname option
      Allow cn-north-1 region SSL certificate to validate hostname
      Use already imported ssl.CERT_NONE
      Add single quotes around progress output filenames
      run-tests: s/'-'/'<stdin>'/
      run-tests: use curl instead of wget
      more curl cleanups
      Use test_curl_HEAD() throughout, add --location flag
      run-tests: detect if curl is on path before using it
      add --storage-class option
      Remove unicode chars from --server-side-encryption-kms-id helptext
      Update copyright year to include 2015
      Version 1.6.0

Mike Klaas (13):
      iterative du, rm
      bugfix
      fixed to conform to test suite
      typo
      slight cleanup of du code
      remove errant print statement
      Code review feedback:      * Print all filenames deleted      * Use
batch mode if max-deletes is set (& remove logic from iterative delete)
 * Fix formatting of aggregate output of the deletion, and elide unless -v
is specified
      _status() inner funtion wasnt really needed
      eliminate recurse() function which is unneeded and simplify
subcmd_bucket_usage
      revert unnecessary change
      remove closure variable capture in batch_del_iterative; fix typo in
subcmd_disk_usage
      stream groups of 1000 (or whatever is available) objects at a time,
and delete that group directly
      fix non-streaming bucket_list()

Mikhail Gusarov (3):
      Fix various typos spot by lintian
      Return non-zero code from 'put' command if some files failed to upload
      Do not loop endlessly if removal of bucket content fails

Paul Wise (1):
      Use user home instead of tmp in examples

Petri Pellinen (1):
      Add support for display names in --acl-revoke (#223)

Petrus Janse van Rensburg (1):
      Update README.md

Przemyslaw Hejman (1):
      Added dry-run to fixbucket command

Ryan Schumacher (1):
      Add ability to set, get and del cors policy.

Tim Anderson (7):
      Add stop_on_error
      Add better exit errors on stop_on_error
      Return PARTIAL in sync transfers
      Correct exit status handling for children
      Error check on multipart upload
      Change Multipart error so it can continue
      Replace ignore_failed_copy with stop_on_error

Yin Jifeng (1):
      FIX #535 mark delay-updates obsolete

ben.sapiro (1):
      basic SSE-KMS support

dzialak (1):
      fix simply error in setlifecycle command

I've put out a really quick update, 1.5.2.  This fixes one big failure on
Ubuntu 14.04, due to their SSL library not yet validating a server's SSL
certificate.  It also fixes some packaging issues (I had added, and have
now removed, python ez_setup, which, as it happens, is more hassle than
it's worth).

github, PyPi, and SourceForge all updated.  Fedora & EPEL testing repos
have 1.5.1.2, and aren't affected by the SSL library failure that broke
Ubuntu, so I don't intend to update those again.

shortlog since 1.5.1:

Matt Domsch (9):
      include ez_setup.py in MANIFEST.in
      Update version to 1.5.1.1
      Change name of README.md in MANIFEST.in
      Update to version 1.5.1.2
      setup.py: add install_requires python-magic
      make bundled s3cmd.spec.in closer to Fedora spec
      Handle unvalidated SSL certificate
      remove ez_setup
      Update version to 1.5.2


Thanks,
Matt

I've pushed out s3cmd 1.5.1.2 today.  It's mostly a bugfix release over
what was in 1.5.0.  The two minor minor point releases were necessary to
fix up PyPi packaging bugs that kept 'pip install s3cmd' from working
correctly.

Sourceforge, github, and PyPi are all updated now.

Shortlog:

Andrew Gaul (2):
      Sort s3cmd ls output by bucket name
      Capitalize command descriptions for consistency

Chris Lamb (1):
      Support relative expiry times in signurl.

Luke Winslow (1):
      Fixed issue with mixed path separators with s3cmd get --recursive on
Windows.

Matt Domsch (51):
      remember git push --tags for release
      fix S3 wildcard certificate checking
      Add docstring explanation into match_hostname_aws()
      Handle headers with spaces in their values properly (#460)
      Don't add --add-headers results into every request
      Fix lack of SSL certificate checking libraries on older python
      set content-type header for stdin from command line or Config()
      Fix multipart abortmp error message
      fix uploads from stdin (#464)
      in S3.send_file(), don't conditionalize on stdin
      Multipart: send 2nd+ chunks from start of buffer
      add test for put from stdin
      add multipart test
      testsuite: add Multipart Put large file from stdin
      use close_fds = True in calls to Popen()
      reduce use of os.system() in run-tests stdin tests
      reduce "Forwarding request to %s' % region" from warning to info
      Fix directory exclusions (#467)
      Correct exclude pattern for python 2.4
      s/sign_url/sign_url_v2/
      sign_url_v2 is in Crypto now, not Utils
      sign_string_v2 is now in Crypto, not Utils
      fix signurl
      run-tests: add sign, signurl tests
      Don't retry in response to HTTP 405 error (#422)
      run-tests: add -c <config file>
      run-tests: add test for [la] command
      setup.py: use install_requires
      it's called python-dateutil in pypi
      Don't crash when a proxy returns an invalid XML error document
      refresh manpage after capitolization patch
      move from distutils to setuputils
      invoke python2 setup.py build ... with python2
      sign uploads to pypi
      update pypi metadata
      report the actual error XML when we can't parse it.
      Fix indentation causing failures
      remove useless debug() and print() message
      update NEWS for 1.5.1
      update PkgInfo and manpage version
      no longer keeping sourceforge git tree in sync
      include ez_setup.py in MANIFEST.in
      Update version to 1.5.1.1
      Change name of README.md in MANIFEST.in
      Update to version 1.5.1.2


Thanks,
Matt

We have released s3cmd version 1.5-rc1, in preparation for a final 1.5.0 
release, with all changes, fixes and improvements implemented since 1.5 
beta.

Thanks a lot to all for your contributions and fixes. Keep them coming!

Special thanks to Matt Domsch for administering and leading the project. 
And to Michal Ludvig for putting together the 1.5-rc1 release.

Year to date, s3cmd has been downloaded 781,248 times from SourceForge.

See:
http://sourceforge.net/projects/s3tools/files/s3cmd/stats/timeline?dates=2014-01-01+to+2014-07-29

You can download the latest s3cmd version from here:
http://s3tools.org/s3cmd

thank you,
M.Bartoli

Hi Matt & all

Thanks a lot for looking after s3cmd and once again my apologies for not
being active in the community. I'm really happy to see s3cmd is alive
and well.

I have merged the mdomsch/s3cmd/merge branch to s3tools/s3cmd/master and
released it as
*s3cmd 1.5.0-beta1*
(https://sourceforge.net/projects/s3tools/files/s3cmd-1.5.0-beta1.tar.gz).
Please download, test and report back your success.

Matt - Quite a long time ago I made you an admin of the github repo
(s3tools/s3cmd) and just recently also gave you admin access to the
SourceForge s3tools project so you can merge directly to the official
master and make new releases if you wish. There are a few steps
involved, we can discuss that offline if you're keen. Or keep updating
your merge branch and ping me when it's ready for a release, I'm sure I
can find the time to roll out a release when the code is ready. I just
don't have enough time for the day to day participation.

Thanks a lot for all your contributions guys, keep up the good work! And
special thanks to Matt for putting it all together :)

Michal

On 27/11/13 04:50, Matt Domsch wrote:
> The pull requests have been piling up for several months, with lots of
> bugs being fixed by 21 different developers.  Michal hasn't had time
> to review, test, and merge these, so here's my attempt to help the
> process along, in hope of getting more people to test and report success/failure.
>
> Appended is the shortlog of all changes pulled into my merge branch since
> the current upstream-master 9ee3bdd320c8.  I've also tried leaving a
> comment on each of the existing pull requests.  There were a few I
> think need a little more work before they can be merged, or that
> require Michal's judgment.
>
> The tree: https://github.com/mdomsch/s3cmd/tree/merge
>   (HEAD commit ae6030cd33d4)
> ZIP archive: https://github.com/mdomsch/s3cmd/archive/merge.zip
> Clone:   git clone git://github.com/mdomsch/s3cmd.git
>   (you want the branch called 'merge')
>
> If this works for you, please leave a comment.  If you were
> particuarly affected by a bug that is now fixed, _please_ let us know
> that as well.
>
> Special thanks to Will Tinsdeall (@ivebeenlinuxed) for reviewing and
> merging six of these patches into his tree for testing this last month.
>
> Thanks,
> Matt Domsch (@mdomsch)
>
>
>

Hi s3cmd users!

Another week passed, another release arrived - *s3cmd 1.5.0-alpha3
<http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-alpha3/s3cmd-1.5.0-alpha3.tar.gz>*

The main feature of this release is _Persistent reusable HTTP
connections_ - an improvement that brings massive speedup for pretty
much all Amazon S3 operations.

Up until now s3cmd used to open a new connection for each file upload,
download, delete, etc. That resulted in quite a big overhead, especially
for short requests like remote file copy or delete. Even worse when
HTTPS was turned on - setting up the SSL channel for each and every
request was a very resource intensive and time consuming exercise.

>From s3cmd 1.5.0-alpha3 on we support a much faster approach - we open
the connection once and use it again and again and again without closing
and re-opening for each subsequent request. That achieves a significant
speedup in all operations.

>From my office on a remote Pacific island with slow ADSL and huge
latency to S3 I get:

- Testsuite run without SSL from 3 min 54 sec down to 2' 54" - that's
26% speedup
- Testsuite run with SSL (i.e. HTTPS) from 5' 59" down to 3' 11" = 47%
speedup
- Uploading 10000 files from 145' 43" down to 57' 43" = 61% speedup
- Deleting 10000 files from 121' 41" down to 33' 6" - that's _72% speedup_!!

Obviously if you are on a fat pipe with a low latency close to an Amazon
datacentre you may not experience such an extreme improvement - I'm keen
to hear how much (if at all) the new connection handling helps in your case!

With regards to other changes in alpha3
- new switch --quiet / -q to suppress all "normal" output from s3cmd.
(Siddarth Prakash)
- s3cmd get now honours umask again - for some time the downloaded files
were created with 0600 permissions - that's now corrected. (Jason Dalton)
- Matt Domsch did a great job fixing small annoying bugs and closing
resolved tickets - thanks!

Please download, test and report any problems
http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-alpha3/s3cmd-1.5.0-alpha3.tar.gz

The most up to date sourcecode is available on GitHub:
https://github.com/s3tools/s3cmd

Also check out our website for further updates:
http://s3tools.org/s3cmd

Have a nice day!

Michal

Hi guys*

**it's **release time again**- welcome s3cmd 1.5.0-alpha2
<http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-alpha2/s3cmd-1.5.0-alpha2.tar.gz>
:)

*The major new feature in this release is *IAM roles support* thanks to
David Kohen and Eric Dowd.

Another new feature is the support for managing bucket policies
(setpolicy, delpolicy and info for showing the policy) from Kota Uenishi.

s3cmd 1.5.0-alpha2 works again with Python 2.4

On top of that we've got many minor fixes and improvements from various
contributors.

Download from here:
http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-alpha2/s3cmd-1.5.0-alpha2.tar.gz

The most up to date sourcecode is available on GitHub:
https://github.com/s3tools/s3cmd

Also check out our website for further updates:
http://s3tools.org/s3cmd

Please test and report any issues you may find.

Enjoy

Michal & contributors

Surprise surprise!

After more than a year of neglecting s3cmd and letting it rot
unmaintained I thought I should spend some time bringing it back to life.

Here you go: s3cmd 1.5.0-alpha1
<http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-alpha1/s3cmd-1.5.0-alpha1.tar.gz>
is now available

Big thanks to Matt Domsch for integrating many pull requests over the
year into his Fedora version of s3cmd. This s3cmd 1.5.0-alpha1 is very
much based on his outstanding work.

And of course many thanks and a big sorry to all the contributors who
submitted new features and bug fixes and were left without a response,
wondering if their work is ever going to be merged back into the
official s3cmd codebase. Sorry guys, this indeed wasn't a prime example
of a project maintainership :(

So what has changed since 1.1.0-beta3 released a year ago?

_s3cmd 1.5.0__-alpha1_
* Server-side copy for hardlinks/softlinks to improve performance  (Matt
Domsch)
* New [signurl] command (Craig Ringer)
* Improved symlink-loop detection (Michal Ludvig)
* Add --delete-after option for sync (Matt Domsch)
* Handle empty return bodies when processing S3 errors.  (Kelly McLaughlin)
* Upload from STDIN (Eric Connell)
* Updated bucket locations (Stefhen Hovland)
* Support custom HTTP headers (Brendan O'Connor, Karl Matthias)
* Improved MIME support (Karsten Sperling, Christopher Noyes)
* Added support for --acl-grant/--acl-revoke to 'sync' command  (Michael
Tyson)
* CloudFront: Support default index and default root invalidation 
(Josep del Rio)
* Command line options for access/secret keys (Matt Sweeney)
* Support [setpolicy] for setting bucket policies (Joe Fiorini)
* Respect the $TZ environment variable (James Brown)
* Reduce memory consumption for [s3cmd du] (Charlie Schluting)
* Rate limit progress updates (Steven Noonan)
* Download from S3 to a temp file first (Sumit Kumar)
* Reuse a single connection when doing a bucket list (Kelly McLaughlin)
* Delete empty files if object_get() failed (Oren Held)

That's a lot of changes all around the codebase. Although s3cmd passes
the testsuite and works in the common situations I'm afraid we may have
missed some corner cases. Please report any problems you experience when
running s3cmd 1.5.0-alpha1 to make the final release as stable as possible.

Download from here:
http://sourceforge.net/projects/s3tools/files/s3cmd/1.5.0-alpha1/s3cmd-1.5.0-alpha1.tar.gz

The most up to date sourcecode is available on GitHub:
https://github.com/s3tools/s3cmd

Also check out our website for further updates:
http://s3tools.org/s3cmd

Now download, test and enjoy! :)

Michal & contributors

Hi all!

I have just released s3cmd 1.1.0-beta3 with two important fixes:

- Multipart now works for [sync] with a limitation in that it doesn't 
compare MD5 sums of the local and remote file. The reason is that Amazon 
doesn't provide a MD5 sum for multipart-uploaded files and therefore we 
have nothing to check the local MD5 sum against. For now s3cmd only 
compares the file sizes in that case, will add a timestamp check later 
as well.

- The new MIME guessing turned out to a bit difficult to initialise 
properly across all the platforms. Not only that the python-magic module 
may not be installed, worse yet there are two incompatible python-magic 
modules currently shipped with Linux recent distros. It's quite a mess 
but hopefully it should now work as expected for everyone.

And obviously a number of minor bugs was fixed as well. Thanks to all 
the reporters and contributors!

Download from here:
http://sourceforge.net/projects/s3tools/files/s3cmd/1.1.0-beta3/s3cmd-1.1.0-beta3.tar.gz/download

Alternatively grab the latest source from the official git repository:
git clone git://github.com/s3tools/s3cmd.git

_**_As with any new release I'm keen to hear your feedback, positive or 
not. Please direct all emails to s3t...@li... 
as usual.

Enjoy s3cmd 1.1.0-beta3 :)

Michal

Hi all!

It's been a looong time since the last major s3cmd release. It's been 
quiet but the project is not dead.

A lot has happened since s3cmd 1.0.0 - most importantly the development 
was moved to GIT and is now primarily hosted on GitHub 
(https://github.com/s3tools/s3cmd). This move has sparked an unexpected 
activity among the users(-now-developers) of s3cmd who begun to 
contribute new features in a much higher rate than I was used to while 
the code was hosted in SourceForge's SVN. s3cmd 1.1.0 will be a true 
community effort - thanks folks!

Here are the major changes...

*Multipart Upload*
It's by far the most requested feature and it's finally here. In this 
beta release it works only for the [put] command, not for [sync] because 
there are some technical difficulties yet to be sorted out.
Multipart is enabled by default and kicks in for files bigger than 15MB. 
You can set this treshold as low as 5MB (Amazon's limit) with 
--multipart-chunk-size-mb=5 or to any other value between 5 and 5120 MB.

Two "management" commands are still missing - [mplist] for listing all 
unfinished multipart uploads and [mpabort] for aborting those 
left-behind ones. S3cmd tries to cleanly abort the upload and free up 
all the resources on the S3 side if it's interrupted for any reason but 
there still may be occasions where an unfinished upload is left behind.

A number of contributors submitted patches adding this functionality 
over the past few months, thank you all! Unfortunately none of 
submissions was complete and clean enough to make it through. In the end 
I used Jerome Leclanche's (Adys) work as a starting point and based my 
work on that. Thanks Jerome!

*CloudFront invalidation*
Another often requested feature. s3cmd now can invalidate objects in 
CloudFront distributions immediately during [sync] with --cf-invalidate 
switch. With this switch all uploaded files will be invalidated from 
CloudFront which makes the content delivery network consistent with the 
primary S3 storage. It takes some time to process the invalidation - use 
[s3cmd cfinvalinfo cf://dist-id] command to query the status. See 
CloudFront documentation 
<http://docs.amazonwebservices.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html> 
for details.

*Static WebSite support*
Jens Braeuer contributed a patch for creating and deleting "static 
websites" in S3/CF. See S3 documentation 
<http://docs.amazonwebservices.com/AmazonS3/latest/dev/WebsiteHosting.html> 
for details.

*Config settings from Environment*
Many users are not comfortable with keeping their access and secret keys 
in the config file. Ori Bar provided a patch that allows any config 
value to be taken from the process environment. For example to obtain 
the access key from the environment put this in your ~/.s3cfg file:
     access_key=$S3_ACCESS_KEY
The S3_ACCESS_KEY name is just an example - use anything you like, just 
prepend it with "$" sign. For instance
     access_key=$ABRACADABRA
is just as valid as $S3_ACCESS_KEY

*MIME options made saner*
The previous semantic of --default-mime-type and --guess-mime-type was a 
bit ... illogical. It actually caught me by surprise a couple of times, 
despite the fact that I was the author of such an ill-logic ;)
I believe it's a lot cleaner now: --guess-mime-type tries to guess the 
MIME type by the extension or, if available, by using the "magic" file 
(python-magic support contributed by Karsten Sperling). If guessing 
fails the type is set to binary/octet-stream or to whatever is set in 
--default-mime-type. Explicit setting including the mime-type parameters 
can be achieved with --mime-type="text/plain; charset=utf-8".

*Download multiple files to stdout*
Rob Wills contributed a support for downloading multiple files to 
standard output. I'm not quite sure what is it good for to be honest ;) 
Anyway the patch was straightforward and so I accepted it.

*s3cmd --configure can test access to a specified bucket
*Listing buckets requires the S3 ListAllMyBuckets permission which is 
typically not available to delegated IAM accounts. With this change, 
s3cmd --configure accepts an (optional) bucket uri as a parameter and if 
provided, the access check will just verify access to this bucket 
individually. Contributed by Mike Repass.

*Minor changes*
Obviously this release also comes with a number of smaller fixes and 
improvements, tweaked config values here and there, stability 
improvements, documentation fixes, etc. Too much to list here individually.

_*Contribute to s3cmd project on GitHub*_
All hands are welcome to contribute to the s3cmd project. Whether you 
can code new features in Python or add/improve documentation you're 
welcome to "fork" the https://github.com/s3tools/s3cmd project, do your 
changes in your private s3cmd workspace and once ready submit them for 
inclusion to the official repository by sending a "Pull Request" from 
the GitHub web interface. I'm not going into more details about Git, 
GitHub workflow, forks, pull requests, etc - it's well out of scope of 
this announcement. Google it, Try it, Like it and Contribute :)

_*Download s3cmd 1.1.0-beta2*_
Download the s3cmd 1.0.0-beta2 package from SourceForge:
http://sourceforge.net/projects/s3tools/files/s3cmd/1.1.0-beta2/s3cmd-1.1.0-beta2.tar.gz/download

Alternatively grab the latest source from the official git repository:
git clone git://github.com/s3tools/s3cmd.git

_*Donate to s3cmd project*_
The s3cmd script is a free software, no licence or fees are required to 
use it. However if you find s3cmd useful and worth a buck please make a 
donation. Any amount, small or big, will be greatly appreciated!
Follow this link for a PayPal or Credit Card donation: 
http://s3tools.org/donate

_*Etc...
*_As with any new release I'm keen to hear your feedback, positive or 
not. Please direct all emails to s3t...@li... 
as usual.

Enjoy s3cmd 1.1.0-beta2 :)

Michal

Hi all s3cmd fans!

The long requested *CloudFront invalidation* support is now available in 
s3cmd-1.1.0-beta1 thanks to a sponsorship from Philipp Winkler and 
Interactive 21 <http://interactive21.de>.

CF Invalidation is integrated with /s3cmd sync/ command:

~$ s3cmd *--cf-invalidate* --acl-public sync /local/path/ 
s3://<bucket>/remote/path/
[... uploading ...]
Created invalidation request for 4 paths
Check progress with: s3cmd cfinvalinfo cf://E9RB4B71C06M7/IJK5N4YQBAG8K

It takes a couple of minutes for the invalidation requests to be 
processed on the Amazon side. Can check the status with s3cmd cfinvalinfo:

~$ s3cmd *cfinvalinfo* cf://E9RB4B71C06M7/IJK5N4YQBAG8K
URI:            cf://E9RB4B71C06M7/IJK5N4YQBAG8K
Status:         InProgress
Created:        2011-04-11T03:05:55.270Z
Nr of paths:    4
Reference:      E9RB4B71C06M7.20110411150554.2741

As there can be up to 3 invalidation requests running in parallel you 
can list them all (using either s3://bucket or cf://dist-id identifier):

~$ s3cmd cfinvalinfo s3://public.example.com
URI:            cf://E9RB4B71C06M7/IJK5N4YQBAG8K
Status: *InProgress*
Created:        2011-04-11T03:05:55.270Z
Nr of paths:    4
Reference:      E9RB4B71C06M7.20110411150554.2741

URI:            cf://E9RB4B71C06M7/I1TA7E8XR7J70P
Status: *Completed*
Created:        2011-04-08T14:59:02.714Z
Nr of paths:    20
Reference:      E9RB4B71C06M7.20110409025901.2531
[...]

There is also a couple of limitations:
- invalidation is only available for sync, there is no dedicated /s3cmd 
cfinval/ command that would invalidate without uploading. Yet.
- the maximum number of paths invalidated in one run is 999 (that's a 
single request size limit in Amazon). If more than 999 paths are to be 
invalidated the list is saved in a file and the user will have to deal 
with that in some other way, possibly using the future /s3cmd cfinval/ 
command.

Download the package from this link:
http://sourceforge.net/projects/s3tools/files/testing/s3cmd-1.1.0-beta1.tar.gz/download
For testing it is /not/ necessary to install s3cmd into the system. 
Simply unpack it and run from that directory. For instance instead of 
invoking the installed s3cmd run /tmp/s3cmd-1.1.0-beta1/s3cmd (if that's 
where you unpacked the tarball).

S3cmd 1.1.0-beta1 should work as expected and there should be no 
regressions against s3cmd 1.0.0 but please test it and let me know.

Enjoy!

Michal

(resending - corrected URL)

Hi everyone!

We have been maintaining installation repositories for RPM-based 
distributions, namely RHEL, Fedora, CentOS and OpenSUSE / SLES for quite 
some time now and the service became pretty popular - only yesterday we 
have seen over 1500 unique IP addresses checking for updates!

However one of the frequent questions from the community - "*Is there a 
repository for Debian / Ubuntu?*" - used to have a negative answer. 
Although both Ubuntu and Debian have s3cmd in their standard 
repositories the versions there are never updated once released which is 
especially annoying for distributions with a long support like Ubuntu LTS.

Now the things have changed - the S3tools project announces an *Official 
s3cmd DEB repository* from which you can always get the latest greatest 
s3cmd for your Debian or Ubuntu.

Follow these steps to enable the repo from command line:

1) Import the repository key:
wget -O- -q http://s3tools.org/repo/deb-all/stable/s3tools.key | sudo 
apt-key add -

2) Add the repo to sources.list:
sudo wget -O/etc/apt/sources.list.d/s3tools.list 
http://s3tools.org/repo/deb-all/stable/s3tools.list

3) Refresh package cache:
sudo aptitude update && sudo aptitude install -V s3cmd

4) Test if it works as expected and let me know if not :)

Although I'm not a DEB packaging expert the repo seems to work fine, 
tested on Ubuntu 10.4 LTS x86 (Lucid Lynx), on Debian 5 amd64 (Lenny)  
and on Debian 6 (Squeeze) running on quite an exotic HP-PA machine. 
Should work just fine with Ubuntu 10.10 and newer and most likely with 
some older Ubuntus too. Success stories are indeed welcome :)

Enjoy it!

Michal

Hi everyone!

We have been maintaining installation repositories for RPM-based 
distributions, namely RHEL, Fedora, CentOS and OpenSUSE / SLES for quite 
some time now and the service became pretty popular - only yesterday we 
have seen over 1500 unique IP addresses checking for updates!

However one of the frequent questions from the community - "*Is there a 
repository for Debian / Ubuntu?*" - used to have a negative answer. 
Although both Ubuntu and Debian have s3cmd in their standard 
repositories the versions there are never updated once released which is 
especially annoying for distributions with a long support like Ubuntu LTS.

Now the things have changed - the S3tools project announces an *Official 
s3cmd DEB repository* from which you can always get the latest greatest 
s3cmd for your Debian or Ubuntu.

Follow these steps to enable the repo from command line:

1) Import the repository key:
wget -O- -q http://s3tools.org/repo/deb-all/stable/s3tools.key 
<http://s3tools.org/repo/deb-py26/stable/s3tools.key> | sudo apt-key add -

2) Add the repo to sources.list:
sudo wget -O/etc/apt/sources.list.d/s3tools.list 
http://s3tools.org/repo/deb-all/stable/s3tools.list

3) Refresh package cache:
sudo aptitude update && sudo aptitude install -V s3cmd

4) Test if it works as expected and let me know if not :)

Although I'm not a DEB packaging expert the repo seems to work fine, 
tested on Ubuntu 10.4 LTS x86 (Lucid Lynx), on Debian 5 amd64 (Lenny)  
and on Debian 6 (Squeeze) running on quite an exotic HP-PA machine. 
Should work just fine with Ubuntu 10.10 and newer and most likely with 
some older Ubuntus too. Success stories are indeed welcome :)

Enjoy it!

Michal

Hello World :)

The time has come, _s3cmd 1.0.0 has been released_!

Four years after the first public version s3cmd is now a stable,
production ready, easy to use piece of software. It supports almost all
features of Amazon S3 and Amazon CloudFront, is designed for use in
shell scripts and automated processes, great both for sensitive
encrypted backups to Amazon S3 as well as for publishing web content via
Amazon CloudFront content delivery network. And all in between.

The list of all currently supported features is very long, let me just
include the changes from s3cmd 0.9.9.91:
* [sync] now supports --no-check-md5
* [sync] now supports bucket-to-bucket synchronisation
* Added [accesslog] command.
* Added access logging for CloudFront distributions using [cfmodify --log]
* Added --acl-grant and --acl-revoke [thx Timothee Groleau]
* Allow s3:// URI as well as cf:// URI as a distribution name for most
CloudFront related commands.
* Support for Reduced Redundancy Storage (--reduced-redundancy)
* Follow symlinks in [put] and [sync] with --follow-symlinks
* Support for CloudFront DefaultRootObject [thx Luke Andrew]
* Network connections now have 10s timeout

_Download_
Obviously this is the best s3cmd release ever - all users are advised to
upgrade. See http://s3tools.org/download for download links or go
directly to the SourceForge download page:
https://sourceforge.net/projects/s3tools/files/s3cmd/

_RPM repositories_
If you run a RedHat, CentOS, Fedora or one of the SuSE distributions the
best way to keep your s3cmd up to date is to install it from our RPM
repository. See http://s3tools.org/repositories for more informations
and instructions.

_Development support_
If you find s3cmd useful, and especially if you use it for your
business, you may want to support further development by giving us a
small donation.
Visit http://s3tools.org/donate for more info.

_Future development_
s3cmd 1.0.0 is an important milestone but we're not yet done. The most
requested features that will be addressed in the near future include:
* Multipart upload - more reliable than current file-at-once and
allowing over 5GB files.
* CloudFront invalidation - integrated with 'sync' as well as standalone
'cfinvalidate' command.
* and a lot more :)

_Etc..._
As with any new release I'm keen to hear your feedback, positive or not.
Direct any emails to s3t...@li... as usually.

Enjoy s3cmd 1.0.0 and have a great day!

Michal

Hi all,

It has been a long time since the last s3cmd release. I hope that means
 that s3cmd is now a pretty stable, mature piece of software and that
we’re on track to 1.0.0 release :-) Therefore take this 0.9.9.91 release
as a -rc1 for 1.0.0.

What's new?
===========

== Added --recursive support for cp and mv ==

It is finally possible to 'rename' buckets by moving the whole contents
from one bucket to another on the server side (ie without downloading to
your server and uploading back to S3).

== Added --include/--exclude and --dry-run for cp, mv, del and setacl ==

All file manipulation commands now understand the standard set of
parameters for including and excluding files, test runs with --dry-run, etc.

== New command 'sign' ==

This is useful for signing for instance POST upload policies in order to
allow uploading to a bucket through an ordinary web page.

== Added --verbatim mode ==

This mode prevents most filenames pre-processing, for instance
converting them to Unicode, converting to XML entities, etc. It's an
invaluable for fixing broken buckets, however successful use assumes
some decent knowledge of Amazon S3 internals. Most users will probably
prefer...

== New command 'fixbucket' ==

New command for fixing invalid filenames in a bucket. Many people
somehow seem to manage to upload files for instance with a backspace
character that later lead to an invalid XML returned from Amazon. This
command should be able to fix such buckets.

Download
--------

Users of RPM-based Linux distributions are advised to grab one of the
RPMs provided in our repositories and eventually add our repos to their
package manager. Details are here: http://s3tools.org/repositories

All others should install from a source package. Get it from
SourceForge: https://sourceforge.net/projects/s3tools/files/



Many thanks to all of you who reported problems encountered in s3cmd
0.9.9. Without you I wouldn't know what needs to get fixed.

And indeed many thanks to those who contributed with a donation
(http://s3tools.org/donations). Your help is much appreciated!

Any questions or feedback? Please send an email to the mailing list:
s3t...@li...

Enjoy!

Michal

Hi all,

over 3 months after the last release of s3cmd it's time for an upgrade
again ;-) As you may have already noticed on SourceForge or Freshmeat,
s3cmd 0.9.9 has finally been released with a load of new features and
improvements.

To learn more about Amazon S3 and s3cmd visit http://s3tools.org/s3cmd

Major new features in s3cmd 0.9.9:
---------------------------------

== Improved [put], [get] and [sync] ==
All these three commands now can do recursive transfer, accept multiple
sources and support wildcards even for the direction from S3 to local
dir. All of them also support --exclude / --include and eventually
--skip-existing filtering. For instance:
	s3cmd put --recursive dir1 dir2 file3.txt s3://bkt/backup/
	s3cmd get --exclude "tmp*.jpg" s3://bkt/backup/*.jpg /tmp/
Be aware that these commands may potentially behave slightly differently
to what you were used to in older s3cmd versions. To make sure that it
does what you meant and that your exclude / include patterns are correct
use --dry-run mode. With that option s3cmd will only print what files
should be transferred without actually transferring them.
Check out http://s3tools.org/s3cmd-sync for more details.

== CloudFront support ==
Amazon recently released their CloudFront Content Delivery Network for
public and thanks to a sponsorship from Joseph Denne s3cmd now can
manage CloudFront for you - the commands are [cfcreate], [cfdelete],
[cfmodify] and [cfinfo]. See more at http://s3tools.org/s3cmd-cloudfront

== Progress meter ==
Uploading or downloading large files often meant many minutes wait
without any way to check how far the transfer has gone. Now we've got a
new progress meter that'll keep you updated on the progress:
	testfile.pdf -> s3://bkt/testfile.pdf  [1 of 1]
	  495616 of 2304411    21% in    4s   112.52 kB/s
When s3cmd is running on a terminal it by default uses the progress
meter. If it's running from a cron job or otherwise without a terminal
it won't. These defaults can be overriden with --progress and
--no-progress switches.

== Commands for copying and moving remote files ==
Andrew Ryan contributed support for Copy and Move operations, accessible
as [cp] and [mv] commands. You can copy or move objects within a bucket
or from one bucket to another, even from US to EU data center. At the
moment these two commands don't support recursive operation or multiple
sources, such a functionality will be added in the upcoming version.

== New command [setacl] for setting ACL on existing objects ==
To enable public access to your files over HTTP they need to have a
"Public ACL". You can do it either during upload with --acl-public
parameter or later on using [setacl] command. Again, sponsored by Joseph
Denne. Use it with --acl-public or --acl-private to set the desired
Access Control.

== New --add-header option ==
One more feature for web hosters - occasionally you may want to set a
specific headers to your files that are to be accessed over HTTP. Such
as "Content-encoding:gzip" for, say, your gzipped javascript, or perhaps
Expires or Cache-control headers (not only) for use with CloudFront CDN.

With --add-header, that is available with [put] and [sync] as well as
with [cp] and [mv], you can do all that:

	s3cmd put blah.js.gz s3://public-bkt/js/ 	\
	     --mime-type "application/javascript"	\
	     --add-header "Content-Encoding: gzip"	\
	     --acl-public

== Recursive [del] and removal of non-empty buckets ==
Many people wanted an easy way to delete a subtrees from S3 or to remove
 non-empty buckets. Both is now possible - [del] understands --recursive
and [rb] honours --force (which in fact will do a recursive delete first
internally):
	~$ s3cmd rb --force s3://bkt-test
	WARNING: Bucket is not empty. Removing all the objects from it
	         first. This may take some time...
	File s3://bkt-test/testfile.txt deleted
	Bucket 's3://bkt-test/' removed

== Support for non-recursive [ls] ==
Listing large buckets with thousands of files used to be pretty
difficult with s3cmd ls. With the non-recursive [ls], which is a default
from now, listings will look like this:

	~$ s3cmd ls s3://bkt/
	                       DIR   s3://bkt/subdir1/
	                       DIR   s3://bkt/subdir2/
	2009-02-11 03:32   2304411   s3://bkt/testfile.pdf

The old "list everything" behaviour is still available with --recursive
option.

== Added --list-md5 option for [ls] ==
One more improvement for [ls] - print out MD5 sums of the remote files:

	~$ s3cmd ls s3://bkt/
	                       DIR                   s3://bkt/subdir1/
	2009-02-11 03:32   2304411 6e4469df...7a8b88 s3://bkt/tst.pdf

== Improvements in handling non-ASCII filenames ==
Ever lasting troubles with non-ASCII characters in filenames should now
be gone (or at least reduced). If you're on a UTF-8 system and all your
files have UTF-8 encoded filenames s3cmd should work just fine. In all
other cases ... let me know ;-)

Your system encoding should be autodetected correctly, but in case of
cron jobs it may eventually fail at times. In such a case --encoding
option to set the encoding explicitly:
	~$ s3cmd put --encoding UTF-8 číča.txt s3://bkt/


Minor features and bugfixes:
----------------------------
* Improved resistance to communication errors (Connection
  reset by peer, etc.)
* Continue [get] of partially downloaded files with --continue
* Fixed GPG (--encrypt) compatibility with Python 2.6.
* Always send Content-Length header to satisfy some http proxies.
* Fixed installation on Windows and Mac OS X.
* Don't print a nasty backtrace on KeyboardInterrupt.

Quite an bunch of new stuff, isn't it?


Download
--------
Users of RPM-based Linux distributions are advised to grab one of the
RPMs provided in our repositories and eventually add our repos to their
package manager. Details are here: http://s3tools.org/repositories

All others should install from a source package. Get it from SourceForge:
https://sourceforge.net/project/showfiles.php?group_id=178907&package_id=218690


Many thanks to all of you who were testing the -pre and -rc releases and
reported back the problems encountered. And indeed many thanks to those
who contributed with a donation (http://s3tools.org/donations). Your
help is much appreciated!


Any questions or feedback? Please send an email to the mailing list:
s3t...@li...

Enjoy!

Michal

Hi all,

s3cmd 0.9.9-rc3 is now available for download. It fixes three problems:

1) S3cmd crashed on upload instead of restarting the failed operation
when Amazon returned Internal Error. The crash message typically was:
	AttributeError: 'S3Error' object has no attribute '_message'
Fixed now.

2) Under some circumstances s3cmd crashed on put / get / sync when there
were no files to transfer.
Fixed now.

3) s3cmd sync with --dry-run always reported files scheduled for
deletion, thus confusing users. Now these files are only printed out
when --delete-removed is also used.

Let me know if you encounter any other problems please.

Download from here:
https://sourceforge.net/project/showfiles.php?group_id=178907&package_id=218690

Enjoy

Michal