csv writer with blank lineterminator breaks quoting

[EricHaszlakiewicz]

BPO	22855
Nosy	@bitdancer

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-11-12.17:05:03.785>
labels = ['type-bug', 'docs']
title = 'csv writer with blank lineterminator breaks quoting'
updated_at = <Date 2014-11-12.17:37:50.605>
user = 'https://bugs.python.org/EricHaszlakiewicz'

bugs.python.org fields:

activity = <Date 2014-11-12.17:37:50.605>
actor = 'r.david.murray'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2014-11-12.17:05:03.785>
creator = 'Eric.Haszlakiewicz'
dependencies = []
files = []
hgrepos = []
issue_num = 22855
keywords = []
message_count = 3.0
messages = ['231083', '231086', '231088']
nosy_count = 3.0
nosy_names = ['r.david.murray', 'docs@python', 'Eric.Haszlakiewicz']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22855'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

Linked PRs

• gh-67044: csv.QUOTE_MINIMAL minor Doc improvements #104118
• gh-67044: Always quote or escape \r and \n in csv.writer() #115741
• [3.12] gh-67044: Always quote or escape \r and \n in csv.writer() (GH-115741) #115866
• [3.11] gh-67044: Always quote or escape \r and \n in csv.writer() (GH-115741) #115867

[EricHaszlakiewicz]

I'm trying to emit a single line of csv without any line terminators, but specifying lineterminator=None results in a "lineterminator must be set" error, and setting lineterminator='' results in lack of quotes around certain fields.

    with open("foo.csv", "wb") as csvfile:
        csvw = csv.writer(csvfile, quoting=csv.QUOTE_MINIMAL, lineterminator='')
        csvw.writerow(["col1","col2\ndata", "col3"])

I expected the contents of the file to be:
col1,"col2
data",col3

It should be possible to change the lineterminator without changing the quoting behavior.
At the very least, the documentation needs to explain better what logic is used to determine whether something gets quoted, and what affects that.

[bitdancer]

If the line terminator is not \n, there is no reason to quote values with \n in them. (Try your code with lineterminator set to 'd' to see what I mean.)

[bitdancer]

Also, it is hard to see how to make this clearer:

csv.QUOTE_MINIMAL
Instructs writer objects to only quote those fields which contain special characters such as delimiter, quotechar or any of the characters in lineterminator.

Hmm. Perhaps it would be a bit clearer if it said "... which contain the special characters delimiter, quotechar, ..."

[serhiy-storchaka]

Note that quotechar is only quoted if doublequote is true.

[serhiy-storchaka]

The reader does not support non-quoted and non-escaped \r and \n, independently from lineterminator. So they should always be quoted or escaped.

[aterrel]

I think this is expected behaviour. There is certainly a quirk around escaping and quoting.

The writer hits \n, but with no escapechar set it quotes the element. If the user escapes the \n for there is no quoting.

>>> with open('file2.csv', 'w') as csvfile:
...     csvw = csv.writer(csvfile, quoting=csv.QUOTE_MINIMAL, lineterminator='')
...     csvw.writerow(["col1","col2\\ndata", "col3"])
...  
$ cat file2.csv 
col1,col2\ndata,col3%   

If one sets the quoting policy to be csv.QUOTE_NONE then an error on the original code reveals that it cannot escape the \n. But if one doesn't want the quoting, one can escape with csv.QUOTE_NONE

>>> with open('file3.csv', 'w') as csvfile:
...     csvw = csv.writer(csvfile, quoting=csv.QUOTE_NONE, lineterminator='')
...     csvw.writerow(["col1","col2\\ndata", "col3"])
$ cat file2.csv 
col1,col2\ndata,col3%   

Unfortunately if one does set a escapechar the quirky behavior emerges:

>>> import csv
... with open('file4.csv', 'w') as csvfile:
...     csvw = csv.writer(csvfile, quoting=csv.QUOTE_MINIMAL, lineterminator='', escapechar='\\')
...     csvw.writerow(["col1","col2\ndata", "col3"])
$ cat  file4.csv
col1,"col2
data",col3%                                                                                                                                                                                   

[serhiy-storchaka]

This is expected. Quoting is used if possible. Setting quoting=csv.QUOTE_NONE or quotechar=None disables quoting, so escaping is used for quoting char.

>>> import csv, sys
>>> csvw = csv.writer(sys.stdout, quoting=csv.QUOTE_NONE, lineterminator='', escapechar='\\')
>>> csvw.writerow(["col1","col2\ndata", "col3"])
col1,col2\
data,col320
>>> csvw = csv.writer(sys.stdout, quotechar=None, lineterminator='', escapechar='\\')
>>> csvw.writerow(["col1","col2\ndata", "col3"])
col1,col2\
data,col320

[aterrel]

@serhiy-storchaka so close this bug as "wont-fix"?

[serhiy-storchaka]

Actually, it was fixed by #115741