0.6.3: testing stalls at test_pack (gitdb.test.test_pack.TestPack) ...

[yarikoptic]

Just mentioned that a sweep of builds I ran a few days back got stalled while building for Debian wheezy at

test_pack (gitdb.test.test_pack.TestPack) ...

yet not sure if wheezy specific or sporadic or what... but just wanted to report if may be there are immediate ideas

[yarikoptic]

Ctrl-C'ing of the next build (so reproducible) lead to following backtrace if it makes any sense

...
  File "/tmp/buildd/python-gitdb-0.6.3/gitdb/test/test_pack.py", line 141, in test_pack
    self._assert_pack_file(pack, version, size)
  File "/tmp/buildd/python-gitdb-0.6.3/gitdb/test/test_pack.py", line 90, in _assert_pack_file
    for obj in pack.stream_iter():
  File "/tmp/buildd/python-gitdb-0.6.3/gitdb/pack.py", line 555, in _iter_objects
    cur_offset += (data_offset - ostream.pack_offset) + ostream.stream.compressed_bytes_read()
  File "/tmp/buildd/python-gitdb-0.6.3/gitdb/stream.py", line 179, in compressed_bytes_read
    self.read(mmap.PAGESIZE)
  File "/tmp/buildd/python-gitdb-0.6.3/gitdb/stream.py", line 211, in read
    size = min(size, self._s - self._br)
KeyboardInterrupt

[Byron]

Thank you.
I just ran the tests on the latest version and they ran through in 25s. However, before the next release I shall run the tests on the source distribution, maybe something is different there.
Just for completeness, it would be interesting to hear which python version you have been using.

[kevin-brown]

Ctrl-C'ing of the next build (so reproducible) lead to following backtrace if it makes any sense

A KeyboardInterrupt is raised whenever you hit CTRL + C when running a Python program. I'm not sure if that is actually related to your original issue though.

[yarikoptic]

this issue seems to be related solely to older stable wheezy and not e.g. more recent to be released jessie. Ubuntu builds are yet to be ran, but meanwhile you could see anything you like about versions of python (2.7.3) and git etc in
http://neuro.debian.net/_files/_buildlogs/python-gitdb/0.6.3/python-gitdb_0.6.3-1~nd70+1_amd64.build

[yarikoptic]

keyboard interrupt track should be close to the place where it was stuck when I pressed it thus I hoped providing information on where that test was stuck

[yarikoptic]

FWIW -- none of the compatible ubuntus was affected, so solely a wheezy issue. I will try to get a better clue

[yarikoptic]

FWIW here is some information from arrogant me since I am not sure about details but it never exits that loop on wheezy:

root@head2:/home/neurodebian/deb/builds/python-gitdb/0.6.3-1/python-gitdb-0.6.3# nosetests -s -v --pdb --pdb-failures /home/neurodebian/deb/builds/python-gitdb/0.6.3-1/python-gitdb-0.6.3/gitdb/test/test_pack.py
test_pack (gitdb.test.test_pack.TestPack) ... ENTERING LOOP
> /home/neurodebian/deb/builds/python-gitdb/0.6.3-1/python-gitdb-0.6.3/gitdb/stream.py(180)compressed_bytes_read()
-> while not self._zip.unused_data and self._cbr != len(self._m):
(Pdb) print self._zip
<zlib.Decompress object at 0x1129d40>
(Pdb) print self._cbr
0
(Pdb) print len(self._m)
3646
(Pdb) n
> /home/neurodebian/deb/builds/python-gitdb/0.6.3-1/python-gitdb-0.6.3/gitdb/stream.py(181)compressed_bytes_read()
-> self.read(mmap.PAGESIZE)
(Pdb) n
> /home/neurodebian/deb/builds/python-gitdb/0.6.3-1/python-gitdb-0.6.3/gitdb/stream.py(180)compressed_bytes_read()
-> while not self._zip.unused_data and self._cbr != len(self._m):
(Pdb) print len(self._m)
3646
(Pdb) print self._cbr
0
(Pdb) n
> /home/neurodebian/deb/builds/python-gitdb/0.6.3-1/python-gitdb-0.6.3/gitdb/stream.py(181)compressed_bytes_read()
-> self.read(mmap.PAGESIZE)
(Pdb) print mmap.PAGESIZE
4096
(Pdb) l
176                 else:
177                     # pass in additional pages, until we have unused data
178                     print "ENTERING LOOP"
179                     import pdb; pdb.set_trace()
180                     while not self._zip.unused_data and self._cbr != len(self._m):
181  ->                     self.read(mmap.PAGESIZE)
182                     print "LEAVING LOOP"
183                     # END scrub-loop default zlib
184                 # END handle stream scrubbing
185
186                 # reset bytes read, just to be sure

(print statements were added by me)

[yarikoptic]

ok -- added a print statement

print "unused %s cbr %s len %s" %(self._zip.unused_data, self._cbr, len(self._m))

at the end of the loop
and here is how it looks on my laptop

$> nosetests -s -v gitdb/test/test_pack.py
test_pack (gitdb.test.test_pack.TestPack) ... unused ½x%ÇÁ   0À�ªØ
lC�ÚBԃ, cbr 43 len 1633
Ø¢x340031 cbr 43 len 1503
unused ¢x cbr 12 len 1457
unused d>xuà´|¢x340031 cbr 43 len 1443
unused ¢x cbr 12 len 1397
unused ½IxuSÉnÛ0=_1·.¨©EÂ@ cbr 43 len 1383
unused fx;϶ cbr 21 len 720
...

and on wheezy:

test_pack (gitdb.test.test_pack.TestPack) ... ENTERING LOOP
unused  cbr 0 len 3646
unused  cbr 0 len 3646
unused  cbr 0 len 3646
unused  cbr 0 len 3646
unused  cbr 0 len 3646
unused  cbr 0 len 3646
unused  cbr 0 len 3646
...

if that is of help -- dig into zip and self._cbr ;)

[Byron]

Thank you, I believe you pinpointed the spot where it hangs quite precisely !
Do you have any pointers on where I can most easily get a (small) image of wheezy ? I will setup a new VM just to reproduce (and hopefully fix) the issue.

[Byron]

I have made some intense cross-testing just to figure out what works and what doesn't work. With a changed logic, I was at least able to get it to work with most configuration (basically I just checked for zlib version 1.2.7 on wheezy, and made a special case for osx and zlib version 1.2.5).

OS	zlib version	Python Version	Works with Old Logic	Works with Changed Logic
Wheezy	1.2.7	2.6,2.7,3.2	2.6 only	✅
OSX 10.10	1.2.5	2.6,2.7.3.4	✅	✅
centos 6.5	1.2.3	2.6	✅	✅
xubuntu 14.04.1 LTS	1.2.8	2.7,3.4	✅	✅
fedora 17	1.2.5	2.7,3.2	❌	✅
windows 7	1.2.3	2.7,3.4	✅	✅

In other words, with the changed logic, it works on all tested platforms, but I expect this to fail in the future, as the actual problem is not solved. Thus I could not tell why the changed logic at this spot is even necessary.

[Byron]

Development videos can be found here:

• Youtube PT 1/2
• Youtube PT 2/2