Assert fails in GitPython 0.3.5 comparing FETCH_HEAD and branches actually fetched

[domino14]

Stack trace:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/fabric/main.py", line 743, in main
    *args, **kwargs
  File "/usr/local/lib/python2.7/site-packages/fabric/tasks.py", line 424, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/usr/local/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
    return self.wrapped(*args, **kwargs)
  File "/var/lib/abc/ramble/venom/fabfile.py", line 286, in deploy_celery
    prepare_deploy(settings, branch_name=branch_name)
  File "/var/lib/abc/ramble/venom/fabfile.py", line 195, in prepare_deploy
    repo_install_path=install_path)
  File "/var/lib/abc/ramble/ramble/utils/git_util.py", line 31, in set_deploy_context
    remote.fetch()
  File "/usr/local/lib/python2.7/site-packages/git/remote.py", line 613, in fetch
    res = self._get_fetch_info_from_stderr(proc, progress or RemoteProgress())
  File "/usr/local/lib/python2.7/site-packages/git/remote.py", line 552, in _get_fetch_info_from_stderr
    fetch_info_lines)
AssertionError: len(....) <= len (...) (elided for length)

I wrote a script to parse the two things that it is trying to compare, from the assert output. FETCH_HEAD right now has 813 lines. The output of git fetch, when I run it manually on the command line (I have not been able to figure out exactly what command it is running, but I'm assuming it's git fetch --all -v - is this correct?) has also 813 lines if you ignore the first two lines: Fetching origin and From github.com:org/repo

However, the very last 4 branches in FETCH_HEAD are missing in the GitPython fetch output, the last of which is this line:

a3a81646dd65dfa4b4412c0f3a5fd4183cbaa2e5 not-for-merge branch 'zendesk' of github.com:org/repo

It can't find the matching line in the fetch output (or for the previous 3):

= [up to date] zendesk -> origin/zendesk

Even though I see the above lines when I do git fetch --all -v, I don't see the last 4 lines in the GitPython assert output. Let me know if you need more info. Is there possibly some length limit? I'm using git 1.8.3.4.

BTW I was using GitPython==0.3.2.RC1 prior to this and it was still failing an assertion, although I'm not sure the assertion was exactly the same.

[Byron]

You can set GIT_PYTHON_TRACE=full in your shell to see all the commands that git-python runs in the background. A quick look revealed that its probably only running git fetch -v though.

git-python version 0.3.3 to 0.3.4 don't have this assertion at all and should therefore just work for you.

The only way I think this can be fixed is if I can reproduce it, so let me post the information here that I put in the referenced issue:

I would be glad if you could help me to reproduce this issue - either by providing me with steps to do it myself or by getting access to the data involved. The latter would be each line in my_progress_handler and the entire contents of the FETCH_HEAD.

Thank you.

[domino14]

Hi Byron, I see the assertion in 0.3.3 here:
https://github.com/gitpython-developers/GitPython/blob/0.3.3/git/remote.py#L550

It's also in a similar place in 0.3.4 so this would also fail for me.

How do I get the lines in my_progress_handler? And please let me know how I can send you this info.

[Byron]

Are you able to use a custom version of git-python in your setup ? If so, I could make a temporary branch that will contain code to write this information to files. These files you could just attach here by dropping them into the issue (or make a gist).
Basically you would have to run a custom version of git-python and reproduce the issue on your system.
It would be easier though if you could tell me how to reproduce the issue myself, is the repo you are fetching publicly accessible ?

[domino14]

I can use a custom version of git-python in the meantime, yes, so I can send you those files. The repo is not publicly accessible so if possible I'd like to send these files to you privately.

[Byron]

Alright, let's do that. You could send me a private message on github with download links. Additionally, you could find-and-replace common portions of both files to obfuscate them, as long as said obfuscation is consistent within each file, and across both files. After all, I don't want your secrets either :).

[Byron]

Oh, and I will make a post here once the branch is ready.

[Byron]

You will find a git-python version able to write the respective information in this branch. Please check the commit message for more information on where the debug data can be found.

Last but not least, you can watch the archived live-stream of the work done on the issue so far on youtube.

[Byron]

@domino14 Release v0.3.6 is due this week, and I would be glad if I could provide a fix to this issue in it. For you it should be as simple as running your testcase/code with the provided branch and send me the files. Feel free to send them by e-mail. Thank you !

[domino14]

sent you an email with the files, thanks!

On Mon, Jan 19, 2015 at 5:00 AM, Sebastian Thiel [email protected]
wrote:

@domino14 https://github.com/domino14 Release v0.3.6 is due this week,
and I would be glad if I could provide a fix to this issue in it. For you
it should be as simple as running your testcase/code with the provided
branch and send me the files. Feel free to send them by e-mail. Thank you !

—
Reply to this email directly or view it on GitHub
#232 (comment)
.

[Byron]

Thank you !
A first glimpse at the data seems to indicate that it just didn't read all the progress data - there seem to be 14 lines missing in the progress output. That's quite a bunch - besides that though, the files seemed to match up.

• Can you tell me if you are using python 2 or 3 ?
• Which platform do you run the code on ?
• Does select.poll exist in your python interpreter ?

[domino14]

When the problem started there was only 1 line missing in the progress
output, but we've added a few more branches since then and it keeps cutting
off more and more from the bottom.

• Python 2.7.8
• CentOS release 6.5 (Final)
• Not sure, I would assume it exists? I did import select, and select.poll
  is a function.

On Mon, Jan 19, 2015 at 12:02 PM, Sebastian Thiel [email protected]
wrote:

Thank you !
A first glimpse at the data seems to indicate that it just didn't read all
the progress data - there seem to be 14 lines missing in the progress
output. That's quite a bunch - besides that though, the files seemed to
match up.

• Can you tell me if you are using python 2 or 3 ?
• Which platform do you run the code on ?
• Does select.poll exist in your python interpreter ?

—
Reply to this email directly or view it on GitHub
#232 (comment)
.

[Byron]

Great ! There are two different branches for parsing process output, and the one using poll() seems to be the source of the issue here.
Thanks the information, as I believe we are getting closer. I also have an idea on how to possibly fix this.

[Byron]

@domino14 Please have a look at the latest state on our issue-related branch, as it contains simplifications which should make your case work natively.
Please be sure you don't catch the Assertion error anymore while testing this one.
Thanks for your feedback !

[Byron]

@domino14 Please also try the latest master branch, as it contains a greatly improved poll() based implementation, that should now work for you.
Please let me know about your findings in both branches - I believe combining both adjustments would be the best solution after all.
Thanks for your feedback.

[Byron]

The archived live-stream can be found on youtube:

• part 1
• part 2

[Byron]

Please note that I have merged improvements made to the issue-related branch into master, without the debug portions.
That way, you would only have to test master to be sure it does work indeed, thus saving some of your time.

[domino14]

Hi Byron, thanks and your live streams are pretty cool - good idea. The current master works without throwing any exceptions. Thanks for looking at this!!

[Byron]

Awesome, I am happy to hear that, in every which way !
If everything works out as planned, the new release will be up on pypi tomorow.
Cheers