Commits

Branch selector

 master

User selector

All users

Datepicker

All time

Commit History

Commits on Dec 4, 2011

• Fixed merge conflict

  theanti9

  committed
  2d186f4

  Copy full SHA for 2d186f4

• Changes to the content processor

  theanti9

  committed
  38593b6

  Copy full SHA for 38593b6

Commits on Nov 27, 2011

• Merge pull request #8 from WilHall/master

  Show description for fafb8c1
  theanti9

  committed
  fafb8c1

  Copy full SHA for fafb8c1

• Added logging facility, ColorStreamHandler, and updated .gitignore

  WilHall

  committed
  d9ed584

  Copy full SHA for d9ed584

• Merge branch 'master' of https://github.com/theanti9/PyCrawler

  Show description for 5db8daf
  WilHall

  committed
  5db8daf

  Copy full SHA for 5db8daf

Commits on Nov 1, 2011

• Work on colors, Added option to rotate sqlite database files

  WilHall

  committed
  ad55cd8

  Copy full SHA for ad55cd8

• Fix the problem that caused it to leave thread pools open after starting and flooding the system with dead processes

  theanti9

  committed
  ca6d7a8

  Copy full SHA for ca6d7a8

• Added user agent header to requests

  theanti9

  committed
  3b7e262

  Copy full SHA for 3b7e262

• lots of fixes

  theanti9

  committed
  dacd344

  Copy full SHA for dacd344

• Created Printer class for color printing

  WilHall

  committed
  4425b4c

  Copy full SHA for 4425b4c

• Adding readme

  theanti9

  committed
  6fdbc75

  Copy full SHA for 6fdbc75

• Complete redo

  theanti9

  committed
  ad8c39f

  Copy full SHA for ad8c39f

Commits on Aug 19, 2011

• Changed the linkregex to the suggestion in gtoffoli's issue #7 temporarily while i restructure/rewrite the entire project

  theanti9

  committed
  c462210

  Copy full SHA for c462210

Commits on Aug 8, 2011

• Updated .gitignore

  theanti9

  committed
  a2abec1

  Copy full SHA for a2abec1

Commits on Jun 16, 2011

• Merge pull request #5 from sjparsons/status

  Show description for f24e79b
  theanti9

  committed
  f24e79b

  Copy full SHA for f24e79b

Commits on Jun 15, 2011

• Status Codes and URL limited

  Show description for ba0ecb9
  sjparsons

  committed
  ba0ecb9

  Copy full SHA for ba0ecb9

Commits on May 8, 2011

• Fixing issue 2, psyco in __main__

  theanti9

  committed
  82ccb6b

  Copy full SHA for 82ccb6b

Commits on Jan 9, 2011

• Fixed links regexp + newline at end of file

  mswietochowski

  committed
  7ce95c0

  Copy full SHA for 7ce95c0

Commits on Nov 23, 2010

• Error fix - couldn't join parse object in urljoin.

  theanti9

  committed
  e14a6d0

  Copy full SHA for e14a6d0

• Error fix - couldn't join parse object in urljoin.

  theanti9

  committed
  ee70c79

  Copy full SHA for ee70c79

Commits on Nov 7, 2010

• Fixed major glitch in url joining.

  theanti9

  committed
  6261c47

  Copy full SHA for 6261c47

Commits on Oct 11, 2010

• Updated readme

  theanti9

  committed
  9da2e16

  Copy full SHA for 9da2e16

• Added robots.txt compliance and some small fixes.

  theanti9

  committed
  3017da5

  Copy full SHA for 3017da5

• Added more comments. Added optional arg 'verbose' as 4th arg to specify if the urls should be printed as they are being crawled.

  theanti9

  committed
  f24055b

  Copy full SHA for f24055b

• Restructed the way the request is made very slightly to add a User-Agent

  theanti9

  committed
  848ce4a

  Copy full SHA for 848ce4a

Commits on May 2, 2010

• Changed syntax error

  theanti9

  committed
  7f8095b

  Copy full SHA for 7f8095b

Commits on May 1, 2010

• logic error debugging

  theanti9

  committed
  9a25e59

  Copy full SHA for 9a25e59

Commits on Apr 30, 2010

• Got rid of beautiful soup as it caused problems with any page that has any malformed html.fixed a few syntax errors

  theanti9

  committed
  2b3f0df

  Copy full SHA for 2b3f0df

• Restructured the crawler. works with the new tables, crawles to limited specified depth. keeps current and parent ids so the gui can display the nesting properly

  theanti9

  committed
  c07b5dd

  Copy full SHA for c07b5dd

• Updated command line args and sqlite table creation. Added id and parent fields to both queue and crawl_index table. Added status table to keep track of whether the crawler has started and finished.

  

  unknown

  committed
  426482e

  Copy full SHA for 426482e

Commits on Apr 24, 2010

• started implementing Beautiful Soup

  theanti9

  committed
  e2251a6

  Copy full SHA for e2251a6

Commits on Apr 22, 2010

• Added simple comments

  theanti9

  committed
  3ed76c2

  Copy full SHA for 3ed76c2

• Added check for a starting point

  theanti9

  committed
  f930b67

  Copy full SHA for f930b67

• Switched continue statements not in the loop to pass statements which were a result of switching the body of the program to a function.

  theanti9

  committed
  1912786

  Copy full SHA for 1912786

• removed threading as it produced no performance gain and also was bad in combination with SQLite according to their documentation.

  theanti9

  committed
  4accdac

  Copy full SHA for 4accdac

Pagination

Previous

Next