A toolchain for bringing web2 to web3. IPFS daemon should be launched. Download enwiki-latest-all-titles to crawler root dir. Basically, there are two main functions provided by crawler tool. The first one is to parse wiki titles and submit links between keywords and wiki pages. Also, crawler has separate command upload-duras-to-ipfs to upload files to local IPFS node. All DURAs are collected under single root unixfs directory.

Features

  • Note: Requires Go 1.12+
  • IPFS daemon should be launched
  • Download enwiki-latest-all-titles to crawler root dir
  • Basically, there are two main functions provided by crawler tool
  • Parse wiki titles and submit links between keywords and wiki pages
  • Upload duras to IPFS

Project Samples

crawler Screenshot 1

Project Activity

See All Activity >

Categories

File Sharing

License

MIT License

Follow crawler

crawler Web Site

Other Useful Business Software
Our Free Plans just got better! | Auth0 Icon
Our Free Plans just got better! | Auth0

With up to 25k MAUs and unlimited Okta connections, our Free Plan lets you focus on what you do best—building great apps.

You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
Try free now
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of crawler!

Additional Project Details

Operating Systems

Linux, Mac, Windows

Programming Language

Go

Related Categories

Go File Sharing Software

Registered

2023-07-27
Report inappropriate content