[Solved] How to build a powerful crawler like google’s? [closed]

by


For Python you could go with Frontera by Scrapinghub

https://github.com/scrapinghub/frontera

https://github.com/scrapinghub/frontera/blob/distributed/docs/source/topics/distributed-architecture.rst

They’re the same guys that make Scrapy.

There’s also Apache Nutch which is a much older project.
http://nutch.apache.org/

1

solved How to build a powerful crawler like google’s? [closed]