distributed-system

[Solved] algorithm for slicing brute force keyspace [closed]

by

Let’s start with a slightly simpler problem. All keys are 8 numeric digits, and you have 10 machines. Simple enough – one machine checks 0???????, another checks 1??????? and so on. Note that this slicing doesn’t care that you’re doing multiprocessing – it’s just allocating ranges by fixing one of the digits. Your version is … Read more

[Solved] How to build a powerful crawler like google’s? [closed]

by

For Python you could go with Frontera by Scrapinghub https://github.com/scrapinghub/frontera https://github.com/scrapinghub/frontera/blob/distributed/docs/source/topics/distributed-architecture.rst They’re the same guys that make Scrapy. There’s also Apache Nutch which is a much older project. http://nutch.apache.org/ 1 solved How to build a powerful crawler like google’s? [closed]