[Solved] How to make this crawler more efficient [closed]

by


Provided your intentions are not nefarious–

As mentioned in the comment, one way to achieve this is executing the crawler in parallel (multithreading)—as opposed to doing one domain at a time.

Something like:

exec('php crawler.php > /dev/null 2>&1 &');
exec('php crawler.php > /dev/null 2>&1 &');
exec('php crawler.php > /dev/null 2>&1 &');
exec('php crawler.php > /dev/null 2>&1 &');
exec('php crawler.php > /dev/null 2>&1 &');

On the server, you can setup a CRON job that will do this automatically, so that you are not running it manually.

1

solved How to make this crawler more efficient [closed]