[Solved] How to launch 100 workers in multiprocessing?

Question

The literally first example on the page you link to works. So I’m just going to copy and paste it here and change two values.

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(100) as p:
        print(p.map(f, range(100)))

EDIT: you just said that you’re using Google colab. I think google colab offers you two cpu cores, not more. (you can check by running !cat /proc/cpuinfo). With 2 cpu cores, you can only execute two pieces of computation at once.

So, if your function is not primarily something that waits for external IO (e.g. from network), then this makes no sense: you’ve got 50 executions competing for one core. The magic of modern multiprocessing is that this means that suddenly, one function will be interrupted, its state saved to RAM, the other function then may run for a while, gets interrupted, and so on.

This whole exchanging of state of course is overhead. You’d be faster just running as many instances your function in parallel as you have cores. Read the documentation on Pool as used above for more information.

Accepted Answer

The literally first example on the page you link to works. So I’m just going to copy and paste it here and change two values.

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(100) as p:
        print(p.map(f, range(100)))

EDIT: you just said that you’re using Google colab. I think google colab offers you two cpu cores, not more. (you can check by running !cat /proc/cpuinfo). With 2 cpu cores, you can only execute two pieces of computation at once.

So, if your function is not primarily something that waits for external IO (e.g. from network), then this makes no sense: you’ve got 50 executions competing for one core. The magic of modern multiprocessing is that this means that suddenly, one function will be interrupted, its state saved to RAM, the other function then may run for a while, gets interrupted, and so on.

This whole exchanging of state of course is overhead. You’d be faster just running as many instances your function in parallel as you have cores. Read the documentation on Pool as used above for more information.