Python: Multiprocessing under Linux and Windows

Ran into an interesting issue with the Python
multiprocessing
library under Windows, where the code that I had written would hang. It
was the equivalent of this:

from multiprocessing import Pool

...

with Pool() as pool:
    results = pool.starmap(foo, arglist)

This was in part because of how Windows was forking / pulling back the
results.

The work-around I used was the following:

if os.name == 'nt':
    from multiprocessing.pool import ThreadPool as Pool
else:
    from multiprocessing import Pool

...

with Pool() as pool:
    results = pool.starmap(foo, arglist)

Which did the trick. This is because ThreadPool uses Python's native
threading (which is Global Interpreter Locked, yes) to handle
multi-processing instead of the OS-level forking / spawning.

Posting this here in case I run into this again.

http://decafbad.net/2021/07/27/python-multiprocessing-under-linux-and-windows/

#misc #Adayinthelife #Python #Computers