Let’s say I have this simple function:
def print_name(name): print(name) time.sleep(10) print('finished sleeping @ ', str(dt.datetime.now()))
I’m trying to use multiprocessing to run this in a loop for several names all at once, like in the below:
from multiprocessing.pool import ThreadPool as Pool names = ['A','B','C','D','E'] with Pool() as pool: for name in names: pool.map(print_name,[name])
However, it doesn’t run simultaneously, it runs one after the other as you can see:
A finished sleeping @ 2022-07-26 11:03:12.394843 B finished sleeping @ 2022-07-26 11:03:22.400343 .......
NB: I’m having to use
ThreadPool instead of
Pool just throws up a random pickling error:
_pickle.PicklingError: Can't pickle <function print_name at 0x144ce5000>: attribute lookup print_name on __main__ failed
I have also seen people talking about
pathos.multiprocessing but that throws up this error
NotImplementedError: pool objects cannot be passed between processes or pickled.
ThreadPool is the only way I can get any form of multiprocessing to at least not throw in an error message. So possibly that’s my problem?
Ultimately I’m hoping to be able to use multiprocessing as I have a big function that takes about 15 mins to run, and I need to run it over a list of about 100 items so multiprocessing would be really handy. But I can’t even get this simple example to work at the minute and I’m a bit stuck, so any help would be really appreciated.
You may want to consider using ProcessPoolExecutor from concurrent.futures but in the meantime, this may help:
import datetime as dt import time from multiprocessing import Pool def print_name(name): print(name) time.sleep(5) print('finished sleeping @ ', dt.datetime.now()) names = ['A','B','C','D','E'] def main(): with Pool() as pool: pool.map(print_name, names) if __name__ == '__main__': main()
A B C D E finished sleeping @ 2022-07-26 11:29:31.277722 finished sleeping @ 2022-07-26 11:29:31.277749 finished sleeping @ 2022-07-26 11:29:31.285708 finished sleeping @ 2022-07-26 11:29:31.292636 finished sleeping @ 2022-07-26 11:29:31.295505