Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Python imap to take multiple functions

I am utilizing multiprocessing using imap, since I need the results in order of the tasks/jobs executed. I just realized that I need a different function for the first task/job. Is there a way to specify 2 functions for the imap (where the first function (say func1) is only applied to the first task/job, and the rest will execute the second function (say func2)?

What I currently have is like:

from multiprocessing import Pool

pools = 2

with Pool(pools) as p:
    for result in p.imap(func2, args): # where args is an iterator/list of params to be passed to func2
        # some more processing here on the result

Which basically before I realized what I needed to do.

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

Is there a way to do what I am looking for with imap? Or would I just have to execute func1 first, then do the imap with the second function. That would work, but I really wanted the first task and the second task to run in parallel (when pools is set to 2 at least).

Assume there are hundreds of tasks/jobs to be performed.

>Solution :

If I understand you correctly you can use itertools.chain:

from time import sleep
from itertools import chain
from multiprocessing import Pool


def func1(args):
    print("func1", args)
    sleep(1)


def func2(args):
    print("func2", args)
    sleep(2)


if __name__ == "__main__":
    pools = 2

    args = [1, 2, 3]

    with Pool(pools) as p:
        for result in chain(p.imap(func1, [args[0]]), p.imap(func2, args[1:])):
            pass

Prints (the first two tasks are executed in parallel):

func1 1  # in parallel with func2
func2 2  # in parallel with func1
func2 3 
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading