1 minute read

Python’s multiprocess.Process makes forking and spawning intuitive - you don’t have to worry about opening a new Python interpreter manually, etc. Here is a snippet that demonstrates the most interesting part of Process:

p = mp.Process(target=fn, args=args, kwargs=kwargs)    
p.start()                                              

In short, the function fn will run in parallel, in a new process. You may pass positional parameters args or named parameters kwargs, but these have to be picklable. To generalize, the Process API allows us to pass functions to the new process, but what if we want to pass and execute an AST Parsetree instead? Let’s say we have the following scenario:

codestring="import ..."
parsetree1=ast.parse(codestring)
parsetree2=mod_parsetree(parsetree1)

p = mp.Process(target=???, args=args, kwargs=kwargs)    
p.start()                                              

What would the target be?

Considering that:

exec(parsetree2)

is what I ultimately want to happen inside the new process, my first approach was to pass exec with parsetree2 as the first argument. However, this raized an error because apparently, parsetrees are not picklable (why not though?).

Since strings are picklable, I quickly found out that the ast module has an unparse function which will return the str equivalent of a given parsetree. Sadly, this function is only available since Python 3.9 but I was bound to Python 3.8. astunparse came to the rescue - it’s a simple library (and beautifully written, check the code btw.) that gives us back the missing unparse functionality. Now we can do:

sudo python3 -m pip install astunparse

and

codestring1="import ..."
parsetree1=ast.parse(codestring)
parsetree2=mod_parsetree(parsetree1)

codestring2=astunparse.unparse(parsetree2)
p = mp.Process(target=exec, args=(codestring2))    
p.start()                                              

And the day is saved! I am not happy that I had to include another dependency, which may or may not work for all possible input cases (e.g. complex programs). I wonder if this could be done in a simpler way:

Warning this is just speculative, not tested

Since multiprocess.Process allows us to pass a function to the other process, we could prepare a special function using functools to execute there. This would look something among the lines of:

def exectree(tree):
    exec(tree):
foreign_function = partial(exectree,parsetree2)
p = mp.Process(target=exec)    
p.start()                                              

Notice that we don’t have to pass any parameters to the other process this time, because they are embedded in the function we are passing.