Subprocess - Simple Guide to Launching a Program as a Child Process in Python¶

Developers many times need to execute programs that are installed on the operating system but are not available through any Python module. They also need to capture their results (output, errors, etc). Many of these programs are easily executable as a command from the shell in Unix/Linux and command prompt on windows. Python provides a module named subprocess which lets us execute these programs by creating an independent child process. It also lets us capture their results. The subprocess module gives full control over program execution. It lets us create a process, capture results, capture errors, communicate with the process and terminate/kill the process if it does not complete within the expected time.

The subprocess module lets us create a child process in two ways using its API.

• run() - This method takes an input command to execute as a list of strings and executes it. It waits for the child process to complete. It returns an instance of CompletedProcess which has information about process results. This is recommended way of executing programs and will work in the majority of cases.
• Popen() - This method takes an input command to execute as a list of strings and executes it. It does not wait for the completion of the process. It immediately returns with the instance of Popen which will be populated with results when the process complete. It does provide methods that can force to wait for the completion of the process. It also provides methods to communicate with the process, terminates the process, etc.

The parameters of run() and Popen() are almost same. The Popen() is very comprehensive and should be only for advanced cases.

As a part of this tutorial, we'll explain how we can create a process to execute a program in Python using subprocess module with very simple and easy-to-understand examples.

Example 1: Create a Process using run() to Run a Command¶

As a part of our first example, we'll create a simple child process using run() method that executes the Linux ls command and returns the result (list of files/directories in the present directory). We can provide commands as a list of strings to run() method.

Our code for this example creates 3 subprocesses each of which executes the same command. Our first call to run() method is without any parameters other than a program to execute. It does not capture any results of the execution of ls command as a process. Our second call to run() method sets capture_output parameter asking it to capture results of execution of program. Our third call to run() method sets stdout and stderr parameters to capture results and errors during execution of program.

We can notice from the output of a call to run() that it returns an instance of CompletedProcess. We have printed output captures in second and third call to run() method using stdout and strerr attributes of CompletedProcess instance. The output returned by the execution is in bytes format hence we are calling decode() method on them to decode output. We are only printing the first 5 lines of output to avoid a lot of output as the current directory has a lot of files. It’s not always necessary to capture output though.

Important Parameters of run() Method¶

• args - It accepts a list of strings specifying commands to execute a particular program as a process. It should the first parameter to the run() method.
• capture_output - It’s a boolean flag indicating to the method that it should capture standard output and errors.
• stdout - It accepts a file-like object as input to which output of the program execution should be directed.
• stderr - It accepts a file-like object as input to which errors that occurred during the program execution should be directed.

The stdout and stderr parameter also accepts value PIPE which is special value available from subprocess module and creates io stream to capture output and errors.

Import Attributes of CompletedProcess Instance¶

• returncode - It returns an integer specifying the status of execution of the child process. The value of 0 indicates successful completion. The value of -N indicates that the process was terminated by signal number N.
• stdout - It holds the standard output of the child process.
• stderr - It holds standard errors of the child process.

Example 2: Create a Process using run() to Run a Command Continued ...¶

As a part of our second example, we are again building on the first example. We are again executing ls command as a child process using run() method but this time introducing more parameters of the method for an explanation.

The first part of the code creates a child process that executes ls command. We have given encoding and errors parameters as new parameters to the method. The second part of the code again executes ls command but explains the usage of text parameter. The third part of the example executes ls command with argument --a which is the wrong argument and it'll fail the command. We have created this example to explain how we can capture errors that occurred during the child process.

Please make a note that we have not used decode() method as a part of these examples because all of them return string output due to their parameters, unlike the previous example.

The output of our first and second parts of the code is exactly the same as in our previous example. The output of the third part of the code successfully captures the error of the child process.

Important Parameters of run() Method¶

• encoding - It accepts string specifying encoding for the standard output and errors.
• errors - It accepts string value specifying what to do when any errors happen during encoding. Below is a list of commonly used values for the parameter. The list of all possible values can be gained by calling command help(codecs.Codec).
  • ignore - It'll ignore encoding errors.
  • strict - It'll raise an error if an encoding error happens.
  • replace - It'll replace errors with suitable matching characters.
• text - This is a boolean parameter which if set to True will capture the output of the child process in text format.

Example 3: Create a Process using run() to Run a Command Continued ...¶

We are again using our third example to demonstrate some important parameters of the run() method.

Our code for this example has two parts. Both parts execute a command passed to it in the shell as shell command is set to True. We even have passed command as a single string rather than a list of strings because shell parameter is set to True. The first part of the code executes ls -l command which lists files and directories in the present directory. The second part of the code executes the command ls -l | grep .py which lists down the list of files and directories which are ending with .py extension.

Important Parameters of run() Method¶

• shell - This parameter accepts boolean value. If set to True will execute command passed to args in the Unix shell. The default value is None.

Example 4: Capture Errors in Error Stream¶

As a part of our fourth example, we are executing a Python script as an independent process using run() method. We are capturing its output as well. We'll be reusing this script in future examples as well for demonstration.

The script has a function named addition which accepts two parameters and returns their sum. It waits for 2 seconds before returning output to mimic a real-life situation that computing sum takes time. We are then calling this function four times in the script. The first three calls are all with integer argument hence completes fine. The last call has one argument as a string hence fails with an error. We have captured the error using stderr parameter. We are then printing both normal output and errors that were captured when running the script as a child process.

Please make a NOTE that the process returns code 1 as there were errors during execution.

addition.py

import time

def addition(a, b):
    time.sleep(2)
    return a + b

if __name__ == "__main__":
    res = addition(10,20)
    print("{} + {} = {}".format(10,20, res))

    res = addition(20,20)
    print("{} + {} = {}".format(20,20, res))

    res = addition(20,30)
    print("{} + {} = {}".format(20,30, res))

    res = addition('20',30)
    print("{} + {} = {}".format('20',30, res))

Example 5: Capture Errors and Output Both in the Same Output Stream¶

We are using our fifth example to explain how we can direct standard errors to standard output. Our code for this example is exactly the same as our previous example with minor changes. We have set stdout to PIPE and stderr to STDOUT. The STDOUT value will inform run() method that directs the error captured as a part of the standard error to the standard output stream.

We can notice from the output captured that the error is now getting printed when printing the standard output stream.

Example 6: Timeout Process¶

As a part of our sixth example, we'll demonstrate how we can timeout the child process if it’s taking more than expected time.

Our code for this example again runs addition.py script from the previous example but this time it calls method run() with timeout parameter set to 4 seconds. It'll timeout the process if it does not complete within 4 seconds and raise TimeoutExpired error. We have wrapped the code into the try-except block to capture errors.

Important Parameters of run() Method¶

• timeout - This parameter accepts a number specifying the number of seconds to wait for the process to complete. If it does not complete within that time then raise TimeoutExpired error. The default is None.

Example 7: Check for Errors in Process¶

As a part of our seventh example, we'll explain how we can check for the errors that occurred while executing the child process.

Our code for this example is exactly the same as our previous example with the only change that we have set check parameter to True and timeout parameter is removed. The execution raises CalledProcessError error which we are capturing and printing error details.

We can notice from the output that it also has captured the output of the code which ran successfully.

Important Parameters of run() Method¶

• check - This parameter accepts boolean value. If it’s set to True then it'll check for the return code of the child process and will raise CalledProcessError error if the return code is not zero. The default is False.

Example 8: Pass Input to the Process¶

We are using our eighth example to demonstrate how we can give input to the child process using input parameter of the run() method. We have created another script for this part of the code.

The script has addition function which is the same as the previous script. The main part of the code asks for the user to enter the value of the parameters of the addition function and then calls the function using these inputs. It does that two times. We are passing data to the child process in two different formats. Once as a string and once as bytes. We have passed values separated by new line characters to indicate that input is completed.

We can notice from the output that it seems to have captured values correctly.

Important Parameters of run() Method¶

• input - It accepts input which will be given to process as input. The default input is accepted as bytes but it accepts input as a string if text or encoding parameters are set.

addition_user_input.py

import time

def addition(a, b):
    time.sleep(2)
    return a + b

if __name__ == "__main__":
    a = input("Enter First  Number : ")
    b = input("Enter Second Number : ")
    res = addition(int(a),int(b))
    print("\n{} + {} = {}".format(a,b, res))

    a = input("Enter First  Number : ")
    b = input("Enter Second Number : ")
    res = addition(int(a),int(b))
    print("\n{} + {} = {}".format(a,b, res))

8.1: String Input¶

This part of the code calls the script and passes input to the script as a string. We have set encoding to utf which will let us give input in string format. If we have set text to True then also we can give input as a string.

8.2: Bytes Input¶

This part of the code explains how we can give bytes as input to the child process. We need to give input in bytes format if we have not set test or encoding parameter.

Example 9: Pass Environment Variables to the Process¶

As a part of our ninth example, we'll demonstrate how we can give environment variables to our child process by setting env parameter of the run() method. We can give different environment variables than the one set by the parent process which started the child process.

We have created a simple script that just gets environment variables by using os module and prints the value of environment variables HOME, USER, and PASS. Our main part of the code calls this script using run() method. It first gets current environment variables mapping and creates a copy of it. It then sets the 3 parameters which we mentioned earlier. It then passes this mapping of environment variables as env parameter of the run() method.

We can notice from the output that it seems to have correctly captured the value of environment variables.

Important Parameters of run() Method¶

• env - It accepts mapping which has the key and value of environment variables that we want to pass to the child process.

environment_variables.py

import os

def print_env_variables():
    for env_var in ["HOME", "USER", "PASS"]:
        print("{} : {}".format(env_var, os.environ[env_var]))

print_env_variables()

Example 10: Simple Example to Create a Process using Popen¶

As a part of our example, we'll explain how we can create a child process using Popen() constructor. The arguments of Popen constructor are almost the same as that of run() method, even it has more parameters than run() method. The majority of our previous examples will work even with Popen constructor as well. The Popen constructor does not have parameters named capture_output, timeout, input, and check, on the other hand, it has other methods to perform the functionalities provided by these parameters.

As we discussed earlier, unlike run() method which waits for the child process to complete, Popen() returns immediately with an instance of type Popen after the process has started. Our code for this part is exactly the same as our previous examples with the only change that we are using Popen() to launch a child process that will run addition.py Python script.

We can notice from the output that it’s almost the same as our previous examples. When we run this example, the Popen() constructor returns immediately. with an instance of Popen instead of waiting for the child process to complete. This is the reason the return code is None because it’s not set yet. It'll be set after the child process has completed.

Example 11: Wait for Process to Complete¶

As a part of our eleventh example, we'll demonstrate how we can instruct the main process to wait for the child process to complete using wait() method of Popen instance. We'll also explain the usage of the poll() method which can be used to check the completion of the child process.

Important Methods of Popen Instance¶

• wait(timeout=None) - This method makes the process wait for the child process to complete. It accepts a parameter named timeout which accepts a number as input. If the child process is not completed within that time frame, it'll raise TimeoutExpired error. It works exactly like timeout parameter of run() method.
• poll() - This method checks whether the child process has completed and sets and returns the return code. If the child process is not complete then it returns None as return code.

Important Attributes of Popen Instance¶

• pid - It returns an integer specifying process ID.
• return_code - It returns return code of the process completion. The return code of 0 means successful process completion else failure.
• stdout - It returns the standard output of the process.
• stderr - It returns a standard error that occurred during the process.

11.1 Wait Forever¶

Our code for this part is almost exactly the same as our previous example with few minor changes. After creating an instance of Popen to run addition.py Python script. We are first calling poll() method to check whether the child process is terminated. We are then calling wait() method asking our main process to wait for the child process to complete. We are then again calling poll() to check for the status of the child process. We have called the wait method without timeout hence it'll wait forever for the child process to complete.

We can notice from the output that the initial return code is None. After the child process completes, we are checking for the return code again and it’s set to 1 after completion (child process had a failure.). When we run the script we can notice from the output that it waits for the child process to complete.

11.2 Timeout After Waiting for 2 Seconds¶

Our code for this part is exactly the same as our previous example with 2 minor changes. The first change is that we have introduced a timeout of 2 seconds inside of the wait() method call. Our second change is that we have wrapped the code inside of the try-except block to capture errors. We can notice from the output that the child process terminates with a time-out error.

Example 12: Communicate with Process¶

As a part of our twelfth example, we'll explain how we can communicate with the child process (send inputs and receive results) using communicate() method of the Popen instance.

Our code for this part simply creates a Popen instance to run addition.py python script. We are then calling communicate() method to collect the standard output and standard error data. We are then printing data collected by the communicate() method. In this example, we have collected standard output and error data from communicate() method whereas, in previous examples, we had collected them from Popen instance.

Important Methods of Popen Instance¶

• communicate(input=None, timeout=None) - This method waits for the process to terminate. Till the process, if terminated, it collects data from its standard output and error streams. It returns a tuple of standard output and error data. We can provide data to the method using input parameter. We can set timeout parameter to time out process after a specified number of seconds.

Example 13: Communicate with the Process Continued ...¶

We are using our thirteenth example to demonstrate how we can pass data to the child process using communicate() method.

13.1: String Input¶

Our code for this part creates an instance of Popen which would run addition_user_input.py Python script as a child process. As part of this call, We have set stdin as well part from stdout and stderr. We have then called communicate() method giving it an input string that will be passed to the script. We have passed text parameter as True to indicate that input should be in string format else input is expected as bytes.

We can notice from the output that it seems to have correctly captured input data.

13.2: Bytes Input¶

Our code for this example is exactly the same as our previous example with few minor changes. We have not set text parameter of the Popen() constructor hence input data is expected as bytes. We have then given input data through communicate() method in bytes format.

When we run this part of the code, it has the same output as the previous example even though input data was given in binary format.

Example 14: Killing the Process¶

As a part of our fourteenth example, we'll explain how we can kill the child process in three different ways by using three different methods of the Popen instance.

Important Methods of Popen Instance¶

• kill() - It kills the child process.
• terminate() - It terminates the process.
• send_signal(signal) - It sends signal given to it to the process.

14.1: Using kill() Method¶

Our code for this example is exactly the same as the 11th example with the addition of few lines of code. We are again running addition.py script which takes 6 seconds to complete. We are asking the main process to sleep for 4 seconds and after that, we are checking whether the child process has completed or not using poll() method. If it's still not completed after 4 seconds then, we kill it using kill() method.

We can notice from the output that after we have killed the process there is no standard output and error data.

14.2: Using terminate() Method¶

Our code for this example is exactly the same as the previous code but with the only difference that we are using, terminate() method to kill the process.

14.3: Using send_signal() Method¶

Our code for this example is exactly the same as our previous two examples with the only change that we are killing the process by using send_signal() method. We are sending a method signal SIGTERM which will kill it.

This ends our small tutorial explaining the API of subprocess module which can be used to launch a program installed on the system as a child process from Python. Please feel free to let us know your views in the comments section.