Share @ LinkedIn Facebook  python, subprocess, unixprocesses

Overview

subprocess module let developer spawn new process from the existing process. It lets the developer connect to input/output/error streams of newly spawned processes and also collects return codes.

Install subprocess

subprocess module comes with python itself and does not need installation.

Simple Way To Start A Subprocess

Currently, the standard way to start a subprocess is to call the run() function of the subprocess module. It was introduced in python v3.5. Prior to version 3.5, it's equivalent function call() was used. It was introduced to create the same API like that of threading and multiprocessing modules. call() function still exists in newer versions for backward compatibility.

run() function takes list of arguments and then runs them as one subprocess. It returns CompletedProcess object when it completes running command. Developer can execute any command which it generally executes in shell/command prompt.

In [1]:
import subprocess
%cd ~/Desktop/
/home/sunny/Desktop
In [2]:
out = subprocess.run(['ls', '-l'])
out, out.stdout, out.stderr
Out[2]:
(CompletedProcess(args=['ls', '-l'], returncode=0), None, None)

We can notice from the above example that run() function returns CompletedProcess object once a command is completed running. We can also see that it just ran command but did not capture output which we need in this case.

In [3]:
out = subprocess.run(["ls", "-l"], stdout=subprocess.PIPE,stderr=subprocess.PIPE)
out
Out[3]:
CompletedProcess(args=['ls', '-l'], returncode=0, stdout=b'total 16\ndrwxrwxr-x 10 sunny sunny 4096 Feb 10 12:14 blogs\ndrwxrwxr-x  8 sunny sunny 4096 Feb 14 21:02 Python\ndrwxrwxr-x  2 sunny sunny 4096 Jan 16 11:49 SMTP\ndrwxrwxr-x 10 sunny sunny 4096 Jan 23 08:04 tutorials\n', stderr=b'')
In [4]:
out.stderr, out.stdout
Out[4]:
(b'',
 b'total 16\ndrwxrwxr-x 10 sunny sunny 4096 Feb 10 12:14 blogs\ndrwxrwxr-x  8 sunny sunny 4096 Feb 14 21:02 Python\ndrwxrwxr-x  2 sunny sunny 4096 Jan 16 11:49 SMTP\ndrwxrwxr-x 10 sunny sunny 4096 Jan 23 08:04 tutorials\n')

As one can see from the above CompletedProcess output that now commands output and errors occurred during execution are captured as part of stdout and stderr attributes of CompletedProcess object. They are captured as byte streams and the developer will need to convert it to string to see it in proper format as below.

In [5]:
print(out.stdout.decode())
total 16
drwxrwxr-x 10 sunny sunny 4096 Feb 10 12:14 blogs
drwxrwxr-x  8 sunny sunny 4096 Feb 14 21:02 Python
drwxrwxr-x  2 sunny sunny 4096 Jan 16 11:49 SMTP
drwxrwxr-x 10 sunny sunny 4096 Jan 23 08:04 tutorials

CompletedProcess object has other attributes as well like returncode and args. returncode will be generally 0 if the command completed successfully else it'll be non-zero. args will be holding command passed as a list to run() method.

In [6]:
out = subprocess.run(['ls', '-z'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print('Return Code : %d'%out.returncode)
print()
print(out.stderr.decode())
Return Code : 2

ls: invalid option -- 'z'
Try 'ls --help' for more information.

CompletedProcess also has method named check_output() which checks returncode and raises CalledProcessError object.CalledProcessError is subclass of SubprocessError and holds error information during execution of command.

In [7]:
out = subprocess.run(['ls', '-z'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out.check_returncode()
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-7-715cd9b76e23> in <module>()
      1 out = subprocess.run(['ls', '-z'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
----> 2 out.check_returncode()

~/anaconda2/envs/p3/lib/python3.6/subprocess.py in check_returncode(self)
    367         if self.returncode:
    368             raise CalledProcessError(self.returncode, self.args, self.stdout,
--> 369                                      self.stderr)
    370
    371

CalledProcessError: Command '['ls', '-z']' returned non-zero exit status 2.
In [8]:
subprocess.run(['python', '--version'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Out[8]:
CompletedProcess(args=['python', '--version'], returncode=0, stdout=b'', stderr=b'Python 3.6.5 :: Anaconda, Inc.\n')

shell argument to run() function lets run command as a shell command. This lets the developer execute commands with pipelines and wildcards. It also lets developers use ~ for referring to the home directory.

In [9]:
out = subprocess.run(['ls', '-l', '|','grep', '*.ipynb'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
print(out.stdout.decode())
blogs
Python
SMTP
tutorials

Popen Class

Popen class is responsible for creation of new sub processes in this module.


Sunny Solanki  Sunny Solanki