Python subprocess
last modified May 25, 2026
In this article, we show how to use the subprocess module to run external commands and manage processes in Python.
The subprocess module is Python's modern tool for launching external commands and interacting with them programmatically. It gives you full control over a child process's input, output, and error streams, and lets you retrieve detailed information such as exit codes and captured output. This makes it the preferred way to run shell commands from Python, whether you need simple command execution or complex pipelines.
As part of the standard library, subprocess requires no installation and supersedes older approaches like os.system and os.spawn*. It provides a safer, more consistent API that works across platforms and supports advanced features such as piping, redirection, and asynchronous process handling.
Main features of the subprocess module:
| Feature | Description | Key API / Argument |
|---|---|---|
| Run external commands | Execute a program and wait for it to complete. This is the recommended approach for most use cases. | subprocess.run() |
| Capture output | Capture standard output (stdout) and standard error (stderr) to use the data within Python. | capture_output=Truestdout=subprocess.PIPE |
| Text / String processing | Automatically decode output from raw bytes into readable strings. | text=Trueencoding="utf-8" |
| Strict error handling | Automatically raise a CalledProcessError if the command fails (returns a non‑zero exit code). |
check=True |
| Inspect results | Check the success, failure, or captured output of a finished command via the CompletedProcess object. |
result.returncoderesult.stdout |
| Asynchronous / Non-blocking | Start a process in the background without waiting for it to finish immediately. | subprocess.Popen() |
| Interact with processes | Send dynamic data to a running process's stdin and read its output safely to avoid deadlocks. | Popen.communicate(input=...) |
| Create pipelines | Chain multiple commands together (similar to the shell | operator) by connecting their streams. |
Popen(..., stdin=p1.stdout) |
Running a simple command
The subprocess.run function is the recommended way to execute
external commands. The following example runs the system ls command
to list files in the current directory.
import subprocess
result = subprocess.run(["ls", "-l"])
print(f"Return code: {result.returncode}")
The command and its arguments are passed as a list of strings. The
run function waits for the command to complete and returns
a CompletedProcess instance.
import subprocess
We import the subprocess module.
result = subprocess.run(["ls", "-l"])
We execute the ls -l command. The command line arguments are
specified as a list, where the first element is the program name and the
following elements are its arguments.
print(f"Return code: {result.returncode}")
We print the return code of the completed process. A zero means success.
$ python main.py total 12 -rw-rw-r-- 1 jano jano 202 May 25 12:00 main.py -rw-rw-r-- 1 jano jano 182 May 25 12:00 data.txt Return code: 0
The output of the command is printed directly to the terminal. If we want to
capture the output inside Python, we must enable
capture_output=True or redirect stdout to a pipe.
Capturing command output
By default, subprocess.run prints the output directly to the
console. To capture the output for further processing, we set
capture_output=True and text=True.
import subprocess
result = subprocess.run(["echo", "Hello from subprocess"],
capture_output=True, text=True)
print(f"stdout: {result.stdout.strip()}")
print(f"stderr: {result.stderr}")
print(f"Return code: {result.returncode}")
The capture_output=True parameter redirects stdout and stderr
into the result.stdout and result.stderr attributes.
The text=True parameter returns the output as a string instead
of bytes.
result = subprocess.run(["echo", "Hello from subprocess"],
capture_output=True, text=True)
We run the echo command and capture its output. With
capture_output=True, the output is stored in the result object.
With text=True, the output is decoded to a string.
print(f"stdout: {result.stdout.strip()}")
print(f"stderr: {result.stderr}")
We access the standard output and standard error of the process. The
strip method removes the trailing newline.
$ python main.py stdout: Hello from subprocess stderr: Return code: 0
Text output vs binary output
The text=True parameter returns the output as a string. Without it,
the output is returned as bytes. This allows you to choose the appropriate
format based on your needs.
import subprocess
cmd = ["printf", "line1\nline2\nline3\n"]
# Default: output is returned as bytes
result_bytes = subprocess.run(cmd, capture_output=True)
print("Bytes mode:")
print(" type:", type(result_bytes.stdout))
print(" value:", result_bytes.stdout)
# Text mode: output is decoded into a Python string
result_text = subprocess.run(cmd, capture_output=True, text=True)
print("\nText mode:")
print(" type:", type(result_text.stdout))
print(" value:", result_text.stdout)
In bytes mode, the output is returned exactly as the underlying process produced it, including escape sequences and raw byte values. In text mode, Python automatically decodes the output using the system's default encoding (usually UTF-8) and returns a normal string. This makes it easier to work with line-based text, while bytes mode is useful when dealing with binary data or custom encodings.
In the first part of the example, the output is a bytes object, which is why it
is prefixed with b'' and contains escape sequences such as
\n. In the second part, text=True tells Python to
decode the output into a regular string, so the newlines are interpreted and
printed normally. Choosing between bytes and text depends on the task: text mode
is convenient for parsing command output, while bytes mode is required for
binary data or when you want to handle decoding manually.
Processing line-based output
Many external commands produce line-oriented text. With text=True,
the output is returned as a normal Python string, which makes it easy to split
into lines and process further. This is convenient when working with tools such
as ls, ps, or grep.
import subprocess
# Run a command and capture its output as text
result = subprocess.run(["ls"], capture_output=True, text=True)
print("Files in the current directory:")
for line in result.stdout.splitlines():
print(" -", line)
The program runs ls and prints each file on a separate line. Because
we enabled text=True, the output is already decoded into a string,
so we can use standard string methods such as splitlines().
result = subprocess.run(["ls"], capture_output=True, text=True)
The text=True parameter tells Python to decode the command's output
into a Unicode string. Without it, the output would be returned as raw bytes.
for line in result.stdout.splitlines():
print(" -", line)
We iterate over the lines of the output and print them. This pattern is common when processing the output of text-based command-line tools.
$ python main.py Files in the current directory: - main.py - data.txt - script.sh - notes.md
Checking for errors
When a command fails, it returns a non-zero exit code. The
check=True parameter makes run raise a
CalledProcessError if the command exits with a non-zero status.
import subprocess
try:
subprocess.run(["ls", "/nonexistent"], check=True, capture_output=True, text=True)
except subprocess.CalledProcessError as e:
print(f"Command failed with exit code {e.returncode}")
print(f"stderr: {e.stderr.strip()}")
The program attempts to list a non-existent directory and catches the resulting error.
try:
subprocess.run(["ls", "/nonexistent"], check=True, capture_output=True, text=True)
except subprocess.CalledProcessError as e:
We wrap the call in a try/except block. With check=True,
a non-zero exit code raises CalledProcessError.
print(f"Command failed with exit code {e.returncode}")
print(f"stderr: {e.stderr.strip()}")
The exception object provides the return code and the captured stderr output.
$ python main.py Command failed with exit code 2 stderr: ls: cannot access '/nonexistent': No such file or directory
Passing input to a process
The input parameter sends data to the process's standard input.
This is useful for feeding data to commands that read from stdin.
import subprocess
data = "apple\nbanana\ncherry\napple\ndate\n"
result = subprocess.run(["sort", "-u"],
input=data, capture_output=True, text=True)
print("Sorted unique lines:")
print(result.stdout)
We pass a multi-line string to the sort -u command, which sorts
the lines and removes duplicates.
data = "apple\nbanana\ncherry\napple\ndate\n"
We prepare a string with several fruit names, one per line. Note that
apple appears twice.
result = subprocess.run(["sort", "-u"],
input=data, capture_output=True, text=True)
The input parameter feeds the string to the process's stdin.
The -u flag tells sort to output only unique lines.
$ python main.py Sorted unique lines: apple banana cherry date
Running a command in a different directory
The cwd parameter specifies the working directory for the
executed command.
import subprocess
import os
current = os.getcwd()
print(f"Current directory: {current}")
result = subprocess.run(["pwd"], capture_output=True, text=True,
cwd="/tmp")
print(f"pwd from /tmp: {result.stdout.strip()}")
result = subprocess.run(["ls", "-d", "/etc/*.conf"], capture_output=True, text=True,
cwd="/tmp")
print("\nConfiguration files in /etc:")
print(result.stdout)
The cwd parameter changes the working directory before the
command executes. The current directory of the Python script itself remains
unchanged.
result = subprocess.run(["pwd"], capture_output=True, text=True,
cwd="/tmp")
We run the pwd command with the working directory set to
/tmp. The command prints /tmp even though the
Python script runs from a different location.
$ python main.py Current directory: /home/jano/Documents/zetcode-remote/python/subprocess pwd from /tmp: /tmp Configuration files in /etc: /etc/adduser.conf /etc/ca-certificates.conf /etc/deluser.conf /etc/host.conf ...
Setting environment variables
The env parameter sets environment variables for the child
process. When provided, it completely replaces the current environment;
to extend it, pass a modified copy of os.environ.
import subprocess
import os
custom_env = os.environ.copy()
custom_env["APP_MODE"] = "production"
custom_env["DEBUG"] = "false"
result = subprocess.run(["bash", "-c", "echo Mode: $APP_MODE; echo Debug: $DEBUG"],
capture_output=True, text=True, env=custom_env)
print(result.stdout)
We create a custom environment by copying the current one and adding our own variables. The child process then uses this modified environment.
custom_env = os.environ.copy() custom_env["APP_MODE"] = "production" custom_env["DEBUG"] = "false"
We copy the current process environment with os.environ.copy()
and add two custom variables.
result = subprocess.run(["bash", "-c", "echo Mode: $APP_MODE; echo Debug: $DEBUG"],
capture_output=True, text=True, env=custom_env)
We run a bash inline script that prints the environment variables. The
env parameter passes our custom environment to the child process.
$ python main.py Mode: production Debug: false
Setting a timeout
The timeout parameter limits the execution time of a command.
If the command does not complete within the specified number of seconds,
a TimeoutExpired exception is raised.
import subprocess
import sys
try:
# This command sleeps for 10 seconds, but we only wait 3
result = subprocess.run(
["sleep", "10"],
capture_output=True, text=True, timeout=3
)
except subprocess.TimeoutExpired as e:
print(f"Command timed out after {e.timeout} seconds")
print(f"Command: {' '.join(e.cmd)}")
The program runs sleep 10 but sets a timeout of 3 seconds.
The command is killed when the timeout expires.
result = subprocess.run(
["sleep", "10"],
capture_output=True, text=True, timeout=3
)
We run the sleep command with a 3-second timeout. Since the
command needs 10 seconds to complete, it will be terminated early.
except subprocess.TimeoutExpired as e:
print(f"Command timed out after {e.timeout} seconds")
The TimeoutExpired exception has a timeout attribute
indicating how many seconds were allowed.
$ python main.py Command timed out after 3 seconds Command: sleep 10
Using shell mode
The shell=True parameter executes the command through the system
shell. This allows shell features like wildcard expansion, piping, and
environment variable expansion.
import subprocess
# Count Python files using shell pipeline and wildcards
cmd = "ls -1 *.py 2>/dev/null | wc -l"
result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
print(f"Number of Python files: {result.stdout.strip()}")
When using shell=True, the command is passed as a single string.
This mode is convenient but should be used carefully with user-supplied input
to avoid shell injection vulnerabilities.
cmd = "ls -1 *.py 2>/dev/null | wc -l"
We define a shell command that uses a wildcard (*.py) and a
pipe (|). These shell features are only available with
shell=True.
result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
The shell=True parameter tells subprocess to execute the command
through /bin/sh. The command is provided as a single string.
$ python main.py Number of Python files: 2
Using Popen for more control
The Popen class provides more control over process execution.
Unlike run, Popen does not wait for the process to
complete, allowing you to interact with it while it runs.
import subprocess
proc = subprocess.Popen(["ping", "-c", "3", "localhost"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True)
print("Process is running...")
stdout, stderr = proc.communicate()
print(f"Exit code: {proc.returncode}")
print("\nOutput:")
print(stdout)
Popen starts the process and returns immediately. The
communicate method then reads all output and waits for the
process to finish.
proc = subprocess.Popen(["ping", "-c", "3", "localhost"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True)
We create a Popen instance to run the ping command.
stdout=PIPE and stderr=PIPE redirect the output
streams so we can capture them.
stdout, stderr = proc.communicate()
The communicate method reads all output from the process and
waits for it to finish. It returns a tuple of (stdout, stderr).
$ python main.py Process is running... Exit code: 0 Output: PING localhost (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.042 ms 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.062 ms 64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.065 ms --- localhost ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2035ms rtt min/avg/max/mdev = 0.042/0.056/0.065/0.010 ms
Reading output line by line
With Popen, we can read output line by line as the process runs,
which is useful for long-running commands where we want to show progress.
import subprocess
proc = subprocess.Popen(["ping", "-c", "4", "localhost"],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True)
for line in proc.stdout:
print(f">> {line}", end="")
proc.wait()
print(f"\nProcess finished with exit code: {proc.returncode}")
We iterate over proc.stdout to read lines as they become
available. The wait method ensures the process has completed
before we check the return code.
proc = subprocess.Popen(["ping", "-c", "4", "localhost"],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True)
We redirect stderr to stdout using stderr=subprocess.STDOUT,
so all output is available through a single stream. The text=True
parameter returns strings rather than bytes.
for line in proc.stdout:
print(f">> {line}", end="")
Iterating over proc.stdout yields lines as the process produces
them. Each line is prefixed with >> for demonstration.
Piping between processes
We can pipe the output of one process directly into another using
Popen. This replicates shell-style piping but without
invoking the shell.
Most shell pipelines hide the complexity of how processes exchange data. When we
recreate a pipeline in Python, we must explicitly manage the file descriptors
that connect the processes. Each Popen call can create a pipe, which is simply a
unidirectional communication channel with a read end and a write end. By wiring
the write end of one process to the read end of another, we reproduce the
behavior of a shell pipeline such as ps aux | grep python. This
approach mimics the shell's behavior of chaining commands with pipes, allowing
us to replicate the same functionality in Python, but with full control
from Python.
import subprocess
ps = subprocess.Popen(["ps", "aux"], stdout=subprocess.PIPE, text=True)
grep = subprocess.Popen(
["grep", "[p]ython"],
stdin=ps.stdout,
stdout=subprocess.PIPE,
text=True,
)
ps.stdout.close()
output, _ = grep.communicate()
ps.wait()
if ps.returncode != 0:
raise subprocess.CalledProcessError(ps.returncode, "ps")
if grep.returncode not in (0, 1):
raise subprocess.CalledProcessError(grep.returncode, "grep")
print("Python processes:")
print(output)
When a Popen object is created, the operating system immediately
starts the child process. There is no additional trigger; both ps
and grep begin running as soon as they are constructed. The
pipeline therefore becomes active before we call communicate(). The
communicate() method only waits for the process to finish and
collects its output; it does not initiate execution.
ps = subprocess.Popen(["ps", "aux"], stdout=subprocess.PIPE, text=True)
grep = subprocess.Popen(
["grep", "[p]ython"],
stdin=ps.stdout,
stdout=subprocess.PIPE,
text=True,
)
We create two processes. The ps process's stdout is connected to
grep's stdin. This creates a direct pipe between them. The
grep process's stdout also piped back to the parent process,
allowing Python to capture and process the filtered results.
The grep process has its standard output redirected to a pipe so
that Python can capture the filtered results. Without
stdout=subprocess.PIPE, the output would go directly to the
terminal, and the parent process would have no access to it. Redirecting
grep's output ensures that the final stage of the pipeline is
available to Python for further processing or display.
ps.stdout.close()
After connecting ps.stdout to grep.stdin, the parent
process still holds an open file descriptor for the read end of the pipe. If
this descriptor remains open, grep will never receive an
end-of-file signal, because the kernel sees that at least one reader still
exists. Closing the parent's copy ensures that once ps
finishes writing, the pipe is fully closed and grep can terminate
normally. This mirrors how the shell closes unused pipe ends when constructing
pipelines.
ps.wait()
The wait() method ensures the parent process waits for the
ps process to complete.
if ps.returncode != 0:
raise subprocess.CalledProcessError(ps.returncode, "ps")
We check the return code of the ps process. If it's non-zero,
we raise an exception.
if grep.returncode not in (0, 1):
raise subprocess.CalledProcessError(grep.returncode, "grep")
We check the return code of the grep process. A return code of
0 indicates a match was found, 1 indicates no match, and any other value
indicates an error.
Suppressing output with DEVNULL
Sometimes we want to discard a process's output entirely.
subprocess.DEVNULL redirects streams to the system's null device.
import subprocess
# Run a command and discard all output
result = subprocess.run(["ping", "-c", "2", "localhost"],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL)
if result.returncode == 0:
print("Ping successful (output discarded)")
else:
print("Ping failed")
Both stdout and stderr are redirected to /dev/null, so nothing
is printed to the console. We only check the return code to determine success.
$ python main.py Ping successful (output discarded)
Running a Python script as a subprocess
The subprocess module can run other Python scripts as child processes. This is useful for parallel execution or isolating tasks.
import sys
import time
name = sys.argv[1]
seconds = int(sys.argv[2])
print(f"{name}: sleeping for {seconds} seconds")
time.sleep(seconds)
print(f"{name}: done")
sys.exit(0)
compute.py is a simple script that takes a name and a number of
seconds to sleep. It prints messages before and after sleeping.
import subprocess
# Run the same Python script with different arguments
result = subprocess.run(
["python3", "compute.py", "task1", "2"],
capture_output=True, text=True
)
print("Task 1 output:")
print(result.stdout)
result = subprocess.run(
["python3", "compute.py", "task2", "1"],
capture_output=True, text=True
)
print("Task 2 output:")
print(result.stdout)
print("All tasks completed")
The main script runs compute.py twice with different arguments.
Each invocation waits for the child process to finish before proceeding.
$ python main.py Task 1 output: task1: sleeping for 2 seconds task1: done Task 2 output: task2: sleeping for 1 seconds task2: done All tasks completed
Getting system information
The following example demonstrates a real-world use case: gathering system information by running multiple shell commands.
import subprocess
def run_cmd(cmd):
"""Run a command and return its stdout, or an error message."""
try:
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
return result.stdout.strip()
except subprocess.CalledProcessError:
return "N/A"
except FileNotFoundError:
return "Command not found"
except Exception as e:
return f"Error: {e}"
info = {
"Hostname": run_cmd(["hostname"]),
"Kernel": run_cmd(["uname", "-r"]),
"Uptime": run_cmd(["uptime", "-p"]),
"Memory": run_cmd(["free", "-h"]),
}
for key, value in info.items():
print(f"{key}:")
print(f" {value}")
print()
A helper function run_cmd safely executes commands and captures
their output. The results are collected into a dictionary and printed in a
structured format.
Asynchronous execution with Popen
Unlike subprocess.run(), which blocks your script until the command
finishes, subprocess.Popen is asynchronous. It starts the process
in the background and returns immediately. This allows you to launch multiple
processes concurrently or continue doing work in your main Python script while
the external commands run.
import subprocess
import sys
import time
print("Starting background tasks...")
# Define cross-platform commands that simulate long-running tasks
cmd1 = [sys.executable, "-c", "import time; time.sleep(2)"]
cmd2 = [sys.executable, "-c", "import time; time.sleep(3)"]
# 1. Start processes asynchronously. Popen does not block.
process1 = subprocess.Popen(cmd1)
process2 = subprocess.Popen(cmd2)
print("Tasks launched. Python is free to do other work!")
# 2. The main thread continues running while subprocesses execute
for i in range(3):
print(f"Main thread working... (second {i+1})")
time.sleep(1)
# 3. Synchronize: Wait for both background processes to complete
code1 = process1.wait()
code2 = process2.wait()
print(f"Process 1 exited with code {code1}")
print(f"Process 2 exited with code {code2}")
Because we used Popen, process1 and
process2 execute at the exact same time. The main Python script
also continues running its own for loop. Finally,
wait() is used to pause the script at the very end until the
background processes wrap up, preventing them from becoming zombie processes.
$ python main.py Starting background tasks... Tasks launched. Python is free to do other work! Main thread working... (second 1) Main thread working... (second 2) Main thread working... (second 3) Process 1 exited with code 0 Process 2 exited with code 0
Downloading large files asynchronously (with wget)
When fetching large files, such as an operating system image, blocking the main
thread means your application freezes until the download finishes. By using
subprocess.Popen, you can start the download in the background and
use process.poll() to periodically check its status, allowing you
to update a UI or perform other tasks.
import subprocess
import time
import sys
# The URL for the FreeBSD 15.0 mini-memstick compressed image
url = "https://download.freebsd.org/releases/amd64/amd64/ISO-IMAGES/15.0/FreeBSD-15.0-RELEASE-amd64-mini-memstick.img.xz"
output_file = "FreeBSD-mini-memstick.img.xz"
# We use wget with -O to specify the output file and -q for quiet mode
cmd = ["wget", "-q", "-O", output_file, url]
print(f"Starting background download of {output_file}...")
print("This may take a minute depending on your connection.\n")
# Start the download process asynchronously
process = subprocess.Popen(cmd)
# Monitor the process while it is running
seconds_elapsed = 0
while process.poll() is None:
# Print a status update to the console on the same line
sys.stdout.write(f"\rDownloading... [{seconds_elapsed} seconds elapsed]")
sys.stdout.flush()
time.sleep(1)
seconds_elapsed += 1
print("\n")
# Once process.poll() is no longer None, the download is finished
if process.returncode == 0:
print("Download completed successfully!")
else:
print(f"Download failed with exit code {process.returncode}")
Because we use sys.stdout.write with a carriage return
(\r), the main thread continuously overwrites the same line in the
terminal. This provides a live "seconds elapsed" counter while the operating
system efficiently handles the heavy lifting of the network download in the
background using wget.
while process.poll() is None:
The while loop continuously checks if the process has finished by
calling process.poll(). If the process is still running,
poll() returns None, and the loop continues. Once the
process completes, poll() returns the exit code, and the loop
exits.
$ python main.py Starting background download of FreeBSD-mini-memstick.img.xz... This may take a minute depending on your connection. Downloading... [126 seconds elapsed] Download completed successfully!
Source
Python subprocess documentation
In this article, we have worked with the Python subprocess module to run external commands, manage process input and output, handle errors, set timeouts, and pipe data between processes.
Author
List all Python tutorials.