Running Commands the Right Way: A Practical Deep Dive into Python's subprocess

Sooner or later, almost every Python program needs to reach outside itself: run git, invoke ffmpeg, call a system tool, or shell out to a script written in another language. Python's answer is the subprocess module. It's powerful, but its API has accumulated layers over the years, and a lot of code in the wild still uses old, fragile patterns like os.system() or shell=True with string interpolation — patterns that are slower, harder to debug, and often a security hole.

This guide walks through the modern, idiomatic way to run external commands: starting with the high-level subprocess.run(), then dropping down to Popen when you need fine-grained control over pipes and streaming. Every example below was tested on Python 3.10, and the concepts apply to any 3.7+ interpreter.

Start with subprocess.run()

Since Python 3.5, subprocess.run() is the one function you should reach for first. It runs a command, waits for it to finish, and returns a CompletedProcess object describing what happened. The single most important rule: pass your command as a list of arguments, not a single string.

import subprocess

result = subprocess.run(
    ["echo", "hello world"],
    capture_output=True,   # collect stdout and stderr
    text=True,             # decode bytes to str using the default encoding
)

print(result.returncode)   # 0
print(repr(result.stdout)) # 'hello world\n'
print(repr(result.stderr)) # ''

Two flags do the heavy lifting here. capture_output=True redirects both streams into the result object instead of letting them print to your terminal. text=True (you may also see it written as encoding="utf-8") decodes the output into strings — without it, stdout and stderr come back as raw bytes, which is occasionally what you want but usually not.

The list form is not just a style preference. When you pass ["echo", "hello world"], Python hands those exact arguments to the operating system with no shell in between. There is no word-splitting, no glob expansion, and no chance of a filename with a space or a semicolon being reinterpreted as something dangerous.

Checking for failure

By default, run() does not raise an exception when the command fails — it just reports a non-zero returncode. If you want a failed command to blow up loudly (which is usually the right default for scripts), pass check=True.

import subprocess

try:
    subprocess.run(
        ["python3", "-c", "import sys; sys.exit(3)"],
        check=True,
    )
except subprocess.CalledProcessError as e:
    print("command failed with code:", e.returncode)  # 3

When check=True and the process exits non-zero, you get a CalledProcessError. If you also captured output, the exception carries e.stdout and e.stderr, so you can log exactly what the failing command printed before it died. This combination — check=True plus capture_output=True — is the backbone of reliable automation scripts.

Feeding input to a command

Many command-line tools read from standard input. The input argument lets you pipe a string (or bytes) straight into the process:

import subprocess

result = subprocess.run(
    ["grep", "b"],
    input="apple\nbanana\ncherry\n",
    capture_output=True,
    text=True,
)
print(repr(result.stdout))  # 'banana\n'

Note the symmetry: because we passed text=True, the input must be a str. If you drop text=True, pass bytes instead. Mixing the two raises a TypeError.

Timeouts: never hang forever

An external command can hang — a network call stalls, a tool waits for input that never comes. Always consider a timeout (in seconds) for anything that could block. When the limit is hit, the child is killed and TimeoutExpired is raised.

import subprocess

try:
    subprocess.run(["sleep", "5"], timeout=1)
except subprocess.TimeoutExpired:
    print("command took too long, aborted")

Where does the output go?

You have precise control over each stream. A few common patterns:

import subprocess

# Discard output entirely (like redirecting to /dev/null)
subprocess.run(["echo", "noisy"], stdout=subprocess.DEVNULL)

# Merge stderr into stdout so you capture both in order
result = subprocess.run(
    ["python3", "-c", "import sys; print('oops', file=sys.stderr)"],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,   # fold stderr into the stdout stream
    text=True,
)
print(repr(result.stdout))  # 'oops\n'

If you just want a command's output as a string and don't care about the rest, the older convenience wrapper subprocess.check_output() is still perfectly fine — it's essentially run(..., check=True, stdout=PIPE).stdout:

import subprocess

version = subprocess.check_output(["python3", "--version"], text=True)
print(version.strip())  # e.g. 'Python 3.10.12'

Pipes without the shell

A frequent reason people reach for shell=True is to build a pipeline like cat file | sort | uniq. But you often don't need a shell at all. Many tools accept the whole job in one invocation:

import subprocess

# Instead of `sort | uniq`, sort has a built-in unique flag
result = subprocess.run(
    ["sort", "-u"],
    input="1\n2\n3\n2\n1\n",
    capture_output=True,
    text=True,
)
print(repr(result.stdout))  # '1\n2\n3\n'

When you genuinely need to connect two separate processes, wire them together with Popen and hand one process's stdout to the next process's stdin:

import subprocess

p1 = subprocess.Popen(["printf", "1\n2\n3\n2\n1\n"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(
    ["sort"],
    stdin=p1.stdout,
    stdout=subprocess.PIPE,
    text=True,
)
p1.stdout.close()          # let p1 receive SIGPIPE if p2 exits early
output, _ = p2.communicate()
print(repr(output))        # '1\n1\n2\n2\n3\n'

The p1.stdout.close() line is easy to forget but matters: after handing the pipe to p2, the parent process should release its own copy so that p1 is correctly notified if p2 stops reading.

Streaming output line by line

For a long-running command you may want to process output as it arrives rather than waiting for the whole thing. Use Popen as a context manager and iterate over its stdout:

import subprocess

with subprocess.Popen(
    ["python3", "-c", "for i in range(3): print('line', i, flush=True)"],
    stdout=subprocess.PIPE,
    text=True,
) as proc:
    for line in proc.stdout:
        print("got:", line.strip())

Using Popen in a with block ensures the pipes are closed and the process is cleaned up even if an exception interrupts the loop.

Controlling the environment and working directory

Two keyword arguments cover most configuration needs. cwd sets the directory the command runs in, and env replaces the child's environment variables. A common idiom is to copy the current environment and add to it, rather than passing a bare dict (which would wipe out PATH and everything else):

import os
import subprocess

result = subprocess.run(
    ["printenv", "MYVAR"],
    capture_output=True,
    text=True,
    env={**os.environ, "MYVAR": "42"},
    # cwd="/path/to/run/in",
)
print(repr(result.stdout))  # '42\n'

The security question: avoid shell=True

You will see shell=True everywhere, and it is the single biggest source of subprocess bugs and vulnerabilities. When shell=True, your command string is handed to /bin/sh -c, which means any shell metacharacters — ;, |, &&, $(), backticks — are interpreted. If any part of that string comes from user input, you have handed the user a remote shell:

# DANGEROUS — never do this with untrusted input
filename = input("file to inspect: ")
subprocess.run(f"cat {filename}", shell=True)
# user types:  notes.txt; rm -rf ~

The list form is immune to this, because the arguments never touch a shell:

# SAFE — the filename is always treated as one literal argument
filename = input("file to inspect: ")
subprocess.run(["cat", filename])

If you have a command as a single string (say, from a config file you trust) and need to split it into an argument list, use shlex.split() instead of a naive str.split() — it respects quotes and escaping the way a shell would:

import shlex

args = shlex.split("ls -la '/some dir/with spaces'")
print(args)  # ['ls', '-la', '/some dir/with spaces']

The honest rule of thumb: reach for shell=True only when you specifically need shell features (globbing, pipes, variable expansion) and the entire command is a hard-coded, trusted constant. In every other case, use the argument list.

Common pitfalls

A few mistakes come up again and again. Passing a string like "echo hi" without shell=True fails with FileNotFoundError, because Python looks for a program literally named "echo hi". Forgetting text=True leaves you comparing bytes to str and wondering why nothing matches. Assuming a program is on PATH can bite you in cron jobs and containers where the environment is minimal — use a full path or an explicit env when in doubt. And capturing gigabytes of output into memory with capture_output=True can exhaust RAM; for huge streams, write to a file object via stdout= or stream line by line with Popen.

Wrap-up and next steps

The mental model is simple once it clicks. Use subprocess.run() with a list of arguments for the overwhelming majority of tasks; add capture_output=True and text=True to get strings back, check=True to fail loudly, and timeout= to avoid hangs. Drop down to Popen only when you need to stream output or connect processes into a pipeline. Keep commands as argument lists, treat shell=True as a last resort, and reach for shlex.split() when you must turn a string into arguments.

From here, worthwhile next steps include reading the CompletedProcess and Popen sections of the official docs, exploring proc.communicate() with a timeout for two-way interaction, and looking at higher-level libraries like plumbum or sh if you build a lot of command pipelines. But you can go a very long way with nothing more than run() and a list of strings.