Shell Out Safely: A Practical Deep Dive into Python's subprocess

Sooner or later every Python script needs to reach outside itself: run git, call ffmpeg, shell out to a system tool, or glue together a small pipeline. The temptation is to grab os.system() and move on. Resist it. The subprocess module is the modern, secure, and far more capable way to launch and communicate with external programs, and once you understand a handful of patterns it becomes one of the most reliable tools in your kit.

This guide walks through the practical core of subprocess: running a command, capturing its output, handling failures, feeding it input, setting timeouts, and the one security rule that matters most. Every example below is runnable on Python 3.7+.

Start with subprocess.run

Since Python 3.5, subprocess.run() is the one function you should reach for by default. It launches a command, waits for it to finish, and hands you back a CompletedProcess object describing what happened.

import subprocess

result = subprocess.run(
    ["echo", "hello world"],
    capture_output=True,
    text=True,
)

print(result.returncode)  # 0
print(repr(result.stdout))  # 'hello world\n'

Three things are worth calling out. First, the command is a list of arguments, not a single string. Second, capture_output=True collects stdout and stderr instead of letting them stream to your terminal. Third, text=True decodes the output as a string; without it you get raw bytes. Getting into the habit of passing text=True saves a lot of .decode() noise.

The list-of-arguments rule (and why the shell is dangerous)

The single most important habit with subprocess is to pass arguments as a list and leave shell=False (the default). When you pass a list, each element is handed to the program exactly as written, with no shell in between to reinterpret spaces, quotes, or special characters.

Compare that with the shell-based form, which is where security incidents are born:

filename = "report.txt; rm -rf ~"

# DANGEROUS: the shell interprets the semicolon as a command separator
subprocess.run(f"cat {filename}", shell=True)

# SAFE: the filename is a single argument, semicolon and all
subprocess.run(["cat", filename])

With shell=True, that crafted filename runs a second, destructive command. With the list form there is no shell to trick — cat simply receives a weird filename and reports that it does not exist. Only use shell=True when you genuinely need shell features like glob expansion or pipes, and never with input you did not construct yourself.

Checking for failure

By default, run() does not raise an exception when a command exits with a non-zero status. That is a common surprise: your script keeps going even though the command failed. You have two choices. Inspect returncode yourself, or pass check=True to have a failure raise CalledProcessError.

try:
    subprocess.run(
        ["python3", "-c", "import sys; sys.exit(3)"],
        check=True,
    )
except subprocess.CalledProcessError as e:
    print(f"Command failed with code {e.returncode}")
    # e.stdout and e.stderr are available too if you captured them

For scripts and automation, check=True is almost always what you want: fail loudly and early rather than silently continuing on bad data.

Capturing stdout and stderr separately

When you capture output, stdout and stderr land in separate attributes so you can tell normal results from diagnostics.

import sys

result = subprocess.run(
    [sys.executable, "-c",
     "import sys; print('the answer'); print('a warning', file=sys.stderr)"],
    capture_output=True,
    text=True,
)

print(repr(result.stdout))  # 'the answer\n'
print(repr(result.stderr))  # 'a warning\n'

Using sys.executable instead of the literal string "python3" guarantees you launch the same interpreter that is running your script — a small detail that prevents confusing bugs inside virtual environments.

If you would rather have stderr merged into stdout (handy when a tool interleaves the two), redirect it explicitly:

result = subprocess.run(
    ["some-tool"],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    text=True,
)

Sending input to a command

Many command-line tools read from standard input. The input parameter feeds a string (or bytes) straight to the process and closes the stream for you.

result = subprocess.run(
    [sys.executable, "-c", "import sys; print(sys.stdin.read().strip().upper())"],
    input="quiet please",
    capture_output=True,
    text=True,
)

print(repr(result.stdout))  # 'QUIET PLEASE\n'

This is the clean way to pass data to tools like grep, jq, or a formatter without writing a temporary file.

Guarding against hangs with timeouts

An external command can hang forever — waiting on a lock, a network socket, or user input that never comes. The timeout parameter puts a ceiling on how long you will wait. If the command overruns, the child is killed and TimeoutExpired is raised.

try:
    subprocess.run(
        [sys.executable, "-c", "import time; time.sleep(5)"],
        timeout=1,
    )
except subprocess.TimeoutExpired:
    print("Command took too long and was terminated")

In any long-running or unattended script, a timeout on external calls is cheap insurance against a process that quietly stalls the whole job.

Convenience wrapper: check_output

When all you want is a command's stdout and you want an error to raise automatically, check_output() is a compact shortcut. It behaves like run(..., check=True, stdout=PIPE) and returns the captured output directly.

version = subprocess.check_output(
    ["python3", "--version"],
    text=True,
).strip()

print(version)  # e.g. 'Python 3.12.3'

Pipelines: when you need Popen

run() covers the vast majority of cases, but it always waits for the command to finish. When you need to wire the output of one process into the input of another — the equivalent of a shell pipe — drop down to Popen, which starts a process without blocking.

p1 = subprocess.Popen(["printf", "apple\nbanana\napple\n"],
                      stdout=subprocess.PIPE)
p2 = subprocess.Popen(["sort"], stdin=p1.stdout,
                      stdout=subprocess.PIPE, text=True)

p1.stdout.close()  # let p1 get a SIGPIPE if p2 exits
output, _ = p2.communicate()
print(output)  # apple\napple\nbanana

That said, before building a shell-style pipeline, ask whether Python can do the middle steps itself. Reaching for sort, grep, and wc through subprocesses is often slower and more fragile than a few lines of pure Python:

import collections

raw = subprocess.run(
    ["printf", "apple\nbanana\napple\n"],
    capture_output=True, text=True,
).stdout

counts = collections.Counter(raw.split())
print(counts.most_common())  # [('apple', 2), ('banana', 1)]

Controlling the environment and working directory

Two keyword arguments handle most real-world configuration needs. cwd runs the command in a specific directory, and env replaces the child's environment variables. A common idiom is to copy the current environment and layer changes on top so you do not accidentally strip out PATH.

import os

result = subprocess.run(
    [sys.executable, "-c", "import os; print(os.environ.get('DEPLOY_ENV'))"],
    capture_output=True,
    text=True,
    cwd="/tmp",
    env={**os.environ, "DEPLOY_ENV": "staging"},
)

print(result.stdout.strip())  # staging

Common pitfalls

Forgetting text=True. Without it, stdout and stderr are bytes, and string comparisons silently fail. Add it unless you are handling binary data on purpose.

Assuming failure raises. A non-zero exit code is ignored unless you pass check=True or inspect returncode. Do not let a failed command masquerade as success.

Splitting command strings by hand. If you have a command as one string and need a list, use shlex.split() rather than .split(), which mishandles quoted arguments: shlex.split('grep "a b" file.txt') gives you the three correct pieces.

Reaching for shell=True out of habit. It is rarely necessary and it is the doorway to injection bugs. Prefer the argument list.

Wrap-up and next steps

The mental model is simple: use subprocess.run() with a list of arguments, add capture_output=True and text=True when you want the output, set check=True so failures are loud, and add a timeout so nothing hangs forever. Keep shell=False unless you have a concrete reason not to. Reserve Popen for the cases where you truly need concurrent processes or streaming pipelines.

From here, explore subprocess.DEVNULL for discarding output you do not care about, Popen.poll() for checking on a background process without blocking, and the shlex module for safely tokenizing command strings. Master these and calling out to the wider system stops being a source of bugs and becomes just another dependable part of your Python toolbox.