Running Shell Commands Safely: A Practical Deep Dive into Python's subprocess

Sooner or later, every Python script needs to talk to the outside world: run git, call ffmpeg, invoke a compiler, or glue together a couple of command-line tools. The standard library's answer is the subprocess module — and it's excellent, once you know which parts to use. Unfortunately, a lot of code in the wild still uses os.system(), ignores exit codes, or reaches for shell=True without understanding the security implications.

In this guide we'll build up from the one function that covers 90% of use cases (subprocess.run()) to streaming output with Popen, wiring commands into pipelines, and handling the classic pitfalls: deadlocks, injection, and hung processes.

Forget os.system() — use subprocess.run()

os.system() gives you an exit code and nothing else: no captured output, no timeout, and it always goes through the shell. Since Python 3.5, subprocess.run() is the recommended high-level interface:

import subprocess

result = subprocess.run(["ls", "-l", "/tmp"])
print(result.returncode)  # 0 on success

Two things to notice. First, the command is a list of arguments, not a single string. Each element is passed to the operating system exactly as-is — no shell parsing, no globbing, no surprises with spaces in filenames. Second, run() returns a CompletedProcess object carrying the return code and (optionally) the output.

Capturing output

By default, the child process writes straight to your terminal. To capture stdout and stderr into Python strings, add capture_output=True and text=True:

result = subprocess.run(
    ["git", "status", "--short"],
    capture_output=True,
    text=True,
)

print("stdout:", result.stdout)
print("stderr:", result.stderr)

text=True decodes the raw bytes using the default encoding, so result.stdout is a str instead of bytes. If you need a specific encoding, pass encoding="utf-8" instead. Without either, you get raw bytes — which is exactly what you want when the output is binary (an image from ffmpeg, say).

Failing loudly: check=True

A silent failure is the worst kind. By default, run() does not raise when the command exits with a non-zero status — it just records the code. In scripts, you almost always want check=True:

try:
    subprocess.run(
        ["git", "clone", "https://example.com/does-not-exist.git"],
        check=True,
        capture_output=True,
        text=True,
    )
except subprocess.CalledProcessError as exc:
    print(f"Command failed with exit code {exc.returncode}")
    print(exc.stderr)

The raised CalledProcessError carries returncode, cmd, stdout, and stderr, so your error handling can log exactly what went wrong. If you'd rather decide later, call result.check_returncode() on the CompletedProcess.

Sending input to a process

Need to feed data to a command's stdin? Use the input parameter:

result = subprocess.run(
    ["grep", "error"],
    input="line one\nan error occurred\nline three\n",
    capture_output=True,
    text=True,
)
print(result.stdout)  # "an error occurred\n"

This is the safe replacement for the old pattern of writing to proc.stdin manually and hoping you don't deadlock.

Timeouts: never hang forever

External commands can stall — a network mount that's gone away, a tool waiting for input you never send. Always set a timeout for anything that could block:

try:
    subprocess.run(["sleep", "60"], timeout=5)
except subprocess.TimeoutExpired as exc:
    print(f"Killed after {exc.timeout} seconds")

When the timeout expires, run() kills the child process and raises TimeoutExpired. In production automation this single argument prevents a whole class of "the cron job has been stuck since Tuesday" incidents.

The shell=True trap

Passing shell=True runs your command through /bin/sh (or cmd.exe on Windows), which enables pipes, wildcards, and environment expansion — and also enables shell injection:

# DANGEROUS — never do this with untrusted input
filename = input("File to delete: ")  # user types: foo.txt; rm -rf ~
subprocess.run(f"rm {filename}", shell=True)

Because the string is handed to the shell, a malicious value executes arbitrary commands. The argument-list form is immune — the filename is passed as one literal argument, semicolons and all:

# Safe: the value is a single argument, never parsed by a shell
subprocess.run(["rm", filename], check=True)

If you have a command written as a string and need to split it correctly (respecting quotes), use shlex.split():

import shlex

cmd = shlex.split('ffmpeg -i "my input.mp4" -vn output.mp3')
subprocess.run(cmd, check=True)

Legitimate uses of shell=True exist — a hard-coded command that genuinely needs shell features — but treat it as a code-review red flag whenever any part of the string comes from outside your program.

Streaming output with Popen

run() waits for the process to finish and hands you everything at once. For long-running commands where you want output line by line — progress from a build, logs from a server — drop down to subprocess.Popen:

from subprocess import Popen, PIPE, STDOUT

with Popen(
    ["ping", "-c", "4", "python.org"],
    stdout=PIPE,
    stderr=STDOUT,   # merge stderr into stdout
    text=True,
) as proc:
    for line in proc.stdout:
        print("live:", line.rstrip())

print("exit code:", proc.returncode)

The with block ensures the pipes are closed and the process is waited on. Iterating over proc.stdout yields lines as they arrive, so you can display progress, parse incrementally, or bail out early.

One pitfall: if you use stdout=PIPE and stderr=PIPE separately and read only one of them, a chatty process can fill the other pipe's buffer and deadlock. Either merge the streams as above, or use proc.communicate(), which reads both safely.

Building pipelines

To replicate ps aux | grep python without a shell, connect one process's stdout to the next one's stdin:

from subprocess import Popen, PIPE

p1 = Popen(["ps", "aux"], stdout=PIPE)
p2 = Popen(["grep", "python"], stdin=p1.stdout, stdout=PIPE, text=True)
p1.stdout.close()  # let p1 receive SIGPIPE if p2 exits early

output, _ = p2.communicate()
print(output)

That said — if the "pipeline" is really just filtering text, do the filtering in Python. Running one process and processing its output with ordinary string methods is simpler and more portable than juggling two processes.

Controlling the environment and working directory

Two more arguments you'll reach for constantly in automation:

import os

subprocess.run(
    ["make", "build"],
    cwd="/home/anton/projects/app",          # run in this directory
    env={**os.environ, "CFLAGS": "-O2"},     # inherit env, add one var
    check=True,
)

Note the {**os.environ, ...} pattern: if you pass env=, the child gets only that mapping. Forgetting to merge in the existing environment (and losing PATH) is a classic head-scratcher.

Common pitfalls, collected

A few final things that bite people regularly. Splitting commands by hand with cmd.split() breaks on quoted arguments — use shlex.split(). On Windows, shell builtins like dir aren't real executables, so they need shell=True (with the caveats above). Forgetting text=True and then calling string methods on bytes is a frequent TypeError source. And checking result.stdout while the actual error message went to result.stderr leads to many confused debugging sessions — when something fails, print both.

Wrap-up and next steps

Reach for subprocess.run() with an argument list, check=True, capture_output=True, text=True, and a timeout — that combination handles the vast majority of real-world needs safely. Drop to Popen only when you need streaming or pipelines, and treat shell=True with the suspicion it deserves.

From here, two natural directions: if you're orchestrating many commands concurrently, look at asyncio.create_subprocess_exec() for async process control; and if your commands are long-running services rather than one-shot tools, a process supervisor (systemd, supervisord) is usually the better home for them than a Python wrapper.