How to run shell command and capturing the output in Python ?

To run a shell command in Python and capture its output, use the subprocess module. Below are multiple methods with detailed examples, including error handling and best practices.

1. Using subprocess.run() (Python 3.5+)

The modern way to execute commands and capture output.

Example 1: Capture Standard Output (stdout)

import subprocess

# Run a command and capture stdout
result = subprocess.run(
    ["ls", "-l"],  # Command and arguments as a list
    capture_output=True,  # Capture stdout and stderr
    text=True  # Return output as a string (not bytes)
)

print("Output:", result.stdout)
print("Error:", result.stderr)
print("Exit Code:", result.returncode)

Output:

Output: total 4
-rw-r--r-- 1 user group  123 Jan 1 00:00 example.txt
Error: 
Exit Code: 0

Example 2: Capture Both stdout and stderr

# Run a command that may fail
result = subprocess.run(
    ["ls", "non_existent_file"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True
)

if result.returncode != 0:
    print(f"Command failed with error: {result.stderr}")
else:
    print(f"Output: {result.stdout}")

Output:

Command failed with error: ls: non_existent_file: No such file or directory

2. Using subprocess.check_output() (Python 2.7+/3.0+)

Simpler way to capture stdout (raises an error on failure).

Example 3: Basic Usage

import subprocess

try:
    output = subprocess.check_output(
        ["echo", "Hello, World!"],
        text=True  # Return string instead of bytes
    )
    print(output.strip())  # "Hello, World!"
except subprocess.CalledProcessError as e:
    print(f"Command failed: {e}")

3. Using subprocess.Popen() (Advanced Control)

For asynchronous execution or complex I/O handling.

Example 4: Capture Output Incrementally

process = subprocess.Popen(
    ["ping", "-c", "4", "google.com"],
    stdout=subprocess.PIPE,
    text=True
)

# Read output line by line
while True:
    line = process.stdout.readline()
    if not line:
        break
    print(line.strip())

process.wait()  # Wait for the process to finish

4. Security Considerations

• Avoid shell=True unless necessary, as it can expose security risks (e.g., shell injection).

  # Risky (user_input could be malicious)
  user_input = "file; rm -rf /"
  subprocess.run(f"ls {user_input}", shell=True)  # ⚠️ Dangerous!

  # Safer alternative (without shell=True)
  subprocess.run(["ls", user_input])  # Treats user_input as a single argument

5. Handling Bytes vs. Strings

By default, output is returned as bytes. Use text=True or universal_newlines=True to decode to strings:

# Without text=True (returns bytes)
result = subprocess.run(["echo", "hello"], capture_output=True)
print(result.stdout)  # b'hello\n'

# With text=True (returns string)
result = subprocess.run(["echo", "hello"], capture_output=True, text=True)
print(result.stdout)  # 'hello\n'

6. Full Example: Run and Log Output

import subprocess

def run_command(command):
    try:
        result = subprocess.run(
            command,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True,
            check=True  # Raise error on non-zero exit code
        )
        return result.stdout
    except subprocess.CalledProcessError as e:
        print(f"Error: {e.stderr}")
        return None

output = run_command(["git", "status"])
if output:
    print(output)

Summary Table

Method	Use Case	Key Features
subprocess.run()	General-purpose execution with full control	Captures stdout/stderr, returns CompletedProcess
subprocess.check_output()	Simple stdout capture, raises on error	Returns output directly
subprocess.Popen()	Advanced use (streaming, async)	Low-level control over process I/O

Common Errors & Fixes

• File Not Found: Ensure the command exists in the system’s PATH.
• Permission Denied: Check execute permissions for the command.
• Command Hangs: Use timeout to prevent indefinite blocking:

  result = subprocess.run(["slow_command"], timeout=30)

By leveraging these methods, you can execute shell commands and handle their output securely in Python!