How to read a file into a variable in shell?

Reading a File into a Variable in Shell: Detailed Guide

There are several methods to read file contents into a variable in shell scripting. Below are the most common techniques with examples:

1. Using Command Substitution ($(<file) or \cat file“)

Best for reading entire files at once (small to medium files).

# Modern bash method (recommended)
content=$(<file.txt)

# POSIX-compatible method
content=$(cat file.txt)

Example:

#!/bin/bash

# Read file into variable
file_content=$(<config.txt)

# Print content
echo "$file_content"

Important: Always quote "$file_content" to preserve formatting.

2. Using read in a Loop

For processing large files line-by-line or storing as an array.

Read into a string (single variable):

while IFS= read -r line; do
    content="$content$line\n"
done < file.txt

Read into an array:

declare -a lines
while IFS= read -r line; do
    lines+=("$line")
done < file.txt

Example:

#!/bin/bash

# Read lines into array
declare -a config_lines
while IFS= read -r line; do
    config_lines+=("$line")
done < servers.txt

# Print line count and third line
echo "Total lines: ${#config_lines[@]}"
echo "Third line: ${config_lines[2]}"

3. Using mapfile/readarray (Bash 4+)

Most efficient for reading files into arrays.

mapfile -t lines < file.txt  # -t removes trailing newlines

Example:

#!/bin/bash

# Read into array
mapfile -t log_lines < /var/log/app.log

# Search for errors
for line in "${log_lines[@]}"; do
  if [[ $line =~ "ERROR" ]]; then
    echo "Found error: $line"
  fi
done

4. Using dd (Binary-Safe Method)

For binary files or precise control over read operations.

content=$(dd if=file.bin bs=1 count=100K status=none)

Example (read first 1KB of a binary):

#!/bin/bash

# Read first 1024 bytes
binary_header=$(dd if=image.png bs=1 count=1024 status=none)

# Check PNG signature
if [[ "$binary_header" =~ ^.PNG ]]; then
  echo "Valid PNG file"
fi

Key Considerations

1. File Size Limitations

• Small files (<100MB): Use $(<file)
• Large files (>100MB): Use while read or mapfile
• Binary files: Use dd or base64

2. Special Characters & Formatting

• Always use IFS= and -r with read:

  while IFS= read -r line; do ... done < file

• Preserve newlines with \n in double-quoted variables

3. Performance Comparison

Method	Speed	Memory	Best For
$(<file)	⭐⭐⭐⭐⭐	High	Small text files
mapfile	⭐⭐⭐⭐	Medium	Large text files
while read	⭐⭐⭐	Low	Huge files
dd	⭐⭐	Medium	Binary data

Full Examples

Example 1: Config File Parser

#!/bin/bash

# Read entire config file
config=$(<app.conf)

# Extract version
if [[ $config =~ "APP_VERSION="([0-9.]+) ]]; then
  ver="${BASH_REMATCH[1]}"
  echo "App Version: $ver"
fi

Example 2: Large Log Processor

#!/bin/bash

# Process large log line-by-line
while IFS= read -r line; do
  if [[ $line == *"HTTP 500"* ]]; then
    echo "Server error: $line" >> errors.log
  fi
done < access.log

Example 3: Binary File Signature Check

#!/bin/bash

# Read first 4 bytes of a file
signature=$(dd if="$1" bs=1 count=4 2>/dev/null)

case "$signature" in
    PK*) echo "ZIP archive" ;;
    ELF*) echo "Executable" ;;
    *) echo "Unknown format" ;;
esac

Common Pitfalls & Fixes

1. Missing newlines:
   Add trailing newline: echo "$content" (not echo $content)
2. Path errors:
   Use absolute paths: $(</path/to/file)
3. Binary corruption:
   For non-text files:

   # Encode to base64
   b64_content=$(base64 -w0 file.bin)

   # Decode later
   echo "$b64_content" | base64 -d > restored.bin

4. Sluggish large file reads:
   Avoid appending in loops:

   # Slow: content+="$line"
   # Fast: Use arrays or process immediately

Final Recommendations

• Text files < 100MB: $(<file)
• Text files > 100MB: mapfile or while read
• Binary files: dd or base64
• POSIX compatibility: Use $(cat file) and while read

By selecting the appropriate method based on your use case, you can efficiently read file contents into variables while preserving data integrity.