Bash

⚙️ Bash Concurrency & Parallelism

Bash can run multiple processes concurrently, allowing you to speed up tasks like downloading files, processing logs, or batch operations. This is useful in automation, scripting, DevOps, and data pipelines.


🧵 1. Run in Background with &

Start any command as a background job:

sleep 5 &
echo "Sleeping in background..."

Use jobs to list background tasks.


⏳ 2. Wait for Background Jobs

The wait command pauses until all background jobs finish.

echo "Starting jobs..."
sleep 3 &    # Job 1
sleep 5 &    # Job 2

wait         # Waits for both to complete
echo "All done"

Wait for a specific PID:

wait 12345

🧮 3. Background Loop Tasks

Example: Download 3 files in parallel

for i in 1 2 3; do
  curl -O http://example.com/file$i.txt &
done
wait

🧰 4. Parallel Processing with xargs -P

xargs can run multiple commands in parallel using -P:

cat urls.txt | xargs -n 1 -P 4 curl -O
Option Description
-n 1 One argument per command
-P 4 Run up to 4 processes in parallel

🚦 5. Using GNU parallel (optional)

Powerful tool for multi-core execution:

cat files.txt | parallel gzip

Install it:

sudo apt install parallel

🔧 6. Monitor Jobs

jobs         # List jobs
fg %1        # Bring job 1 to foreground
bg %1        # Resume job in background

🚫 7. Kill Background Jobs

Store PID and kill manually:

my_script.sh &
pid=$!
sleep 5
kill $pid

📌 Best Practices

Tip Why
Use wait after & Prevent premature script exit
Use xargs -P Efficient for batch workloads
Limit background jobs Avoid resource overuse
Redirect output Prevent mixed stdout/stderr

📚 References


📁 File: advance/concurrency.md
Author: Kashif Alam