Linux Processes Continued: Deadlocks, Pipes, and Memory Monitoring

Introduction

In a previous post, I looked at process states, zombie processes, and basic communication between processes. In this part, I go a bit deeper. I tested what happens when processes get stuck in a deadlock, how pipes can also cause blocking, and how to monitor system memory and process usage using built-in Linux tools.

Creating a Deadlock with File Locks

A deadlock happens when two processes are waiting on each other and neither can continue. To demonstrate this, I created two scripts.

delock1.sh:

#!/bin/bash
exec 3>file1.lock
flock -x 3

echo "Locked file1.lock and waiting for file2.lock..."
sleep 5

exec 4>file2.lock
flock -x 4

echo "Acquired file2.lock, completing..."

delock2.sh:

#!/bin/bash
exec 4>file2.lock
flock -x 4

echo "Locked file2.lock and waiting for file1.lock..."
sleep 5

exec 3>file1.lock
flock -x 3

echo "Acquired file1.lock, completing..."

Running both at the same time causes a classic deadlock. Script 1 locks file1.lock and waits for file2.lock. Script 2 locks file2.lock and waits for file1.lock. Both processes are now stuck forever.

Finding and Fixing the Deadlock

To see what was happening, I used:

lsof | grep .lock

This shows which processes are holding the lock files. Both scripts were holding one lock and waiting for the other. To fix the deadlock, I killed one of the processes:

kill -9 <PID>

Once one process is killed, the other continues and finishes.

Deadlock with Pipes

I also tested a similar situation using named pipes.

mkfifo pipe1 pipe2

process1.sh:

#!/bin/bash
echo "Process 1: Writing to pipe1 and waiting for pipe2..."
echo "Message from Process 1" > pipe1
cat pipe2
echo "Process 1: Completed."

process2.sh:

#!/bin/bash
echo "Process 2: Writing to pipe2 and waiting for pipe1..."
echo "Message from Process 2" > pipe2
cat pipe1
echo "Process 2: Completed."

When both scripts run at the same time, they both get stuck. Each process writes, then waits to read, but both are waiting at the same time so neither proceeds. This is another form of deadlock, this time through communication rather than locks.

lsof +D .
kill -9 <PID>

Checking System Memory

To understand how the system is using memory, I used a few commands. For a basic overview:

free -m

This shows total memory, used memory, free memory, and cache. To see memory activity over time:

vmstat 1 5

This updates every second and shows memory usage, CPU activity, and system processes. For detailed memory information:

cat /proc/meminfo

Viewing Memory Usage by Process

To find which processes are using the most memory:

ps aux --sort=-%mem | head -10

To inspect memory used by a specific process:

pmap <PID>

This shows the memory regions used by that process.

Swap and Cache Behavior

To check swap usage:

swapon --show
free -m

High swap usage usually means the system is low on RAM. To view buffers and cache:

cat /proc/meminfo | grep -E "Buffers|Cached|Dirty"

To clear cache for testing purposes:

sync
echo 3 | sudo tee /proc/sys/vm/drop_caches

Running free -m again after this shows the difference clearly.

Monitoring Memory Over Time

To continuously watch memory:

watch -n 1 free -m

This updates every second and makes it easy to see changes in real time. vmstat 1 gives a more detailed live view.

Checking for Zombie Processes

To check if zombie processes exist:

ps aux | grep Z

top also shows a zombie count at the top of the output. The real fix for a zombie is to handle the parent process correctly, but for testing you can remove it by killing the parent:

kill -9 <parent_pid>

Conclusion

This builds on basic process concepts and shows what happens when things go wrong. Deadlocks show how processes can block each other completely, and pipes can cause the same problem if both sides wait at the same time. Memory tools like free, vmstat, and top help understand what the system is doing in real time. These are simple commands, but they come up constantly when debugging performance issues or stuck processes.