How to Use strace to Trace Linux System Calls

Every useful thing a Linux process does goes through a system call: opening a file, reading a socket, forking a child, waiting on a lock. strace intercepts those calls and prints them, with arguments and return values, as they happen. That makes it the fastest way to answer questions the application's own logs won't: which config file is it actually reading, why does it say "permission denied", where exactly is it hanging, and what is it waiting on.

This guide covers installing strace, reading its output, the filters that keep it readable, attaching to running processes, timing, the per-syscall summary, and a set of recipes for the incidents strace is best at. It ends with the one caveat that matters in production: strace slows the traced process down a lot, and there are lighter tools when that is a problem.

Install strace

# Debian / Ubuntu
sudo apt update && sudo apt install -y strace

# RHEL, AlmaLinux, Rocky, Fedora
sudo dnf install -y strace

# Alpine
sudo apk add strace

# Arch
sudo pacman -S strace

strace -V prints the version. Anything from 5.x onward has every flag used below.

First run: trace a command

strace ls /etc/hostname

You get several dozen lines ending in something like:

openat(AT_FDCWD, "/etc/hostname", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOTDIR (Not a directory)
openat(AT_FDCWD, "/etc/hostname", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=7, ...}) = 0
write(1, "/etc/hostname\n", 14)         = 14
close(3)                                = 0
exit_group(0)                           = ?

Each line is one system call: the name, its arguments (decoded — flags, structs, strings), = and the return value. A negative return is shown with the errno name and text, which is the part you usually care about.

By default strace prints to stderr, which is why it interleaves with the program's own output. Redirect with -o (below) when the noise matters.

Make the output readable

Filter by system call or family

strace -e trace=openat,read,write ls /etc        # specific calls
strace -e trace=file curl https://example.com    # everything that takes a path
strace -e trace=network curl https://example.com # sockets, connect, send/recv
strace -e trace=process ./run.sh                 # fork, clone, execve, wait
strace -e trace=signal ./daemon                  # signal delivery and handlers

%file, %network, %process, %signal, %desc (file descriptors) and %memory are the families you will reach for most. -e trace=!futex,clock_gettime excludes noisy calls instead.

Show full strings

strace truncates string arguments at 32 bytes. For paths, SQL, HTTP headers and the like, raise it:

strace -s 512 -e trace=write ./app

Resolve file descriptors to paths

strace -y -e trace=read,write ./app
# read(3</var/lib/app/config.yaml>, "...", 4096) = 1203

-y turns read(3, …) into read(3</path/to/file>, …). -yy also decodes sockets into TCP:[10.0.0.5:44321->10.0.0.9:5432], which is what you want when debugging connections.

Write to a file

strace -o /tmp/app.trace -f ./app
less /tmp/app.trace

With -ff strace writes one file per process (app.trace.<pid>), which is easier to read for anything that forks.

Attach to a running process

Most real debugging is on a process that is already running and misbehaving:

pidof nginx            # or: pgrep -f 'php-fpm: pool www'
sudo strace -p 1842 -f -s 256 -yy

Press Ctrl+C to detach; the process continues unaffected.

• -p PID attaches to that process (repeatable: -p 1842 -p 1843).
• -f follows children created after you attach — essential for anything pre-forked (nginx workers, php-fpm, Postgres backends, uWSGI).
• You usually need sudo: on most distributions kernel.yama.ptrace_scope=1 means you can only attach to your own descendants without CAP_SYS_PTRACE.

Timing: find where the time goes

strace -T ./app          # append <0.000123> — time spent inside each call
strace -r ./app          # relative timestamp since the previous call
strace -tt ./app         # absolute wall-clock time with microseconds
strace -ttt ./app        # epoch seconds, easy to correlate with other logs

Combine -T with -e trace=network and you can see a connect() that takes 5 s (a firewall dropping SYNs) or a poll() on the DNS socket that times out.

The summary table: -c

strace -c -f ./app
# % time     seconds  usecs/call     calls    errors syscall
# ------ ----------- ----------- --------- --------- ----------------
#  61.02    0.482913         312      1547        23 futex
#  18.44    0.145966          48      3041           read
#   9.70    0.076771        1535        50        50 connect

-c suppresses the per-call lines and prints counts, errors and time per syscall at the end. Two patterns jump out immediately: a syscall with many errors (50 connect calls, 50 errors — something is unreachable and the app is retrying), and a syscall dominating time (futex — the app is mostly waiting on locks, not doing I/O).

Recipes

Which config file does it actually read?

strace -f -e trace=openat,stat,access -o /tmp/cfg.trace myapp --check-config
grep -E '\.(conf|yaml|yml|ini|json|toml)' /tmp/cfg.trace

Lines returning = -1 ENOENT are the paths it tried; the one returning a file descriptor is the one it used. This settles "but I edited the config" arguments in thirty seconds.

Why "permission denied"?

strace -f -e trace=file -o /tmp/perm.trace sudo -u www-data php /var/www/app/cron.php
grep EACCES /tmp/perm.trace

You get the exact path and the exact call (openat, mkdir, unlink, connect on a Unix socket), which tells you whether to fix a file mode, a directory mode or a socket owner.

Where is it hanging?

Attach to the stuck PID with timestamps and watch the last line:

sudo strace -p "$(pidof myapp)" -f -tt -yy

If the output stops at read(7<TCP:[10.0.0.5:51234->10.0.0.20:5432]>, … the process is waiting on the database. If it stops at futex(…, FUTEX_WAIT, …) it is waiting on a lock inside the process — go look at threads (-f shows which). If it stops at wait4(…) it is waiting for a child.

Slow startup = slow DNS?

strace -e trace=network -T -yy ./app 2>&1 | grep -E ':53\b'

A connect() to port 53 followed by poll() calls with multi-second <…> timings means the resolver is the problem, not the app. The fix is usually /etc/resolv.conf or a local DNS cache.

What does a shell script actually run?

strace -f -e trace=execve -s 200 ./deploy.sh

Every execve is a command the script spawned, with its full argument vector — including the ones built from variables you couldn't see.

Use it responsibly in production

strace stops the traced process on every syscall to read its state. For a syscall-heavy process (a web server, a database) that can mean a 10–100× slowdown while attached. Rules of thumb:

• Attach briefly, with a tight -e trace= filter, and detach as soon as you have what you need.
• Prefer -c for a few seconds over a full trace for minutes.
• For continuous or low-overhead tracing use perf trace, bpftrace or sysdig, which observe syscalls from the kernel side without stopping the process.
• Never leave strace attached to a production process unattended.

Troubleshooting strace itself

• strace: attach: ptrace(PTRACE_SEIZE, …): Operation not permitted — Yama's ptrace_scope (check sysctl kernel.yama.ptrace_scope). Run strace as root, or temporarily sudo sysctl kernel.yama.ptrace_scope=0. Inside a container you also need the SYS_PTRACE capability (docker run --cap-add SYS_PTRACE …).
• Output is empty but the program ran — the program's own output went to stdout and strace's trace to stderr; use -o file. Or the interesting work happened in a child process; add -f.
• Strings are cut off with … — raise -s.
• Thousands of futex/clock_gettime lines — exclude them: -e trace=!futex,clock_gettime,nanosleep.
• strace: Can't stat 'myapp': No such file or directory — strace resolves the program via PATH like a shell; give a relative or absolute path (./myapp).

Summary

1. strace -f -o trace.txt command or sudo strace -f -p PID to attach.
2. Narrow with -e trace=file|network|process|… and raise -s for full strings.
3. -yy to see what each file descriptor is, -T/-tt to see where the time goes.
4. -c for a quick count-and-errors summary before you read anything line by line.
5. Keep sessions short on production hosts; use perf trace or bpftrace for anything sustained.

Related Resources

• Troubleshoot Linux using dmesg — the kernel's side of the story when strace shows a syscall failing with an unexpected error
• How to check running processes on Ubuntu Server — find the PID to attach to and the children worth following
• How to install and use tcpdump — when strace shows a connect() that never completes, tcpdump shows whether the packets even leave the box
• Server monitoring — catch the CPU, memory and I/O symptoms that send you reaching for strace, before users do