Debugging mindset

Debugging mindset

Logs and Troubleshooting

Learn where logs live, how journalctl works, and how to debug permission errors, package issues, boot failures, networking, and disk space calmly.

Audience
Everyone who will eventually see an error
Reading time
22 min

A safe troubleshooting mindset

Troubleshooting is not guessing commands until something changes. It is narrowing the problem: what changed, what layer is failing, what evidence exists, and what is the smallest safe test?

Write down exact errors before changing things. Prefer read-only inspection first. Make backups before editing configuration. Change one variable at a time.

Where logs live

Modern systemd distributions store many logs in the journal, queried with journalctl. Traditional logs also appear under /var/log. Applications may log to files, the journal, containers, or their own data directories.

Where logs live
Log sourceWhere to look
systemd servicesjournalctl -u service-name
Current bootjournalctl -b
Kernel messagesjournalctl -k or dmesg
Traditional logs/var/log
Application-specific logsApplication docs, /var/log/app, /var/lib/app, container logs

journalctl basics

journalctl can filter by service, boot, priority, time, executable, user, and more. Targeted filters are usually more useful than scrolling through everything.

Current boot
journalctl -b
Service logs
journalctl -u nginx --since "30 minutes ago"
Follow logs
journalctl -f
Errors and worse
journalctl -p err -b

Common Linux problems and first checks

Common Linux problems and first checks
ProblemFirst checks
Permission deniedwhoami, id, ls -l, directory execute bits, sudo need
Package install failsNetwork, repository metadata, held packages, disk space, conflicting repos
Service will not startsystemctl status, journalctl -u, config syntax, port conflicts
Boot issuejournalctl -xb, systemctl --failed, fstab entries, recent kernel or driver changes
Networking issueip addr, ip route, DNS test, ping, curl, ss, firewall
Disk fulldf -h, du -h -d 1, logs, caches, snapshots, container images

How to read error messages

An error often contains the failing operation, target path or service, reason, and sometimes the file and line number. Do not search only the final generic line if there is a more specific line above it.

For permission errors, identify the user, target file, parent directories, and desired operation. For network errors, separate DNS errors from connection refused, timeout, TLS failures, and HTTP status codes.

Practical debugging steps

  1. 1Reproduce the issue once and copy the exact error.
  2. 2Identify the layer: file, permission, package, service, network, disk, boot, or application.
  3. 3Inspect state with read-only commands first.
  4. 4Check logs close to the time of failure.
  5. 5Make a reversible change.
  6. 6Test again and compare evidence.
  7. 7Keep notes so you can undo or explain what changed.