6.4. Application and Service Troubleshooting
Application problems live above the OS layer — the hardware, filesystem, and network are all fine, but the application itself is misbehaving. Diagnosing these requires understanding what the application does and reading its logs intelligently.
💡 First Principle: Application troubleshooting follows the same elimination methodology as system troubleshooting, but the scope is narrower. Start with logs (what does the application say happened?), then check dependencies (is the database it connects to available?), then check resources (is it out of memory? file descriptors?), then check configuration (has something changed?).
Application-layer failures are the most common source of production incidents — and the hardest to diagnose because the symptoms (HTTP 500, slow responses, job hangs) can have dozens of causes. A systematic approach prevents wasted hours chasing the wrong layer.
⚠️ Common Misconception: If the service process is running, the application is working. A process can be running but deadlocked, out of file descriptors, or unable to reach its database — all producing failures despite showing as active in systemctl status.
Application failures are the most disruptive incidents: users are affected immediately, the cause isn't visible in infrastructure metrics, and without a systematic approach an engineer can spend hours checking the wrong layer.
Think of application troubleshooting like a stack of filters: OS → process → network → config → code. Work from the bottom up — confirm the process is running, confirm it can reach its dependencies, confirm its configuration is valid, then look at application-level logs. Jumping straight to code review when the service isn't even starting wastes time.
