How to monitor web servers and their performance

Why monitor Web servers?

Web servers are among the most important components in modern IT infrastructures. They host the websites, web services, and web applications that we use on a daily basis. Social networking, media streaming, software as a service (SaaS), and other activities wouldn’t be possible without the use of web servers. And with the advent of cloud computing and the movement of more services online, web servers and their monitoring are only becoming more important. Given the extensive usage of Web servers, Sysadmins and SREs should monitor web servers as a key aspect for performance. 

What to monitor on Web Servers?

There are dozens of web servers, but the two most popular are Apache and NGINX. And irrespective of the web server in use, apart from the host metrics on which the web server is hosted like - Uptime, CPU Usage, Memory Usage, Cache, Threads etc - you will need to monitor metrics related to the Web server connections themselves and some of the more important ones are:

• Request rate: The number of requests that the server receives over time. High request rates could indicate a recent increase in traffic.
• Response rate / Error codes: The number of requests that the server is able to handle and respond over time and to monitor any errors in response.
• Response time: The time to send a response to a request. High response times could indicate problems with the web server, host, or website resources, and result in frustrated users.
• Response size / Bandwidth: The amount of data delivered with each response (typically measured in bytes). Smaller responses use less network bandwidth and load faster for users, especially over limited and mobile connections.
• Active connections: The number of requests currently being fulfilled. Too many active connections can exhaust the host’s available network ports, preventing new users from connecting.

Web Server Log Collector in Netdata

Both Apache and NGINX report internal metrics via an HTML page. Apache has the mod_status module, and NGINX has the ngx_http_stub_status_module module. The Netdata plugins for NGINX Monitoring and Apache Monitoring generate metrics and alerts from these pages. But if you can provide Netdata access to their log files, you can get even more insights via the generic Web Server Log Monitoring collector.

Once the collector is set up, it comes with default summary charts (which can be modified on the custom dashboards in the future) and gives a high level indication of how the web server(s) are performing.

Request Rate: Now, going into the individual metrics let us look at the most important ones and the rate of “requests” being handled by the Web servers determines the load and an indication of whether sufficient computing resources are allocated to the service if the cpu, memory, etc. are showing signs of exhaustion. 

If you are monitoring multiple Web servers, you can further group by “instance” to see the requests being handled by each of your web servers. Now, the dimension names look funny. That’s because our front-end does some funky things to keep the length of each dimension within a certain width. When you hover over the chart, you see the fully expanded dimension names. 

Response rate / Error Codes: The next important metric to monitor is the “response” rate to compare if the request and response rates are in order. And in addition the error codes on responses will provide a good idea of any issues in serving the requests.

Response time and Response size: In addition, monitoring the “request processing time” (response time) and “bandwidth” (response size) gives a realistic view of the web server's capability of serving multiple requests and can point to any latencies on specific web server instances.

Type of Requests: In the world of web servers, it is also crucial to understand the types of requests and clients to allocate the necessary resources/networking.

The Web server log collector also comes with default alerts which can be further customized to your infrastructure’s requirements. You can view all the active alerts and configured alerts on the Alerts tab.

For more information on configuring the Web server log Collector using Netdata and setting up custom alerts look into the enclosed links.

Let us hear from you

If you haven’t already, sign up now for a free Netdata account!

We’d love to hear from you – if you have any questions, complaints or feedback please reach out to us on Discord or Github.

Happy Troubleshooting!