aiScaler for no-overhead error reporting
Posted by Max Robbins on October 25th, 2010You might need a method to allow reporting of assorted error conditions. For example, your web site might offer videos, breaking news and other APIs that are invoked by AJAX or some other form of client-side Javascript. When these APIs fail to respond, you would like to have some way to report the error condition so that it can be recognized, logged and acted upon.
Normally, accomplishing this requires writing custom server-side code that would receive the error notification and execute some form of reporting, logging and alerting function. However, outside of having to write such code, you might face another danger when following down this path: when in error state, your site will be receiving large number of error message and that in itself, can cause further deterioration of service which would result in even more of error conditions being reported. In just few quick seconds you whole site just might melt down.
aiScaler offers an easy way to accomplish the required functionality, without requiring any custom code to be written or any code to be executed by your web, application or database servers. It is aiScaler that receives, logs and alerts on error condition, without ever forwarding any of the traffic to the rest of your infrastructure.
Reporting using a dedicated error reporting website.
To accomplish this, setup a dummy website in aiScaler, for example error.acmenews.com . Identify the site as dummy website via dummy_website website-level configuration directive. You don’t need to specify any origin server for websites defined as dummy, as no requests ever are sent to origin servers for such websites.
Setup the site to alert on when number of requests exceed the desired threshold – for example set alert_req_sec_max to alert when this website receives more than 2 requests per second.
Setup the client side code to execute a simple HTTP GET against this website, passing arbitrary URL and parameters, as part of the request. For example, to report a problem with your News API, you might request http://error.acme.com/app=newsapi&errorcode=123&client=premier . It is up to you just what parameters you want to pass via such a request.
Now, when an error condition is detected, a request is made to the specified URL. aiScaler receives the request and logs it. When number of such requests exceed the configurable threshold, an alert is sent out. So there you have it: without writing any of the server side logic and without placing any extra load on your infrastructure, you now can receive, log and alert on error conditions – all courtesy of aiScaler.
It doesn’t have to be only the client-side logic that sends such error-reporting requests. You can use method of your choosing to execute such requests – including server-side code, custom script etc. You can also redirect clients to the error-reporting URL in case of a problem that you discover when executing some logic. Again, you’re only limited by your imagination in how this functionality can be deployed.
You can train your personnel to look at the aiScaler log file for such error reporting web sites, when the error condition is reported and alerted by aiScaler, to see the additional information that is embedded inside of the error reporting URLs (such as application, error code and client information in the example above).
To assist in rapidly detecting the error condition, aiScaler lights up the respective web site name in red, in its self-refreshing web monitor screen, when it detect that RPS for a dummy website exceeds the set limit. This is in addition to regular email alerting on the error condition – which should be configured as per instructions in Automated Alerting and Monitoring section of this manual. Specifically, you must provide the email address to send the alert to, at either or both global and website levels.
Reporting using pattern attribute.
You can set a bad_response pattern flag. When aiScaler matches a request to such pattern, the response is reported as bad response (same as say 5xx response code) – so you can monitor for such bad responses, log and alert on them.
Here’s an example explaining how you can use this feature. Some sites have dedicated pages setup for common errors – such as PNF (page-not-found) and similar. So let’s imagine there is such a dedicated error page called 404error.html . The page is requested/redirected to in response to assorted error conditions, as detected by assorted server and client side code.
By setting bad_response flag for matching pattern, you will configure aiScaler to report and optionally, alert on, whenever this page is requested, providing for 0-cost monitoring and alerting of this error condition.
Please note that setting of bad_response flag doesn’t affect anything else in terms of request/response handling logic, its sole purpose is for reporting and optional alerting of matching traffic.