Would you like to discuss your environment with a knowledgable engineer?
An all too common web site failure is running out of space in log partition. It can happen due to a failure to rotate and offload log files, unexpected spike in traffic and similar conditions.
A normal reaction to such out-of-space condition is for web server to abort serving requests (exit) and hope that some kind of monitoring system catches such down condition, someone notices the alert, logs in and rectifies the problem. All the while the users are receiving error message. Clearly less than desirable situation.
If it is more important to you to continue serving user requests than it is to make sure each user request is logged, you can instruct aiScaler to ignore write failures via stay_up_on_write_error global setting.
AiCache server provides several options for automated or on-demand log rotation
aiScaler provides automated, on-the-fly, size-based access log file rotation. This feature is typically used to cut down size of the access log files to some reasonable value, as it might be easier to deal with 10 files of 100MB each, as opposed to a single file of 1GB. To enable, simply set in the cfg file:
Be careful with those zeroes – the size is specified in bytes !
When max_log_file_size is set, aiScaler rotates present log file upon it reaching the maximum size. The renamed access file has a timestamp added to the name of the file. The size-based rotation is a zero-downtime operation and happens automagically.
To execute on-the-fly, on-demand log file rotation, send SIGUSR1 to aiScaler’s process ID at time of your choosing – manually or using a script driven by Linux cron facility. For example:
pkill -USR1 aicache
Simply add this command to be scheduled by Linux cron facility at time that is convenient for you. aiScaler’s log file rotation – size-based or on demand, is a zero-overhead and zero-downtime operation, so feel free to do it on schedule that is convenient for you, there’s no performance impact. Please note that aiScaler rotates log files after the rotation is requested, upon receiving first log-enabled request.
You can also tell aiScaler to rotate the log files from the CLI, using rotatelog command. After logging into CLI
file rotation is not cluster-wide, it is only the aiScaler instance that the command is directed at, that executes the rotation.