filter errors from text based log files such as php, nginx, apache etc.
elog can find and filter specified error logs from your applications, web servers and any other text based log files, store the logs into MongoDB (>= 2.1.0). Then you can access the logs from MongoDB via a web interface with some filter options.
elog contains elog-client and elog-server, elog-client can filter and push logs to elog-server via http requests, elog-server is a kind of web server with expressjs.
$ [sudo] npm -g install elog
client settings: elog-client
It's a standard JSON file, You need to specify 1 or more log files for each app and api like below.
"apps":"name": "app name""file": "/tmp/php_errors.log""interval_time": 5000"position": 0"rules":"include":"error" "i" "LOG_ERROR""Notice" "LOG_WARN""exclude": "Primary script unknown""name": "nginx""file": "/usr/local/nginx/logs/error.log""interval_time": 5000"position": 0"rules":"include":"error" "LOG_ERROR""alert" "LOG_WARN""exclude":"api":"key": "mykey""url": ""
for each app, there are 5 parameters:
- name: name of your app of the log file
- file: log file path
- interval_time: every number of seconds to check new logs
- position: read data from log file in the specified positon after elog-client is started
- rules: define some rules to filter logs, it contains "include" and "exclude":
- rules['include']: it's an array, each element is also an array which contains 2 elements, the first one is for build regular expression, it could be a string or an array. The sedond parameter is log level (LOG_FATAL, LOG_ERROR, LOG_WARN, LOG_INFO, LOG_DEBUG), logs will be processed if matched.
- rules['exclude']: it's an array, logs will be excluded if matched any rules (regular expression) list here.
Also, you need to define an api key and url like above, the api key is just a random string which need to match the server side api key settings.
server settings: elog-server
"api_key": "mykey""http":"host": "localhost""port": 3339"mongodb":"port": 27017"host": "localhost""database": "elog""collection": "logs""web":"title": "elog""limit_per_page": 100"refresh_time": 10000
The server side settings is also a standand JSON file:
- api_key: api key for the authentication, elog-client should send the same api key to match this one when push logs.
- http: define a host and port to start a web server.
- mongodb: mongodb related settings
- web: web page related settings
Actually, you can generate a default config file with the following commands:
# mkdir /etc/elog # elog-server show-config > /etc/elog/server.json # elog-client show-config > /etc/elog/client.json
$ nohup elog-client /etc/elog/client.json > /var/log/elog-client.log &
$ nohup elog-server /etc/elog/server.json > /var/log/elog-server.log &
If something went wrong, you can check the log files you specified such as above, otherwise, you can go and visit http://localhost:3339 to see error logs.
reload client or server
In case if you changed some configuration, we can reload the settings without shutdown the client or the server process, just reload it:
$ elog-server reload # server side $ elog-client reload # client side
stop client and server
$ elog-server stop $ elog-client stop
run without parameters
$ elog-server $ elog-client
You can see all of available commands.
You need to update elog-server:
$ elog-server /etc/elog/server.json update all
To see all of update options, run "elog-server --help".
JSON configuration check
$ node /etc/elog/client.json No output if there is no errors.
Read log files failed
Please check if the log file is exists or the current user has read permission to read the file.
Run client and server in different servers
You can run multiple elog-clients for each server which has log files to process, run one elog-server in one server to receive the requests from elog-clients. elog-server can also working with Nginx http proxy, it's just a normal HTTP server.
we are using mocha with should for the test, run test in the elog directory:
$ mocha -r should --compilers coffee:coffee-script
or run it with jake
$ jake test
- It doesn't work with logs with multiple bytes.
- It only can process logs line by line.
- It's only tested on linux/unix system, especially for the reload and stop commands.
- php, nginx and apache