ざっくりとした設定
<source>
type config_expander
<config>
type tail
path /some/path/to/log
pos_file /some/path/to/pos/file
tag some.tag
☆
</config>
</source>
以下だらだら挙げる各ログ名の下にある斜体の設定が☆の部分に入る
Apache
ssl_access_log
format apache2[ログ]
xx.xx.xx.xx - - [30/Dec/2013:16:49:34 +0900] "GET / HTTP/1.1" 304 -
xx.xx.xx.xx - - [30/Dec/2013:16:49:49 +0900] "GET /assets/jquery-i18n/messages_ja.properties?_=1388389789290 HTTP/1.1" 200 2589
2013-12-30T07:49:34Z some.tag {"host":"xx.xx.xx.xx","user":null,"method":"GET","path":"/","code":304,"size":null,"referer":null,"agent":null}
2013-12-30T07:49:49Z some.tag {"host":"xx.xx.xx.xx","user":null,"method":"GET","path":"/assets/jquery-i18n/messages_ja.properties?_=1388389789290","code":200,"size":2589,"referer":null,"agent":null}
[メモ]
標準で用意されてるapache2を使えたので一番楽できた
ssl_error_log
format /^\[(?<time>[^\]]*)\] \[(?<level>[^\]]*)\] \[client (?<host>[^\]]*)\] (?<message>.*)$/
[ログ]
time_format %a %b %d %H:%M:%S %Y
[ログ]
[Mon Dec 30 17:53:50 2013] [error] [client xx.xx.xx.xx"] Premature end of script headers: hogehoge
[fluentd出力]
[メモ]
host部分を取る方法が泥臭い
2013-12-30T08:53:50Z some.tag {"level":"error","host":"xx.xx.xx.xx"","message":"Premature end of script headers: hogehoge"}
[メモ]
host部分を取る方法が泥臭い
ssl_request_log
format /^\[(?<time>[^\]]*)\] (?<host>[^ ]*) (?<ssl_protocol>[^ ]*) (?<ssl_cipher>[^ ]*) "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<size>[^ ]*)$/
time_format %d/%b/%Y:%H:%M:%S %z
[ログ]
[30/Dec/2013:17:53:44 +0900] xx.xx.xx.xx TLSv1.2 DHE-RSA-AES128-GCM-SHA256 "GET /assets/jquery-i18n/messages_ja.properties?_=1388393624678 HTTP/1.1" 2589
[30/Dec/2013:17:53:50 +0900] xx.xx.xx.xx TLSv1.2 DHE-RSA-AES128-GCM-SHA256 "GET /hogehoge HTTP/1.1" 534
[fluentd出力]
2013-12-30T08:53:44Z some.tag {"host":"xx.xx.xx.xx","ssl_protocol":"TLSv1.2","ssl_cipher":"DHE-RSA-AES128-GCM-SHA256","method":"GET","path":"/assets/jquery-i18n/messages_ja.properties?_=1388393624678","size":"2589"}
2013-12-30T08:53:50Z some.tag {"host":"xx.xx.xx.xx","ssl_protocol":"TLSv1.2","ssl_cipher":"DHE-RSA-AES128-GCM-SHA256","method":"GET","path":"/hogehoge","size":"534"}
[メモ]
こんなもんかな
access_log（combined + %{X-Forwarded-For}i）
format /^(?<host>[^ ]*) (?<remotelog>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<status>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" "(?<forwardedfor>[^\"]*)"$/
time_format %d/%b/%Y:%H:%M:%S %z
xx.xx.xx.xx - - [30/Dec/2013:16:49:32 +0900] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36" "-"
xx.xx.xx.xx - - [30/Dec/2013:17:18:19 +0900] "GET /admin/sign_in HTTP/1.1" 200 2038 "http://www.baidu.com" "Mozilla/5.0 (X11; U; Linux i686) Web-Security/1.0(it's for a research study,if you have questions,plz contact me liangw@cs.wisc.edu)" "-"
2013-12-30T07:49:32Z some.tag {"host":"xx.xx.xx.xx","remotelog":"-","user":"-","method":"GET","path":"/","status":"304","size":"-","referer":"-","agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36","forwardedfor":"-"}
2013-12-30T08:18:19Z some.tag {"host":"xx.xx.xx.xx","remotelog":"-","user":"-","method":"GET","path":"/admin/sign_in","status":"200","size":"2038","referer":"http://www.baidu.com","agent":"Mozilla/5.0 (X11; U; Linux i686) Web-Security/1.0(it's for a research study,if you have questions,plz contact me liangw@cs.wisc.edu)","forwardedfor":"-"}
[メモ]
baiduアク禁にしとくか
error_log
format /^(?<message>.+)$/
[ログ]
[ pid=25565 thr=140417271351328 file=ext/apache2/Hooks.cpp:841 time=2013-12-30 17:53:50.689 ]: The backend application (process 13501) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing bugs in this application.
[fluentd出力]
2013-12-30T08:53:50Z some.tag {"message":"[ pid=25565 thr=140417271351328 file=ext/apache2/Hooks.cpp:841 time=2013-12-30 17:53:50.689 ]: The backend application (process 13501) did not send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing bugs in this application."}
[メモ]
スタックトレースがあるので空行を削る以外の加工を諦めた
System
/var/log/messages
format syslog
[ログ]
Dec 30 17:15:01 xx.xx.xx.xx yum[14369]: Updated: 2:tar-1.26-27.21.amzn1.x86_64
[fluentd出力]
2013-12-30T08:15:01Z some.tag {"host":"xx.xx.xx.xx","ident":"yum","pid":"14369","message":"Updated: 2:tar-1.26-27.21.amzn1.x86_64"}
[メモ]
標準でsyslogが用意されていたので助かった
/var/log/secure
format /^(?<time>[^ ]* [^ ]* [^ ]*) (?<host>[^ ]*) (?<message>.*)$/
time_format %b %d %H:%M:%S
[ログ]
Dec 30 17:15:53 xx.xx.xx.xx runuser: pam_unix(runuser:session): session opened for user td-agent by ec2-user(uid=0)
Dec 30 17:15:54 xx.xx.xx.xx runuser: pam_unix(runuser:session): session closed for user td-agent
[fluentd出力]
2013-12-30T08:15:53Z some.tag {"host":"xx.xx.xx.xx","message":"runuser: pam_unix(runuser:session): session opened for user td-agent by ec2-user(uid=0)"}
2013-12-30T08:15:54Z some.tag {"host":"xx.xx.xx.xx","message":"runuser: pam_unix(runuser:session): session closed for user td-agent"}
[メモ]
スペースで分けられたtimeに初めて対処。ここのsyslogの項に学んだ
/var/log/cron
format /^(?<time>[^\]]*) (?<host>[^ ]*) (?<process>[^\[]*)\[(?<process_id>[0-9]+)\]: (?<message>.*)$/
time_format %b %d %H:%M:%S
Dec 30 17:05:01 xx.xx.xx.xx CROND[14223]: (root) CMD (/usr/sbin/logrotate /etc/logrotate.d/ooo)
Dec 30 17:10:01 xx.xx.xx.xx CROND[14297]: (root) CMD (/usr/lib64/sa/sa1 1 1)
[fluentd出力]
2013-12-30T08:05:01Z some.tag {"host":"xx.xx.xx.xx","process":"CROND","process_id":"14223","message":"(root) CMD (/usr/sbin/logrotate /etc/logrotate.d/ooo)"}
2013-12-30T08:10:01Z some.tag {"host":"xx.xx.xx.xx","process":"CROND","process_id":"14297","message":"(root) CMD (/usr/lib64/sa/sa1 1 1)"}
[メモ]
サンプル不足
/var/log/yum.log
format /^(?<time>[^\]]*) (?<action>[^:]*): (?<package>[^ ]*)$/
time_format %b %d %H:%M:%S
[ログ]
Dec 30 17:15:01 Updated: 2:tar-1.26-27.21.amzn1.x86_64
[fluentd出力]
2013-12-30T08:15:01Z some.tag {"action":"Updated","package":"2:tar-1.26-27.21.amzn1.x86_64"}
[メモ]
とりあえず仕掛けた。サンプル不足
/var/log/maillog
format /^(?<time>[^\]]*) (?<host>[^ ]*) (?<process>[^\[]*)\[(?<process_id>[0-9]+)\]: (?<message>.*)$/
time_format %b %d %H:%M:%S
[ログ]
Dec 30 16:58:42 xx.xx.xx.xx sendmail[13943]: NOQUEUE: SYSERR(ec2-user): can not chdir(/var/spool/mqueue/): Permission denied
[fluentd出力]
2013-12-30T07:58:42Z some.tag {"host":"xx.xx.xx.xx","process":"sendmail","process_id":"13943","message":"NOQUEUE: SYSERR(ec2-user): can not chdir(/var/spool/mqueue/): Permission denied"}
[メモ]
これでいいのか。サンプル不足
App（my app）
myapp.log (tab separated)
format tsv
keys user_id, ref_user_id, request_url, params,executed_at
time_key executed_at
[ログ]
28 22 https://xx.xx.xx.xx/18 {"controller"=>"users", "action"=>"detail", "id"=>"18"} 2013-12-30 08:26:57 UTC
2013-12-30T08:26:57Z some.tag {"user_id":"28","ref_user_id":"22","request_url":"https://xx.xx.xx.xx/users/18","params":"{\"controller\"=>\"users\", \"action\"=>\"detail\", \"id\"=>\"18\"}"}
[メモ]
初めてのtsv。問題なくtime_keyも解釈できた様子
0 件のコメント:
コメントを投稿