这是一个创建于 574 天前的主题，其中的信息可能已经有所发展或是发生改变。
由于我的网站访问人数越来越多，nginx 的日志大小也在指数级增长，想问一下大家有没有 nginx 日志分析的工具，即我将 nginx 日志复制出来，然后通过该工具进行数据的相关分析（错误类型、各地区 IP 访问量等）
 |
1
hasdream 2023-05-02 10:08:57 +08:00
goaccess
|
 |
2
julyclyde 2023-05-02 10:36:23 +08:00
如果真的是指数级那估计是被攻击了
应该线性增长才对啊 |
 |
3
seers 2023-05-02 10:43:30 +08:00
一般都是用 prometheus
|
 |
4
bjzhush 2023-05-02 10:46:12 +08:00
不用这么麻烦，直接 awk sort 几个组合一下，按 IP 排序访问数量就知道了
|
 |
5
tonlmy 2023-05-02 10:46:23 +08:00
logstalgia
|
 |
6
LxnChan OP |
 |
7
PolarBears 2023-05-02 11:50:24 +08:00
默认格式就 goaccess 吧,如果有条件的话还是建议配置 nginx 日志格式为 json 格式然后丢 elasticsearch 上分析吧
|
 |
8
tiga99 2023-05-02 17:32:37 +08:00
如果资源足够，可以将日志写到 es ，用 grafana 做展示；参考博客： https://www.xiaoleizhang.com/index.php/archives/120/
如果资源不足或者不想这么复杂，可以将 nginx 重新编译一下，加上 nginx-module-vts 和 geoip2 模块，大概效果如下： ```text # HELP nginx_vts_filter_bytes_total The request/response bytes # TYPE nginx_vts_filter_bytes_total counter # HELP nginx_vts_filter_requests_total The requests counter # TYPE nginx_vts_filter_requests_total counter # HELP nginx_vts_filter_request_seconds_total The request processing time in seconds counter # TYPE nginx_vts_filter_request_seconds_total counter # HELP nginx_vts_filter_request_seconds The average of request processing times in seconds # TYPE nginx_vts_filter_request_seconds gauge # HELP nginx_vts_filter_request_duration_seconds The histogram of request processing time # TYPE nginx_vts_filter_request_duration_seconds histogram # HELP nginx_vts_filter_cache_total The requests cache counter # TYPE nginx_vts_filter_cache_total counter nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="unknow",direction="in"} 7332314 nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="unknow",direction="out"} 504487933 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="1xx"} 0 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="2xx"} 25577 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="3xx"} 6 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="4xx"} 4 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="unknow",code="5xx"} 0 nginx_vts_filter_request_seconds_total{filter="a.example.com",filter_name="unknow"} 0.000 nginx_vts_filter_request_seconds{filter="a.example.com",filter_name="unknow"} 0.000 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="miss"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="bypass"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="expired"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="stale"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="updating"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="revalidated"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="hit"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="unknow",status="scarce"} 0 nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="四川省",direction="in"} 3647380 nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="四川省",direction="out"} 5444493 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="1xx"} 0 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="2xx"} 12939 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="3xx"} 0 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="4xx"} 6441 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="四川省",code="5xx"} 0 nginx_vts_filter_request_seconds_total{filter="a.example.com",filter_name="四川省"} 0.000 nginx_vts_filter_request_seconds{filter="a.example.com",filter_name="四川省"} 0.000 nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="0.100"} 19380 nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="0.200"} 19380 nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="0.500"} 19380 nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="1.000"} 19380 nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="3.000"} 19380 nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="5.000"} 19380 nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="10.000"} 19380 nginx_vts_filter_request_duration_seconds_bucket{filter="a.example.com",filter_name="四川省",le="+Inf"} 19380 nginx_vts_filter_request_duration_seconds_sum{filter="a.example.com",filter_name="四川省"} 0.000 nginx_vts_filter_request_duration_seconds_count{filter="a.example.com",filter_name="四川省"} 19380 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="miss"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="bypass"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="expired"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="stale"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="updating"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="revalidated"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="hit"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="四川省",status="scarce"} 0 nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="overseas",direction="in"} 2160 nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="overseas",direction="out"} 4680 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="1xx"} 0 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="2xx"} 18 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="3xx"} 0 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="4xx"} 0 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="overseas",code="5xx"} 0 nginx_vts_filter_request_seconds_total{filter="a.example.com",filter_name="overseas"} 0.000 nginx_vts_filter_request_seconds{filter="a.example.com",filter_name="overseas"} 0.000 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="miss"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="bypass"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="expired"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="stale"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="updating"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="revalidated"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="hit"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="overseas",status="scarce"} 0 nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="江西",direction="in"} 14156529 nginx_vts_filter_bytes_total{filter="a.example.com",filter_name="江西",direction="out"} 19690820 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="1xx"} 0 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="2xx"} 74493 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="3xx"} 0 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="4xx"} 1000 nginx_vts_filter_requests_total{filter="a.example.com",filter_name="江西",code="5xx"} 0 nginx_vts_filter_request_seconds_total{filter="a.example.com",filter_name="江西"} 0.000 nginx_vts_filter_request_seconds{filter="a.example.com",filter_name="江西"} 0.000 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="miss"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="bypass"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="expired"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="stale"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="updating"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="revalidated"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="hit"} 0 nginx_vts_filter_cache_total{filter="a.example.com",filter_name="江西",status="scarce"} 0 ``` |
 |
9
davidyin 2023-05-02 18:25:24 +08:00 via Android
Awstat
|
 |
10
eroko 2023-05-02 19:21:10 +08:00
我们用的是 loki
|
 |
13
changdig 2023-05-03 22:04:13 +08:00
我是简单写了个 py 脚本去处理 nginx 日志然后写库或者写 csv 文件
|
 |
14
killva4624 2023-05-04 10:50:38 +08:00
elk 或者 loki ，把日志结构化入库分析。
|
Select Language