通过Nginx采集后台服务接口访问日志

通过Nginx采集后台服务接口访问日志

采用的是openresty1.15版本。

步骤一，Nginx配置

在nginx的nginx.conf中配置以下参数：

• 日志格式化配置

  在http或https配置中添加log_format，将访问格式化为json结构。如下：

   log_format operatelog escape=json '{"log_id":"$logid","user_id":"$http_userId","system_id":"$http_systemId","client_ip":"$remote_addr","server_ip":"$server_addr","proxypass":"$upstream_addr","request_time":$sibat_timestamp,"request_method":"$request_method","request_status":$status,"cost_time":"$upstream_response_time","user_agent":"$http_user_agent","request_uri":"$uri","referer":"$http_referer","request_param":"$request_body"}';
  

  其中http_userId和http_systemId是来自于解析前端header携带的token。

  • 配置日志记录ID

  在server配置中添加以下

  set_by_lua $operate_timestamp "return tonumber(ngx.var.msec)*1000";
  set_by_lua $operate_rdm "return math.random(100000000,os.time())";
  set $logid "$operate_timestamp:$operate_rdm:$remote_addr";
  

  完整示例：

   server {
          listen       80;
          server_name  localhost;
          access_log  logs/access.log  main;
         
          set_by_lua $operate_timestamp "return tonumber(ngx.var.msec)*1000";
          set_by_lua $operate_rdm "return math.random(100000000,os.time())";
          set $logid "$operate_timestamp:$operate_rdm:$remote_addr";        
   
          location / {
              root   html;
              index  index.html index.htm;
          }
   }
  

• 配置location

  在location下添加logId和access_log

  proxy_set_header logId $logid;
  access_log /data/nginx/logs/operate.log operatelog ;
  

  完整示例：

          location ~/api/test/ {
                  proxy_pass http://127.0.0.1:7000;
                  add_header Access-Control-Allow-Origin *;
                  
  				proxy_set_header logId $logid;
                  access_log /data/nginx/logs/operate.log operatelog ;
          }
  

步骤二，日志切分脚本

• 编辑Nginx日志切分脚本split_log.sh

  #!/bin/bash  
  
  # Nginx 日志文件所在的目录  
  BASE_PATH=/data/nginx/logs
  
  ## Nginx 分割后日志所在位置
  SPLIT_PATH=${BASE_PATH}/split
  
  # 获取当前时间
  TIME=$(date +"%Y%m%d%H%M%S")
  # 创建存放分割文件的文件夹
  if [ ! -d "${SPLIT_PATH}" ]; then
          echo "创建文件夹"
  	mkdir ${SPLIT_PATH}
  fi
  ## 移动文件  
  mv ${BASE_PATH}/operate.log ${SPLIT_PATH}/access_${TIME}.log
  
  ## 向 Nginx 主进程发送 USR1 信号。USR1 信号是重新打开日志文件
  kill -USR1 $(cat /usr/local/openresty/nginx/logs/nginx.pid)
  

• 设置Nginx日志切分脚本定时执行

  执行命令crontab -e，添加

  */1 * * * * sh /data/nginx/logs/split_log.sh
  

步骤三，部署日志读取存储程序

监测/data/nginx/logs/split/文件夹下的文件创建事件，读取处理json数据文件存储到数据库。
数据库表示例

 create table public.tbl_nginx_access_log
  (
      log_id         varchar(256),
      user_id        varchar(32),
      system_id      varchar(32),
      client_ip      varchar(64),
      server_ip      varchar(64),
      proxypass      varchar(64),
      request_time   bigint,
      request_method varchar(32),
      request_status varchar(32),
      cost_time      varchar(32),
      user_agent     varchar(1024),
      request_uri    varchar(1024),
      referer        varchar(1024),
      request_param  text
  );
  comment on table public.tbl_nginx_access_log is 'nginx访问日志';
  comment on column public.tbl_nginx_access_log.log_id is '日志ID';
  comment on column public.tbl_nginx_access_log.user_id is '用户ID';
  comment on column public.tbl_nginx_access_log.system_id is '系统ID';
  comment on column public.tbl_nginx_access_log.client_ip is '客户端IP';
  comment on column public.tbl_nginx_access_log.server_ip is '服务端IP';
  comment on column public.tbl_nginx_access_log.proxypass is '代理服务IP';
  comment on column public.tbl_nginx_access_log.request_time is '访问时间';
  comment on column public.tbl_nginx_access_log.request_method is '请求方法';
  comment on column public.tbl_nginx_access_log.request_status is '请求状态';
  comment on column public.tbl_nginx_access_log.cost_time is '耗时';
  comment on column public.tbl_nginx_access_log.user_agent is '客户端信息';
  comment on column public.tbl_nginx_access_log.request_uri is '请求接口';
  comment on column public.tbl_nginx_access_log.referer is '请求来源';
  comment on column public.tbl_nginx_access_log.request_param is '请求参数';