[日志]es的增删改查

最新推荐文章于 2026-01-15 10:37:26 发布

原创最新推荐文章于 2026-01-15 10:37:26 发布 · 2.4k 阅读

本内容遵循CC 4.0 BY-SA版权协议

这篇博客介绍了如何在Elasticsearch中进行创建索引、过滤索引日期、删除超过10天和上个月的索引等操作。内容包括使用命令行和Logstash创建索引，结合shell和grep进行索引日期过滤，以及手动安装ELK栈的最佳实践。参考了阮一峰的教程来维护节点健康和管理索引。

创建索引

命令行:

curl -XPUT 192.168.6.103:9200/customer-%{+yyyy.MM.dd}  #反馈格式错误

通过logstash

output{
    if [type] == "system"{
        elasticsearch{
            hosts => ["192.168.14.136:9200"]
            index => "system-%{+YYYY.MM.dd}"  #这样是可以的
        }
    }
}

过滤索引日期

用命令行的好处是可以结合shell grep等过滤

curl -XGET http://192.168.14.132:9200/_cat/indices | awk -F" " '{print $3}' | awk -F"-" '{print $NF}' | egrep "[0-9]*\.[0-9]*\.[0-9]*" | sort | uniq  | sed 's#.#-#g'

删除超过10天的索引

日期比较
es索引过滤
前提是es index命令以日期为后缀…

#!/bin/bash

###################################
#删除早于十天的ES集群的索引
###################################
function delete_indices() {
    comp_date=`date -d "10 day ago" +"%Y-%m-%d"`
    date1="$1 00:00:00"
    date2="$comp_date 00:00:00"

    t1=`date -d "$date1" +%s`
    t2=`date -d "$date2" +%s`

    if [ $t1 -le $t2 ]; then
        echo "$1时间早于$comp_date，进行索引删除"
        #转换一下格式，将类似2017-10-01格式转化为2017.10.01
        format_date=`echo $1| sed 's#-#\.#g'`
        curl -XDELETE http://es-cluster-ip:9200/*$format_date
    fi
}

curl -XGET http://192.168.14.132:9200/_cat/indices | awk -F" " '{print $3}' | awk -F"-" '{print $NF}' | egrep "[0-9]*\.[0-9]*\.[0-9]*" | sort | uniq  | sed 's#.#-#g' | while read LINE
do
    #调用索引删除函数
    delete_indices $LINE
done

删除上个月的索引

#!/usr/bin/env bash

## 删除上个月的索引
curl -XDELETE 'http://127.0.0.1:9200/logstash-2016-07-*'

## 删除上个月的索引
cat es-index-clear.sh
#/bin/bash
#es-index-clear
#获取上个月份日期
LAST_DATA=`date -d "last month"+%Y-%m`
#删除上个月份所有的索引
curl -XDELETE'http://127.0.0.1:9200/*-'${LAST_DATA}'-*'

crontab -e
0 1 5 * * /script/es-index-clear.sh


# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed

手动安装elk

物理机安装elk之前的优化操作
sudo sysctl -w vm.max_map_count=262144

make it persistent:
$ vim /etc/sysctl.conf
vm.max_map_count=262144



## es常用操作参考: http://www.cnblogs.com/lishouguang/p/4560930.html
## 备份,扩容等脚本,有点老,但是思路可以参考,https://github.com/gregbkr/docker-elk-cadvisor-dashboards

http://192.168.14.133:9200/_cat/health?v   #查看集群状态
http://192.168.14.133:9200/_cat/nodes?v    #查看节点状态
http://192.168.14.133:9200/_cat/indices?v  #查看index列表

#创建index
curl -XPUT http://vm1:9200/customer?pretty

#添加一个document
[es@vm1 ~]$ curl -XPUT vm1:9200/customer/external/1?pretty -d '{"name":"lisg"}'

#检索一个document
[es@vm1 ~]$ curl -XGET vm1:9200/customer/external/1?pretty

#删除一个document
[es@vm1 ~]$ curl -XDELETE vm1:9200/customer/external/1?pretty

#删除一个type
[es@vm1 ~]$ curl -XDELETE vm1:9200/customer/external?pretty

#删除一个index
[es@vm1 ~]$ curl -XDELETE vm1:9200/customer?pretty

#POST方式可以添加一个document，不用指定ID
[es@vm1 ~]$ curl -XPOST vm1:9200/customer/external?pretty -d '{"name":"zhangsan"}'

#使用doc更新document
[es@vm1 ~]$ curl -XPUT vm1:9200/customer/external/1?pretty -d '{"name":"lisg4", "age":28}'

#使用script更新document（1.4.3版本动态脚本是被禁止的）
[es@vm1 ~]$ curl -XPOST vm1:9200/customer/external/1/_update?pretty -d '{"script":"ctx._source.age += 5"}'

手动安装elk

useradd elk
cd /usr/local/src/
tar xf elasticsearch-5.6.4.tar.gz -C /usr/local/
tar xf kibana-5.6.4-linux-x86_64.tar.gz -C /usr/local/

ln -s /usr/local/elasticsearch-5.6.4 /usr/local/elasticsearch
ln -s /usr/local/kibana-5.6.4-linux-x86_64 /usr/local/kibana

chown -R elk. /usr/local/elasticsearch
chown -R elk. /usr/local/elasticsearch/
chown -R elk. /usr/local/kibana
chown -R elk. /usr/local/kibana/


mkdir /data/es/{data,logs} -p
chown -R elk. /data

修改es配置
0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"


echo 'http.cors.enabled: true' >> /usr/local/elasticsearch/config/elasticsearch.yml
echo 'http.cors.allow-origin: "*"' >> /usr/local/elasticsearch/config/elasticsearch.yml

sed -i 's#\#network.host: 192.168.0.1#network.host: 0.0.0.0#g' /usr/local/elasticsearch/config/elasticsearch.yml
sed -i 's#\#network.host: 192.168.0.1#network.host: 0.0.0.0#g' /usr/local/kibana/config/kibana.yml




修改内核:
vim /etc/security/limits.conf
*               soft    nproc           65536
*               hard    nproc           65536
*               soft    nofile          65536
*               hard    nofile          65536


sysctl -w vm.max_map_count=262144
sysctl -p

nohup /bin/su - elk -c "/usr/local/elasticsearch/bin/elasticsearch" > /data/es/es-start.log 2>&1 &
nohup /bin/su - elk -c "/usr/local/kibana/bin/kibana" > /data/es/kibana-start.log 2>&1 &

docker run -d -v /etc/localtime:/etc/localtime --restart=always -p 9100:9100 mobz/elasticsearch-head:5

最佳实战

参考: http://www.ruanyifeng.com/blog/2017/08/elasticsearch.html
- 维护:
- 节点
- 健康
- 索引

增加/更新索引/记录

  PUT  创建索引(仅固定id)
  POST 创建记录(多了随机id)

获取

  GET account/person/1
      account/person/_search

删除

  DELETE

查询:

    accounts/person/_search
    {
      "query" : { "match" : { "desc" : "管理" }},
      "from": 1, #从1开始偏移1个
      "size": 1  #默认返回10条
    }
    #或
    GET accounts/person/_search
    {
      "query": {"match": {
        "name": "maotai 2"
      }}
    }
    #且
    GET accounts/person/_search
    {
      "query": {
        "bool": {
          "must": [
            { "match": { "name": "maotai" } },
            { "match": { "name": "2" } }
          ]
        }
      }
    }

GET _cat/indices
GET _cat/health
GET _cat/nodes


如果不指定id,用POST
如果指定id,  用PUT

## 增加删除索引
PUT app1
PUT app2
DELETE app1
DELETE app1,app2,app3

## 添加记录-id随机--(以下两条属于同一个索引,但字段可以不同)
POST elc/user
{
  "username": "maotai",
  "gender": "male",
  "desc":"IT egineering...."
}


## 添加/更新记录-id不随机(多次添加属于更新)
POST elc/user/1
{
  "username": "maotai",
  "gender": "male",
  "desc": "IT egineering....",
  "part": "jishubu",
  "salary": "hello world"
}