docker 安装elk(单节点)
创建docker网络
docker network create -d bridge elastic
拉取elasticsearch 8.4.3版本
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.4.3 也可能是这个 docker pull elasticsearch:8.4.3
第一次执行docker脚本
docker run -it \ -p 9200:9200 \ -p 9300:9300 \ --name elasticsearch \ --net elastic \ -e es_java_opts="-xms1g -xmx1g" \ -e "discovery.type=single-node" \ -e lang=c.utf-8 \ -e lc_all=c.utf-8 \ elasticsearch:8.4.3
注意第一次执行脚本不要加-d这个参数，否则看不到服务首次运行时生成的随机密码和随机 enrollment token
拷贝日志中的以下内容，备用
✅ elasticsearch security features have been automatically configured! ✅ authentication is enabled and cluster connections are encrypted. ℹ️ password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`): =hjjcu=tj1ordtljbwpv ℹ️ http ca certificate sha-256 fingerprint: 9204867e59a004b04c44a98d93c4609937ce3f14175a3eed7afa98ee31bbd4c2 ℹ️ configure kibana to use this cluster: • run kibana and click the configuration link in the terminal when kibana starts. • copy the following enrollment token and paste it into kibana in your browser (valid for the next 30 minutes): eyj2zxiioii4ljqumyisimfkcii6wyixnziumjiumc4yojkymdaixswizmdyijoiotiwndg2n2u1owewmdrimdrjndrhothkotnjndywotkzn2nlm2yxnde3nwezzwvkn2fmytk4zwuzmwjizdrjmiisimtlesi6img0bgnvskfcykjnr1bqqxrtb3vzonpccjzqmutzvfhhb1vds2paazrhrhcifq== ℹ️ configure other nodes to join this cluster: • copy the following enrollment token and start new elasticsearch nodes with `bin/elasticsearch --enrollment-token` (valid for the next 30 minutes): eyj2zxiioii4ljqumyisimfkcii6wyixnziumjiumc4yojkymdaixswizmdyijoiotiwndg2n2u1owewmdrimdrjndrhothkotnjndywotkzn2nlm2yxnde3nwezzwvkn2fmytk4zwuzmwjizdrjmiisimtlesi6imhzbgnvskfcykjnr1bqqxrtb3vlojrzwlfkn1jiuk5pcvjqztlsx2p6lxcifq== if you're running in docker, copy the enrollment token and run: `docker run -e "enrollment_token= " docker.elastic.co/elasticsearch/elasticsearch:8.4.3`
创建相应目录并复制配置文件到主机
mkdir -p apps/elk8.4.3/elasticsearch # 这个cp命令是在 /home/ubuntu目录下执行的 docker cp elasticsearch:/usr/share/elasticsearch/config apps/elk8.4.3/elasticsearch/ docker cp elasticsearch:/usr/share/elasticsearch/data apps/elk8.4.3/elasticsearch/ docker cp elasticsearch:/usr/share/elasticsearch/plugins apps/elk8.4.3/elasticsearch/ docker cp elasticsearch:/usr/share/elasticsearch/logs apps/elk8.4.3/elasticsearch/
删除容器
docker rm -f elasticsearch
修改apps/elk8.4.3/elasticsearch/config/elasticsearch.yml
vim apps/elk8.4.3/elasticsearch/config/elasticsearch.yml
添加
- 增加：xpack.monitoring.collection.enabled: true
- 说明：添加这个配置以后在kibana中才会显示联机状态，否则会显示脱机状态
启动elasticsearch
docker run -it \ -d \ -p 9200:9200 \ -p 9300:9300 \ --name elasticsearch \ --net elastic \ -e es_java_opts="-xms1g -xmx1g" \ -e "discovery.type=single-node" \ -e lang=c.utf-8 \ -e lc_all=c.utf-8 \ -v /home/ubuntu/apps/elk8.4.3/elasticsearch/config:/usr/share/elasticsearch/config \ -v /home/ubuntu/apps/elk8.4.3/elasticsearch/data:/usr/share/elasticsearch/data \ -v /home/ubuntu/apps/elk8.4.3/elasticsearch/plugins:/usr/share/elasticsearch/plugins \ -v /home/ubuntu/apps/elk8.4.3/elasticsearch/logs:/usr/share/elasticsearch/logs \ elasticsearch:8.4.3
启动验证
https://xxxxx:9200/ 用户名:elastic 密码在第一次启动时保存下来的信息中查找
kibana
安装kibana
docker pull kibana:8.4.3
启动kibana
docker run -it \ --restart=always \ --log-driver json-file \ --log-opt max-size=100m \ --log-opt max-file=2 \ --name kibana \ -p 5601:5601 \ --net elastic \ kibana:8.4.3
初始化kibana鉴权凭证
http://xxxx:5601/?code=878708
注意：
在textarea中填入之前elasticsearch生成的相关信息,注意这个token只有30分钟的有效期，如果过期了只能进入容器重置token,进入容器执行
/bin/elasticsearch-create-enrollment-token -s kibana --url "https://127.0.0.1:9200"
kibana验证
将服务端的log中输出的验证码，输入到浏览器中，我这里是628503
创建kibana目录并copy相关配置信息
mkdir apps/elk8.4.3/kibana # 这个cp命令是在 /home/ubuntu目录下执行的 docker cp kibana:/usr/share/kibana/config apps/elk8.4.3/kibana/ docker cp kibana:/usr/share/kibana/data apps/elk8.4.3/kibana/ docker cp kibana:/usr/share/kibana/plugins apps/elk8.4.3/kibana/ docker cp kibana:/usr/share/kibana/logs apps/elk8.4.3/kibana/ sudo chown -r 1000:1000 apps/elk8.4.3/kibana
修改apps/elk8.4.3/kibana/config/kibana.yml
### >>>>>>> backup start: kibana interactive setup (2024-03-25t07:30:11.689z)
#
# ** this is an auto-generated file **
#
# default kibana configuration for docker target
#server.host: "0.0.0.0"
#server.shutdowntimeout: "5s"
#elasticsearch.hosts: [ "http://elasticsearch:9200" ]
#monitoring.ui.container.elasticsearch.enabled: true
### >>>>>>> backup end: kibana interactive setup (2024-03-25t07:30:11.689z)
# this section was automatically generated during setup.
i18n.locale: "zh-cn"
server.host: 0.0.0.0
server.shutdowntimeout: 5s
# #这个ip一定是elasticsearch的容器ip，可使用docker inspect | grep -i ipaddress
elasticsearch.hosts: ['https://your ip:9200']
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.serviceaccounttoken: aaeaawvsyxn0awmva2liyw5hl2vucm9sbc1wcm9jzxnzlxrva2vulte3mteznte4mta5ndm6zhz1r3m5cv9rrlc2nmq3de9wawm0qq
elasticsearch.ssl.certificateauthorities: [/usr/share/kibana/data/ca_1711351811685.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://your ip:9200'], ca_trusted_fingerprint: 5e7d9fe48c485c2761f9e7a99b9d5737e4e34dc55b9bf6929d929fb34d61a11a}]
删除容器并重启
docker rm -f kibana docker run -it \ -d \ --restart=always \ --log-driver json-file \ --log-opt max-size=100m \ --log-opt max-file=2 \ --name kibana \ -p 5601:5601 \ --net elastic \ -v /home/ubuntu/apps/elk8.4.3/kibana/config:/usr/share/kibana/config \ -v /home/ubuntu/apps/elk8.4.3/kibana/data:/usr/share/kibana/data \ -v /home/ubuntu/apps/elk8.4.3/kibana/plugins:/usr/share/kibana/plugins \ -v /home/ubuntu/apps/elk8.4.3/kibana/logs:/usr/share/kibana/logs \ kibana:8.4.3
logstash
logstash拉取镜像
docker pull logstash:8.4.3
启动
docker run -it \ -d \ --name logstash \ -p 9600:9600 \ -p 5044:5044 \ --net elastic \ logstash:8.4.3
创建目录并同步配置文件
mkdir apps/elk8.4.3/logstash # 这个cp命令是在 /home/ubuntu目录下执行的 docker cp logstash:/usr/share/logstash/config apps/elk8.4.3/logstash/ docker cp logstash:/usr/share/logstash/pipeline apps/elk8.4.3/logstash/ sudo cp -rf apps/elk8.4.3/elasticsearch/config/certs apps/elk8.4.3/logstash/config/certs sudo chown -r 1000:1000 apps/elk8.4.3/logstash
修改配置apps/elk8.4.3/logstash/config/logstash.yml
http.host: "0.0.0.0" xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.hosts: [ "http://your ip:9200" ] xpack.monitoring.elasticsearch.username: "elastic" # 第一次启动elasticsearch是保存的信息中查找 l3wkr6rotik_dbqzbr8c xpack.monitoring.elasticsearch.password: "l3wkr6rotik_dbqzbr8c" xpack.monitoring.elasticsearch.ssl.certificate_authority: "/usr/share/logstash/config/certs/http_ca.crt" # 第一次启动elasticsearch是保存的信息中查找 5e7d9fe48c485c2761f9e7a99b9d5737e4e34dc55b9bf6929d929fb34d61a11a xpack.monitoring.elasticsearch.ssl.ca_trusted_fingerprint: "5e7d9fe48c485c2761f9e7a99b9d5737e4e34dc55b9bf6929d929fb34d61a11a"
修改配置apps/elk8.4.3/logstash/pipeline/logstash.conf
input {
beats {
port => 5044
}
}
filter {
date {
# 因为我的日志里，我的time字段格式是2024-03-14t15:34:03 08:00 ，所以要使用以下两行配置
match => [ "time", "iso8601" ]
target => "@timestamp"
}
json {
source => "message"
}
mutate {
remove_field => ["message", "path", "version", "@version", "agent", "cloud", "host", "input", "log", "tags", "_index", "_source", "ecs", "event"]
}
}
output {
elasticsearch {
hosts => ["https://your ip:9200"]
index => "douyin-%{ yyyy.mm.dd}"
ssl => true
ssl_certificate_verification => false
cacert => "/usr/share/logstash/config/certs/http_ca.crt"
ca_trusted_fingerprint => "第一次启动elasticsearch是保存的信息中查找e924551c1453c893114a05656882eea81cb11dd87c1258f83e6f676d2428f8f2"
user => "elastic"
password => "第一次启动elasticsearch是保存的信息中查找uknx8px1yrmyiht30quc"
}
}
删除容器并重新启动
docker rm -f logstash docker run -it \ -d \ --name logstash \ -p 9600:9600 \ -p 5044:5044 \ --net elastic \ -v /home/ubuntu/apps/elk8.4.3/logstash/config:/usr/share/logstash/config \ -v /home/ubuntu/apps/elk8.4.3/logstash/pipeline:/usr/share/logstash/pipeline \ logstash:8.4.3
filebeat
filebeat拉取镜像
sudo docker pull elastic/filebeat:8.4.3
filebeat启动
docker run -it \ -d \ --name filebeat \ --network elastic \ -e tz=asia/shanghai \ elastic/filebeat:8.4.3 \ filebeat -e -c /usr/share/filebeat/filebeat.yml docker run -d --name filebeat \ -v /home/linyanbo/docker_data/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml \ -v /home/linyanbo/docker_data/filebeat/data:/usr/share/filebeat/data \ -v /var/logs/:/var/log \ --link elasticsearch:elasticsearch \ --network elastic \ --user root \ elastic/filebeat:8.4.3
设置开机启动
docker update elasticsearch --restart=always
配置文件
filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/logs/duty-admin/spring.log/crmduty-admin-2024-07-12.log
fields:
log_source: oh-promotion
fields_under_root: true
multiline.pattern: ^\d{4}-\d{1,2}-\d{1,2}
multiline.negate: true
multiline.match: after
scan_frequency: 5s
close_inactive: 1h
ignore_older: 24h
output.logstash:
hosts: ["your ip:5044"]
logstash.conf
input {
beats {
port => 5044
}
}
filter {
# mutate {
# split => {"message"=>" "}
# }
mutate {
add_field => {
"mm" => "%{message}"
}
}
}
output {
elasticsearch {
hosts => ["https://your ip:9200"]
#index => "duty-admin%{ yyyy.mm.dd}"
index => "duty-admin%{ yyyy}"
ssl => true
ssl_certificate_verification => false
cacert => "/usr/share/logstash/config/certs/http_ca.crt"
ca_trusted_fingerprint => "9204867e59a004b04c44a98d93c4609937ce3f14175a3eed7afa98ee31bbd4c2"
user => "elastic"
password => "=hjjcu=tj1ordtljbwpv"
}
}
output {
stdout {
codec => rubydebug
}
}
elasticsearch.yml
cluster.name: "docker-cluster" network.host: 0.0.0.0 #----------------------- begin security auto configuration ----------------------- # # the following settings, tls certificates, and keys have been automatically # generated to configure elasticsearch security features on 11-07-2024 05:54:41 # # -------------------------------------------------------------------------------- # enable security features xpack.security.enabled: true # 说明：添加这个配置以后在kibana中才会显示联机状态，否则会显示脱机状态 xpack.monitoring.collection.enabled: true xpack.security.enrollment.enabled: true # enable encryption for http api client connections, such as kibana, logstash, and agents xpack.security.http.ssl: enabled: true keystore.path: certs/http.p12 # enable encryption and mutual authentication between cluster nodes xpack.security.transport.ssl: enabled: true verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 #----------------------- end security auto configuration -------------------------
kibana.yml
### >>>>>>> backup start: kibana interactive setup (2024-07-11t06:09:35.897z)
#
# ** this is an auto-generated file **
#
# default kibana configuration for docker target
#server.host: "0.0.0.0"
#server.shutdowntimeout: "5s"
#elasticsearch.hosts: [ "http://elasticsearch:9200" ]
#monitoring.ui.container.elasticsearch.enabled: true
### >>>>>>> backup end: kibana interactive setup (2024-07-11t06:09:35.897z)
# this section was automatically generated during setup.
server.host: 0.0.0.0
server.shutdowntimeout: 5s
elasticsearch.hosts: ['https://your ip:9200']
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.serviceaccounttoken: aaeaawvsyxn0awmva2liyw5hl2vucm9sbc1wcm9jzxnzlxrva2vulte3mja2nzgxnzu2mzu6bu5rr25uquvsawexbudhq2tsodrmzw
elasticsearch.ssl.certificateauthorities: [/usr/share/kibana/data/ca_1720678175894.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://your ip:9200'], ca_trusted_fingerprint: 9204867e59a004b04c44a98d93c4609937ce3f14175a3eed7afa98ee31bbd4c2}]
总结
以上为个人经验，希望能给大家一个参考，也希望大家多多支持。
