目錄EFK1. 日志系統(tǒng)2. 部署ElasticSearch2.1 創(chuàng)建handless服務(wù)2.2 創(chuàng)建sts3. 部署kibana4. 部署ilogtail（docker-compose）4.1 編寫docker-compose4.2 配置ilogtail采集4.3 查看容器采集的日志4.4 采集
這是一個日志收集系統(tǒng)，日志收集屬于可觀測性體系
可觀測性體系
監(jiān)控
日志
鏈路追蹤
ELK現(xiàn)在用的少，原因是
- jruby（java+ruby）
- 語法復(fù)雜：重量級日志采集
- 性能差
EFK
PLG
我們這里部署的架構(gòu)是
ilogtail ---> kafka ---> logstash ---> elasticsearch ---> kibana
使用ilogtail采集日志寫入到kafka消息隊列里，再由logstash從消息隊列里讀取日志寫入到 es，最后再由kibana做展示
至于第三個環(huán)節(jié)為什么是logstash而不是ilogtail是因為，ilogtail要往es里面寫日志會需要配置es的認(rèn)證密碼，但我們是沒有給es配置用戶名和密碼的，所以采用logstash
[root@master EFK]# vim es-svc.yaml
kind: Service
apiVersion: v1
metadata:
name: elasticsearch
namespace: logging
labels:
app: elasticsearch
spec:
selector:
app: elasticsearch
clusterIP: None
ports:
- port: 9200
name: rest
- port: 9300
name: inter-node
[root@master EFK]# vim es-sts.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: es
namespace: logging
spec:
serviceName: elasticsearch
replicas: 1
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
initContainers:
- name: initc1
image: busybox
command: ["sysctl","-w","vm.max_map_count=262144"]
securityContext:
privileged: true
- name: initc2
image: busybox
command: ["sh","-c","ulimit -n 65536"]
securityContext:
privileged: true
- name: initc3
image: busybox
command: ["sh","-c","chmod 777 /data"]
volumeMounts:
- name: data
mountPath: /data
containers:
- name: elasticsearch
image: swr.cn-east-3.myhuaweicloud.com/hcie_openeuler/elasticsearch:7.17.1
resources:
limits:
cpu: 1000m
requests:
cpu: 100m
ports:
- containerPort: 9200
name: rest
protocol: TCP
- containerPort: 9300
name: inter-node
protocol: TCP
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
env:
- name: cluster.name
value: k8s-logs
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: cluster.initial_master_nodes
value: "es-0"
- name: discovery.zen.minimum_master_nodes
value: "2"
- name: discovery.seed_hosts
value: "elasticsearch"
- name: ES_JAVA_OPTS
value: "-Xms512m -Xmx512m"
- name: network.host
value: "0.0.0.0"
volumeClaimTemplates:
- metadata:
name: data
labels:
app: elasticsearch
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
應(yīng)用yaml文件
[root@master EFK]# kubectl create ns logging
[root@master EFK]# kubectl apply -f .
service/elasticsearch create
statefulset.apps/es create
[root@master EFK]# kubectl get pods -n logging
NAME READY STATUS RESTARTS AGE
es-0 1/1 Running 0 46s
pod顯示running就是部署好了
我直接將所有需要的資源放在一個yaml文件里面
apiVersion: v1
kind: ConfigMap
metadata:
namespace: logging
name: kibana-config
labels:
app: kibana
data:
kibana.yml: |
server.name: kibana
server.host: "0.0.0.0"
i18n.locale: zh-CN
elasticsearch:
hosts: ${ELASTICSEARCH_HOSTS}
---
apiVersion: v1
kind: Service
metadata:
name: kibana
namespace: logging
labels:
app: kibana
spec:
ports:
- port: 5601
type: NodePort
selector:
app: kibana
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kibana
namespace: logging
labels:
app: kibana
spec:
selector:
matchLabels:
app: kibana
template:
metadata:
labels:
app: kibana
spec:
containers:
- name: kibana
image: swr.cn-east-3.myhuaweicloud.com/hcie_openeuler/kibana:7.17.1
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 1
requests:
cpu: 1
env:
- name: ELASTICSEARCH_URL
value: http://elasticsearch:9200 # 寫handless的名字
- name: ELASTICSEARCH_HOSTS
value: http://elasticsearch:9200 # 寫handless的名字
ports:
- containerPort: 5601
volumeMounts:
- name: config
mountPath: /usr/share/kibana/config/kibana.yml
readOnly: true
subPath: kibana.yml
volumes:
- name: config
configMap:
name: kibana-config
查看端口并訪問
[root@master EFK]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch ClusterIP None 9200/TCP,9300/TCP 17m
kibana NodePort 10.104.94.122 5601:30980/TCP 4m30s
kibana的nodeport端口是30980，我們來訪問
這樣就算部署好了，接下來需要部署日志采集工具
因為Fluentd配置復(fù)雜，所以我這里采用ilogtail來采集日志
我們先使用docker-compose的方式部署，最后整個平臺搭建起來之后我們再將ilogtail部署到k8s集群里
[root@master ilogtail]# vim docker-compose.yaml
version: "3"
services:
ilogtail:
container_name: ilogtail
image: sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:2.0.4
network_mode: host
volumes:
- /:/logtail_host:ro
- /var/run:/var/run
- ./checkpoing:/usr/local/ilogtail/checkpoint
- ./config:/usr/local/ilogtail/config/local
啟動容器
[root@master ilogtail]# docker-compose up -d
[root@master ilogtail]# docker ps |grep ilogtail
eac545d4da87 sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:2.0.4 "/usr/local/ilogtail…" 10 seconds ago Up 9 seconds ilogtail
這樣容器就啟動了
[root@master ilogtail]# cd config/
[root@master config]# vim sample-stdout.yaml
enable: true
inputs:
- Type: input_file # 文件輸入類型
FilePaths:
- /logtail_host/var/log/messages
flushers:
- Type: flusher_stdout # 標(biāo)準(zhǔn)輸出流輸出類型
OnlyStdout: true
[root@master config]# docker restart ilogtail
/logtail_host/var/log/messages：這里是這個地址的原因是我們將宿主機的 / ，掛載到了容器內(nèi)的logtail_host，所以我們宿主機產(chǎn)生的日志會在容器的/logtail_host/var/log/messages這個目錄下
配置文件寫好之后我們還需要重啟容器讓他讀取配置，所以有一個restart
[root@master config]# docker logs ilogtail
2024-06-30 11:16:25 {"content":"Jun 30 19:16:22 master dockerd[1467]: time=\"2024-06-30T19:16:22.251108165+08:00\" level=info msg=\"handled exit event processID=9a8df40981b3609897794e50aeb2bde805eab8a75334266d7b5c2899f61d486e containerID=61770e8f88e3c6a63e88f2a09d2683c6ccce1e13f6d4a5b6f79cc4d49094bab4 pid=125402\" module=libcontainerd namespace=moby","__time__":"1719746182"}
2024-06-30 11:16:25 {"content":"Jun 30 19:16:23 master kubelet[1468]: E0630 19:16:23.594557 1468 kubelet_volumes.go:245] \"There were many similar errors. Turn up verbosity to see them.\" err=\"orphaned pod \\\"9d5ae64f-1341-4c15-b70f-1c8f71efc20e\\\" found, but error not a directory occurred when trying to remove the volumes dir\" numErrs=2","__time__":"1719746184"}
可以看到，宿主機的日志已經(jīng)被成功采集了，宿主機的日志會被封裝到content里，如果沒有看到輸出的日志的話， 需要進入到容器內(nèi)部查看一個叫做ilogtail.LOG的文件 ，而不能使用docker logs ilogtail
[root@master config]# cp sample-stdout.yaml docker-stdout.yaml
# 為了避免同時輸出到標(biāo)準(zhǔn)輸出而導(dǎo)致的日志雜亂，我們臨時將sample-stdout關(guān)掉
[root@master config]# cat sample-stdout.yaml
enable: false # 將這里改為false
inputs:
- Type: input_file # 文件輸入類型
FilePaths:
- /logtail_host/var/log/messages
flushers:
- Type: flusher_stdout # 標(biāo)準(zhǔn)輸出流輸出類型
OnlyStdout: true
[root@master config]# cat docker-stdout.yaml
enable: true
inputs:
- Type: service_docker_stdout
Stdout: true # 采集標(biāo)準(zhǔn)輸出
Stderr: false # 不采集錯誤輸出
flushers:
- Type: flusher_stdout
OnlyStdout: true
[root@master config]# docker restart ilogtail
ilogtail
2024-06-30 11:24:13 {"content":"2024-06-30 11:24:10 {\"content\":\"2024-06-30 11:24:07 {\\\"content\\\":\\\"2024-06-30 11:24:04.965 [INFO][66] felix/summary.go 100: Summarising 12 dataplane reconciliation loops over 1m3.4s: avg=3ms longest=12ms ()\\\",\\\"_time_\\\":\\\"2024-06-30T11:24:04.965893702Z\\\",\\\"_source_\\\":\\\"stdout\\\",\\\"_container_ip_\\\":\\\"192.168.200.200\\\",\\\"_image_name_\\\":\\\"calico/node:v3.23.5\\\",\\\"_container_name_\\\":\\\"calico-node\\\",\\\"_pod_name_\\\":\\\"calico-node-hgqzr\\\",\\\"_namespace_\\\":\\\"kube-system\\\",\\\"_pod_uid_\\\":\\\"4d0d950c-346a-4f81-817c-c19526700542\\\",\\\"__time__\\\":\\\"1719746645\\\"}\",\"_time_\":\"2024-06-30T11:24:07.968118197Z\",\"_source_\":\"stdout\",\"_container_ip_\":\"192.168.200.200\",\"_image_name_\":\"sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:2.0.4\",\"_container_name_\":\"ilogtail\",\"__time__\":\"1719746647\"}","_time_":"2024-06-30T11:24:10.971474647Z","_source_":"stdout","_container_ip_":"192.168.200.200","_image_name_":"sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:2.0.4","_container_name_":"ilogtail","__time__":"1719746650"}
能夠正�？匆娙罩揪驼f明日志采集沒有問題，接下來我們部署kafka，用來接收ilogtail的日志，注意將日志采集關(guān)掉，不然你的虛擬機磁盤很快就會滿
kafka作為消息隊列，會有消費者和生產(chǎn)者，生產(chǎn)者在這里就是ilogtail，也就是將日志寫入到kafka，消費者就是logstash，從kafka里面讀取日志寫入到es
Apache kafka是分布式的，基于發(fā)布/訂閱的容錯消息系統(tǒng)，主要特性如下
高吞吐，低延遲：可以做到每秒百萬級的吞吐量，并且延遲低(其他的消息隊列基本也都可以)
持久性，可靠性：消息會被持久化到本地磁盤，支持?jǐn)?shù)據(jù)備份防止數(shù)據(jù)丟失，并且可以配置消息有效期，以便消費者可以多次消費
kafka官方不支持docker部署，我們可以使用第三方的鏡像
version: '3'
services:
zookeeper:
image: quay.io/3330878296/zookeeper:3.8
network_mode: host
container_name: zookeeper-test
volumes:
- zookeeper_vol:/data
- zookeeper_vol:/datalog
- zookeeper_vol:/logs
kafka:
image: quay.io/3330878296/kafka:2.13-2.8.1
network_mode: host
container_name: kafka
environment:
KAFKA_ADVERTISED_HOST_NAME: "192.168.200.200"
KAFKA_ZOOKEEPER_CONNECT: "192.168.200.200:2181"
KAFKA_LOG_DIRS: "/kafka/logs"
volumes:
- kafka_vol:/kafka
depends_on:
- zookeeper
volumes:
zookeeper_vol: {}
kafka_vol: {}
[root@master kafka]# docker run -d --rm -p 9000:9000 \
-e KAFKA_BROKERCONNECT=192.168.200.200:9092 \
-e SERVER_SERVLET_CONTEXTPATH="/" \
quay.io/3330878296/kafdrop
部署好之后就可以使用web界面查看了，部署web界面的原因是我們將日志寫入到kafka之后可以直接使用web界面查看也沒有寫入進去，比kafka命令行更加的直觀
在瀏覽器輸入ip:9000
[root@master config]# cd /root/ilogtail/config
[root@master config]# cp sample-stdout.yaml kafka.yaml
[root@master config]# vim kafka.yaml
enable: true
inputs:
- Type: input_file
FilePaths:
- /logtail_host/var/log/messages
flushers:
- Type: flusher_kafka_v2
Brokers:
- 192.168.200.200:9092
Topic: KafkaTopic
[root@master config]# docker restart ilogtail
ilogtail
這個時候我們再回到web界面就會出現(xiàn)一個topic
點進去可以查看有哪些日志被寫入進去了
能看見日志就沒問題了，接下來部署logstash
logstash會從kafka讀取消息然后寫入到es里面去
[root@master ~]# mkdir logstash
[root@master ~]# cd logstash
[root@master logstash]# vim docker-compose.yaml
version: '3'
services:
logstash:
image: quay.io/3330878296/logstash:8.10.1
container_name: logstash
network_mode: host
environment:
LS_JAVA_OPTS: "-Xmx1g -Xms1g"
volumes:
- /etc/localtime:/etc/localtime:ro
- /apps/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
- /apps/logstash/pipeline:/usr/share/logstash/pipeline
- /var/log:/var/log
docker-compose寫好之后先不要著急啟動，因為我們給他掛載的配置文件還沒有啟動
現(xiàn)在編寫配置文件
[root@master logstash]# mkdir /apps/logstash/{config,pipeline}
[root@master logstash]# cd /apps/logstash/config/
[root@master config]# vim logstash.yml
pipeline.workers: 2
pipeline.batch.size: 10
pipeline.batch.delay: 5
config.reload.automatic: true
config.reload.interval: 60s
寫好這個文件之后我們啟動這個logstash容器
[root@master logstash]# /root/logstash
[root@master logstash]# docker-compose up -d
[root@master logstash]# docker ps |grep logstash
60dfde4df40d quay.io/3330878296/logstash:8.10.1 "/usr/local/bin/dock…" 2 minutes ago Up 2 minutes logstash
啟動之后就沒問題了
Logstash官方文檔地址
我們要使用logstash輸出日志到es的話就需要到pipeline里面去寫一些規(guī)則
[root@master EFK]# cd /apps/logstash/pipeline/
[root@master pipeline]# vim logstash.conf
input {
kafka {
# 指定kafka地址
bootstrap_servers => "192.168.200.200:9092"
# 從哪些topic獲取數(shù)據(jù),要寫已經(jīng)存在topic
topics => ["KafkaTopic"]
# 從哪個地方開始讀取，earliest是從頭開始讀取
auto_offset_reset => "earliest"
codec => "json"
# 當(dāng)一個logstash中有多個input插件時,建議每個插件定義一個id
# id => "kubernetes"
# group_id => "kubernetes"
}
}
filter {
json {
source => "event.original"
}
mutate {
remove_field => ["event.original","event"]
}
}
output {
elasticsearch {
hosts => ["http://192.168.200.200:9200"]
index => "kubernetes-logs-%{+YYYY.mm}"
}
}
[root@master EFK]# kubectl expose pod es-0 --type NodePort --port 9200 --target-port 9200
service/es-0 exposed
[root@master EFK]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch ClusterIP None 9200/TCP,9300/TCP 3h38m
es-0 NodePort 10.97.238.173 9200:32615/TCP 2s
kibana NodePort 10.106.1.52 5601:30396/TCP 3h38m
這里他將9200映射到了本地的32615端口，所以我們將logstash的地址改到32615
output {
elasticsearch {
hosts => ["http://192.168.200.200:32615"]
index => "kubernetes-logs-%{+YYYY.mm}"
}
}
然后重啟logstash
[root@master pipeline]# docker restart logstash
點擊stack management
點擊索引管理，會看到有索引存在就是正常
點擊索引模式，創(chuàng)建索引
機器學(xué)習(xí)：神經(jīng)網(wǎng)絡(luò)構(gòu)建（下）
閱讀華為Mate品牌盛典：HarmonyOS NEXT加持下游戲性能得到充分釋放
閱讀實現(xiàn)對象集合與DataTable的相互轉(zhuǎn)換
閱讀算法與數(shù)據(jù)結(jié)構(gòu) 1 - 模擬
閱讀5. Spring Cloud OpenFeign 聲明式 WebService 客戶端的超詳細(xì)使用
閱讀Java代理模式：靜態(tài)代理和動態(tài)代理的對比分析
閱讀Win11筆記本“自動管理應(yīng)用的顏色”顯示規(guī)則
閱讀本站所有軟件，都由網(wǎng)友上傳，如有侵犯你的版權(quán)，請發(fā)郵件[email protected]
湘ICP備2022002427號-10 湘公網(wǎng)安備：43070202000427號© 2013~2025 haote.com 好特網(wǎng)
