- A+
所属分类：linux技术
一、CentOS 7.9 安装 elasticsearch-7.8.1
地址
- https://github.com/medcl/elasticsearch-analysis-ik
- https://www.elastic.co/guide/en/logstash/7.8/index.html
二、安装准备
JDK 11+：由于elasticsearch 运行需要JDK环境，如果 JDK 低于 1.8 会有一些问题。
future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk1.8.0_333/jre] does not meet this requirement
版本需要和elasticserach版本一致，否则会报错
三、安装ElasticSearch
1 解压
tar -zxvf /opt/software/elastic/elasticsearch-7.8.1-linux-x86_64.tar.gz -C /opt/elastic/2 进入elasticsearch目录中
目录中自带了jdk
3 修改配置：jvm.options
因为默认的jvm参数-Xms和Xmx都是1G。自己搭的虚拟机可能内存分配不多，会导致启动失败，根据自己的机器情况合理分配。
vim /opt/elastic/elasticsearch-7.8.1/config/jvm.options ################################################################ # Xms represents the initial size of total heap space # Xmx represents the maximum size of total heap space -Xms512m -Xmx512m ################################################################4 修改配置：elasticsearch.yml
vim /opt/elastic/elasticsearch-7.8.1/config/elasticsearch.yml # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /opt/elastic/es/data # 数据存放位置 # # Path to log files: # path.logs: /opt/elastic/es/logs # 日志存放位置 # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 192.168.0. # 开放外网访问 # # Set a custom port for HTTP: # http.port: 9200 # 设置端口 # # For more information, consult the network module documentation.
五、ES尝试启动
1 require 必须 Java 11
2 运行elasticsearch不能使用root用户启动
查看代码
[root@ecs-65685 bin]# /opt/elastic/elasticsearch-7.8.1/bin/elasticsearch future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk1.8.0_333/jre] does not meet this requirement future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk1.8.0_333/jre] does not meet this requirement [2022-10-11T17:40:42,664][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ecs-65685] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.1.jar:7.8.1] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.1.jar:7.8.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.1.jar:7.8.1] Caused by: java.lang.RuntimeException: can not run elasticsearch as root at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.1.jar:7.8.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.1.jar:7.8.1] ... 6 more uncaught exception in thread [main] java.lang.RuntimeException: can not run elasticsearch as root at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) at org.elasticsearch.cli.Command.main(Command.java:90) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) For complete error details, refer to the log at /opt/elastic/es/logs/elasticsearch.log [root@ecs-65685 bin]# 2022-10-11 09:40:42,993994 UTC [6056] ERROR CLogger.cc@311 Cannot log to named pipe /tmp/elasticsearch-1369077120224648390/controller_log_5893 as it could not be opened for writing 2022-10-11 09:40:42,994085 UTC [6056] INFO Main.cc@104 Parent process died - ML controller exiting
3 创建一个用户
groupadd es # 创建一个组 useradd es -g es # 用户属于es组 passwd es # 密码 # R（递归）分配目录下所有文件/目录的拥有者 chown -R es:es /opt/elastic/elasticsearch-7.8.1 # 授权存储数据权限 chmod -R 777 /opt/elastic/es
4 再次启动es
/opt/elastic/elasticsearch-7.8.1/bin/elasticsearch 
[1] 说你的系统太low了，给爷增加点增加虚拟内存，至少262144
vim /etc/sysctl.conf vm.max_map_count=655360 # 使设置的虚拟内存生效 [root@ecs-65685 config]# sysctl -p vm.swappiness = 0 net.core.somaxconn = 1024 net.ipv4.tcp_max_tw_buckets = 5000 net.ipv4.tcp_max_syn_backlog = 1024 vm.max_map_count = 655360
[2] 解决：在elasticsearch的config目录下，修改 elasticsearch.yml配置文件
# 取消注释，并保留一个节点 # skipping cluster bootstrapping as local node does not match bootstrap requirements: [node-1] cluster.initial_master_nodes: ["node-1"]
5 再次启动es
/opt/elastic/elasticsearch-7.8.1/bin/elasticsearch /opt/elastic/elasticsearch-7.8.1/bin/elasticsearch -d6 访问
curl 192.168.0...:9200
公网地址：114.116...:9200
注意：放行9200
五、Kibana安装
1 下载
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.8.1-linux-x86_64.tar.gz2 解压
tar -zxvf /opt/software/elastic/kibana-7.8.1-linux-x86_64.tar.gz -C /opt/elastic/3 配置 kibana.yml
vim /opt/elastic/kibana-7.8.1-linux-x86_64/config/kibana.yml # 端口 server.port: 5601 # 开放外网访问 server.host: "0.0.0.0" # es的服务器 elasticsearch.hosts: ["http://192.168.0...:9200"]
4 启动
查看代码
# 如果是要用root用户启动 就在后面加 --allow-root ，要么就切换用户执行 [root@ecs-65685 bin]# /opt/elastic/kibana-7.8.1-linux-x86_64/bin/kibana Kibana should not be run as root. Use --allow-root to continue. [root@ecs-65685 bin]# /opt/elastic/kibana-7.8.1-linux-x86_64/bin/kibana --allow-root log [17:31:37.776] [warning][plugins-discovery] Expect plugin "id" in camelCase, but found: apm_oss log [17:31:37.787] [warning][plugins-discovery] Expect plugin "id" in camelCase, but found: triggers_actions_ui log [17:32:03.147] [info][plugins-service] Plugin "visTypeXy" is disabled. log [17:32:03.148] [info][plugins-service] Plugin "endpoint" is disabled. log [17:32:03.148] [info][plugins-service] Plugin "ingestManager" is disabled. log [17:32:03.149] [info][plugins-service] Plugin "lists" is disabled. log [17:32:05.837] [warning][config][deprecation] Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0." log [17:32:05.904] [info][plugins-system] Setting up [94] plugins: [taskManager,licensing,observability,eventLog,encryptedSavedObjects,code,usageCollection,ossTelemetry,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaLegacy,devTools,translations,uiActions,statusPage,share,newsfeed,mapsLegacy,mapsLegacyLicensing,kibanaUtils,kibanaReact,inspector,embeddable,advancedUiActions,embeddableEnhanced,drilldowns,indexPatternManagement,esUiShared,discover,charts,bfetch,expressions,data,home,cloud,console,consoleExtensions,apm_oss,searchprofiler,painlessLab,grokdebugger,management,upgradeAssistant,reporting,licenseManagement,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,watcher,advancedSettings,telemetryManagementSection,fileUpload,dataEnhanced,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,features,security,snapshotRestore,transform,ingestPipelines,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,inputControlVis,savedObjects,navigation,lens,graph,maps,visualize,dashboard,dashboardEnhanced,savedObjectsManagement,spaces,actions,case,alerting,alertingBuiltins,triggers_actions_ui,infra,monitoring,logstash,uptime,ml,siem,apm] log [17:32:05.914] [warning][config][encryptedSavedObjects][plugins] Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml log [17:32:06.058] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml log [17:32:06.059] [warning][config][plugins][security] Session cookies will be transmitted over insecure connections. This is not recommended. log [17:32:06.109] [warning][actions][actions][plugins] APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml. log [17:32:06.122] [warning][alerting][alerting][plugins][plugins] APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml. log [17:32:06.149] [info][monitoring][monitoring][plugins] config sourced from: production cluster log [17:32:06.150] [warning][monitoring][monitoring][plugins] X-Pack Monitoring Cluster Alerts will not be available: undefined log [17:32:06.188] [info][crossClusterReplication][plugins] Your basic license does not support crossClusterReplication. Please upgrade your license. log [17:32:06.189] [info][plugins][watcher] Your basic license does not support watcher. Please upgrade your license. log [17:32:06.196] [info][kibana-monitoring][monitoring][monitoring][plugins] Starting monitoring stats collection log [17:32:06.393] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations... log [17:32:06.412] [info][savedobjects-service] Starting saved objects migrations log [17:32:06.447] [info][savedobjects-service] Creating index .kibana_task_manager_1. log [17:32:06.450] [info][savedobjects-service] Creating index .kibana_1. log [17:32:06.899] [info][savedobjects-service] Pointing alias .kibana_task_manager to .kibana_task_manager_1. log [17:32:06.947] [info][savedobjects-service] Pointing alias .kibana to .kibana_1. log [17:32:06.999] [info][savedobjects-service] Finished in 552ms. log [17:32:07.022] [info][savedobjects-service] Finished in 577ms. log [17:32:07.025] [info][plugins-system] Starting [72] plugins: [taskManager,licensing,observability,eventLog,encryptedSavedObjects,code,usageCollection,ossTelemetry,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaLegacy,translations,share,discover,bfetch,expressions,data,home,cloud,console,consoleExtensions,apm_oss,searchprofiler,painlessLab,grokdebugger,management,upgradeAssistant,reporting,licenseManagement,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,watcher,advancedSettings,fileUpload,dataEnhanced,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries,rollup,visTypeTimelion,features,security,snapshotRestore,transform,ingestPipelines,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,inputControlVis,lens,graph,visualize,dashboard,savedObjectsManagement,spaces,actions,case,alerting,alertingBuiltins,infra,monitoring,logstash,uptime,ml,siem,apm] log [17:32:09.117] [info][status][plugin:kibana@7.8.1] Status changed from uninitialized to green - Ready log [17:32:09.121] [info][status][plugin:elasticsearch@7.8.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch log [17:32:09.121] [info][status][plugin:elasticsearch@7.8.1] Status changed from yellow to green - Ready log [17:32:09.123] [info][status][plugin:xpack_main@7.8.1] Status changed from uninitialized to green - Ready log [17:32:09.133] [info][status][plugin:monitoring@7.8.1] Status changed from uninitialized to green - Ready log [17:32:09.136] [warning][plugins][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in kibana.yml log [17:32:09.138] [warning][plugins][reporting] Chromium sandbox provides an additional layer of protection, but is not supported for Linux Centos 7.9.2009 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'. log [17:32:09.139] [warning][reporting] Enabling the Chromium sandbox provides an additional layer of protection. log [17:32:10.356] [info][status][plugin:reporting@7.8.1] Status changed from uninitialized to green - Ready log [17:32:10.383] [info][status][plugin:spaces@7.8.1] Status changed from uninitialized to green - Ready log [17:32:10.385] [info][status][plugin:security@7.8.1] Status changed from uninitialized to green - Ready log [17:32:10.391] [info][status][plugin:dashboard_mode@7.8.1] Status changed from uninitialized to green - Ready log [17:32:10.393] [info][status][plugin:beats_management@7.8.1] Status changed from uninitialized to green - Ready log [17:32:10.438] [info][status][plugin:maps@7.8.1] Status changed from uninitialized to green - Ready log [17:32:10.446] [info][plugins][taskManager][taskManager] TaskManager is identified by the Kibana UUID: 66a6ab9a-0339-47c1-9fa8-ef059a59152e log [17:32:10.449] [info][status][plugin:task_manager@7.8.1] Status changed from uninitialized to green - Ready log [17:32:10.451] [info][status][plugin:encryptedSavedObjects@7.8.1] Status changed from uninitialized to green - Ready log [17:32:10.454] [info][status][plugin:apm_oss@7.8.1] Status changed from uninitialized to green - Ready log [17:32:10.456] [info][status][plugin:console_legacy@7.8.1] Status changed from uninitialized to green - Ready log [17:32:10.459] [info][status][plugin:region_map@7.8.1] Status changed from uninitialized to green - Ready log [17:32:10.464] [info][status][plugin:ui_metric@7.8.1] Status changed from uninitialized to green - Ready log [17:32:10.470] [info][listening] Server running at http://192.168.0...:5601 log [17:32:10.901] [info][server][Kibana][http] http server running at http://192.168.0...:56015 浏览器访问：IP:5601
六、elasticsearch-analysis-ik 分词器插件安装
作用：如果直接使用Elasticsearch的分词器在处理中文内容的搜索时，ES会将中文词语分成一个一个的汉字。当用Kibana作图，按照term来分组的时候，也会将一个汉字单独分成一组。这对于我们的使用是及其不方便的，因此我们引入es之中文的分词器插件es-ik就能解决这个问题。
1 elasticsearch-analysis-ik分词器下载地址：https://github.com/medcl/elasticsearch-analysis-ik/releases
2 创建分词器目录
分词器安装在ES目录下即可，所有在ES目录下创建分词器目录
mkdir /opt/elastic/elasticsearch-7.8.1/plugins/ik3 解压安装包
unzip /opt/software/elastic/elasticsearch-analysis-ik-7.8.1.zip -d /opt/elastic/elasticsearch-7.8.1/plugins/ik/4 重新对ES目录授权
chown -R es:es /opt/elastic/elasticsearch-7.8.1
5 关闭ES和Kibana
可以先通过Kill命令杀掉ES和Kibana进程
ps -ef | grep elastic kill -96 启动
/opt/elastic/elasticsearch-7.8.1/bin/elasticsearch -d /opt/elastic/kibana-7.8.1-linux-x86_64/bin/kibana此时的ES日志是可以查看到是有加载分词器插件的
七、Logstash安装
1 下载
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.8.1.tar.gz2 解压
tar -zxvf /opt/software/elastic/logstash-7.8.1.tar.gz -C /opt/elastic/3 配置 logstash.yml
vim /opt/elastic/logstash-7.8.1/config/logstash.yml
系统配置不高可以适当减少
4 启动
- -t 测试配置文件是否正确。
-
-f 指定logstash的配置文件。
-
-e 格式是后面跟字符串，这个字符串就被当做是logstash的配置，如果""，那么默认使用stdin作为输出。
logstash启动命令位于安装路径的bin目录中，直接启动会报错，需要按照如下方式提供参数。
命令行启动日志中看到 "Successfully started Logstash API endpoint {:port=>9600}" ，就说明启动成功。
上面命令行定义的是一个标准输入插件（即 stdin）和一个标准输出插件（即 stdout）。
意思就是从命令行提取输入，并从命令行直接将提取的数据输出，可以测试下，启动后如下图，界面在等待输入。
/opt/elastic/logstash-7.8.1/bin/logstash -e "input {stdin {}} output {stdout {}}"
5 连接 Elasticsearch
logstash可以通过命令行并使用 -e 参数传入配置字符串，指定了标准输入 stdin 插件和 stdout 插件。
但在实际应用中，通常使用配置文件指定插件，配置文件的语法形式与命令行相同，要使用的插件名称来指定，一般配置文件均放置到部署目录的config目录中，配置中找到名为 logstash-sample.conf 的文件，也可新建配置文件
启动配置文件 logstash-sample.conf
vim /opt/elastic/logstash-7.8.1/config/logstash-sample.conf input { file { type => "systemlog-xyz" #标签 path => "/opt/elastic/logstash/logs/xyz" #采集点 start_position => "beginning" #开始收集点 stat_interval => "2" #扫描间隔时间，默认是1s，建议5s } } output { elasticsearch { hosts => ["192.168.0.98:9200"] #指定信息发送到es index => "app-xyz-log-%{+YYYY.MM.dd}" #定义格式 } file { path => "/opt/elastic/logstash/logs/xyz/xyz.log" } }6 启动命令
/opt/elastic/logstash-7.8.1/bin/logstash -f ../config/logstash-sample.conf -t /opt/elastic/logstash-7.8.1/bin/logstash -f ../config/logstash-sample.conf
八、Filebeat安装
报错
1 failed; error='Not enough space
虚拟机内存不足，修改虚拟机分配内存大小的参数是，-Xmx和-Xms
[es@ecs-65685 ~]$ /opt/elastic/elasticsearch-7.8.1/bin/elasticsearch Exception in thread "main" java.lang.RuntimeException: starting java failed with [1] output: # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 549755813888 bytes for committing reserved memory. # An error report file with more information is saved as: # logs/hs_err_pid12013.log error: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f38ee000000, 549755813888, 0) failed; error='Not enough space' (errno=12) at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:126) at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88) at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59) at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:137) at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95) [es@ecs-65685 ~]$
2 Permission denied
es用户启动 elasticsearch时，es目录没有权限：chmod -R 777 /opt/elastic/es
[es@ecs-65685 ~]$ /opt/elastic/elasticsearch-7.8.1/bin/elasticsearch 2022-10-11 23:54:04,818 main ERROR RollingFileManager (/opt/elastic/es/logs/elasticsearch_deprecation.log) java.io.FileNotFoundException: /opt/elastic/es/logs/elasticsearch_deprecation.log (Permission denied) java.io.FileNotFoundException: /opt/elastic/es/logs/elasticsearch_deprecation.log (Permission denied) at java.base/java.io.FileOutputStream.open0(Native Method) at java.base/java.io.FileOutputStream.open(FileOutputStream.java:291) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:234) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:155) 
3 logstash直接启动会报错
[root@ecs-65685 ~]# /opt/elastic/logstash-7.8.1/bin/logstash OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/opt/elastic/logstash-7.8.1/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long) WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Sending Logstash logs to /opt/elastic/logstash-7.8.1/logs which is now configured via log4j2.properties ERROR: Pipelines YAML file is empty. Location: /opt/elastic/logstash-7.8.1/config/pipelines.yml usage: bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG] bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG] bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG] bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace] bin/logstash -V [--log.level fatal|error|warn|info|debug|trace] bin/logstash --help [2022-10-12T14:31:11,184][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
