Fluentd 홈페이지 소개
https://docs.fluentd.org/installation
여러가지 방법에 대해 설명되어 있음
pre-install
NTP 서버 구축
# 기존 ntpd 제거
systemctl stop ntpd
systemctl disable ntpd
yum remove ntp
# chrony 설치
yum install chrony
# 기존 서버 목록은 주석 처리
#server 0.rhel.pool.ntp.org iburst
#server 1.rhel.pool.ntp.org iburst
#server 2.rhel.pool.ntp.org iburst
#server 3.rhel.pool.ntp.org iburst
# 한국 공용 타임서버 목록 설정
server 1.kr.pool.ntp.org
server 0.asia.pool.ntp.org
server 2.asia.pool.ntp.org
# 내부 네트워크에서 이 서버를 타임서버로 참조하기 위한 설정
allow 192.168.2.0/24
# 방화벽
firewall-cmd --add-service=ntp --permanent
firewall-cmd --reload
# chronyd 시작
systemctl start chronyd
systemctl enable chronyd
# 동기화 확인
[root@tlog-collect ~]# chronyc tracking
Reference ID : A9FEA97B (169.254.169.123)
Stratum : 4
Ref time (UTC) : Tue Nov 10 05:32:04 2020
System time : 0.000000639 seconds fast of NTP time
Last offset : +0.000000399 seconds
RMS offset : 0.000000725 seconds
Frequency : 4.559 ppm slow
Residual freq : +0.000 ppm
Skew : 0.010 ppm
Root delay : 0.000274238 seconds
Root dispersion : 0.000264288 seconds
Update interval : 32.1 seconds
Leap status : Normal
[root@tlog-collect ~]#Increase the Maximum Number of File Descriptors
/etc/security/limits.conf
root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536Optimize the Network Kernel Parameters
/etc/sysctl.conf
net.core.somaxconn = 1024
net.core.netdev_max_backlog = 5000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_wmem = 4096 12582912 16777216
net.ipv4.tcp_rmem = 4096 12582912 16777216
net.ipv4.tcp_max_syn_backlog = 8096
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535
Install RPM Package
Download and Install
# td-agent 4
$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent4.sh | sh
... ...
=========================================================================================================================================================================================================================================
Package Arch Version Repository Size
=========================================================================================================================================================================================================================================
Installing:
td-agent x86_64 4.0.1-1.el7 treasuredata 13 M
Transaction Summary
=========================================================================================================================================================================================================================================
Install 1 Package
Total download size: 13 M
Installed size: 58 M
Downloading packages:
td-agent-4.0.1-1.el7.x86_64.rpm | 13 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : td-agent-4.0.1-1.el7.x86_64 1/1
Created symlink from /etc/systemd/system/multi-user.target.wants/td-agent.service to /usr/lib/systemd/system/td-agent.service.
prelink detected. Installing /etc/prelink.conf.d/td-agent-ruby.conf ...
Verifying : td-agent-4.0.1-1.el7.x86_64 1/1
Installed:
td-agent.x86_64 0:4.0.1-1.el7
Complete!
Installation completed. Happy Logging!rpm 파일 직접 다운로드
wget http://packages.treasuredata.com.s3.amazonaws.com/4/redhat/7/x86_64/td-agent-4.1.1-1.el7.x86_64.rpm
yum localinstall ./td-agent-4.1.1-1.el7.x86_64.rpm
td-agent daemon 시작
[root@tlog-collect yum.repos.d]# systemctl start td-agent.service
[root@tlog-collect yum.repos.d]# systemctl status td-agent.service
● td-agent.service - td-agent: Fluentd based data collector for Treasure Data
Loaded: loaded (/usr/lib/systemd/system/td-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2020-11-10 06:17:09 UTC; 4s ago
Docs: https://docs.treasuredata.com/articles/td-agent
Process: 5326 ExecStart=/opt/td-agent/bin/fluentd --log $TD_AGENT_LOG_FILE --daemon /var/run/td-agent/td-agent.pid $TD_AGENT_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 5332 (fluentd)
CGroup: /system.slice/td-agent.service
├─5332 /opt/td-agent/bin/ruby /opt/td-agent/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid
└─5335 /opt/td-agent/bin/ruby -Eascii-8bit:ascii-8bit /opt/td-agent/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid --under-supervisor
Nov 10 06:17:08 tlog-collect systemd[1]: Starting td-agent: Fluentd based data collector for Treasure Data...
Nov 10 06:17:09 tlog-collect systemd[1]: Started td-agent: Fluentd based data collector for Treasure Data.
Ruby 버전 upgrade
일부 plugin 의 경우 ruby 버전이 centos 에 기본으로 깔린 2.0.0 에서는 실행이 안되는 경우가 있다 이런 경우 ruby 버전을 upgrade 하여야 한다. 다음은 ruby 업그레이드 절차이다.
dependency 설치
sudo yum install git-core zlib zlib-devel gcc-c++ patch readline readline-devel libyaml-devel libffi-devel openssl-devel make bzip2 autoconf automake libtool bison curl sqlite-devel
rbenv / ruby-build
curl -fsSL https://github.com/rbenv/rbenv-installer/raw/HEAD/bin/rbenv-installer | bash
ruby path 설정
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc
source ~/.bashrc
설치가능 버전 확인
[root@tlog-sinker ~]# rbenv install -l
2.6.7
2.7.3
3.0.1
jruby-9.2.19.0
mruby-3.0.0
rbx-5.0
truffleruby-21.1.0
truffleruby+graalvm-21.1.0default 사용 버전 설치
[root@tlog-sinker ~]# rbenv install 2.7.3| [root@tlog-sinker ~]# rbenv install 2.7.3 [root@tlog-sinker ~]# rbenv global 2.7.3 |
설치버전 확인
[root@tlog-sinker ~]# ruby --version
ruby 2.7.3p183 (2021-04-05 revision 6847ee089d) [x86_64-linux]참고
'DATA COLLECTOR > Fluentd' 카테고리의 다른 글
| Fluentd config Multi process workers (0) | 2021.10.01 |
|---|---|
| Fluentd config High Availability (0) | 2021.10.01 |
| Fluentd config source (0) | 2021.10.01 |
| Fluentd config directives, section (0) | 2021.10.01 |
| Fluentd 소개 (0) | 2021.10.01 |
