Fluentd Fluent-bit在k8s中的日志采集

发表于 | 分类于
字数统计: 594 | 阅读时长 ≈ 3

Fluentd Fluent-bit在k8s中的日志采集

  1. fluent-bit – fluentd – kafka – elasticsearch
  2. fluentd – fluentd – kafka
  3. fluent-bit – kafka

1. fluent-bit – fluentd – kafka – elasticsearch

1.1 fluent-bit to fluentd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    warning

[INPUT]
    Name         tail
    Tag          zz-test
    Path         /work/admin/logs/*/*.log
    Db           /tmp/fluent-bit.db
    Format      json
    Mem_Buf_Limit  1MB
    Interval_Sec 2
    Refresh_Interval 2

[FILTER]
    Name record_modifier
    Match *
    Record hostname ${HOSTNAME}
    #Record product Awesome_Tool

[OUTPUT]
    Name         forward
    Match        *
    Time_Key        @timestamp
    Host         172.17.0.4
    #Host         39.97.7.185
    Port         24224
    Retry_Limit  False

1.2 fluentd to kafka

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>

<source>
  @type monitor_agent
  @id monitor_agent_input
  port 24220
</source>

#<match zz-test*>
#  @type stdout
#</match>

<filter zz-test*>
  @type parser
  format json
  key_name log
</filter>

<match zz-test*>
  @type               kafka
  brokers             kafka2.feiersmart.local:9092
  default_topic       zz-test
  default_message_key zz-test
  output_data_type    json
  max_send_retries    3
  required_acks       -1
  compression_codec   gzip
  max_send_retries    3
  <inject>
    tag_key tag
    time_key time
  </inject>
</match>

1.3 fluentd to elasticsearch

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
<source>     
  @type kafka                                                                                   
  brokers kafka2.feiersmart.local:9092,kafka3.feiersmart.local:9092,kafka1.feiersmart.local:9092
  format json
  <topic>        
    topic zz-test
  </topic>
</source>
               
<match zz-test>      
  @type elasticsearch 
  include_tag_key true
  tag_key _key       
  time_key @timestamp   
  include_timestamp true 
  reconnect_on_error true
  reload_on_failure true  
  reload_connections false
  id_key _hash                                                                                
  hosts bj-es1.feiersmart.local:9200,bj-es2.feiersmart.local:9200,bj-es3.feiersmart.local:9200
  index_name zz-test-
  type_name zz-test     
  include_timestamp true                 
  time_key_exclude_timestamp false
  utc_index false    
  request_timeout 15s   
  reload_on_failure true            
  bulk_message_request_threshold  -1
            
    <buffer>      
      @type memory     
      flush_interval 5s   
      flush_thread_count 4
      retry_type periodic
      retry_forever true
      retry_wait 10s       
      chunk_limit_size 16Mb
      queue_limit_length 1 
      total_limit_size 16Mb
    </buffer>
</match>

2. fluentd – fluentd – kafka

2.1 fluentd to fluentd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>

<source>
  @type monitor_agent
  @id monitor_agent_input
  port 24220
</source>

<match zz-test*>
  @type            forward
  #compress         gzip
  send_timeout 60s
  recover_wait 10s
  hard_timeout 60s
  <server>
  host             kafka2.feiersmart.local
  port 24224
  </server>
</match>

2.2 fluentd to kafka

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>

<source>
  @type monitor_agent
  @id monitor_agent_input
  port 24220
</source>

<filter zz-test*>
  @type parser
  format json
  key_name log
</filter>

<match zz-test*>
  @type               kafka
  brokers             kafka2.feiersmart.local:9092
  default_topic       zz-test
  default_message_key zz-test
  #output_data_type    attr:all
  output_data_type    json
  max_send_retries    3
  compression_codec   gzip
  max_send_retries    3
  <inject>
    tag_key tag
    time_key time
  </inject>
</match>

3. fluent-bit – kafka

3.1 fluent-bit to kafka

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    warning

[INPUT]
    Name         tail
    Tag          zz-test
    Path         /work/admin/logs/*/*.log
    Db           /tmp/fluent-bit.db
    Interval_Sec 2

[FILTER]
    Name    record_modifier
    Match zz-test
    Record hostname ${HOSTNAME}
    Record type zz-test

[OUTPUT]
    Name         kafka
    Match        *
    Brokers      39.97.7.185
    Port         9092
    Topics       zz-test
    Timestamp_Key @timestamp
    Retry_Limit  False
    Format       json
    rdkafka.log.connection.close false
    rdkafka.queue.buffering.max.kbytes 2048000
    #rdkafka.request.required.acks 1
-------------本文结束感谢您的阅读-------------
分享到： 收藏夹 复制网址 邮件 微信 QQ空间 腾讯微博 豆瓣 一键分享 更多