Linux data

This page is designed to help customers leverage the Forward Events integration within their Anvilogic account for FluentBit.

Pre-Reqs

• Anvilogic account

• Snowflake data repository connected to your Anvilogic account

• FluentBit installed

Setting up FluentBit Config

1. Anvilogic will provide a S3 bucket and the corresponding access keys/ids (note these change for each integration) when you create a forward events integration in your Anvilogic deployment.

   1. 
2. Create a credential file on the machine that fluentBit can read from. For example, /home/<username>/creds . Inside the file please paste the following config with your specific access key/id

[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

3. Since our credentials are already updated in the /home/<username>/creds file, we need to configure the service config file for Fluent Bit and set the path to this credential file (see image for reference). To do that, fire up your favorite text editor and edit the fluent-bit.service file located at /usr/lib/systemd/system/fluent-bit.service.

   1. Environment="AWS_SHARED_CREDENTIALS_FILE=/home/<username>/creds"

4. Then run the following commands in a terminal window

   1. sudo systemctl daemon-reload

   2. sudo systemctl start fluent-bit

5. Next we need to configure fluentbit to read our logs and send them to S3. In this example, I will be looking at apache2 access logs and sending them to S3.

[INPUT]
    Name              tail
    Tag               apache
    Path              /var/log/apache2/access.log
    Parser            apache2
    Mem_Buf_Limit     50MB

[OUTPUT]
    Name              s3
    Match             *
    bucket            avl-raw-prod-s3-111-12345678/sdi_custom_data-0
    region            us-east-1
    use_put_object    On
    Store_dir         /tmp/fluent-bit/s3
    s3_key_format     /$TAG/%Y/%m/%d/%H/%M/%S

Once you have pasted the above config into your fluentBit.conf file (typically located at /etc/fluent-bit/fluent-bit.conf)

• NOTE: You can also edit or add any of your own customer parsers for logs by editing the parser.conf file at /etc/fluent-bit/

• Once you have edited your fluent-bit.conf, please restart the fluentBit service sudo systemctl restart fluent-bit

  • You can validate that your config is working by heading to /tmp/fluent-bit/s3/ and looking inside that folder.

6. You can now confirm that data has landed in your snowflake account.

   1. 

      

Please update the input section of this example config to fit your exact needs.