filebeat #19

  • By filebeat-charmers
  • Latest version (#19)
  • xenial, trusty, bionic
  • Stable
  • Candidate
  • Beta
  • Edge

Description

As the next-generation Logstash Forwarder, Filebeat tails logs and quickly
sends this information to Logstash for further parsing and enrichment or to
Elasticsearch for centralized storage and analysis.


Overview

Filebeat is a lightweight, open source shipper for log file data. As the
next-generation Logstash Forwarder, Filebeat tails logs and quickly sends this
information to Logstash for further parsing and enrichment or to Elasticsearch
for centralized storage and analysis.

Usage

Filebeat can be added to any principal charm thanks to the wonders of being
a subordinate charm. The following example will deploy an ubuntu log source
along with the elk stack so we can visualize our log data.

juju deploy ~elasticsearch-charmers/bundle/elk-stack
juju deploy xenial/filebeat
juju deploy xenial/ubuntu
juju add-relation filebeat:beats-host ubuntu
juju add-relation filebeat logstash

Deploying the minimal Beats formation

If you do not need log buffering and alternate transforms on data that is
being shipped to ElasticSearch, you can simply deploy the 'beats-core' bundle
which stands up Elasticsearch, Kibana, and the known working Beats
subordinate applications.

juju deploy ~containers/bundle/beats-core
juju deploy xenial/ubuntu
juju add-relation filebeat:beats-host ubuntu
juju add-relation topbeat:beats-host ubuntu

Changing what is shipped

By default, the Filebeat charm is setup to ship everything in:

/var/log/*/*.log
/var/log/*.log

If you'd rather target specific log files:

juju config filebeat logpath=/var/log/mylog.log

Testing the deployment

The applications provide extended status reporting to indicate when they are
ready:

juju status

This is particularly useful when combined with watch to track the on-going
progress of the deployment:

watch juju status

The message for each unit will provide information about that unit's state.
Once they all indicate that they are ready, you can navigate to the kibana
url and view the streamed log data from the Ubuntu host.

juju status kibana --format=yaml | grep public-address

Navigate to http://<kibana-ip>/ in a browser and begin creating your
dashboard visualizations.

Upgrading filebeat

Upgrades are handled at both the charm and apt repository levels. Use
upgrade-charm to get the latest charm code on all filebeat units:

juju upgrade-charm filebeat

Apt repositories are scanned any time the install_sources config changes. If
a new version of filebeat is found in the configured repository, juju status
will instruct operators to run the reinstall action. This action must be
run on each filebeat unit:

juju run-action --wait filebeat/0 reinstall

The reinstall action will stop the filebeat service, purge the apt package,
and reinstall the latest version available from the configured repository.

Scale Out Usage

As a subordinate charm, filebeat will scale when additional principal units are
added. For example, adding ubuntu units that are related to filebeat will
automatically install and configure filebeat for the new unit(s).

juju add-unit ubuntu

To monitor additional applications, simply relate the filebeat subordinate:

juju add-relation filebeat:beats-host my-charm

Contact Information

Need Help?

Configuration

kafka_topic
(string) Topic name. Format strings are allowed. https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html#_literal_topic_literal
%{[type]}
max_bytes
(int) Maximum number of bytes a single log event can have. Default 10MB
10485760
kafka_hosts
(string) A comma separated list of kafka output hosts in addition to those from relations.
kube_logs
(boolean) Add a prospector to ship logs from Kubernetes pods
kafka_topics
(string) Expert setting topics filter. https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html#_literal_topics_literal
fields
(string) Space seperated list of key:value that the prospector will assign as field to each beat
package_status
(string) The status of service-affecting packages will be set to this value in the dpkg database. Valid values are "install" and "hold".
install
logpath
(string) Space separated log paths to monitor. Can contain wildcards.
/var/log/*.log /var/log/*/*.log
logging_to_syslog
(boolean) Send filebeat logs to syslog https://www.elastic.co/guide/en/beats/filebeat/master/configuration-logging.html#_literal_to_syslog_literal
True
harvester_buffer_size
(int) Defines the buffer size every harvester uses when fetching the file
16384
kafka_version
(string) Expert setting kafka version. https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html#_literal_version_literal
exclude_lines
(string) A list of regular expressions to match the lines that you want Filebeat to exclude. https://www.elastic.co/guide/en/beats/filebeat/5.3/configuration-filebeat-options.html#exclude-lines
[]
extra_packages
(string) Space separated list of extra deb packages to install.
logstash_ssl_key
(string) Private SSL key data (base64 encoded) for connecting security to logstash.
logstash_ssl_cert
(string) Public SSL certificate data (base64 encoded) for connecting securely to logstash.
logstash_hosts
(string) A comma separated list of logstash output hosts in addition to those from relations.
install_keys
(string) repository key
D88E42B4
install_sources
(string) apt repository to fetch beats from
deb https://artifacts.elastic.co/packages/5.x/apt stable main
exclude_files
(string) A list of regular expressions to match the files that you want Filebeat to ignore. https://www.elastic.co/guide/en/beats/filebeat/5.3/configuration-filebeat-options.html#exclude-files
[".gz$"]