riemann.elasticsearch

Forwards events to Elasticsearch.

default-bulk-formatter

(default-bulk-formatter {:keys [es-index type es-action index-suffix]})

Returns a function which accepts an event and formats it for the Elasticsearch bulk API.

Options :

  • :es-index Elasticsearch index name (without suffix).
  • :type Type to send to index.
  • :es-action Elasticsearch action, for example “index”.
  • :index-suffix Index suffix, for example “-yyyy.MM.dd”.

Each event received by the function can also have these keys (which override default options), and an optional es-id key.

elasticsearch

(elasticsearch opts & maybe-formatter)

Returns a function which accepts an event and sends it to Elasticsearch. Custom event formatter can be provided as optional second argument.

Options:

  • :es-endpoint Elasticsearch, default is "http://127.0.0.1:9200".
  • :es-index Index name, default is “riemann”.
  • :index-suffix Index-suffix, default is “-yyyy.MM.dd”.
  • :type Type to send to index, default is “event”.
  • :username Username to authenticate with.
  • :password Password to authenticate with.

Example:

(elasticsearch
  ; ES options
  {:es-endpoint "https:example-elastic.com"
   :index-suffix "-yyyy.MM"}
  ; (optional) custom event formatter
  (fn [event]
    (let
      [newtags (concat (:tags event) ["extra-tag"])]
      (merge event {:tags newtags}))))

elasticsearch-bulk

(elasticsearch-bulk opts)

Returns a function which accepts an event (or a list of events) and sends it to Elasticsearch using the Bulk API. Custom event formatter can be provided using the :formatter key.

A formatter is a function which accepts an event.

Event time is mandatory.

Events should have this format :

{:es-action "index"
 :es-metadata {:_index "test"
               :_type "type1"
               :_id "1"}
 :es-source {:field1 "value1"}}

:es-action is the action (create, update, index, delete), :es-metadata the document metadata, and es-source the document source.

More informations about the Elasticsearch bulk API: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html

If a formatter is specified, events will be formatted using it. You can then send events not respecting the previous format if the specified formatter converts them to it.

Options:

  • :es-endpoint Elasticsearch, default is "http://127.0.0.1:9200".
  • :username Username to authenticate with.
  • :password Password to authenticate with.
  • :formatter Fn taking an event and returning it with the ES Bulk API format
  • :http-options Http options (like proxy). See https://github.com/dakrone/clj-http for option list

gen-request-bulk-body

(gen-request-bulk-body events)

Takes a list of events, generates the body request for Elasticsearch

gen-request-bulk-body-reduce

(gen-request-bulk-body-reduce result elem)

Reduction fn used in gen-request-bulk-body to generate the body request