Elasticsearch Monitoring

Monitor your Elasticsearch cluster performance by knowing details on cluster status, nodes and shards details, JVM stats, and more.

Start 30-day free trial Try now, sign up in 30 seconds

Install and configure the Elasticsearch plugin to monitor the open source, distributed document store and search engine. It depends strongly on Apache Lucene, a full text search engine in Java. Keep a pulse on the performance of the Elasticsearch environment to ensure you are up to date with the internals of your working cluster.

This document details how to configure the Elasticsearch plugin and the monitoring metrics for providing in-depth visibility into the performance, availability, and usage stats of Elasticsearch clusters.

Performance Metrics

Active shards

The active_shards indicates the number of primary shards in your cluster. This is an aggregate total across all indices.

Intializing shards

The initializing_shards is the number of shards that are being freshly created.

Number of nodes/data nodes

The number of nodes/data nodes in the cluster is represented by the metric number_of_nodes and number_of_data_nodes respectively. Data nodes hold data and perform data related operations such as CRUD, search and aggregations.

Relocating shards

The relocating_shards is the number of shards that are currently moving from one node to another node.

Active primary shards

The active_primary_shards is an aggregate total of all shards across all indices, including replica shards.

Unassigned shards

From the initializing position, the shards move to a state of unassigned, as the master node starts to assign shards to the nodes in the cluster. The unassigned_shards exist in the cluster state, but can’t be found in the cluster itself. Being in the unassigned position for a long time could be a warning for an unstable cluster.

Cluster status

The status of the cluster is represented by Red: 0, Green: 1 and Yellow: 2. Cluster status in green means that all primary and replica shards are allocated. Being yellow indicates that atleast one replica shard is unallocated or missing. The cluster status being red means one or more primary shards have not been assigned.

JVM metrics

Elasticsearch runs on Java Virtual Machine (JVM) and one of the ways through which it uses the RAM on your nodes is via JVM heap. The metric jvm_mem_pool_old_used_perc is the average of each node's JVM memory usage (in percentage) of old generation in the Garbage Collection (GC). Metrics jvm_gc_old_coll_time and jvm_gc_old_coll_count give the GC time (in milli seconds) and count of old generation in all the nodes since last poll (5 minutes by default).

Memory and CPU usage

As Elasticsearch depends on the machine it is installed, it is critical to monitor CPU and memory usage. Monitoring CPU usage for each of your node types help in studying the distribution of workload between the nodes. Metrics including free (mem_free), used (mem_used), shared (shared_mem), resident (resident_mem), total virtual memory (virtual_mem) help to keep an eye on memory usage and understand how it loads and impacts the cluster.

Quick installation

If you're using Linux servers, use the Elasticsearch plugin installer that checks the prerequisites and installs the plugin with a bash script. You don't need to manually set up the plugin if you're using the installer.

Execute the command below in the terminal to run the installer and follow the instructions displayed on-screen:

wget https://raw.githubusercontent.com/site24x7/plugins/master/elasticsearch/installer/Site24x7ElasticSearchPluginInstaller.sh && sudo bash Site24x7ElasticSearchPluginInstaller.sh

Bulk installation

The Elasticsearch plugin supports bulk installation across multiple servers through automation using Ansible. To install the plugin using Ansible, refer to the Elasticsearch Ansible playbook.

Standard installation

If you're not using Linux servers or want to install the plugin manually, follow the steps below.

Prerequisites

  • Download and install the Site24x7 server monitoring agent (Linux | Windows) in the network or on the specific host in which the Apache instance is running.

Installation

  • Create a folder named elasticsearch.
  • Download the elasticsearch.py and the elasticsearch.cfg files from our GitHub repository, and place them in the elasticsearch folder.

    For Linux servers, you can execute the following commands to download the files:

    wget https://raw.githubusercontent.com/site24x7/plugins/master/elasticsearch/elasticsearch.py && sed -i "1s|^.*|#! $(which python3)|" elasticsearch.py
    wget https://raw.githubusercontent.com/site24x7/plugins/master/elasticsearch/elasticsearch.cfg
  • To check if the plugin is working, execute the command below with appropriate arguments and check for a valid JSON output with applicable metrics and their corresponding value.
    python3 elasticsearch.py --host "host" --port "port no" --username "elasticsearch username" --password "elasticsearch password" --ssl_option "YES/NO"
  • Add the applicable configurations in the elasticsearch file.
    [elasticsearch]
    host = "localhost"
    port = 9200
    username = "elasticsearch_username"
    password = "elasticsearch_password"
    ssl_option = "No" #Yes if you are using https
    cafile = "None"
  • Follow the steps in this article to learn how to run the Python script on a Windows server. You don't need to do this for Linux.
  • Move the elasticsearch folder to the Site24x7 server monitoring plugins directory. For Linux: /opt/site24x7/monagent/plugins/ For Windows: C:\Program Files (x86)\Site24x7\WinAgent\monitoring\plugins\

The agent will automatically execute the plugin within five minutes and display performance data in Site24x7.

To view the plugin monitor and associated performance charts:

  • Log in to Site24x7.
  • Navigate to Plugins and click the required monitor.

Plugin Contribution

Feel free to contribute to our existing plugin and come up with suggestions or feedback on our Community.

Other Plugin Integrations

Troubleshooting Tips