Install OpenSearch

OpenSearch (or Elasticsearch) is the primary datastore for AxonOps — it stores logs, configuration, events, dashboards, and, by default, metrics. One of the two is always required.

For larger deployments (more than 8 monitored nodes), a dedicated Cassandra cluster can be added to store the metrics; OpenSearch remains required for everything else. See Prerequisites for guidance.

Cluster sizing

AxonOps stores agent logs, events, configuration, and dashboards in OpenSearch. Cluster sizing depends on the volume of data ingested, which is driven by the number of monitored nodes, log verbosity, and retention periods. In most deployments, a 3–6 node OpenSearch cluster is sufficient. For larger environments or longer retention periods, contact AxonOps for a sizing review.

Supported versions

These instructions target OpenSearch 3.x. Confirm the OpenSearch version supported by your AxonOps Server release against the Compatibility Matrix before installing.

Installation

Select the OS Family

Execute the following commands to setup OpenSearch for your OS. Replace the OPENSEARCH_INITIAL_ADMIN_PASSWORD value with a strong password of your own:

sudo apt-get update
sudo apt-get -y install lsb-release ca-certificates curl gnupg2

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://artifacts.opensearch.org/publickeys/opensearch-release.pgp \
  | sudo gpg --dearmor -o /etc/apt/keyrings/opensearch.gpg

echo "deb [signed-by=/etc/apt/keyrings/opensearch.gpg]\
 https://artifacts.opensearch.org/releases/bundle/opensearch/3.x/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/opensearch-3.x.list

sudo apt-get update
sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD='<StrongPassword!1>' apt-get install -y opensearch

OpenSearch Configuration

Configuration File Locations

Package installations of OpenSearch use /etc/opensearch/ for configuration, with the main settings in /etc/opensearch/opensearch.yml and JVM options in /etc/opensearch/jvm.options.

All commands on this page assume OpenSearch was installed via a Debian or RPM package. For other installation methods and the corresponding paths, see the OpenSearch installation documentation.

Increase Bulk Queue Size

Increase the bulk queue size of OpenSearch by running the following command:

echo 'thread_pool.write.queue_size: 2000' \
    | sudo tee --append /etc/opensearch/opensearch.yml

Increase Heap Size

Increase the default heap size of OpenSearch by editing /etc/opensearch/jvm.options.

Set Xmx and Xms to no more than 50% of the machine's physical RAM.

OpenSearch requires memory for purposes other than the JVM heap and it is important to leave available memory (RAM) space for this.

Example

If you have 16 GB of physical RAM, change the settings from:

$ sudo grep 'Xm' /etc/opensearch/jvm.options
-Xms1g
-Xmx1g

to:

$ sudo grep 'Xm' /etc/opensearch/jvm.options
-Xms8g
-Xmx8g

In the above example, we set the minimum and maximum heap size to 8 GB.

Increase Log Compression

Set the following index codec by running the following command:

echo 'index.codec: best_compression' \
    | sudo tee --append /etc/opensearch/opensearch.yml

Increase Number of Available Memory Maps

OpenSearch uses an mmapfs directory by default to store its indices.

The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions.

To increase the limits, run the following command:

sudo sysctl -w vm.max_map_count=262144

To make this change persist across reboots run this command:

echo "vm.max_map_count = 262144" \
    | sudo tee --append /etc/sysctl.d/10-opensearch.conf

Increase Number of File Descriptors

Note: This section is only required for non-package installations. Debian and RPM packages already use the intended value.

OpenSearch needs max file descriptors system settings to be at least 65536, which can be updated with this command:

sudo mkdir -p /etc/security/limits.conf.d
echo 'opensearch  -  nofile  65536' \
    | sudo tee --append /etc/security/limits.conf.d/opensearch.conf

Start OpenSearch

sudo systemctl enable opensearch.service
sudo systemctl start opensearch.service

After a short period of time, it is possible to verify that the OpenSearch node is running by sending an HTTPS request to port 9200 on localhost (the security plugin enables TLS and authentication by default):

curl -k -u admin:<admin-password> "https://localhost:9200"

Securing OpenSearch

Unlike Elasticsearch, OpenSearch bundles the security plugin and enables TLS and authentication by default. The administrator password is set during installation through the OPENSEARCH_INITIAL_ADMIN_PASSWORD environment variable (OpenSearch 2.12 and later).

Create a Dedicated Role

Create a dedicated AxonOps role and user rather than connecting as admin. The role requires the following permissions:

Cluster permissions:
- cluster_monitor
- cluster_manage_index_templates
- Index State Management (ISM) permissions for index lifecycle management (the OpenSearch equivalent of Elasticsearch ILM)
Index permissions, applied to the AxonOps indices:
- read
- write
- create_index
- manage
- indices:admin/mapping/auto_put

The index permissions should be applied to the following indices:

orgs
orgname_*
- Where the orgname prefix should match the org_name value in the AxonOps server and agent config files.

Roles and users can be created with the Security REST API, or by editing internal_users.yml, roles.yml, and roles_mapping.yml and applying them with securityadmin.sh. See the OpenSearch access control documentation for the exact mechanics.

Disabling Security (small / isolated deployments only)

For a small, network-isolated deployment, the security plugin may be disabled by adding the following to /etc/opensearch/opensearch.yml and restarting OpenSearch:

plugins.security.disabled: true

Warning

Disabling the security plugin removes TLS and authentication. Only do this on a trusted, isolated network. AxonOps must then be configured to connect over plain HTTP without credentials.