UnixTime

OpenSearch Enterprise Implementation Guide

In this article
OpenSearch Enterprise Implementation Guide

OpenSearch is a distributed search and analytics engine. It can power application search, log and trace analysis, security analytics, vector retrieval, and retrieval-augmented generation. It is not a transactional system of record, a backup system, or a compliance certificate.

That distinction defines production architecture. A working container proves only that a process started. An enterprise OpenSearch service must also preserve cluster state, shard availability, access boundaries, audit evidence, recovery paths, and query relevance while nodes, zones, credentials, mappings, and software versions change.

This guide uses OpenSearch 3.7 as the current reference release, verified against the project documentation on July 10, 2026. OpenSearch is an OpenSearch Software Foundation project released under the Apache License 2.0; it is not an Apache Software Foundation project.

The implementation decisions that matter first

Do not begin with docker compose up. Begin with the operating decision.

Decision Question to answer Failure when ignored
Workload Product search, telemetry, security analytics, vector search, or mixed? One cluster gets incompatible latency, retention, and memory demands.
Data authority Can the source system replay every indexed document? OpenSearch silently becomes the only copy of business data.
Recovery What are the recovery point and recovery time objectives? Snapshots exist but cannot restore the service in time.
Failure domain Which node, host, zone, or region failures must be tolerated? Replicas are placed inside the same outage boundary.
Access model Which identities can read, write, administer, export, and change security? A broad role bypasses document or field restrictions.
Delivery How are images, plugins, mappings, policies, and security changes promoted? Production drifts from the reviewed configuration.
Evidence Which logs and test records prove the service operated as designed? Compliance becomes screenshots and assertions instead of evidence.

The rest of the design follows from these answers.

How to use this guide and series

This page is the pillar. It contains the minimum complete path from workload selection to production evidence. Focused follow-up parts can deepen the areas that change fastest without turning this page into a vendor-specific runbook:

  1. Cluster engineering: shard strategy, capacity tests, data tiers, query behavior, and recovery.
  2. Platform deployment: Docker Swarm, the Kubernetes Operator, managed services, and platform-specific upgrades.
  3. Security and assurance: identity, encryption, audit pipelines, ISO 27001, SOC 2, NIS2, and NIST evidence.
  4. AI search operations: embeddings, hybrid retrieval, RAG, evaluation, model connectors, and agent observability.

This pillar remains the control point. A deep dive should refine it, not contradict it.

Where OpenSearch fits

OpenSearch is strongest when the workload benefits from inverted indexes, distributed aggregation, approximate nearest-neighbor search, or time-oriented analytics.

Use case OpenSearch capability Primary engineering constraint
Application and product search Full-text queries, facets, filters, ranking, aliases Relevance testing and zero-downtime schema evolution.
Log and observability analytics Data Prepper, OpenTelemetry ingestion, PPL, dashboards, alerting Sustained ingest, retention cost, field-cardinality control.
Security analytics Sigma-based detections, findings, correlation, alerts Evidence integrity, false-positive ownership, protected audit storage.
Vector and hybrid search knn_vector, HNSW or IVF, semantic and hybrid pipelines Native memory, recall, filtering, embedding lifecycle.
RAG and conversational search Retrieval pipelines, ML Commons, model connectors, agents Authorization-preserving retrieval, grounding, model risk, cost.
Operational analytics Aggregations, anomaly detection, near-real-time exploration It is near-real-time, not an ACID reporting database.

OpenSearch is a bad fit when the application needs multi-row transactions, strict relational integrity, authoritative writes, low-cost object retention, or simple key-value access. Keep PostgreSQL, an object store, a data lake, or the source application authoritative and make the index replayable.

Enterprise cluster architecture

The cluster is easier to operate when management, request processing, storage, and recovery are treated as different planes.

Reference architecture Requests enter through a controlled edge; recovery exits the cluster.

Cluster managers coordinate state. Workload nodes process traffic. Data nodes own shards. Snapshots live in a separate failure domain.

Application search APIs, websites, catalogs
Telemetry ingestion OpenTelemetry, Data Prepper
Operators Dashboards, automation, SRE
Traffic boundary TLS ingress, authentication, authorization, rate limits, and request logging
OpenSearch cluster One cluster state, multiple workload planes
Management plane Dedicated cluster managers
  • Manager 01
  • Manager 02
  • Manager 03

Election, mappings, shard allocation, and authoritative cluster state.

Request plane Coordinating, ingest, search, and ML nodes
  • Coordinate
  • Transform
  • Search / ML

Fan-out, aggregation, pipelines, model inference, and workload isolation.

Data plane Zone-aware shard placement
  • Hot data
  • Warm data
  • Search replicas

Primary and replica shards stay on persistent storage across failure domains.

Recovery plane Versioned snapshots in external object storage

Restore tests, retention locks, audit export, and a documented recovery objective.

Cluster-manager nodes protect cluster state

The elected cluster manager coordinates mappings, index creation, shard allocation, and the authoritative cluster state. Production clusters should use three dedicated cluster-manager-eligible nodes spread across failure domains. They need stable CPU, memory, and network behavior, but they should not receive user search or bulk-ingest traffic.

Sending application traffic to cluster managers couples query pressure to cluster governance. Under load, the service can then lose the very control plane needed to recover it.

Coordinating and ingest nodes isolate request pressure

Every node can coordinate requests, but dedicated coordinating nodes are useful when fan-out, aggregations, or high client concurrency create heap pressure. Ingest nodes are useful when pipelines transform documents before indexing. ML nodes isolate model inference and AI workloads from data nodes.

Small clusters can combine roles. Large or high-risk clusters should split them when measurements show contention. Role separation is not a maturity badge; it is a response to measured failure modes.

Data nodes own shard performance

Data nodes store Lucene shards and perform indexing and search work. Their design depends on the workload:

  • Hot nodes need fast storage and enough CPU for active indexing and queries.
  • Warm nodes trade latency for lower storage cost on less active data.
  • Search nodes or search replicas can isolate read-heavy workloads.
  • Vector-heavy nodes need explicit native-memory budgeting in addition to JVM heap.

Do not mix unrelated tenants just because the cluster can hold them. A security-log workload with long retention and bursty ingest has different failure economics from customer-facing product search.

Shards, replicas, and failure domains

A shard is a unit of distribution and recovery. More shards are not automatically more scalable. Each shard consumes heap, file descriptors, metadata, recovery bandwidth, and operator attention.

Use this design sequence:

  1. Estimate daily indexed bytes after mappings and compression.
  2. Define retention by data class, not one global number.
  3. Define peak indexing throughput and query concurrency.
  4. Choose a provisional shard size and count.
  5. Place primary and replica copies across the required zones.
  6. Benchmark with production-shaped data and queries.
  7. Test recovery while the cluster is under representative load.

Replica count is an availability and read-capacity decision. A replica on the same host or zone does not satisfy a host- or zone-loss objective. Use allocation awareness so primary and replica shard copies are distributed across real failure domains.

Cluster color is only a starting signal:

  • Green: all primary and replica shards are assigned.
  • Yellow: primaries are available, but at least one replica is missing. In production, redundancy is reduced.
  • Red: at least one primary shard is unavailable. Some data or operations are unavailable.

A green cluster can still be failing its service objective through high latency, rejected writes, stale snapshots, disk pressure, or incorrect results.

Size from workload evidence, not a fixed VM table

There is no credible universal OpenSearch server size. The old rule “4 CPUs, 64 GB RAM, 2 TB disk” hides the workload assumptions that matter.

Start with these inputs:

Input Measure Why it changes the design
Ingest Average and peak documents or bytes per second Determines CPU, bulk behavior, refresh cost, and hot-tier capacity.
Search Query mix, concurrency, p50/p95/p99 latency Determines coordinating pressure, cache value, and replica demand.
Aggregations Cardinality, bucket count, result size Can dominate heap and trigger circuit breakers.
Retention Indexed bytes per day by data class Drives data tiers, lifecycle policy, and snapshot cost.
Recovery Largest shard and available network bandwidth Determines how long a failed node or zone takes to recover.
Vectors Count, dimensions, engine, m, replicas, compression Determines native-memory consumption and recall/latency tradeoffs.
Availability Host, zone, or region loss objective Determines copies, placement, node count, and platform cost.

Use OpenSearch Benchmark with an official workload or a custom workload that preserves your mappings, document sizes, query distribution, and target throughput. A benchmark that sends maximum traffic with synthetic documents does not predict a production service objective.

For vector workloads, calculate native memory separately. HNSW memory grows with vector count, dimension, graph parameter m, and replicas. The OpenSearch documentation provides engine-specific formulas; use them before deciding how much RAM remains for the filesystem cache and JVM.

Linux and JVM baseline

The host is part of the database. Container orchestration does not remove kernel behavior.

Heap and native memory

Start with equal minimum and maximum heap values and test from there:

Terminal window
OPENSEARCH_JAVA_OPTS="-Xms8g -Xmx8g"

Half of container memory is a useful initial estimate for conventional data nodes, not a universal target. Leave memory for the Linux page cache, direct buffers, memory-mapped files, and native vector indexes. A vector node can fail from native-memory pressure while JVM metrics still look acceptable.

Swap and memory locking

Swapping JVM pages creates unpredictable latency. Disable swap on dedicated hosts or prove that the deployment’s memory-lock configuration is effective:

Terminal window
sudo swapoff -a
grep -E 'VmLck|VmSwap' /proc/$(pgrep -o -f opensearch)/status

bootstrap.memory_lock=true is not evidence by itself. Verify the process and container limits after deployment.

Memory maps and file descriptors

OpenSearch uses many memory-mapped files and open descriptors. Set and verify host policy explicitly:

Terminal window
sudo sysctl -w vm.max_map_count=262144
sysctl vm.max_map_count
ulimit -n

Persist the setting through the host configuration system. A value applied in an interactive shell disappears during rebuilds and says nothing about other nodes.

Transparent Huge Pages: background and production policy

Transparent Huge Pages (THP) let the Linux kernel back eligible virtual memory with larger pages automatically. On common x86 systems, normal pages are 4 KiB and a large THP is typically 2 MiB. Larger pages can reduce translation lookaside buffer pressure and page faults. The tradeoff is allocation, zeroing, reclaim, and compaction work that can create latency spikes.

That tradeoff matters to OpenSearch because the service combines a long-lived JVM, memory-mapped Lucene segments, direct buffers, page-cache pressure, and sometimes large native vector structures. The correct policy is therefore about latency distribution, not a claim that huge pages are always slow.

Understand the two controls

Linux exposes separate policy for allocation and defragmentation:

Terminal window
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

The active value appears in brackets.

  • always lets the kernel attempt THP broadly and can perform synchronous compaction.
  • madvise limits aggressive allocation to memory regions that explicitly request huge pages.
  • never disables THP allocation for the selected page size.
  • The defrag policy controls whether allocation can trigger direct compaction or rely more on background work.

Checking only enabled is incomplete. khugepaged can also collapse pages in the background, and kernel behavior evolves. Record the kernel version and both policies with benchmark evidence.

For latency-sensitive search clusters, begin with never as the conservative baseline. Benchmark madvise on the exact kernel, JDK, OpenSearch version, heap, and workload before changing it. Compare p99 and p99.9 search latency, indexing latency, GC pauses, major faults, direct compaction, CPU, and throughput.

Do not enable always because average throughput improved while tail latency regressed. Do not disable THP fleet-wide because an unrelated database guide said so. The accepted setting belongs in the host baseline and performance evidence.

Persistent systemd policy

Use a host-level unit with explicit ordering and verification:

[Unit]
Description=Set Transparent Huge Page policy for OpenSearch hosts
DefaultDependencies=no
After=local-fs.target
Before=opensearch.service docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'
ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/defrag'
ExecStart=/bin/sh -c 'grep -q "\[never\]" /sys/kernel/mm/transparent_hugepage/enabled'
ExecStart=/bin/sh -c 'grep -q "\[never\]" /sys/kernel/mm/transparent_hugepage/defrag'
[Install]
WantedBy=multi-user.target

Enable and verify it:

Terminal window
sudo systemctl daemon-reload
sudo systemctl enable --now opensearch-thp.service
systemctl status opensearch-thp.service

THP is a host kernel setting. Changing it inside an unprivileged OpenSearch container will fail, and granting the container enough privilege to change host memory policy expands the attack surface. Configure every Swarm or Kubernetes worker through image provisioning, cloud-init, Ansible, a managed node policy, or an approved privileged host agent.

Choose a deployment model

Deployment decision Choose the operating model before choosing the manifest.

The platform changes who owns failure recovery. It does not remove the need for it.

  1. 01

    Docker Compose

    Use when
    Development, integration tests, and controlled single-host installations.
    You still own
    You own the host, storage, certificates, upgrades, backups, and recovery.

    It is not a high-availability control plane.

  2. 02

    Docker Swarm

    Use when
    Small self-managed estates with strong Docker operations and fixed stateful nodes.
    You still own
    You own shard-safe placement, host labels, local volumes, secrets, and rolling changes.

    Swarm rescheduling does not move local shard data.

  3. 03

    Kubernetes Operator

    Use when
    Self-managed clusters that need node pools, PVCs, disruption budgets, and reconciliation.
    You still own
    You own Kubernetes, the operator release, storage classes, security, and restore tests.

    Kubernetes availability does not guarantee OpenSearch health.

  4. 04

    Managed OpenSearch

    Use when
    Teams buying an operational boundary from AWS, Aiven, Instaclustr, or another provider.
    You still own
    The provider operates part of the platform; you still own data, access, schemas, and evidence.

    Service limits and API differences are architecture constraints.

The same OpenSearch version behaves differently when storage, scheduling, identity, and recovery are owned by different platforms. Pick the operating boundary deliberately.

Docker Compose: development and controlled single-host use

Docker Compose is useful for local development, CI integration tests, and a controlled single-host installation where downtime and host loss are explicitly accepted. Pin the image version and never expose the demo security configuration to an untrusted network.

services:
opensearch:
image: opensearchproject/opensearch:3.7.0
environment:
discovery.type: single-node
bootstrap.memory_lock: "true"
OPENSEARCH_JAVA_OPTS: "-Xms2g -Xmx2g"
OPENSEARCH_INITIAL_ADMIN_PASSWORD: "${OPENSEARCH_INITIAL_ADMIN_PASSWORD:?required}"
volumes:
- opensearch-data:/usr/share/opensearch/data
ports:
- "127.0.0.1:9200:9200"
volumes:
opensearch-data:

This remains a demonstration security setup. Before production, replace demo certificates, remove demo users and roles, define trusted certificate authorities, and move secrets out of environment files.

Docker Swarm: a stateful design with explicit limits

Swarm can run OpenSearch, but it does not provide an OpenSearch-aware operator. It can restart a task without understanding shard allocation, cluster-manager quorum, or whether local data followed the task. This will not scale operationally unless node identity and storage placement are deliberate.

For a small resilient cluster:

  1. Prepare at least three Swarm worker nodes in separate failure domains.
  2. Apply the same kernel, time synchronization, disk, and security baseline to each node.
  3. Label each stateful node and pin one OpenSearch task to each label.
  4. Use persistent local or block storage with a documented replacement procedure.
  5. Put TLS certificates and private keys in Swarm secrets; keep non-secret configuration in Swarm configs.
  6. Do not publish node transport port 9300 or REST port 9200 directly to the internet.
  7. Update one node at a time only after checking cluster health and snapshot state.

Label the nodes:

Terminal window
docker node update --label-add opensearch.slot=1 swarm-node-a
docker node update --label-add opensearch.slot=2 swarm-node-b
docker node update --label-add opensearch.slot=3 swarm-node-c

The following stack shows the scheduling and storage pattern. The referenced opensearch-production.yml and TLS secrets must exist before deployment and must contain a non-demo Security plugin configuration.

x-environment: &environment
cluster.name: enterprise-search
discovery.seed_hosts: os-node-1,os-node-2,os-node-3
cluster.initial_cluster_manager_nodes: os-node-1,os-node-2,os-node-3
node.roles: cluster_manager,data,ingest
bootstrap.memory_lock: "true"
OPENSEARCH_JAVA_OPTS: "-Xms8g -Xmx8g"
x-deploy: &deploy
mode: replicated
replicas: 1
update_config:
parallelism: 1
order: stop-first
failure_action: pause
restart_policy:
condition: on-failure
delay: 20s
resources:
reservations:
cpus: "4"
memory: 16G
limits:
memory: 16G
x-node: &node
image: opensearchproject/opensearch:3.7.0
networks: [opensearch-cluster]
configs:
- source: opensearch_yml
target: /usr/share/opensearch/config/opensearch.yml
environment: *environment
deploy: *deploy
services:
os-node-1:
<<: *node
hostname: os-node-1
environment:
<<: *environment
node.name: os-node-1
volumes:
- /srv/opensearch/data:/usr/share/opensearch/data
deploy:
<<: *deploy
placement:
constraints: [node.labels.opensearch.slot == 1]
os-node-2:
<<: *node
hostname: os-node-2
environment:
<<: *environment
node.name: os-node-2
volumes:
- /srv/opensearch/data:/usr/share/opensearch/data
deploy:
<<: *deploy
placement:
constraints: [node.labels.opensearch.slot == 2]
os-node-3:
<<: *node
hostname: os-node-3
environment:
<<: *environment
node.name: os-node-3
volumes:
- /srv/opensearch/data:/usr/share/opensearch/data
deploy:
<<: *deploy
placement:
constraints: [node.labels.opensearch.slot == 3]
networks:
opensearch-cluster:
driver: overlay
attachable: false
configs:
opensearch_yml:
external: true

The snippet is a topology template, not a complete security bundle. Run docker stack config and inspect the fully merged service definitions before deployment.

For larger clusters, split three lightweight cluster managers from data and ingest services. Do not simulate stateful replicas by scaling one data service across arbitrary workers. Give every data node stable identity, storage, and placement.

Kubernetes: use the OpenSearch Operator

Plain StatefulSets can run OpenSearch, but the official OpenSearch Kubernetes Operator adds cluster-specific reconciliation, node pools, TLS handling, safe drain behavior, and rolling changes. That reduces manifest code; it does not outsource operations.

Install a pinned operator release through its Helm repository, then define an OpenSearchCluster custom resource. A minimal role-separated starting point looks like this:

apiVersion: opensearch.org/v1
kind: OpenSearchCluster
metadata:
name: enterprise-search
namespace: search
spec:
general:
serviceName: enterprise-search
version: "3.7.0"
drainDataNodes: true
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
securityContext:
allowPrivilegeEscalation: false
privileged: false
security:
tls:
transport:
generate: true
perNode: true
http:
generate: true
dashboards:
enable: true
version: "3.7.0"
replicas: 2
nodePools:
- component: managers
replicas: 3
diskSize: "20Gi"
roles: ["cluster_manager"]
jvm: "-Xms1g -Xmx1g"
resources:
requests: { cpu: "500m", memory: "2Gi" }
limits: { cpu: "2", memory: "2Gi" }
persistence:
pvc:
storageClass: encrypted-rwo
accessModes: ["ReadWriteOnce"]
- component: data-hot
replicas: 3
diskSize: "500Gi"
roles: ["data", "ingest"]
jvm: "-Xms8g -Xmx8g"
resources:
requests: { cpu: "4", memory: "16Gi" }
limits: { cpu: "8", memory: "16Gi" }
persistence:
pvc:
storageClass: encrypted-rwo
accessModes: ["ReadWriteOnce"]

Before production, add required pod anti-affinity across hosts and zones, PodDisruptionBudgets, priority classes, network policies, approved ingress, externally managed admin credentials, certificate rotation, snapshot repositories, and restore tests. Verify the exact operator release against the target OpenSearch version; the main branch documentation can describe behavior not yet present in your installed operator.

Kubernetes can keep a pod running while OpenSearch is red, a shard is unavailable, or the cluster has no recent snapshot. Monitor both control planes.

Managed cloud services

Managed OpenSearch is an operating model, not one product.

Model Examples Provider typically owns You still own
Provider-native managed service Amazon OpenSearch Service Service hosts, selected patching, replacement, platform integrations Domain policy, VPC design, roles, mappings, retention, queries, evidence, cost.
Cross-cloud managed OpenSearch Aiven, Instaclustr Cluster operations, backups or replacement according to plan and SLA Data classification, tenant access, service limits, restore acceptance, vendor assurance.
Operator on managed Kubernetes EKS, GKE, AKS plus OpenSearch Operator Kubernetes control plane according to provider boundary OpenSearch, operator, nodes, storage, certificates, upgrades, backups, performance.
Cloud VMs or bare metal EC2, Compute Engine, Azure VMs, private cloud Physical infrastructure according to provider boundary Almost the entire OpenSearch operating model.

Amazon OpenSearch Service is the provider-native option on AWS and supports VPC deployment, fine-grained access control, encryption, and Multi-AZ designs. Aiven and Instaclustr offer managed OpenSearch across supported cloud environments. Compare API restrictions, plugin availability, version lag, snapshot export, encryption-key ownership, audit-log export, maintenance windows, network connectivity, data residency, and exit procedures before selecting a service.

Provider compliance reports describe the provider’s control boundary. They do not certify your index permissions, retention policies, model connectors, or incident process.

Index lifecycle, backup, and recovery

An enterprise cluster needs three separate mechanisms:

  1. Replicas protect availability during a node or zone failure.
  2. Index State Management (ISM) controls rollover, tier movement, retention, and deletion.
  3. Snapshots protect recovery from corruption, accidental deletion, failed upgrades, and cluster loss.

Replicas are not backups. Deleting an index deletes its primary and replica copies. Store snapshots in a separate failure and administrative domain such as object storage, and limit who can delete both the live data and its recovery copies.

Define and test:

  • snapshot frequency and maximum accepted data loss;
  • snapshot retention and immutability requirements;
  • restore time for the largest required dataset;
  • configuration, templates, ISM policies, roles, and Dashboards objects needed after restore;
  • a clean-room or alternate-cluster restore procedure;
  • evidence that restore tests meet the stated recovery objective.

OpenSearch nodes cannot be downgraded in place. Before a rolling upgrade, take a current external snapshot and prove the restore path. “Rollback” means standing up a compatible cluster and restoring data, not simply changing the container tag back.

CI/CD for OpenSearch

OpenSearch delivery includes more than an image. Treat the following as one versioned release bundle:

  • OpenSearch and Dashboards image digests;
  • plugin versions and checksums;
  • opensearch.yml and JVM settings;
  • TLS and identity integration references;
  • index and component templates;
  • ingest and search pipelines;
  • ISM policies;
  • Security plugin roles and mappings;
  • alerting, dashboards, and monitors;
  • snapshot and restore procedures.
Delivery pipeline Promote evidence, not mutable configuration

The same reviewed image digest and configuration bundle should move from test to staging to production.

  1. 01

    Define and validate

    Review mappings, templates, lifecycle policies, security roles, plugins, and the target version as code.

    No secrets in Git; schema, policy, and manifest checks pass.
  2. 02

    Build and prove

    Pin the image, generate an SBOM, scan it, launch an isolated cluster, and run search, ingest, backup, and restore tests.

    The exact image digest and configuration bundle pass integration tests.
  3. 03

    Promote and observe

    Deploy to staging, rehearse the rolling change, promote the same digest, and monitor recovery and service objectives.

    Production change requires green health, a current snapshot, approval, and rollback evidence.

Required pipeline gates

Gate What it should prove Release blocker
Manifest validation Compose, Swarm, Helm, CRDs, and policies render correctly Unknown fields, missing secrets, mutable tags.
Supply chain Image digest, plugins, packages, and SBOM are reviewed Critical actionable vulnerability or unverified plugin.
Integration Secure startup, identity, role boundaries, ingest, queries, and aliases work Demo credentials, authorization bypass, failed query contract.
Data migration New mappings and pipelines work on representative data Incompatible mapping or unreplayable transform.
Performance Target throughput and tail latency stay within budget Regression beyond the accepted error budget.
Recovery Snapshot completes and restores into an isolated cluster No usable restore point or recovery objective miss.
Change Rolling procedure keeps cluster health and service objectives Red health, unassigned primaries, unsafe concurrent restart.

A production deployment sequence should look like this:

Terminal window
# Before change
curl --fail --cacert ca.pem -u "$OS_USER:$OS_PASSWORD" \
'https://search.example.com/_cluster/health?wait_for_status=green&timeout=60s'
# Create and verify a named pre-change snapshot
curl --fail --cacert ca.pem -u "$OS_USER:$OS_PASSWORD" \
-X PUT 'https://search.example.com/_snapshot/production/pre-change-2026-07-10?wait_for_completion=true'
# Deploy one node or node pool at a time through the platform controller.
# Then verify node count, shard assignment, write path, search path, and SLOs.

Do not place admin certificates or cluster-admin passwords in CI variables available to untrusted branches. Use workload identity or a protected deployment runner, short-lived credentials, environment approval, and a separate break-glass path.

Security architecture

OpenSearch Security provides encryption, authentication, access control, and audit logging. These are building blocks. Production security depends on how they are configured and evidenced.

Network and TLS

  • Keep REST and transport endpoints on private networks.
  • Require TLS for client traffic and transport traffic.
  • Replace demo certificates and trust roots.
  • Verify hostnames and certificate chains.
  • Separate public ingress certificates from node transport identity.
  • Restrict egress from ML connectors, webhooks, and plugins.
  • Rotate certificates through a tested process; OpenSearch supports certificate hot reload when configured.

Authentication and authorization

Integrate with an enterprise identity provider through OIDC, SAML, LDAP, Active Directory, client certificates, or a managed-service identity plane. Separate human, application, ingestion, automation, and break-glass identities.

Use least-privilege roles at cluster, index, document, field, and tenant levels. Be careful with document-level security (DLS) and field-level security (FLS): these features constrain reads. A user with write or delete permissions can still modify data they cannot retrieve. Separate read-filtered roles from write roles.

Administrative boundaries

The most sensitive capabilities are not ordinary index writes:

  • changing Security plugin configuration;
  • using an admin certificate;
  • managing cluster settings and plugins;
  • deleting indexes or snapshots;
  • changing audit destinations;
  • registering remote model connectors;
  • exporting large result sets.

Require stronger identity, approval, logging, and review for these paths. A single shared admin account destroys attribution.

Audit logging

OpenSearch audit logging is disabled by default. Enable the event categories needed for authentication, denied access, privilege use, security-index changes, and compliance monitoring.

For high-assurance systems, do not store the only audit copy in the cluster being audited. Export audit events to a separate OpenSearch cluster, append-only logging service, or controlled evidence store. Otherwise a cluster outage, administrator, or storage failure can remove both the event and the proof.

Audit volume is a capacity and privacy concern. Define retention, masking, access, alerting, and review ownership before enabling broad request-body logging.

Compliance mapping without overclaiming

OpenSearch can implement controls and produce evidence. It cannot make an organization ISO 27001 certified, SOC 2 compliant, NIS2 compliant, or aligned with NIST by itself.

Assurance loop A control becomes credible when intent, runtime proof, and review stay connected

Configuration is only one input. Evidence must show the control operated over the required period and exceptions were handled.

  1. 01

    Control intent

    Define the risk, owner, control objective, scope, retention period, recovery target, and accepted failure conditions.

    A framework label without a control owner and test method is not a control.
  2. 02

    Runtime evidence

    Capture access decisions, configuration changes, health, snapshots, restore tests, incidents, and deployment approvals.

    Evidence is time-bound, attributable, tamper-resistant, and stored outside the system it proves.
  3. 03

    Review and improve

    Test control operation, investigate exceptions, accept residual risk explicitly, and feed lessons into the next release.

    A dashboard is not assurance; independent review must challenge the evidence.
Framework or obligation Relevant OpenSearch control surface Evidence to retain Important limit
ISO/IEC 27001 and 27002 Asset ownership, access roles, identity lifecycle, cryptography, logging, monitoring, backup, change management Approved design, role review, certificate inventory, audit events, restore tests, change records Certification assesses the ISMS and control operation, not the product name.
SOC 2 Trust Services Criteria Security, availability, confidentiality, processing integrity, privacy controls Period evidence for access, monitoring, incidents, capacity, recovery, change, and vendor oversight A provider SOC report covers only the provider’s stated system and boundary.
NIS2 Article 21 Risk analysis, incident handling, continuity, supply-chain security, vulnerability handling, cryptography, access control, MFA Risk treatment, incident timeline, supplier review, recovery exercise, vulnerability and patch evidence Applicability and national implementation require legal and regulatory review.
NIST CSF 2.0 Govern, Identify, Protect, Detect, Respond, Recover outcomes Target profile, asset and risk records, control telemetry, incident and recovery evidence CSF describes outcomes; it does not prescribe one OpenSearch architecture.
NIST SP 800-53 environments Access control, audit, configuration, system integrity, contingency planning Control-specific implementation statements and test results Control selection depends on system categorization and authorization boundary.

For ISO/IEC 27001:2022, commonly relevant controls include inventory of information and assets, access control and identity management, cloud-service security, incident management, evidence collection, ICT readiness, records protection, configuration management, backup, logging, monitoring, network security, cryptography, and change management. Map only the controls actually selected in the Statement of Applicability and explain how OpenSearch supports them.

For NIS2, OpenSearch may support detection, investigation, evidence, and reporting, but the directive also requires organizational measures, supply-chain risk, continuity, vulnerability handling, governance, and incident notification. A SIEM dashboard is not an NIS2 program.

AI, vector search, and RAG use cases

OpenSearch can store vectors alongside text and metadata, run approximate nearest-neighbor search, combine lexical and semantic retrieval, connect to external models, and support RAG workflows. The enterprise use case is not “add embeddings.” It is controlled retrieval with measurable quality and bounded authority.

AI retrieval path Enterprise RAG starts with governed content and ends with evaluation

Every stage must preserve provenance, access restrictions, and an observable decision trail.

  1. 01

    Ingest and classify

    Chunk source content, attach ownership and access metadata, generate embeddings, and retain the original text and provenance.

    Rejected, private, expired, or unowned content never enters the retrieval corpus.
  2. 02

    Retrieve and rank

    Combine lexical, semantic, and metadata filters, then rerank candidates against a tested relevance set.

    Authorization filters survive every query path and recall is measured, not assumed.
  3. 03

    Generate and evaluate

    Send bounded context to the model, capture citations and trace data, and evaluate groundedness, latency, and cost.

    The model cannot turn retrieved text into authority or bypass an action approval.

High-value AI use cases

  1. Semantic and hybrid search: combine keyword precision with vector similarity for product, support, legal, or knowledge search.
  2. Retrieval-augmented generation: retrieve approved internal sources and provide grounded context to an LLM.
  3. Recommendations and similarity: find related products, incidents, documents, images, or cases.
  4. Security investigation: retrieve similar alerts, incidents, techniques, and historical response evidence.
  5. Observability for AI agents: store model calls, tool traces, token usage, latency, errors, and execution graphs.
  6. Search relevance evaluation: compare ranking changes against judged query sets and controlled LLM-assisted evaluation.

AI-specific architecture risks

  • Access-control loss: vector retrieval returns a semantically similar document the user is not authorized to read.
  • Data poisoning: untrusted content changes retrieval or model behavior.
  • Prompt injection: retrieved text is treated as an instruction rather than untrusted data.
  • Connector exfiltration: a remote model connector sends sensitive context outside the approved boundary.
  • Embedding drift: a model change makes old and new vectors incomparable.
  • Unmeasured relevance: demos look convincing while recall, precision, and grounding are poor.
  • Native-memory exhaustion: HNSW graphs and replicas exceed the off-heap budget.

Separate ML and search workloads when they contend for memory or CPU. Store embedding model identity, vector dimension, chunking policy, source version, classification, owner, and retention metadata with the indexed content. Use aliases or versioned indexes when changing embedding models so rollback does not require mixing incompatible vectors.

Evaluate retrieval independently from generation. Measure recall at k, nDCG or another ranking metric, authorization-filter correctness, p95/p99 latency, index cost, model cost, groundedness, citation accuracy, and refusal behavior. An LLM answer score cannot diagnose a retrieval failure by itself.

Observability and operational readiness

Monitor service behavior, not only process uptime.

Signal What it reveals Example action threshold
Cluster-manager elections and pending tasks Control-plane instability Investigate any unexpected election or sustained pending queue.
Unassigned primaries and replicas Availability and redundancy loss Primary loss is urgent; replica loss consumes resilience budget.
JVM pressure, GC, and circuit breakers Heap contention Correlate sustained pressure with query, ingest, or mapping changes.
Disk watermarks and shard growth Imminent allocation or write blocks Add capacity or reduce retention before flood-stage behavior.
Search and indexing p95/p99 User-facing service objective Roll back or shed load when the error budget is consumed.
Rejected thread-pool operations Saturation and queue pressure Reduce concurrency, scale the constrained role, or fix query behavior.
Snapshot age and restore-test age Recovery confidence Alert when either exceeds the approved recovery policy.
Audit-pipeline lag or failure Evidence loss Treat as a security monitoring incident, not a cosmetic warning.
Query relevance regression Product correctness Block ranking or model promotion when judged metrics fall.

Data Prepper is the preferred OpenSearch ingestion tool for complex server-side pipelines. For observability, use OpenTelemetry collectors to receive telemetry and Data Prepper to transform and route logs and traces. Keep ingestion backpressure away from application request paths and define dead-letter handling for rejected data.

Runbooks should cover at least:

  • loss of a data node, cluster manager, host, and zone;
  • disk watermark and read-only index recovery;
  • unassigned shards and allocation explanations;
  • failed snapshot and failed restore;
  • certificate expiry and identity-provider outage;
  • accidental index or security-role change;
  • slow query, rejected bulk requests, and mapping explosion;
  • compromised admin credential or model connector;
  • rolling upgrade failure and alternate-cluster restore.

Production acceptance checklist

Do not call the service production-ready until the evidence exists.

  1. The workload, data owner, service owner, security owner, and recovery owner are named.
  2. The source of truth can replay indexed data, or the accepted exception is documented.
  3. Cluster-manager quorum and shard copies span required failure domains.
  4. Capacity is based on representative benchmarks and recovery tests.
  5. Swap, memory maps, file descriptors, THP, time, and storage policy are verified on every host.
  6. Images and plugins are pinned, scanned, and promoted by digest.
  7. Demo certificates, demo users, and shared administrator credentials are removed.
  8. REST and transport traffic use approved TLS and private network paths.
  9. Least-privilege roles are tested, including DLS/FLS and write-path exceptions.
  10. Audit logging is enabled, exported, retained, and monitored according to risk.
  11. ISM policies, snapshots, restore tests, and recovery objectives are proven.
  12. Rolling change and failed-change procedures are rehearsed.
  13. SLOs cover availability, latency, ingest, rejection, recovery, and evidence health.
  14. AI retrieval preserves authorization and passes relevance and grounding tests.
  15. Compliance mappings point to owned controls and period evidence, not product features.

Final definition

An enterprise OpenSearch platform is a replayable search and analytics service with explicit workload boundaries, stable cluster state, zone-aware shards, controlled delivery, least-privilege access, independent recovery, and evidence that those controls continue to operate.

If the design only explains how to start containers, it is not a production design.

References

OpenSearch architecture and operations

  1. OpenSearch 3.7 documentation and use cases
  2. OpenSearch version history
  3. Creating and tuning an OpenSearch cluster
  4. OpenSearch node and system settings
  5. Docker installation guidance
  6. Rolling upgrades and restart procedure
  7. Snapshot and restore
  8. OpenSearch Benchmark workloads
  9. OpenSearch Data Prepper

Platforms and host behavior

  1. OpenSearch Kubernetes Operator user guide
  2. Docker Swarm services and placement constraints
  3. Docker Swarm secrets
  4. Linux kernel Transparent Huge Page documentation
  5. Red Hat guidance for managing Transparent Huge Pages
  6. Amazon OpenSearch Service operational best practices
  7. Aiven for OpenSearch high availability

Security, compliance, and AI

  1. OpenSearch Security
  2. OpenSearch TLS configuration
  3. OpenSearch document-level security
  4. OpenSearch field-level security
  5. OpenSearch audit logging
  6. OpenSearch AI search
  7. OpenSearch vector search
  8. OpenSearch RAG tutorials
  9. ISO/IEC 27001:2022 overview
  10. ISO/IEC 27002:2022 overview
  11. AICPA System and Organization Controls resources
  12. Directive (EU) 2022/2555, NIS2
  13. NIST Cybersecurity Framework 2.0
profile image of Hassan El-Masri

Hassan El-Masri

Hassan El-Masri is a Security Strategist with over 20 years of information technology, vast knowledge in many networking areas, and cybersecurity traversing several industries, including a specialization in security research, identity access management, and deception technologies. Mr. El-Masri has extensive experience designing solutions for complex information security problems.

Follow him on mastodon.

Read all posts of Hassan