Skip to main content

Elasticsearch

Overview​

Elasticsearch is a DataHub utility or metadata-focused integration.

The DataHub integration for Elasticsearch covers metadata entities and operational objects relevant to this connector. It also captures stateful deletion detection.

Concept Mapping​

While the specific concept mapping is still pending, this shows the generic concept mapping in DataHub.

Source ConceptDataHub ConceptNotes
Platform/account/project scopePlatform Instance, ContainerOrganizes assets within the platform context.
Core technical asset (for example table/view/topic/file)DatasetPrimary ingested technical asset.
Schema fields / columnsSchemaFieldIncluded when schema extraction is supported.
Ownership and collaboration principalsCorpUser, CorpGroupEmitted by modules that support ownership and identity metadata.
Dependencies and processing relationshipsLineage edgesAvailable when lineage extraction is supported and enabled.

Module elasticsearch​

GA

Important Capabilities​

CapabilityStatusNotes
Detect Deleted Entities✅Enabled by default via stateful ingestion.
Platform Instance✅Enabled by default.

Overview​

The elasticsearch module ingests metadata from Elasticsearch into DataHub. It is intended for production ingestion workflows and module-specific capabilities are documented below.

This plugin extracts the following:

  • Metadata for indexes
  • Column types associated with each index field

Prerequisites​

Before running ingestion, ensure network connectivity to the source, valid authentication credentials, and read permissions for metadata APIs required by this module.

Authentication​

The connector supports one of the following authentication methods per recipe:

  • Basic authentication: set username and password.
  • API key: set api_key to an [id, api_key] pair or a base64-encoded id:api_key string.
  • AWS IAM (SigV4): for Amazon OpenSearch Service domains, set aws_config. Every request is signed with AWS Signature Version 4.

AWS IAM authentication cannot be combined with basic authentication or an API key. aws_config.aws_region is required and must match the domain's region. Credentials resolve from one of:

  • explicit aws_access_key_id/aws_secret_access_key,
  • an aws_profile,
  • an aws_role to assume, using base credentials from the default AWS credential chain (environment variables, AWS_PROFILE, instance profile, ECS task role). aws_role cannot be combined with aws_profile or explicit keys, because the role would not be assumed; to assume a role from a profile, set AWS_PROFILE in the environment and configure only aws_role,
  • or the default AWS credential chain when none of the above is set.

Assumed-role credentials are renewed automatically before they expire, so long-running ingestion does not need special handling.

Amazon OpenSearch Service domains only accept HTTPS, and signed requests must only go to a verified domain certificate, so aws_config requires use_ssl: True and verify_certs: True (use ca_certs for a custom certificate authority). Point host at the domain endpoint on port 443, for example search-my-domain-abc123.us-east-1.es.amazonaws.com:443.

The IAM principal needs es:ESHttpGet on the domain (for example arn:aws:es:us-east-1:123456789012:domain/my-domain/*); the connector only issues read requests. If the domain uses fine-grained access control, also map the principal (as a backend role) to an OpenSearch role with these permissions:

  • Cluster permissions: cluster_monitor, indices:admin/template/get, indices:admin/index_template/get (template reads are cluster-level actions in OpenSearch security)
  • Index permissions on the indices to ingest: indices_monitor, indices:admin/get, indices:admin/aliases/get

OpenSearch Serverless collections are not supported.

Install the Plugin​

pip install 'acryl-datahub[elasticsearch]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
type: "elasticsearch"
config:
# Coordinates
host: 'localhost:9200'

# Credentials: use one of basic auth, api_key, or aws_config
username: user # optional
password: pass # optional
# api_key: ["id", "api_key"] # or a base64-encoded "id:api_key" string

# AWS IAM (SigV4) auth for Amazon OpenSearch Service domains.
# Replaces username/password and api_key. Credentials resolve from explicit keys,
# aws_profile, aws_role, or the default AWS credential chain.
# Requires use_ssl: True and verify_certs: True below; use host '<domain-endpoint>:443'.
# aws_role is assumed from the default credential chain and cannot be combined
# with aws_profile or explicit keys.
# aws_config:
# aws_region: us-east-1
# aws_role: "arn:aws:iam::123456789012:role/DataHubOpenSearchRead" # optional

# SSL support (use_ssl and verify_certs must be True when using aws_config)
use_ssl: False
verify_certs: False
ca_certs: "./path/ca.cert"
client_cert: "./path/client.cert"
client_key: "./path/client.key"
ssl_assert_hostname: False
ssl_assert_fingerprint: "./path/cert.fingerprint"

# Options
url_prefix: "" # optional url_prefix
env: "PROD"
index_pattern:
allow: [".*some_index_name_pattern*"]
deny: [".*skip_index_name_pattern*"]
ingest_index_templates: False
index_template_pattern:
allow: [".*some_index_template_name_pattern*"]

sink:
# sink configs

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
ca_certs
One of string, null
Path to a certificate authority (CA) certificate.
Default: None
client_cert
One of string, null
Path to the file containing the private key and the certificate, or cert only if using client_key.
Default: None
client_key
One of string, null
Path to the file containing the private key if using separate cert and key files.
Default: None
host
string
The elastic search host URI.
Default: localhost:9200
ingest_index_templates
boolean
Ingests ES index templates if enabled.
Default: False
password
One of string(password), null
The password credential.
Default: None
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
ssl_assert_fingerprint
One of string, null
Verify the supplied certificate fingerprint if not None.
Default: None
ssl_assert_hostname
boolean
Use hostname verification if not False.
Default: False
url_prefix
string
There are cases where an enterprise would have multiple elastic search clusters. One way for them to manage is to have a single endpoint for all the elastic search clusters and use url_prefix for routing requests to different clusters.
Default:
use_ssl
boolean
Whether to use SSL for the connection or not.
Default: False
username
One of string, null
The username credential.
Default: None
verify_certs
boolean
Whether to verify SSL certificates.
Default: False
env
string
The environment that all assets produced by this connector belong to
Default: PROD
api_key
One of array, string, null
API Key authentication. Accepts either a list with id and api_key (UTF-8 representation), or a base64 encoded string of id and api_key combined by ':'.
Default: None
api_key.string
string
aws_config
One of AwsConnectionConfig, null
AWS connection settings for IAM authentication against an Amazon OpenSearch Service domain. When set, every request is signed with AWS Signature Version 4 using the resolved credentials (explicit keys, profile, role assumption, or the default credential chain). aws_region, use_ssl: True and verify_certs: True are required. Cannot be combined with username/password or api_key, and aws_role cannot be combined with aws_profile or explicit keys. OpenSearch Serverless collections are not supported.
Default: None
aws_config.aws_access_key_id
One of string, null
AWS access key ID. Can be auto-detected, see the AWS boto3 docs for details.
Default: None
aws_config.aws_advanced_config
object
Advanced AWS configuration options. These are passed directly to botocore.config.Config. max_pool_connections defaults to 25 rather than botocore's 10; raise it when more threads than that share one client, e.g. dbt artifact_read_concurrency above 25.
aws_config.aws_endpoint_url
One of string, null
The AWS service endpoint. This is normally constructed automatically, but can be overridden here.
Default: None
aws_config.aws_profile
One of string, null
The named profile to use from AWS credentials. Falls back to default profile if not specified and no access keys provided. Profiles are configured in ~/.aws/credentials or ~/.aws/config.
Default: None
aws_config.aws_proxy
One of string, null
A set of proxy configs to use with AWS. See the botocore.config docs for details.
Default: None
aws_config.aws_region
One of string, null
AWS region code.
Default: None
aws_config.aws_retry_mode
Enum
One of: "legacy", "standard", "adaptive"
Default: standard
aws_config.aws_retry_num
integer
Number of times to retry failed AWS requests. See the botocore.retry docs for details.
Default: 5
aws_config.aws_secret_access_key
One of string(password), null
AWS secret access key. Can be auto-detected, see the AWS boto3 docs for details.
Default: None
aws_config.aws_session_token
One of string(password), null
AWS session token. Can be auto-detected, see the AWS boto3 docs for details.
Default: None
aws_config.read_timeout
number
The timeout for reading from the connection (in seconds).
Default: 60
aws_config.aws_role
One of string, array, null
AWS roles to assume. If using the string format, the role ARN can be specified directly. If using the object format, the role can be specified in the RoleArn field and additional available arguments are the same as boto3's STS.Client.assume_role.
Default: None
aws_config.aws_role.union
One of string, AwsAssumeRoleConfig
aws_config.aws_role.union.RoleArn ❓
string
ARN of the role to assume.
aws_config.aws_role.union.ExternalId
One of string, null
External ID to use when assuming the role.
Default: None
collapse_urns
CollapseUrns
collapse_urns.urns_suffix_regex
array
List of regex patterns to remove from the name of the URN. All of the indices before removal of URNs are considered as the same dataset. These are applied in order for each URN.
The main case where you would want to have multiple of these if the name where you are trying to remove suffix from have different formats.
e.g. ending with -YYYY-MM-DD as well as ending -epochtime would require you to have 2 regex patterns to remove the suffixes across all URNs.
collapse_urns.urns_suffix_regex.string
string
index_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
index_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
index_template_pattern
AllowDenyPattern
A class to store allow deny regexes.

Patterns are matched against the start of the string only, not the entire
string - a pattern does not need to match to the end to be considered a match.
For example, the pattern "prod" matches "prod", "prod_east", and "production".
To require an exact match, anchor your pattern explicitly, e.g. "^prod$".
index_template_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
profiling
ElasticProfiling
profiling.enabled
boolean
Whether to enable profiling for the elastic search source.
Default: False
profiling.operation_config
OperationConfig
profiling.operation_config.lower_freq_profile_enabled
boolean
Whether to do profiling at lower freq or not. This does not do any scheduling just adds additional checks to when not to run profiling.
Default: False
profiling.operation_config.profile_date_of_month
One of integer, null
Number between 1 to 31 for date of month (both inclusive). If not specified, defaults to Nothing and this field does not take affect.
Default: None
profiling.operation_config.profile_day_of_week
One of integer, null
Number between 0 to 6 for day of week (both inclusive). 0 is Monday and 6 is Sunday. If not specified, defaults to Nothing and this field does not take affect.
Default: None
stateful_ingestion
One of StatefulIngestionConfig, null
Stateful Ingestion Config
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False

Capabilities​

Use the Important Capabilities table above as the source of truth for supported features and whether additional configuration is required.

Limitations​

Module behavior is constrained by source APIs, permissions, and metadata exposed by the platform. Refer to capability notes for unsupported or conditional features.

Troubleshooting​

If ingestion fails, validate credentials, permissions, connectivity, and scope filters first. Then review ingestion logs for source-specific errors and adjust configuration accordingly.

Code Coordinates​

  • Class Name: datahub.ingestion.source.elastic_search.ElasticsearchSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for Elasticsearch, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.