Updating DataHub

This file documents any backwards-incompatible changes in DataHub and assists people when migrating to a new version.

Next

Draft release notes for the upcoming version. Add entries here as breaking changes, deprecations, and notable fixes land on master; promote this section to a versioned heading (for example ## v1.7.1 or ## v1.8.0) at release cut.

Requirements:

• CLI / Python SDK: (TBD)
• Helm Chart: (TBD)

Breaking Changes

• (GMS / Search V3 system metadata) Cross-aspect system metadata on Search V3 (*index_v3) moves out of _search._system_* to a root _system object: _system.created.time, _system.created.actor, _system.modified.time, _system.modified.actor, _system.lastObserved, _system.runId, and _system.lastRunId. Per-aspect _systemMetadata is unchanged. Shared search labels are root fields (name, qualifiedName, description, columns, other), not _search.*. This is not the root runId, systemCreated, or systemIndexModified fields. An existing V3 index cannot take the new object, the removed _search fields, or the changed copy_to in place. Until it is rebuilt, the search engine rejects writes to it. Action: none if Search V3 write (ELASTICSEARCH_ENTITY_INDEX_V3_ENABLED) was never turned on. Otherwise run system-update once with ELASTICSEARCH_INDEX_BUILDER_MAPPINGS_REINDEX=true to rebuild the V3 indices, then run RestoreIndices before turning V3 reads on (ELASTICSEARCH_ENTITY_INDEX_V3_KEYWORD_READ_ENABLED, ELASTICSEARCH_ENTITY_INDEX_V3_SEMANTIC_READ_ENABLED). If ELASTICSEARCH_ENTITY_INDEX_V3_MAPPING_CONFIG points at your own copy of the V3 mapping file, remove _search and add the root _system object and the shared label fields before that run. See Search and Graph Reindexing.

• (GMS / Search V3 structured property search) Search V3 no longer copies structured property values into _search.structuredProperties. A string, rich_text, or urn property is full-text indexed only when its definition has a searchConfiguration. queryByDefault controls whether an unqualified search includes it. boostScore is the weight of its .text subfield (0.8 when unset) and subfieldBoostScores.stemmed is the weight of .stemmed (half of .text when unset). copyTo optionally names shared fields (name, qualifiedName, description, columns, other) or a root field that feeds one. Each value is copied into that shared field, so a name query can match a business name, and the stored name is unchanged. Unset copies nowhere. excludeFromFullTextSearch still applies to Search V2 and V2.5 and is ignored by Search V3. Properties that previously matched because every eligible value was copied in stop matching until their definitions set searchConfiguration, and queryByDefault: true for an unqualified search. Action: set that configuration on the properties that should be searchable. Removing the old copy_to, adding or changing copyTo, or gaining text and stemmed subfields, is a structured-property mapping mismatch, so ENABLE_STRUCTURED_PROPERTIES_SYSTEM_UPDATE plus ELASTICSEARCH_INDEX_BUILDER_MAPPINGS_REINDEX rebuilds the affected indices and fills the opted-in subfields. Rebuilding does not restore full-text search for properties that have no searchConfiguration. This supersedes the Search V3 structured property note later in this section.

• #20476 (GMS / Search V3 autocomplete) With Search V3 keyword reads on (ELASTICSEARCH_ENTITY_INDEX_V3_KEYWORD_READ_ENABLED=true, or the V2 indices turned off), autocomplete returns no suggestions and no entities. This is the search bar typeahead, GraphQL autocomplete, and POST /entities?action=autocomplete. Full-text search is unchanged. Search V3 no longer maps _search.autocomplete, and enableAutocomplete does not feed a field autocomplete reads. V2 keyword reads still autocomplete as before. GMS counts each empty response as autocompleteUnsupported and logs the reason once. Action: none if Search V3 write was never turned on. Otherwise leave V2 keyword reads on if you need typeahead. An index that already mapped _search.autocomplete picks up the removal only by rebuilding, the same system-update run as the Search V3 index mappings note below (ELASTICSEARCH_INDEX_BUILDER_MAPPINGS_REINDEX=true).

• #20468 (Authorization / Invite tokens) Invite token entities (inviteToken) now require the Manage User Credentials platform privilege to create, read, update or delete through the OpenAPI, Rest.li and GraphQL patchEntity entity APIs, matching the existing GraphQL invite token endpoints. Generic entity privileges such as Edit Entity no longer grant this access. Invite tokens redeemed through acceptRole and /auth/signUp are now also checked against their stored encrypted value. Tokens encrypted with the legacy v1 algorithm keep working with the default SECRET_SERVICE_V1_ALGORITHM_ENABLED=true, and are rejected when it is set to false. Action: grant Manage User Credentials to any principal that manages invite tokens through the entity APIs. If you have set SECRET_SERVICE_V1_ALGORITHM_ENABLED=false and existing invite links stop working, regenerate them with Invite Users under Settings > Users & Groups.

• #20460 (Search) Data process instance created and incident startedAt are dates in search. They were numbers holding epoch milliseconds. Saved filters or sorts that compared those fields as numbers need a date comparison after the next reindex. On Search V3, incident startedAt copies into sourceCreatedAt. Incident created stays a date on Search V2, where it was an epoch count, and Search V3 does not index it. Chart and dashboard creation times and ML model and model group times are newly searchable. Fields that share a search label use one existence boolean: description is hasDescription (assertions previously used hasAssertionDescription), tags are hasTags, and glossary terms are hasGlossaryTerms, including schema fields. The stored aspect JSON is unchanged. Action: reindex after upgrade if you filter or sort on those fields, or filter on assertion description, tag, or glossary-term presence. Aspects whose search mapping changed have a new schemaVersion: chartInfo, dashboardInfo, dataProcessInstanceProperties, incidentInfo, mlModelProperties, mlModelGroupProperties, assertionInfo, and apiSignature.

• #20352 (GMS / Entity versioning writes) Writing the versionProperties aspect (OpenAPI, Rest.li, ingestion MCPs, and the linkAssetVersion GraphQL mutation) now requires Edit Entity on the referenced version set when the write links the entity into that version set, creates it, or changes its latest version, matching what linkAssetVersion already required. When the written entity supersedes the current latest version, Edit Entity on that previous latest version is also required, because it is updated to no longer be the latest. This is enforced at the aspect layer by VersionPropertiesAuthorizationValidator. Previously ingest-time authorization covered only the entity being written, while the resulting version set update and previous-latest update ran unchecked. Re-ingesting an unchanged link that does not change the latest version needs no extra privileges, and system writes are exempt. Action: grant any non-admin automation that writes versionProperties Edit Entity on the version sets it writes to, and on the current latest version of any version set where it publishes a newer version. Toggle via metadataChangeProposal.validation.aspectAuthorization.versionProperties.enabled (default true).

• #20343 (Ingestion / dbt) dbt source freshness assertions now attach to the warehouse dataset named with target_platform_instance. Previously that dataset was named with the dbt platform_instance (unset by default), so whenever the two values differed, including when only one of them was set, the assertion attached to a warehouse dataset that does not exist. The assertions keep their urns and run history; only the dataset they attach to changes. Run events recorded before the upgrade still name the old dataset, but no longer count toward any dataset's health. Action: none. After the next ingestion run, freshness assertions and their health appear on the real warehouse table; no cleanup is needed.

• #20368 (Authorization) Moving an entity between domains is now actually checked against the new domain. A non-system writer that changes the domains of an entity that already has domains must be allowed to edit it in both its current and its proposed domains; this includes clearing them, so a writer scoped only to the current domain can no longer remove an entity's domains. Assigning an entity its first domains is still checked against the proposed domains only. Previously an authorization result cached earlier in the same request could answer the check for the proposed domains, so a writer with domain-scoped rights could move an entity into a domain it cannot edit. Writers who can edit entities in both domains, writers with grants that are not domain-scoped, and system actors are unaffected. Action: domain-scoped writers who move entities between domains, or clear an entity's domains, need edit rights covering the result.

• #20335 (GMS / Ingestion rollback) A hard rollback of an ingestion run (the UI Rollback button, or datahub ingest rollback --nuke) no longer deletes an entity the run created when something else has written to that entity since: another ingestion run, or an edit such as a tag, owner or description. The run's own aspects are still rolled back, and the entity stays with what others wrote. Before, the whole entity was deleted, including those later writes. Kept entities are listed as unsafe entities in the rollback response. The CLI's --nuke warning still reports the other runs' aspects on those entities as deleted, but they are kept. Action: none. To remove such an entity completely, delete it explicitly.

• #20382 (Ingestion / Airflow plugin) acryl-datahub now requires click>=8.3.3 and anyio>=4.14.2. An install that also passes Apache Airflow's published constraint file fails to resolve, because that file pins older releases of both. Action: rewrite those two lines before installing. See Installing with Airflow constraints. Installs that do not use Airflow's constraint file are unchanged.

• #19473 (Ingestion / dbt) manifest_path now treats *, ?, [ and ] as glob characters. A single-project recipe whose path literally contains one of them changes behaviour after upgrading. For a local or object-store path the recipe runs in multi-project mode: its dbt platform instance becomes the manifest's project name, so its dbt URNs change. For an HTTP(S) URL, a glob character in the URL path (not the query string) is unsupported, so the run ingests no project and reports a failure. Rename such a path before upgrading; bracket escaping does not help, since an escaped path is still a glob. Other recipes are unaffected. Code that subclasses DBTSourceBase or calls dbt source internals must change:

  • The abstract load_nodes() is replaced by load_projects(), which yields one DBTProject (nodes, exposures, metrics and that project's platform instance) per dbt project. load_exposures() and load_metrics() are removed; return those on the DBTProject instead.
  • DBTCoreSource.loadManifestAndCatalog() and DBTCoreSource.load_file_as_json() are removed. Use datahub.ingestion.source.dbt.dbt_artifacts.load_file_as_json.
  • Report fields: DBTSourceReport.catalog_generated_at is removed (the timestamp is now per project); duplicate_sources_dropped and duplicate_sources_references_updated default to 0 instead of None; DBTCoreReport.run_results_paths_expanded is now a LossyList instead of an optional list.

  Action: a subclass that implemented load_nodes() should implement load_projects() and yield one DBTProject. The custom-properties mapping that load_nodes() returned as its second value goes in DBTProject.artifact_props.

• #20349 (Authorization) The legacy Rest.li snapshot ingest endpoints (/entities?action=ingest and /entities?action=batchIngest) now require Edit Entity to update an entity that already exists, matching ingestProposal. Previously Create Entity alone was enough to overwrite an existing entity's aspects through these endpoints. Creating new entities with Create Entity is unchanged, and nothing changes when REST API authorization is disabled. Action: grant Edit Entity to any principal that uses the legacy snapshot endpoints with only Create Entity to update existing entities.

• #20312 (Authorization) When a write gives an existing entity its first domains (for example, an ingestion run that attaches a domain to a container it created earlier), metadata policies are now evaluated against the entity's stored platform instance, container, tags, glossary terms and structured properties together with the proposed domains. Previously those fields could read as empty for that request, so a policy filtered on one of them could deny a write it should allow (EQUALS) or allow a write it should deny (NOT_EQUALS); this applied to every aspect of that entity in the same request. Policies whose actors are resource owners were not affected. Custom authorizer plugins that use the request's resolved-spec cache now see the entity's stored fields during these writes and no longer see the proposed domains, which matches plugins that do not use the cache. Action: review metadata policies that combine Edit Entity or Create Entity with a NOT_EQUALS filter on platform instance, container, tag, glossary term or structured property; writes they let through when an entity received its first domains are now denied.

• (GMS / System Update / Documents) On the first upgrade, live documents whose parent document is already gone are soft-deleted. This runs once. A document tree with more than 10,000 descendants, or more than 100 levels, is logged and left in place. Action: set systemUpdate.deleteOrphanedDocumentChildren.enabled=false to turn that cleanup off. To retry a tree that was left in place, delete nested documents until it is under those caps, set SYSTEM_UPDATE_DELETE_ORPHANED_DOCUMENT_CHILDREN_REPROCESS=true for the next upgrade, then unset it afterward.

• #20278 (Authorization / Documents) The default Editor role no longer has the Manage Documents privilege. Editors can still create, edit, and move documents, but can only delete documents they own. A new default policy lets document owners delete their own documents. Action: if Editors need to delete any document, add Manage Documents or Delete for documents to a policy for the Editor role.

• #20111 (Ingestion / CLI) The datahub recipe commands (probe, test-connection, validate), first shipped in 1.7.0.14, now report more failures on the exit code that says who has to act: 2 your input, 3 the source, 1 DataHub, the connector or a missing package. Scripts and agents that branch on these codes will see these changes:

  • Now exit 2 (fix your input): a name the source does not list, such as a table name in the wrong case or a Mode space that does not exist (these could come back as an empty result with exit 0); a probe filter --name or --parent that is masked as *** because it equals a secret (it used to be judged); a recipe connection URL whose scheme names no installed SQLAlchemy dialect, or is not a URL (was 3); a probe sql query the database rejects as wrong, such as a misspelled column (SQLSTATE class 42, was 3); and a probe sql query holding a comment or an optimizer hint, which is now refused before it runs.
  • Now exit 1: a database driver or other Python package that is not installed; the error names it (was 3). A connector that exits on its own (SystemExit) during probe run or test-connection is now exit 3 rather than whatever status it exited with, often 0.
  • Now exit 3: a response from the source that cannot be decoded or validated (was 2).
  • Recipe variables: a dotted reference such as ${gms.server} no longer reads ~/.datahubenv. Like datahub ingest, it now resolves to an empty string, and datahub recipe validate warns about it. A reference that takes part of a value (${X:0:8}, ${#X}) is refused, because only whole values can be hidden in the output.

  Action: replace ${gms.server} and ${gms.token} in recipes with ${DATAHUB_GMS_URL} and ${DATAHUB_GMS_TOKEN}; reference whole variables only; remove comments from probe sql queries; and update any script that treats an empty probe result with exit 0 as "nothing there", since a wrong name is now exit 2. See the CLI guide for the exit codes.

• #20163 (Search / refresh interval) Timeseries aspect indices and the usage-event index now default to 60 seconds instead of following ELASTICSEARCH_INDEX_BUILDER_REFRESH_INTERVAL_SECONDS. Entity search, graph, and system metadata stay on that variable (default 3 seconds). Per-entity ELASTICSEARCH_INDEX_REFRESH_INTERVAL_ENTITIES and per-aspect ELASTICSEARCH_INDEX_REFRESH_INTERVAL_ASPECTS replace those defaults. A deployment that raised it keeps the raised value for those three only; timeseries and usage move to 60 seconds on the next system-update. Action: to keep the previous interval, set ELASTICSEARCH_INDEX_REFRESH_INTERVAL_TIMESERIES_SECONDS and ELASTICSEARCH_INDEX_REFRESH_INTERVAL_USAGE_SECONDS. A refresh_interval entry in ELASTICSEARCH_INDEX_BUILDER_SETTINGS_OVERRIDES is ignored, and a cluster index.refreshIntervalSeconds no longer changes the Elasticsearch refresh interval — use index.refreshIntervals on that cluster. The change is a settings update, not a reindex. Quickstart keeps timeseries and usage at 3 seconds.

• (Ingestion / SAP Datasphere) The connector now discovers Views and Analytic Models via the design-time dwaas-core listing by default (discover_unexposed_views: true), not only the consumption catalog. That closes dangling lineage edges to intermediate modelling views that are not Expose for Consumption. Runs may take longer (extra list + CSN calls) and emit more datasets. Action: none if you want the fuller inventory. Set discover_unexposed_views: false for the previous catalog-only discovery.

• (Actions / Kafka) The DataHub Actions Kafka event source now applies Kafka client properties from KAFKA_PROPERTIES_* environment variables (for example KAFKA_PROPERTIES_SECURITY_PROTOCOL, KAFKA_PROPERTIES_SASL_MECHANISM, KAFKA_PROPERTIES_OAUTH_CB), the variables the Helm chart already sets on the actions pod. Previously they were ignored unless the action recipe copied each one into connection.consumer_config, so pipelines such as the bundled ingestion executor connected with PLAINTEXT on SASL/SSL/MSK-IAM clusters. Values in consumer_config still take precedence, and Java-only or schema-registry properties that librdkafka rejects are skipped. Action: none for most deployments. If the actions pod has KAFKA_PROPERTIES_* variables that do not describe how it should connect, remove them, override them in consumer_config, or set DATAHUB_ACTIONS_KAFKA_ENV_PROPERTIES_ENABLED=false to restore the old behaviour. See Kafka Event Source.

• (GMS / MAE / MCE / System Update) DataHub no longer produces or consumes DataHubUpgradeHistory_v1 (or the pgQueue equivalent). GMS, MAE, and MCE startup no longer wait for that marker. New code can serve mixed schemaVersion SQL via ZDU read (and write) mutators; MigrateAspects already runs as a non-blocking sweep after pods are live. Compose still starts those services after blocking system-update (depends_on / service_completed_successfully). Helm/Argo pre-hooks remain the place to finish blocking index work before a rolling update. Pipelines that render the chart and apply the YAML themselves must run those same three phases in order. See Upgrading with rendered manifests. helm rollback is not a reason to keep this topic: a rolled-back image reads its own indices, the {gitVersion}-{revision} payload often does not change unless Helm bumps DATAHUB_REVISION, and dedicated ZDU rollback work owns a real downgrade path. Existing clusters do not need to delete the Kafka topic; leftover messages with internal schema id 8 will not deserialize and can be ignored. Unused env vars DATAHUB_UPGRADE_HISTORY_* and BOOTSTRAP_SYSTEM_UPDATE_WAIT_FOR_SYSTEM_UPDATE are no-ops and can be dropped from values. Internal schema id 8 is reserved (RESERVED_8) and must not be reused.

• (Ingestion / dbt) A dbt semantic model is a logical structure defined in YAML in your dbt project, not a table in the warehouse, and the connector was handling it as a physical one. Two things change as a result:

  • The dataset urn changes. A semantic model was named <database>.<schema>.<semantic_model_name>, using the database and schema of the dbt model it sits on. It is now named from dbt's own unique id, semantic_model.<project>.<semantic_model_name>, the same way exposures are already named. This also fixes a silent data-loss bug: when a semantic model has the same name as the model it is built on - which is dbt's own documented convention (- name: orders with model: ref('orders')) - the two produced an identical urn, and the semantic model, emitted last, overwrote the model's real warehouse schema, subtype and properties with no warning. Where that was happening, the model's metadata was already being lost on every run.
  • The warehouse-side entities disappear. A semantic model no longer produces a dataset on the target platform, a sibling relationship, or lineage into one. Those entities described tables that never existed.

  Both the old dbt-side datasets and the warehouse-side phantoms are orphaned by this change. Action: re-ingest with stateful_ingestion.enabled: true, which soft-deletes both; without it they are left behind with no owner. Governance authored on the old urns - owners, tags, glossary terms, documentation, and column-level tags and terms - does not move to the new urn, and no automated migration is provided; re-apply it if you need it. Separately, materialized_node_pattern no longer filters semantic models, since they have no materialized location; use node_name_pattern, which matches the dbt unique id.

• #19930 (Ingestion / MSSQL) A stored procedure whose body only calls other procedures now emits a dataJobInputOutput aspect. Its lineage was previously discarded, because the keep-check looked only at the dataset arrays. The aspect is a full upsert on the default incremental_lineage: false, so lineage added by hand on such a procedure is overwritten on every run. This happens without opting in. Action: set incremental_lineage: true on the MSSQL recipe if procedures in your instance carry lineage added through the UI; see the entry under Other Notable Changes for what else that flag changes.

• #19931 (GMS / all APIs) The corpUserCredentials aspect on corpuser entities, which stores native users' password hashes, encrypted salts and password-reset tokens, now requires the Manage User Credentials platform privilege to read. Previously any authenticated user could read it through the GraphQL aspects field, OpenAPI (/openapi/v1, /v2, /v3 entity and aspect reads) and Rest.li (/entitiesV2, /entitiesVersionedV2, /aspects), because read authorization was enforced per entity and corpuser is viewable by everyone. Requests for the aspect alone now return 403 (or an empty list in GraphQL); whole-entity reads succeed with the aspect omitted. Admins hold this privilege by default. Action: none unless an integration reads this aspect with a non-admin token, in which case grant that token's user Manage User Credentials.

• (GMS / Logging) GMS no longer writes DEBUG-level logs to gms.debug.log by default. Debug logging has a reasonably high performance cost and is rarely useful outside of active troubleshooting. Local dev/debug Docker Compose stacks (e.g. ./gradlew quickstartDebug) are unaffected and keep it enabled. Action: to restore the old behavior elsewhere, set the JVM system property -Dlogging.appender.debug_file.level=DEBUG on the GMS process (for example via JAVA_OPTS in your Helm values or Docker Compose env).

• (GMS / GraphQL patchEntity) patchEntity and patchEntities now take the entity type used for authorization from the target URN rather than from the optional entityType argument. A request whose entityType disagrees with the URN, or whose URN cannot be parsed, is rejected as unauthorized instead of being checked as the claimed type. Previously a grant scoped to one entity type could be matched against a URN of a different type by supplying the wrong entityType. Action: none for clients that omit entityType or pass the URN's actual type; fix any client that passes a mismatched value. In addition, the mutations now accept the entity type's own management privilege as an alternative to Edit Entity (for example Manage Policies for dataHubPolicy, Manage Secrets for dataHubSecret), matching the dedicated mutations for those types; Edit Entity on the target remains sufficient for every type.

• #19898 (Ingestion / DataHub Lite) DataHub Lite's storage engine is now pluggable, and the default for new instances is SQLite rather than DuckDB. Both engines are selected by the lite.type config key (sqlite or duckdb) and share the same schema, CLI, and REST API. DuckDB is unchanged but has moved out of the datahub-lite extra into its own: acryl-datahub[datahub-lite,duckdb]. Because SQLite ships with Python, acryl-datahub[datahub-lite] now installs on platforms with no DuckDB wheel (Linux armv7l, aarch64). The default database file for new instances is ~/.datahub/lite/datahub.db; ~/.datahub/lite/datahub.duckdb is untouched. The two file formats are not interchangeable: a DuckDB file cannot be opened with type: sqlite, or the other way around. Action: existing DuckDB users keep their instance and config as-is, but must add the duckdb extra to their install; without it, datahub lite reports that the duckdb type is disabled and names the extra. Users of the datahub-lite sink who relied on the default path get a new, empty datahub.db unless they pin type: duckdb and file: ~/.datahub/lite/datahub.duckdb in the recipe; the CLI warns when it creates a fresh store next to an existing *.duckdb file. To switch an existing instance to SQLite, run datahub lite export --file lite_export.json, then datahub lite init --type sqlite and datahub lite import --file lite_export.json. Exact-flavor searches are portable between the two: both engines support the -> and ->> JSON operators (SQLite 3.38+).

• #20325 (Ingestion / Python SDK) The cloud pip extra has been removed from acryl-datahub. It was an undocumented alias that only installed the separate acryl-datahub-cloud package. Note that pip install 'acryl-datahub[cloud]' does not fail on a removed extra — pip prints a warning and installs acryl-datahub without the cloud package. Action: install the package directly instead: pip install acryl-datahub-cloud. It pins the matching acryl-datahub version, so it can be the only package you name.

• #20082 (GMS / Semantic search) Semantic search now checks that each query embedding has the vector dimension configured for the active model (semanticSearch.models.<model>.vectorDimension) and fails the query with a clear error when it does not, instead of sending a mismatched vector to the search engine. By default, changing vectorDimension does not rebuild an existing semantic index, so a deployment whose setting no longer matches its embedding model may have kept working against an index built with the model's real dimension. After upgrading, its semantic queries fail with Embedding provider returned <N> dimensions for model '<model>'; configured mapping expects <M>. Action: none if semantic search is off or the dimension setting matches the model. Otherwise set the vectorDimension of the model named in the error to the size that model returns: ELASTICSEARCH_SEMANTIC_VECTOR_DIMENSION (Helm: global.datahub.semantic_search.vectorDimension) for text_embedding_3_large, LOCAL_EMBEDDING_VECTOR_DIMENSION for nomic_embed_text. For gemini_embedding_001 on Vertex AI, whose default output (768) never matched its default mapping (3072), set VERTEX_AI_EMBEDDING_OUTPUT_DIMENSIONALITY=3072; it sets both, so the existing index mapping stays valid.

• #19899 (Sigma ingestion) A Sigma run now fails when one of its entity listings (workspaces, workbooks, datasets, data models, or the /files metadata behind workbooks and datasets) cannot be read. Previously such a run succeeded with a warning and then soft-deleted everything that listing would have returned, as if it had been removed from Sigma. Action: treat a newly failing run as a real problem, not new noise — the report names the failing listing and what to do about it. The likeliest cause is /v2/datasets, since Sigma ended dataset support on 2026-09-15; once that endpoint returns 404/410 for you, set ingest_datasets: false, which stops ingesting Sigma Datasets and soft-deletes the ones you already have. A token without Data Model access also starts failing, on a 403 from /v2/dataModels that used to be only a warning: grant it that scope, or set ingest_data_models: false.

• #19880 (SAP Analytics Cloud ingestion) Stories and applications no longer browse under a language-specific root folder. SAC returns its built-in public root folder in a resource's ancestorPath localized to the tenant/resource language (for example Public in English and Öffentlich in German), which previously split one physical root into a separate browse-path tree per language. The first ancestorPath segment is now rewritten to a single canonical label (Public by default, configurable via public_root_folder_name), so all public content shares one Public/... tree. Only a first segment matching public_root_folder_aliases (defaults to Public / Öffentlich, case-insensitive) is rewritten; other shared roots that SAC can also return (such as a team/workspace root like Default) are left unchanged rather than merged into the public tree. User-created sub-folders are unchanged. Action: if you have a folder_pattern rule, a saved browse-path filter, or a bookmarked browse URL that targets a localized root such as Öffentlich, rewrite it against the canonical Public/... root, because folder_pattern is now matched against this canonicalized browse path and a stale localized rule silently stops matching. If your tenant's public root uses a language whose label is not in public_root_folder_aliases, add it there. Alternatively, set public_root_folder_name to null to keep the previous localized behavior. If you have no such rules, no action is required.

• #19516 (Ingestion / Data lakes) Data lake ingestion now skips files whose extension is not listed in file_types. Compressed files are checked by their inner extension; files such as archive.gz require default_extension. Action: Add required extensions to file_types, or set default_extension for extensionless or compressed files.

• #19881 (SAP Analytics Cloud ingestion) SAC model datasets are no longer placed under the meaningless t > 4 browse tree. SAC exposes no folder for a model (most are private/embedded models created by a data import), so GMS previously derived a browse path by splitting the dotted internal dataset name (t.4.<hash>:<hash>), producing a t > 4 tree. Models now browse under a single synthetic Models root that the connector invents (SAC has no such folder). folder_pattern does not apply to model browse paths. Action: Re-ingest to move existing models out of the old t/4 tree, and update any saved browse-path filter or bookmarked browse URL that pointed at the old model tree.

• (GMS / Search backend) Elasticsearch 7.x is no longer a supported search backend. DataHub requires Elasticsearch 8.x or OpenSearch 2.x or 3.x. Elasticsearch 9.x is not a certified backend: auto-detect still maps a 9.x cluster onto the Elasticsearch 8 client, but there is no ES9 CI coverage and known gaps remain (for example Analytics query DSL). GMS fails startup if it detects a 7.x cluster, if ELASTICSEARCH_SHIM_ENGINE_TYPE is set to ELASTICSEARCH_7 / ES7, or if OpenSearch is running Elasticsearch compatibility mode (compatibility.override_main_response_version, which makes GET / report 7.10.2). That compatibility mode is not treated as OpenSearch 2/3. Action: upgrade Elasticsearch 7 clusters to 8.x (or OpenSearch 2/3) before this DataHub version; on OpenSearch, turn compatibility mode off so the cluster reports its real 2.x/3.x version. Helm values that still set engine type ELASTICSEARCH_7 must be changed to auto-detect or a supported type.

• (GMS / Lineage) Chart, dashboard, and dataset may take a Metric as an upstream; those edges are stored on the shared upstreamMetrics aspect (ConsumesMetric). OpenAPI and Rest.li writes of upstreamMetrics require Edit Lineage or Edit Entity on the consumer and each destination Metric. A Dataset cannot both consume a Metric via upstreamMetrics and appear on that Metric's metricUpstreams.datasetUpstreams. Action: automation that sent unsupported updateLineage pairs (for example Metric as the downstream, or DataJob↔Metric) will now receive an error instead of a silent no-op. Use metricUpstreams to attach a dataset to a Metric. Grant Edit Lineage to any client that writes upstreamMetrics outside the UI.

• #19815 (Sigma ingestion) Sigma ended support for datasets as a data source on 2026-09-15, and a dataset-backed workbook element no longer returns SQL. The connector now recovers a Sigma Dataset's warehouse table from Sigma's connection metadata instead, so Sigma Dataset lineage URNs are built from connection_to_platform_map rather than chart_sources_platform_mapping. Three things to know:

  • env and platform_instance move. Without a connection_to_platform_map entry the URN uses the recipe's own env and no platform instance, where the SQL route used the mapping's.
  • Adding an entry at all makes its env authoritative, and that field defaults to PROD. So an entry added only to fix casing will move URNs to PROD unless you set env on it too.
  • Identifier casing changes on platforms other than Snowflake. The SQL route lower-cased every platform except BigQuery and DB2; this route lower-cases only Snowflake, so a Postgres table Sigma reports as Analytics.Public.Orders is emitted with that casing rather than analytics.public.orders. Setting convert_urns_to_lowercase: true explicitly restores the lower-cased spelling — but only for Sigma Dataset table names, and never for BigQuery or DB2, whose identifiers are case-sensitive. Data Model and workbook warehouse-table lineage are unchanged, and column identifiers on this route are not folded, so the same physical table can appear lower-cased via a Sigma Dataset and case-preserved via a Data Model in one run. Note a chart reading through a Sigma Dataset does pick up the flag on the table half of its inputFields, since that URN comes from this route.

  Action: add the Sigma connectionId to connection_to_platform_map with the env and platform_instance your warehouse connector used, plus convert_urns_to_lowercase: true if you want the old spelling for those table names. The connector warns when it detects a mapping whose env or platform_instance no longer applies.

• #18982 (GMS / Groups) removeGroupMembers now removes members of a group that never finished migrating to native membership. Such members hold only the legacy groupMembership aspect; previously the mutation stripped only nativeGroupMembership, reported success, and left the user in the group with their access intact. Action: none required. Automation that removed members from an unmigrated group and relied on them staying will now see them actually removed.

• (GMS / Search V3 document IDs) Search V3 Elasticsearch/OpenSearch documents now use a SHA-256 hex of the entity URN as _id instead of the URL-encoded URN. The full URN remains a document field (urn). V2 indices are unchanged. Action: none if Search V3 write was never enabled. If an environment already wrote V3 documents under URL-encoded _ids, hashed IDs are not an in-place rewrite — run RestoreIndices with -a clean (or reindex into a fresh index) so existing V3 docs are rewritten and the old URL-encoded documents are removed. A fill/reindex without clean leaves the previous _id in place, so updates create a duplicate. See Search and Graph Reindexing.

• #20195, #20196, #20197, #20218 (GMS / Search V3 index mappings) Every Search V3 entity document (*index_v3) stores its aspects' searchable fields at the top level, as keywords, numbers and dates that filters, facets, sorts and browse read with the V2 field names, as well as under _aspects (structured properties only at the top level). Full-text search reads a few shared fields instead, name, qualifiedName, description, columns, other (structured properties are searched on their own fields; see the note at the top of this section), the only analyzed full-text fields of a V3 index (see "Shared search fields on Search V3" in Extending the Metadata Model). V3 keyword reads run the Stage 1 query of #20213 and its light query of #20216 over these shared fields, so V3 ranks differently than that query on per-field indices: it has no word-gram clauses, which rank a name higher for holding two to four of the query's words next to each other, the shared fields' weights replace @Searchable boost scores in its word matches (a match of a root field's whole value, such as an exact name, keeps them), the light query cannot leave out ldap, flowId, jobId, cluster or glossary term definitions, which share a field with other values, and an id is matched analyzed only through other, which the fully qualified name match reads along with qualifiedName. On V3, matched fields are found from the document's values after the query and only list the names, descriptions, tags, glossary terms and urn a hit matched on, never a column, a document's text or a custom property (so the bundled excludeColumnHighlight field configuration changes nothing there). The word-gram factors and partial.urnFactor of the search configuration have no effect on V3, field configurations do not remove the urn from V3 search and add or remove whole shared fields on V3 (removing a field that lands in other changes nothing), and custom query configurations that name .delimited, .ngram or word-gram subfields match nothing on V3. A structured /q query that names a URN field, such as /q tags:pii, matches that field's whole value on V3, where V2 matched its parts; use a wildcard (tags:*pii*) or a plain query instead. The _search.tier_N fields are no longer created, and searchIndexed is still accepted but has no effect. An existing V3 index cannot pick this up in place. Until it is rebuilt, the search engine rejects writes to it. system-update logs an error naming the index, and while V3 serves reads (V3 keyword or semantic reads on, or the V2 indices off) it fails until the index is rebuilt or those reads are turned off. Until every document is rewritten, V3 reads miss the fields of documents written before the upgrade, so filters, facets and full-text search can leave them out, and the soft-delete, hidden-lifecycle-stage and latest-version exclusions do not apply to them; keep V3 keyword and semantic reads (ELASTICSEARCH_ENTITY_INDEX_V3_KEYWORD_READ_ENABLED, ELASTICSEARCH_ENTITY_INDEX_V3_SEMANTIC_READ_ENABLED) off until RestoreIndices has finished. Deployments that turned the V2 indices off (ELASTICSEARCH_ENTITY_INDEX_V2_ENABLED=false) always read V3, so their system-update fails until the rebuild, and their search misses older documents until RestoreIndices has finished. Action: none if Search V3 write (ELASTICSEARCH_ENTITY_INDEX_V3_ENABLED) was never turned on. Otherwise run system-update once with ELASTICSEARCH_INDEX_BUILDER_MAPPINGS_REINDEX=true to rebuild the V3 indices, then run RestoreIndices so every document is rewritten in the new shape, with -a clean if V3 documents were written under URL-encoded IDs (see the Search V3 document IDs note above). The reindex flag applies to every index with a pending mapping change, not only V3. If ELASTICSEARCH_ENTITY_INDEX_V3_MAPPING_CONFIG points at your own copy of the V3 mapping file, remove its tier_* dynamic template and the copy_to on urn first, because the tier analyzers are gone; ELASTICSEARCH_ENTITY_INDEX_V3_ANALYZER_CONFIG must not be empty, and if it or a cluster's own ELASTICSEARCH_CLUSTERS_SECONDARY_ANALYZER_CONFIG (which falls back to the main file when blank) points at your own copy of the V3 analyzer file, that copy needs the v3_* analyzers, the v3_word_tokenizer and the v3_underscore_parts filter of the bundled search_entity_analyzer_config.yaml, since V3 indices cannot be created without them. If you upgraded to a release that has only some of these changes, run the same rebuild and RestoreIndices again after upgrading to the release that has all of them. system-update reports a V3 index without the shared search fields the same way, as an error that stops it while V3 serves reads; until the rebuild, V3 full-text search finds only exact names and urns. Autocomplete returns nothing whenever V3 keyword reads are on (see the Search V3 autocomplete note above). An index rebuilt before legacy browse moved to V3 still maps browsePaths as a keyword, with no hierarchy analyzer and no depth, so legacy browse on V3 returns nothing until the rebuild and RestoreIndices have run. To roll back to an earlier version afterwards, delete the *index_v3 indices before starting it (its system-update creates them again in its own shape, since it cannot copy documents in the new shape into them), then run RestoreIndices. See Search and Graph Reindexing.

• #20214 (GMS / Search V3 structured property search) Superseded by the Search V3 structured property note at the top of this section. Previously, full-text search matched string, rich_text and urn structured property values by copying them into _search.structuredProperties at weight 0.8 (see "Shared search fields on Search V3" in Extending the Metadata Model); that intermediate light query also read the shared field for a single word or a dotted name. Matched fields did not list a structured property. excludeFromFullTextSearch kept a property out of that shared field; Search V3 no longer uses it. The rebuild described below filled the shared field. Action: none if Search V3 write (ELASTICSEARCH_ENTITY_INDEX_V3_ENABLED) was never turned on. Otherwise run system-update once with both ENABLE_STRUCTURED_PROPERTIES_SYSTEM_UPDATE=true and ELASTICSEARCH_INDEX_BUILDER_MAPPINGS_REINDEX=true, then run RestoreIndices. System-update then maps every structured property from its definition and reindexes each V3 index whose structured property copy_to differs, which copies the values already indexed into the new field (see the structured property copy_to reindex note below); RestoreIndices also indexes values written before their property was mapped. Both variables are needed: without the first, system-update keeps each index's live property mappings, and without the second it does not reindex. Together they also reindex any other index, V2 included, whose mappings changed or whose structured property mappings differ from the definitions. When this ships in the same upgrade as the Search V3 index mappings change above, add ENABLE_STRUCTURED_PROPERTIES_SYSTEM_UPDATE=true to that change's system-update run; its RestoreIndices covers this change as well. If that reindex fails, recreate the V3 indices instead: turn V3 keyword and semantic reads off; stop everything that writes to search (GMS, the MAE and MCE consumers, and ingestion), because a write after the delete creates a V3 index without its mappings; delete the V3 indices by name, listing them with GET _cat/indices/*index_v3*, which also shows the timestamped indices that an earlier reindex left behind each alias (Elasticsearch 8 rejects wildcard deletes by default); run system-update with ENABLE_STRUCTURED_PROPERTIES_SYSTEM_UPDATE=true; then start the services again and run RestoreIndices. Deployments with the V2 indices off have no search results until that RestoreIndices finishes. See Search and Graph Reindexing.

• #19673 (pgQueue / empty-poll sleep) Idle pgQueue consumers now back off exponentially (1s doubling to 5s) instead of a tight fixed sleep. DATAHUB_PGQUEUE_EMPTY_POLL_SLEEP_MS and DATAHUB_PGQUEUE_MCL_EMPTY_POLL_SLEEP_MS (YAML postgres.pgQueue.consumerPoll.emptyPollSleepMillis / mclEmptyPollSleepMillis) are removed and are not read as aliases. Use shared DATAHUB_PGQUEUE_EMPTY_POLL_SLEEP_MIN_MS (floor) plus per-topic ceilings DATAHUB_PGQUEUE_MAE_USAGE_EMPTY_POLL_SLEEP_MS, DATAHUB_PGQUEUE_MAE_MCL_EMPTY_POLL_SLEEP_MS, DATAHUB_PGQUEUE_MCE_MCP_EMPTY_POLL_SLEEP_MS, and DATAHUB_PGQUEUE_PE_EMPTY_POLL_SLEEP_MS. On the Python actions pgQueue event source, poll_interval_seconds is removed; set empty_poll_sleep_min_millis / empty_poll_sleep_max_millis. Action: delete the old env keys and recipe field. Set the new min/max knobs only if you need non-default idle sleep.

• #19673 (pgQueue / maintenance cron) postgres.pgQueue.maintenance.intervalSeconds must map to a constant pg_cron cadence: a minute interval that divides 60, an hour interval that divides 24, or exactly 86400 (1 day). Values such as 5 hours or 7 minutes fail SqlSetup (90 minutes and multi-day were already rejected). Action: use 1, 2, 3, 4, 5, 6, 10, 12, 15, 20, or 30 minutes; 1, 2, 3, 4, 6, 8, or 12 hours; or 1 day.

• #19549 (Observability / Kafka queue time) KAFKA_CONSUMER_PERCENTILES is removed, and DataHub no longer configures client-side percentile gauges for messaging_queue_time_seconds. Micrometer 1.13+ already ignored those percentiles when a histogram is enabled, so quantile= series were not exported. Histogram _bucket / _count series and SLO buckets are unchanged. Action: delete KAFKA_CONSUMER_PERCENTILES from your deploy config (it has no effect). If a dashboard or alert queries messaging_queue_time_seconds{quantile="0.99"} (or any other quantile label), switch it to histogram_quantile() over messaging_queue_time_seconds_bucket, or to an SLO bucket rate such as 1 - rate(..._bucket{le="64800"}[5m]) / rate(..._count[5m]). See Monitoring.

• #19554 (CLI / Python) Optional extras that pull unstructured / unstructured-ingest (notion, confluence, unstructured, datahub-documents) now pin patched releases that fix CVE-2026-71428. Those libraries require Python 3.11+, so installing those extras on Python 3.10 will fail at pip install. They are no longer included in acryl-datahub[all], so pip install 'acryl-datahub[all]' will not install Notion, Confluence, unstructured, or documents ingestion. Core acryl-datahub and [all] remain Python 3.10+. Managed ingestion still installs the extra from the recipe source type. The full acryldata/datahub-ingestion image still ships those sources (it installs [all,datahub-documents,notion,confluence]); slim and locked images do not. Action: If you use Notion, Confluence, unstructured, or documents ingestion from a local CLI or venv, install the matching extra on Python 3.11 or later (for example pip install 'acryl-datahub[datahub-documents]' or 'acryl-datahub[all,notion,confluence,datahub-documents]'). If you previously relied on [all] alone for those sources, add the extra explicitly. No extra install is needed on the full ingestion image.

• #19554 (agent-context / Snowflake UDFs) Generated Snowflake Python UDFs now set RUNTIME_VERSION = '3.11' instead of '3.10'. Re-applying the generated SQL runs CREATE OR REPLACE FUNCTION and replaces existing 3.10 UDFs. Action: Re-apply the generated UDF SQL after upgrading, and confirm the Snowflake account allows Python 3.11 UDFs.

• #19573 (Spark lineage / Databricks Unity Catalog) acryl-spark-lineage now bundles OpenLineage 1.53. When Databricks Unity Catalog is enabled (spark.databricks.unityCatalog.enabled=true) and DataHub symlink resolution remains enabled (spark.datahub.disableSymlinkResolution=false, the default), table symlinks now use a stable unity-catalog namespace and the fully qualified <catalog>.<schema>.<table> name. With the default spark.datahub.metadata.dataset.hivePlatformAlias=hive, the logical dataset URN therefore changes from urn:li:dataset:(urn:li:dataPlatform:hive,<schema>.<table>,<env>) to urn:li:dataset:(urn:li:dataPlatform:hive,<catalog>.<schema>.<table>,<env>); a configured platform instance or different Hive platform alias is preserved in the corresponding URN. Existing lineage and dataset references against the old URN are not migrated automatically. Action: after deploying the upgraded agent, re-run affected Spark workloads to emit lineage against the catalog-qualified URNs, verify the new edges, then remove stale edges or old logical dataset entities using your normal lineage and entity cleanup process. Jobs with symlink resolution disabled continue to use the physical storage identity and are unaffected. Two related notes: DataHub's Unity Catalog ingestion source uses the databricks platform, so set spark.datahub.metadata.dataset.hivePlatformAlias=databricks if you want Spark lineage to attach to the datasets that source ingests. Because these symlinks no longer carry a storage-location namespace, spark.datahub.metadata.dataset.connections.<olNamespace> entries keyed on that location no longer apply to Unity Catalog tables; the global platform instance and env still do.

• (GMS / Timeseries read authorization) Dataset profile, usage, and operations timeseries (and dashboard usage statistics) now require the matching View Dataset Profile / Usage / Operations privileges on Rest.li and OpenAPI as well as GraphQL. Dedicated timeseries APIs (getTimeseriesAspectValues, OpenAPI timeseries scroll, getTimeseriesStats with a URN filter) also require Get Timeseries Aspect API when REST API authorization is enabled. Single-aspect GET/HEAD of those aspects returns 403 when the viewer lacks the privilege; assembled entity GET omits the aspect. Aggregating a mapped sensitive aspect without a URN filter is denied. Action: grant the View Dataset Profile / Usage / Operations privileges (or rely on the default view-dataset-sensitive policy) to any API client that previously read these aspects with only entity GET. Chart usage statistics are unchanged.

• #19445 (GMS / Structured Properties) Hard-deleting a structured property now requires soft-deleting it first. Hard deletion of an active structured property — or direct deletion of its propertyDefinition aspect — is rejected with an error, uniformly across the UI, GraphQL, OpenAPI, Rest.li, and the CLI. This closes a silent footgun: hard deletion leaves the property's qualified name behind in the entity search index mappings, so the name cannot be reused for a new property until the affected indices are reindexed. The UI delete flow now performs both steps automatically. Ingestion rollback (including datahub ingest rollback --nuke) and internal system operations are unaffected. Action: where automation previously issued a single hard delete, soft-delete first and then hard-delete: datahub delete --urn "urn:li:structuredProperty:<id>" --soft, then datahub delete --urn "urn:li:structuredProperty:<id>" --hard (or the equivalent two API calls: set status.removed=true, then delete the entity). Soft deletion is reversible, so it can also simply be left in place. To reuse a name burned by an earlier hard delete, remove the leftover mappings by running SystemUpdate with ELASTICSEARCH_INDEX_BUILDER_MAPPINGS_REINDEX=true — see Index Mappings Cleanup.

• #19490 (GMS / Structured Properties search indexing) The structuredProperties object in entity search indexes is now mapped dynamic: false, and creating a structured property definition immediately adds an explicit field mapping to the search index of each entity type the property declares. Previously values were indexed via Elasticsearch dynamic mapping, which could lock a wrong text mapping and break search filters and facets on the property. New indexes refuse to dynamic-map structured property values; existing indexes keep their current mappings until reindexed. A value written for a property that declares no matching entity type — or written before its definition — is stored in _source but not searchable until the mapping exists and the document is next written. Re-saving a structured property definition re-applies (and repairs) its index mappings, since the mapping update is an idempotent put-mapping over the property's full declared entity-type set. Action: none for standard deployments. If you run with ENABLE_STRUCTURED_PROPERTIES_HOOK=false, structured property values on newly created indexes are no longer searchable at all (previously they were dynamic-mapped); re-enable the hook and re-save affected property definitions, or run SystemUpdate with ENABLE_STRUCTURED_PROPERTIES_SYSTEM_UPDATE=true to rebuild the mappings (re-enabling the hook only maps definitions created or updated afterward).

• (GMS / Role and group membership writes) Writes to the aspects that grant a user privileges are now authorized at the aspect layer, uniformly across GraphQL, OpenAPI, and Rest.li, by PrivilegeGrantAuthorizationValidator. Adding a role to a user or group (roleMembership) requires Manage Policies — the privilege the Assign Role UI already demands. Adding a user to a group (groupMembership / nativeGroupMembership) requires Edit Group Members, and adding a corpGroup owner (ownership) requires Edit Owners; when the acting user is the one being added, both require Manage Users & Groups instead. Only additions are checked — removals and re-ingestion of unchanged membership always pass, and system/bootstrap writes are exempt. Previously entity-level authorization ignored the aspect name, so Edit Entity on a user (or on a group you own, via the Asset Owners policy) was enough to grant that user or group the Admin role. Action: grant Manage Policies to any automation that writes roleMembership outside the UI — ingestion recipes, scripts, or raw MCP emitters. Ingestion that only syncs group membership (LDAP, Okta, Azure AD, datahub user upsert) needs Edit Group Members or Manage Users & Groups, not Manage Policies. Group owners can no longer add themselves to groups they own, or grant those groups roles. Toggle via metadataChangeProposal.validation.aspectAuthorization.privilegeGrant.enabled (default true).

• (GMS / Analytics API) The raw analytics search endpoint POST /openapi/v2/analytics/datahub_usage_events/_search now requires the Analytics API access privilege (GET_ANALYTICS_PRIVILEGE) or Manage System Operations. Previously it accepted View Analytics, which every user holds through the default "All Users" policy, even though the endpoint forwards a caller-supplied search request directly to the search engine. View Analytics continues to grant the in-app analytics dashboard. Analytics API access is held by Admins by default. Action: grant Analytics API access to any service account or script that calls this endpoint.

• (GMS / Compliance form assignment writes) The batchAssignForm, batchRemoveForm, and createDynamicFormAssignment GraphQL mutations now require the Manage Compliance Forms platform privilege, the same privilege the create / update / delete form mutations already required. The same rule is enforced at the aspect layer by FormAssignmentAuthorizationValidator: changing which forms are associated with an asset (adding or removing entries in the forms aspect) or writing a form's dynamicFormAssignment aspect requires Manage Compliance Forms. Writes that only change prompt completion or verification state of an already-assigned form are unaffected, so assignees can still complete forms. System writes, including the dynamic assignment hook, are exempt. Previously these mutations performed no authorization check. Admins and Editors hold Manage Compliance Forms by default. Action: grant Manage Compliance Forms to any automation (for example datahub forms upsert or scripts writing the forms aspect) that assigns or unassigns forms outside the UI. Toggle via metadataChangeProposal.validation.aspectAuthorization.formAssignment.enabled (default true).

• #19745 (GMS / Asset summary settings writes) The updateAssetSettings GraphQL mutation and any write to the assetSettings aspect (OpenAPI, Rest.li, ingestion MCPs) now require Edit Entity or Manage Asset Summary on the target asset, enforced at the aspect layer by AssetSettingsAuthorizationValidator. Previously the GraphQL mutation performed no authorization check, so any authenticated user could change the asset-summary template of any asset. Admins, Editors, and asset owners hold Manage Asset Summary by default, and REST writes already required Edit Entity, so legitimately privileged callers are unaffected. Note that OpenAPI and Rest.li still authorize the write as a generic entity update, so Manage Asset Summary alone is not sufficient there; only the UI / GraphQL path honors the narrower privilege. System writes are exempt. Action: grant Manage Asset Summary (or Edit Entity) on the relevant assets to any automation that writes assetSettings outside the UI. Toggle via metadataChangeProposal.validation.aspectAuthorization.assetSettings.enabled (default true).

• #19295 (GMS / Search indexing) GMS refuses to start when it is consuming MCLs (MAE_CONSUMER_ENABLED or MCL_CONSUMER_ENABLED) and both UI indexing paths are off (PRE_PROCESS_HOOKS_UI_ENABLED=false and PRE_PROCESS_HOOKS_REPROCESS_ENABLED unset/false). Standalone MAE consumers apply the same check in UpdateIndicesHook. That combination stored UI-created entities without writing search documents (ingestion sources, execution requests, and other GraphQL creates never appeared in the UI). Action: leave PRE_PROCESS_HOOKS_UI_ENABLED at its default true, or set PRE_PROCESS_HOOKS_REPROCESS_ENABLED=true if you intentionally disabled the GMS fast path. Set the same pair on GMS and MAE (embedded GMS and standalone MAE). Helm charts that previously set both MAE env vars to false when GMS preprocess was on will fail MAE startup after this version — upgrade the chart with this GMS/MAE image. Kubernetes system-update scale-down cannot leave this invalid state: when it flips preprocess off it also sets MAE_CONSUMER_ENABLED=false in the same Deployment patch (and scales standalone MAE to zero). v1.7+ Helm defaults to ZDU, which skips scale-down entirely. See #19119.

• (GMS / Access Tokens) Creating access tokens with no expiry (NO_EXPIRY / "Never") is disabled by default. Allowed finite durations are configured as a comma-separated ISO-8601 list (ACCESS_TOKEN_ALLOWED_DURATIONS, default PT1H,P1D,P7D,P30D,P90D,P180D,P365D). Exactly one of GraphQL duration (enum) or durationIso (ISO-8601 string) must be provided on create; use duration: NO_EXPIRY only when ACCESS_TOKEN_ALLOW_NO_EXPIRY=true. Already-issued tokens (including never-expiring ones) keep working — this is create-time policy only. Remote Executor token creation in Settings is shown to users with Manage Metadata Ingestion, independent of this never-expire policy. Action: To allow never-expire tokens again, set ACCESS_TOKEN_ALLOW_NO_EXPIRY=true. To customize available TTLs, set ACCESS_TOKEN_ALLOWED_DURATIONS (e.g. PT1H,P1D,P30D,P1Y). Month/year use fixed approximations (P1M = 30 days, P1Y = 365 days).

• (Ingestion / Kafka Connect) SQL Server (mssql) is now treated as a three-level platform (database.schema.table) in has_three_level_hierarchy. JDBC sink connectors default the schema to dbo when schema.name is unset, so sink URNs change from database.table to database.dbo.table. JDBC source connectors now take the three-level branch too: when the schema cannot be resolved from the connector config, the dataset is dropped (with a warning) instead of emitting a two-level URN. Oracle JDBC sinks now emit schema.table (database_name omitted); schema comes from schema.name / db.schema if set, otherwise connection.user. Previously a missing schema could produce a bare table URN. Action: none if your SQL Server assets were ingested with schema in the URN (the default). If a source connector starts warning Could not find schema for table, set schema.name (or equivalent) on the connector, or use generic_connectors to force the mapping. For Oracle, ingest without add_database_name_to_urn, or map the sink with generic_connectors using a database.schema.table target_dataset.

• (Ingestion / Elasticsearch) acryl-datahub[elasticsearch] now requires opensearch-py 3.x. The 3.x Python client still talks to OpenSearch 2 clusters; 2.x of the client is no longer installed. Action: none if you install via the extra. If you pin opensearch-py yourself, use >=3.0.0,<4.0.0.

• (Ingestion / SQL Profiling) The legacy Great Expectations SQL profiler has been removed. The SQLAlchemy profiler — the default since v1.1.0 and at feature parity for dataset- and column-level metrics — is now the only SQL profiler. The profiling.method config option and the acryl-datahub[profiling-ge] install extra no longer exist. Recipes that still set profiling.method (ge or sqlalchemy) emit a deprecation warning and ignore the field, profiling with SQLAlchemy instead; for Unity Catalog, profiling.method: ge is now rejected (use sqlalchemy or analyze). Action: remove profiling.method from your recipes and drop the profiling-ge extra from your install command. This is unrelated to — and does not affect — the separate Great Expectations integration (acryl-datahub[great-expectations] / the datahub-gx-plugin package) that ingests GX validation results into DataHub.

• #19891 (Ingestion / SQL Profiling) The profiling config classes have been renamed to drop their Great Expectations prefix, now that the GE profiler is gone: GEProfilingConfig is now ProfilingConfig and GEProfilingBaseConfig is now ProfilingBaseConfig, both moved from datahub.ingestion.source.ge_profiling_config to datahub.ingestion.source.profiling.config. Unity Catalog's UnityCatalogGEProfiler is now UnityCatalogProfiler (moved from datahub.ingestion.source.unity.ge_profiler to datahub.ingestion.source.unity.profiler). No recipe changes are needed — every profiling.* config option keeps its name and behavior. Action: only if you maintain a custom ingestion source that imports these classes directly, update the import path and class name.

• #18944 (GMS / OpenAPI entity create authorization) OpenAPI createEntity / createGenericEntities (and aligned create/patch batch paths) now authorize each URN in the request after building the MCP batch, using entity existence to choose Create vs Edit privileges. A principal with only Create Entity can no longer overwrite an existing entity via OpenAPI create/UPSERT. Type-level create checks alone are no longer sufficient for write authorization. OpenAPI createAspect authorizes via the batch-item change type: default aspect create-if-not-exists (ChangeType.CREATE) requires Edit Entity; inventing a missing entity (createIfEntityNotExists=true → CREATE_ENTITY) is existence-aware Create vs Edit. Action: Grant Edit Entity (or an equivalent edit privilege) to callers that update existing entities or use default createAspect; keep create-only policies for true creates. See Policies.

• #18944 (GMS / Domain-scoped create & domain writes) Ingest-time authorization now evaluates domain-scoped Create Entity / Edit Entity policies against proposed domains when establishing domains on a new entity or on an entity that has no domains aspect yet (DomainWriteAuthorizationValidator, plus proposed-domain seeding on RestLi/OpenAPI ingest). Domain-separated writers must include matching domains in the create (or first domains) write. Unscoped elevated ingestion policies are unchanged. Toggle via metadataChangeProposal.validation.aspectAuthorization.domainWrite.enabled (default true). Action: Domain-scoped write policies no longer need a separate unscoped create policy for the initial write when domains are proposed in-batch; update docs/runbooks that assumed domain filters could not authorize create.

• #18944 (GMS / Domain-scoped Edit on domains PATCH) ChangeType.PATCH always requires Edit Entity (never Create Entity). Domain-scoped Edit on a domains PATCH must allow both the before and after domain membership when domains already exist; when establishing first domains via PATCH, Edit is matched against the after domains only. Sync requests re-check inside the DB transaction after patch apply; async MCE consume skips user domain auth (authorized on the API thread before Kafka). Action: Grant Edit on both source and destination domains before moving assets between domains with a domains PATCH.

• #18699 (Ingestion / MySQL, MariaDB, Doris, TiDB profiling) profiling.profile_table_row_limit and profiling.profile_table_size_limit are now enforced on these sources. Previously both were accepted and ignored, so any value already set in your recipe will start filtering profiles on upgrade — re-check it before you upgrade. Both default to null on these sources rather than the shared 5M-row / 5GB defaults, so nothing changes unless you set them.

• #19099 (Ingestion / sql-queries) A run where nothing at all succeeds — every input line unparseable, or every query failing — now reports a failure and exits non-zero, instead of completing successfully with nothing to show for it. Individual bad rows are still skipped and counted as before, and a single malformed row no longer aborts the whole run. The enable_lazy_schema_loading option has been removed (it was never read), and two report fields were renamed: num_queries_processed_sequential is now num_queries_processed, and num_temp_tables_detected is now num_temp_table_matches. Action: remove enable_lazy_schema_loading from your recipe. If you re-ingest a partial query log and want to avoid replacing existing lineage, set incremental_lineage: true.

• #18548 (Python SDK / Circuit breakers) datahub.api.graphql.BaseApi and its Assertion / Operation subclasses now query GMS through DataHubGraph instead of a gql client, so they support OAuth, PAT, system auth and mTLS via DatahubClientConfig. The transport= constructor argument is replaced by graph= (a DataHubGraph), the .client attribute by .graph, GraphQL errors raise GraphError instead of gql's TransportQueryError, and passing both datahub_token and datahub_auth is rejected. Four behaviours come with the shared client, all of which the gql transport lacked: (1) an unset timeout now means a 30-second connect and read timeout instead of an unbounded wait; (2) failed requests are retried up to 4 times, POST included — report_operation is append-only and the server stamps its timestamp, so a request that fails after the write landed can add a duplicate operation row; (3) DataHub Cloud URLs go through fixup_gms_url (:8080 dropped, http upgraded to https, /gms appended), where the gql transport posted to <host>/api/graphql verbatim; and (4) ambient credentials are honoured — DATAHUB_AUTH_TYPE env OAuth, and, when neither a token nor datahub_auth is supplied, the DATAHUB_SYSTEM_CLIENT_ID / DATAHUB_SYSTEM_CLIENT_SECRET pair, which turns a previously unauthenticated client into one acting as the system user. BaseApi logs the credential source it resolved at INFO. Action: code that constructed these classes with a custom gql transport must pass a DataHubGraph instead. Set timeout on the API or circuit-breaker config (timeout_sec in an Airflow connection extra, which the operators forward as timeout) if a poke must wait longer than 30 seconds on a slow GMS, and check that a client meant to stay unauthenticated does not run somewhere the system-client environment variables are set. Declarative auth on a circuit breaker is a config-level option: the Airflow operators build their config from the connection, which cannot express an AuthConfig, so OAuth for them comes from the DATAHUB_AUTH_TYPE environment variables with an empty connection password. The acryl-datahub[circuit-breaker] extra is now empty (it no longer pulls in gql); installing it still works, so no recipe or install change is required.

• (Ingestion SDK / Connector support status) Connector support tiers are now Alpha, Beta, and GA, replacing Certified, Incubating, and Testing. Tiers are assigned from production adoption and code ownership rather than by the connector author, so some connectors moved between tiers in this release; connector docs and the integrations directory show the new badges. The old enum names were removed, not aliased: a custom source decorated with @support_status(SupportStatus.CERTIFIED) now fails at import with AttributeError. They were removed deliberately — kept as aliases, CERTIFIED would have silently published a brand-new connector as GA. Action: in any ingestion source you maintain outside this repository, replace SupportStatus.CERTIFIED with SupportStatus.GA, SupportStatus.INCUBATING with SupportStatus.BETA, and SupportStatus.TESTING with SupportStatus.ALPHA. Connector registries generated by older versions are still read correctly, so no regeneration is required.

• (Logging / Log line format) The default console/file pattern and the Loki aggregator appender's message pattern for datahub-frontend, datahub-upgrade (system-update), and GMS (metadata-service) no longer include the caller's source line number. Log lines now read %logger{36} - %msg%n instead of %logger{36}:%L - %msg%n. Action: update any log-parsing regex, dashboard, or alert rule that matches on a logger:lineNumber token in these services' logs to expect the logger name alone. Deployments with a custom logback.xml are unaffected; mae-consumer and mce-consumer did not include %L and are unchanged.

• #19905 (Ingestion / openapi) The get_token recipe field is now a typed object (request_type: get|post, url_complement) instead of a free-form dict. An unrecognized key, a missing request_type/url_complement, or a request_type: get whose url_complement omits the {username}/{password} placeholders now fails recipe validation instead of failing later (or silently) at ingestion time. get_token is also now mutually exclusive with token/bearer_token. Action: none if your recipe already used get_token: {request_type: ..., url_complement: ...} with a single auth method (the common case). Fix any recipe that relied on a malformed get_token being silently ignored, or that set more than one of token/bearer_token/get_token.

• #19906 (Ingestion / openapi) forced_examples values must now be lists of scalars (str/int/float/bool); a null or object entry is rejected instead of being silently stringified as "None" in the composed URL (booleans are coerced via int, so true/false become "1"/"0"). schema_resolution_max_depth is now bounded to 1..100 (previously any integer was accepted); 0 or a value above 100 now fails validation. Action: none if your recipe used scalar forced_examples values and left schema_resolution_max_depth within 1..100. Fix any recipe relying on the old lenient behavior.

• #19430 (Ingestion / BigQuery) enable_legacy_sharded_table_support: false now takes effect reliably. The flag is applied by overwriting process-wide state on BigqueryTableIdentifier, but only the true branch ever assigned it, so in a process that had already constructed a BigQuery source with the flag enabled (the default) — for example an executor running several recipes — a later source setting it to false was silently ignored and kept collapsing sharded tables to the base name. Action: if you set enable_legacy_sharded_table_support: false and saw it apply inconsistently, it will now apply on every run; expect sharded tables to move to their _yyyymmdd URNs on the runs where it previously failed to take effect.

• #16478 (GMS / Structured Properties) Policies that set the experimental privilegeConstraints filter to scope Edit Entity or Edit Entity Properties (today only usable to constrain tag URNs) now also apply that filter to structured-property writes, enforced by StructuredPropertyPrivilegeConstraintsValidator. Previously structured-property writes ignored privilegeConstraints entirely. Because a tag-URN filter never matches a urn:li:structuredProperty:... sub-resource, a policy with privilegeConstraints set can now deny structured-property writes that were previously allowed under it. Action: before upgrading, review any policy that sets the experimental privilegeConstraints field and confirm it should also restrict structured-property writes; if not, add a separate unconstrained policy granting Edit Entity / Edit Entity Properties for structured properties. Toggle via metadataChangeProposal.validation.privilegeConstraints.enabled (default true).

• (Actions / Kafka) A DataHub Actions pipeline whose Kafka event source cannot reach the cluster (wrong security.protocol, bad SASL credentials, unreachable bootstrap servers) now fails instead of running silently. The source retries its first connection for 60 seconds, logging the underlying client error at ERROR, and then stops the pipeline. Once no pipeline in the datahub-actions process is running, the process now exits: with code 1 if any pipeline failed, so the container runtime restarts it, and with code 0 if every pipeline finished cleanly (for example a DataHub Cloud event source with kill_after_idle_timeout: true, which previously left the process running with nothing to do). Previously the process logged Action Pipeline with name '...' is now running., consumed nothing, and stayed up, with librdkafka FAIL lines as the only symptom. A healthy source now also logs Kafka event source for pipeline '...' connected to Kafka at .... Action: none for correctly configured deployments. If an actions pod starts crash-looping after upgrading, its Kafka connection settings were already wrong; the logs now say why. See Kafka Event Source.

• #17678 (Ingestion / Trino) For catalogs listed in catalog_to_connector_details, an explicit connector_database now always builds a three-tier (database.schema.table) connector-source URN, taking precedence over the two-tier connector list (clickhouse, hive, glue, mysql, iceberg, and now oracle). Previously the two-tier list was checked first, so connector_database was silently ignored for those connectors and the URN was always two-tier (schema.table). This changes the Siblings and UpstreamLineage target URNs for any two-tier catalog that also sets connector_database. For Oracle catalogs the configured connector_database is folded to lowercase when it is all-uppercase, mirroring the identifier normalization the oracle source applies, so a value copied verbatim from Oracle (e.g. ORCLPDB1) still matches the native URN; other platforms use the value as given. Action: drop connector_database from catalog_to_connector_details entries for two-tier connectors to keep the previous two-tier URNs. Keep it only where the native platform really is ingested three-tier — for example Oracle ingested with add_database_name_to_urn: true, or StarRocks reached through Trino's mysql connector.

• #17678 (Ingestion / Trino) oracle and starrocks are now part of the known connector-platform mapping, so Trino catalogs backed by them emit Siblings and UpstreamLineage to the native platform instead of being skipped. Because Trino lowercases identifiers in information_schema, catalogs whose source identifiers are quoted/mixed-case (common for Oracle) will generate lowercased URNs that do not match the case-preserving URNs produced by the native oracle ingestion, which creates unmatched sibling/lineage targets. Action: if you ingest an Oracle- or StarRocks-backed Trino catalog and rely on unquoted (uppercase) or mixed-case source identifiers, confirm the generated URNs line up with your native ingestion before upgrading — see the Trino source docs for the case, two- vs three-tier, and StarRocks-via-mysql caveats.

• #20315 (Ingestion / Glue) Tables shared through a Lake Formation database-level resource link now get an upstream lineage edge to the shared table in the owning account, the same as table-level resource links already did. They are also now named with your own Glue ingestion's platform instance. Previously, when catalog_to_platform_instance mapped the owning account, these tables were named with the owning account's instance combined with your local link database name, so they matched neither account's real dataset. If your link database had the same name as the owning database, they could overwrite the owner's dataset. Action: keep the owning account in catalog_to_platform_instance so the new lineage edge points at the owner's dataset. If you used that mapping with database-level resource links, the affected datasets move to new URNs under your own platform instance. With stateful ingestion enabled, the old datasets are normally removed on the next run, but this is best-effort: if the move changes more than stateful_ingestion.fail_safe_threshold percent of the entities (default 75%), the run skips stale-entity removal and reports a failure. In that case, raise the threshold for one run or delete the old datasets manually. Without stateful ingestion, delete them manually. Caution: if your link database had the same name as the owning database, the old URN is the owner's own dataset. Stateful ingestion may then soft-delete the owner's dataset until the owner's next Glue ingestion run restores it, and you must not delete that URN manually. Separately, a resource link of either kind (table-level links included) whose owner resolves to the link's own URN, because the owning account is not in catalog_to_platform_instance (or is mapped to the same platform_instance/env) and the link's database and table names match the owner's, is now reported as a "Resource link owner catalog not mapped" warning instead of a debug log, so those runs finish as succeeded with warnings. As before, no lineage is emitted for these links. Action: map the owning account to the platform_instance/env its own Glue ingestion uses, or give one of the ingestions a distinct platform_instance.

• #20282 (Ingestion / Databricks) Unity Catalog usage no longer looks up tables in DataHub while processing query history. Usage statistics, operations and dataset status are written only for tables the recipe itself ingests, as in the Snowflake, BigQuery and Redshift connectors. A usage-only recipe (include_tables: false and include_views: false) is instead scoped by its catalogs, catalog_pattern, schema_pattern and table_pattern (and, for hive_metastore tables, include_hive_metastore), and writes usage for those tables without taking ownership of them, so it never changes their status or soft-deletes them. Tables from other catalogs still appear as subjects on Query entities; a query that touches only another catalog's tables is emitted by the recipe that ingests that catalog when it uses the system-tables usage path with include_queries and include_query_usage_statistics enabled. With include_column_usage_stats or the API usage path, column-level usage of SELECT * over a table another recipe ingests is no longer resolved, because that table's schema is not looked up. Lineage and usage URNs for tables the recipe does not ingest now keep a mixed-case platform_instance as written instead of lowercasing it, matching the URNs the connector emits. With the deprecated include_metastore: true, usage is no longer written: it went to URNs that matched no ingested table, and the run now reports a warning instead. Action: if you kept metadata out of DataHub with deny-all patterns to collect only usage, set include_tables: false and include_views: false instead; such a recipe now emits no usage and reports a warning. If stateful ingestion is enabled, run a recipe once with stateful_ingestion.ignore_old_state: true, then remove the setting, when it is a usage-only or deny-all recipe (upgrading from any earlier version), or when you run several Databricks recipes and are upgrading from 1.7.0.14 or 1.7.0.15rc1. Checkpoints written by those setups can list datasets that other recipes own; ignoring the old state once replaces the checkpoint without soft-deleting them or failing the run on fail_safe_threshold. Stale datasets the recipe itself owns are not removed during that one run.

• #20350 (Ingestion / Hive, Presto) The hive and presto sources no longer use PyHive (acryl-pyhive). Hive and Spark Thrift Server connections now go through impyla, and Presto connections through presto-python-client. Existing recipes keep working: the PyHive-style options.connect_args (auth, kerberos_service_name, http_path, ssl_cert, configuration for Hive; protocol, requests_kwargs, session_props and the Kerberos* arguments for Presto) are translated to the new drivers. A recipe using the rarely needed thrift_transport or requests_session arguments, or a requests_kwargs key other than verify, cert, auth or timeout, now fails with an error naming the unsupported argument. The databricks+pyhive scheme of the hive source has been removed, along with the databricks-dbapi dependency. Action: move Databricks recipes that use scheme: databricks+pyhive to the Databricks source (set include_hive_metastore for legacy hive_metastore tables). If you installed acryl-pyhive yourself for Hive Metastore Kerberos authentication, it is no longer needed.

Known Issues

Potential Downtime

• (GMS / Structured Properties / search indices) With ENABLE_STRUCTURED_PROPERTIES_SYSTEM_UPDATE=true, ENABLE_STRUCTURED_PROPERTIES_COPY_TO_MISMATCH_REINDEX=true (the default), and ELASTICSEARCH_INDEX_BUILDER_MAPPINGS_REINDEX=true, system-update BuildIndices reindexes entity search indices when an existing structured-property field's copy_to targets differ from the definition-driven mapping (for example adding customFullTextSearchFields). A put-mapping does not copy values already in the index into the new target, so those values stay out of full-text search until the reindex. All three flags are required. Mappings reindex defaults to off, so turning on system update alone does not start this reindex. Action: Expect longer system-update runtime on the first upgrade if any structured-property copy_to targets disagree with the index. Set ENABLE_STRUCTURED_PROPERTIES_COPY_TO_MISMATCH_REINDEX=false to skip that reindex and leave other structured-property system-update behavior intact.

Deprecations

• #20460 (Metadata model) Audit stamps that only repeat the DataHub write are deprecated in favor of systemMetadata.aspectCreated and systemMetadata.aspectModified (Search V3 copies _system.created.time, _system.created.actor, _system.modified.time, and _system.modified.actor). This covers incidents, views, ownership types, lifecycle stage types, page templates, page modules, files, secrets, posts, documents, APIs, agent skills, and AI agents, plus the existing deprecations on domains, groups, and structured property definitions. The fields remain and are still indexed on Search V2. Search V3 does not index them. Action: read and write the system metadata stamps for these records. Do not add new writers that depend on the deprecated fields.

• (Frontend / /admin) GET /admin and GET /health on the Play port (9002) are deprecated as health checks. They still return 200 GOOD (or 503 during graceful shutdown) so existing monitors keep working, but they run on the Play connection table and can time out while the process is up. Use GET /health/live and GET /health/ready on MANAGEMENT_SERVER_PORT (default 4319). Action: point Kubernetes liveness and readiness, Docker healthchecks, and any other prober at those management paths. Do not add new checks against /admin. The route will be removed in a later release after charts and monitors have moved.

• (OpenAPI) OpenAPI v2 entity and relationship APIs (/openapi/v2/entity, /openapi/v2/relationship, including generated typed entity endpoints under /openapi/v2/entity) are deprecated in favor of /openapi/v3/entity and /openapi/v3/relationship. They remain available; plan removal in a future release after a deprecation notice period. Timeseries (/openapi/v2/timeseries), platform entity ingest (/openapi/v2/platform/entities/v1), and timeline (/openapi/v2/timeline/v1) under v2 are not deprecated by this change.

• (Metadata Model / Semantic Model) semanticModelInfo.datasets is deprecated. The field remains in the aspect for backwards-read of already-ingested data, but @Relationship / @Searchable annotations have been stripped so it no longer writes graph or search edges. Membership is member-side only: datasets use semanticModelProperties.semanticModel (IsPartOf), metrics use metricInfo.semanticModel (ModeledBy, non-lineage). Canonical metric lineage is Metric → Semantic Model Dataset → Physical Dataset via metricUpstreams.datasetUpstreams plus each logical dataset's upstreamLineage. Action: After upgrade, re-ingest your Semantic Models so metricUpstreams is populated with per-metric routing by the connector. Metric-to-dataset lineage will be missing between the upgrade and the next ingest run. Legacy Snowflake Semantic Views: run datahub migrate snowflake-semantic-views first. New writes must not populate semanticModelInfo.datasets. See Semantic Model and Metric.

Other Notable Changes

• (GMS / System Update / Search V3 indices) Search V3 entity indices now go through the same zero-downtime path as V2. With ELASTICSEARCH_ENTITY_INDEX_V3_ENABLED=true, the blocking system update creates each missing V3 index empty behind its alias, and a non-blocking step then fills it from the database one entity type at a time, checkpointing as it goes, so an interrupted run resumes where it stopped. Catch-up replays the writes that landed during the fill, and a later V3 mapping change is copied with _reindex and swapped like a V2 index. An index whose documents were written under a different document-id scheme is rebuilt into a fresh index and swapped in once the fill completes; an index that owned the convention name directly is first copied to a physical index the alias can be swapped back to; writes that land on the original during that copy reach the new index through catch-up, and the retained copy as well once the consumers have noticed the swap, through the rollback dual-write that repeats each replayed write, unless the copy is flagged as holding another scheme's ids. Whether the index a rebuild leaves behind keeps receiving rollback writes after the swap depends on its ids, and because no V3 index created before this version records a document-id scheme, the question arises on the first migration of every existing V3 index, not only when the hasher changes: an index that records another scheme, or that records none and whose sampled documents carry ids other than the running hasher's, or whose sample could not be read, receives none, since the running hasher's ids would sit next to the old ones, and a rollback to it returns to its pre-swap content; an index that records no scheme but whose sampled documents all carry the running hasher's ids, the common case for an index the current writer created, keeps receiving them. The sample is a hundred documents chosen at random, so a small minority written under another scheme can go unseen; rollback writes then sit beside those documents' old copies, which is the state the index was already in before the rebuild. Catch-up rebuilds each replayed entity from what primary storage holds and also removes from its document the aspects storage no longer has, reading the aspect names the documents carry with one request per five hundred entities, so an aspect removed while the fill was writing is dropped once the entity is replayed. Filled documents record the entity's creation time from its key aspect's row. Known limitation of the fill: an entity hard-deleted, or an aspect removed, during the few seconds in which the fill is writing the page that holds it can be written back by that page, since fill writes merge into the index and catch-up re-derives its window from storage, where a deletion leaves no row; the window is one page per entity, and, unless a later write of that entity lands in the catch-up window, the next write to that entity or the orphan consistency check repairs it. Each V3 index records on itself whether its migration, the fill and the catch-up that follows it, has finished, since the upgrade's own record of it belongs to the deploying version: a deploy that finds an index whose migration an earlier deploy left unfinished, in either half, rebuilds it into a fresh index, and the partial index it leaves behind receives no rollback writes. A non-blocking run that finds a fill left failed replays the indices that are ready, skips that one, and then reports failure, so the state is visible; the next blocking system update plans the fill again. Two settings tune the fill: ELASTICSEARCH_BUILD_INDICES_MAX_BACKFILL_HOURS (a wall-clock budget per run; 0, the default, means no budget; a run that reaches it ends successfully with the fill still in progress and the next run continues it) and ELASTICSEARCH_BUILD_INDICES_BACKFILL_BATCH_SIZE (rows per page, default 5000). Rollback dual-writes for a swapped V3 index now write the V3 document shape to the swapped-out index. Search V3 can also run on its own OpenSearch cluster (ELASTICSEARCH_CLUSTERS_SECONDARY_URI with ELASTICSEARCH_COMPONENT_CLUSTER_SEARCH_V3=secondary); the fill then populates it there.

• (datahub-upgrade / LoadIndices) LoadIndices reads the aspect table in bounded pages that keep each entity's aspects together, instead of one open result set, and accepts two more arguments: v3BackfillMode (baseline, the default, or catchup) and skipConversionErrors (true counts and skips an aspect that fails conversion instead of failing the run; off by default). See LoadIndices.

• #20483 (Ingestion / SAP Datasphere) When the ingestion principal can see a space in the catalog but is not a member of it, the dwaas-core object listing returns HTTP 403 and the run now records a failure. This is intentional: a failed run suppresses stale-entity soft-deletion, so a space that cannot be listed never has its previously-ingested objects removed. The trade-off is that a persistent membership gap fails every run, and keeps stale-entity removal off for the whole pipeline, until it is resolved. Action: add the ingestion user/client as a member of the space in SAP Datasphere, or exclude the space with space_pattern if it is not meant to be ingested.

• (Frontend / Domain icons) The domain (and glossary) icon picker now saves Phosphor icons (IconLibrary.PHOSPHOR) instead of Material UI. Existing displayProperties.icon values with iconLibrary: MATERIAL keep working: the UI maps Material icon names to Phosphor equivalents at render time. New picks are stored as Phosphor component names with style: regular. Action: none; re-save an icon only if you want the stored library/name updated to Phosphor.

• #20213 (GMS / Search V3 keyword ranking) With Search V3 keyword reads on (ELASTICSEARCH_ENTITY_INDEX_V3_KEYWORD_READ_ENABLED=true, or the V2 indices turned off), keyword search runs a new query, the Stage 1 query that DataHub Cloud runs as Search V2.5, and returns different results than the V2 query. This covers full-text search, structured (/q) queries, scroll, lineage search and browse with a query. Outside structured (/q) queries, which keep their boolean logic, terms are matched with OR instead of AND, short typos are tolerated (one edit for words of 5-6 characters, two for longer words), a synonym from the default synonym file that writes the same name another way (stg / staging, not glue / athena) also matches exact names unless the query is quoted, and a single word of 5 or more characters also matches inside names and titles (*word*). Search operators (-, +, |, parentheses, ~, *) are treated as plain text, so logging -snowflake also matches snowflake; use a structured query (/q) for boolean logic. Exact name matches rank above all partial matches, and the best matching field sets an entity's score instead of every matching field adding to it. A URN or a storage path (s3://, gs://, hdfs://) is matched on the identity fields and, as on V2, by an all-terms query over every field searched by default, which also finds entities that reference the URN. A custom search configuration's boolQuery wraps the new query: its should clauses only add to the score. Prefix matching covers only those identity fields, so custom no longer finds a customer_id column. With the default search configuration, a quoted query matches the name, title, qualified name, id and URN fields, and the description only for 4 or more words, without typos or matches inside names. To stay within OpenSearch's default limit of 1024 clauses (indices.query.bool.max_clause_count), a long query keeps only the words that fit (about 16 on datasets), a single word with more parts than that, such as a deep dotted path, keeps only its leading parts, and its typo tolerance narrows as it grows: short queries keep it in full, and queries of about 10 or more words on datasets are matched without it. The same query serves scroll requests (scrollAcrossEntities, the Python SDK's get_urns_by_filter, datahub delete --query), so a query there can select many more entities than before, even a single word: words are matched with OR, words of 5 or more characters also match words one or two edits away, a single word also matches inside names, and the parts of an identifier such as temp_table match separately. datahub delete --query prints the number of matches and asks before deleting unless --force is set. Action: none for V2 keyword reads, which are unchanged. With V3 keyword reads on, check the match count of scripted scroll or delete queries, and leave search.allow_expensive_queries at its default of true: when it is false, the search engine rejects the new query's fuzzy and wildcard clauses. To go back to the V2 query, turn V3 keyword reads off; a deployment that turned the V2 indices off must turn them on again and rebuild them first.

• #20216 (GMS / Search V3 light-first search) With Search V3 keyword reads on, a full-text search sorted by relevance first runs a light query without the fuzzy, wildcard and synonym-priority clauses, and runs the full query only when the light query matches nothing. The response always comes from one of the two queries: its results, total and facets all count the matches of the query that served it, so a total can be smaller than the full query's. A single word of ASCII letters, digits, _ and -, or a dotted name, searches only the name and identity fields (name, title, qualified name, id, display name, URN) on the light query (any other word, such as an email address, searches every field like a query of several words), so when any entity's name matches, entities that hold the word only in a description, an edited name, a column or another field are not returned. A word made of parts joined by _ or - is a light match where a name holds any one of its parts, so a column name such as customer_order_line_items returns the entities whose names hold customer, order, line or items, and nothing when no name holds a part, since it has 4 or more parts (DataHub Cloud recovers column matches with a separate query that OSS does not have); queries of several words search the analyzed text of every field except glossary term definitions and a few rarely useful fields (ldap, flowId, jobId, cluster), and not URN-valued fields such as tags, glossary terms and domains, or keyword fields such as email addresses. A word that mixes letters and digits without _ or -, such as orders2017, is a light match only where a name holds it whole or holds all of its parts. The ID and long-name stops apply after the light query, so an ID held only in a description or custom property also returns nothing when no name holds one of its parts, and one held only in a keyword field such as an email address returns nothing when the light query finds nothing. Facet counts, including the filter dropdowns and the browse sidebar, come from searches that the light query serves like the results page, while browse folders always come from the full query, so a sidebar count can be lower than the sum of its folders. A Column Name filter runs the full query directly. A search result download uses scroll, which runs the full query, so it can hold more rows than the search showed. When the light query finds nothing, the full query is not run for a query without spaces that holds 6 or more consecutive digits (an ID or hash) or has 4 or more parts separated by _, -, ., / or : (a long name). Searches sorted by another field, quoted, structured (/q) and URN or path queries, and scroll requests, run the full query directly. Explain (/explainSearchQuery and the includeExplain search flag) explains the query that served the search; /explainSearchQuery decides on the explained entity type alone, and explains the full query for a scroll id or keep-alive, or a size of 0, since those searches never run the light query. Lineage searches with a query use the light query too; above 50,000 relationships they search in batches, and each batch picks its own query. Action: none; V2 keyword reads are unchanged.

• #19955 (Ingestion / ClickHouse) ClickHouse query-log ingestion now uses the tables and columns resolved by system.query_log for SELECT usage and deduplicates repeated query shapes before aggregation. Runs with include_usage_statistics enabled may report different usage totals and field counts. Runs with include_query_log_lineage enabled may report different lineage edges. Filtered objects, resolved columns, and repeated executions are now handled more accurately. Action: none. No recipe changes are required.

• #20222 (Ingestion / ClickHouse) A new include_config_file_dictionaries option (off by default) ingests dictionaries defined in ClickHouse server config files (XML or YAML) as datasets, so dictGet lineage points at the real dictionary instead of a placeholder. It needs two extra grants for the DataHub user: SELECT ON system.dictionaries and SHOW DICTIONARIES ON *.*. With the option on, a failed read of system.dictionaries fails the run. Action: none. To use it, grant both and set include_config_file_dictionaries: true.

• #20164 (Ingestion / datahub-documents) The source's skip check now hashes the embedding provider and model resolved from the server, not the recipe's local embedding block. Recipes without a local block (the default) previously skipped documents hashed while semantic search was off as unchanged after it was turned on, and never re-embedded after a server-side model change. A failed lookup of the server's embedding config now fails the run, instead of silently skipping embedding. Action: none. After upgrading the CLI, deployments with semantic search enabled re-embed every document once.

• (GMS / search) Searches that send no filter at all (for example a keyword-only searchAcrossEntities, search, or scrollAcrossEntities call, or the OpenAPI entity search endpoints) now honor SearchFlags.filterNonLatestVersions (default true), so they no longer return superseded members of a version set. Previously the latest-version clause was only added when the request carried a filter, so the same query returned older versions without filters and hid them with any filter. Entities that are not versioned are unaffected. Action: none; set filterNonLatestVersions: false to keep returning every version.

• (GMS / search) Existing Data Product membership is now backfilled into the asset-side dataProducts search field by ResyncDataProductAssetsStep. The previous path (re-UPSERT dataProductProperties via MigrateAspects / REPROCESS_DATA_PRODUCT_ASSETS, introduced in v1.7.0) was a no-op: GMS suppresses MCLs for identical payloads, so DataProductAssetsSideEffect never wrote member assets and the Data Product search filter stayed empty for pre-existing memberships. The step now reads stored properties, synthesizes RESTATE items, and applies DataProductAssetsSideEffect directly. It is enabled by default (BOOTSTRAP_SYSTEM_UPDATE_DATA_PRODUCT_ASSETS_ENABLED=true) and skips once its versioned upgrade marker has SUCCEEDED; REPROCESS_DATA_PRODUCT_ASSETS=true forces a re-run. Action: none — the next system update writes the missing aspect. New env vars: BOOTSTRAP_SYSTEM_UPDATE_DATA_PRODUCT_ASSETS_ENABLED, BOOTSTRAP_SYSTEM_UPDATE_DATA_PRODUCT_ASSETS_DELAY_MS, BOOTSTRAP_SYSTEM_UPDATE_DATA_PRODUCT_ASSETS_LIMIT.

• (GMS / Hard delete) New setting RELIABLE_HARD_DELETE_ENABLED (featureFlags.reliableHardDelete, off by default). With it on, a whole-entity hard delete deletes only what existed when the request arrived: if an aspect was written in between, the entity stays with that newer data. This applies to the Rest.li delete action without an aspect name, the OpenAPI v3 and v2 DELETE .../entity/{entityName}/{urn} without aspects or clear, the typed OpenAPI v2 entity delete (DELETE /openapi/v2/entity/{entity}/{urn}), the deprecated OpenAPI v1 DELETE /openapi/entities/v1/ with soft=false (which captures every listed entity before deleting any), and hard deletes GMS makes through its entity client (the GraphQL delete mutations). Responses are unchanged, except that a hard delete written to while it runs now fails with a message to try the delete again instead of returning success; reference cleanup found through the graph (tags, glossary terms, domains, owners and similar), where a caller does it, also no longer overwrites a concurrent edit to the referencing aspect; the search-based cleanup for forms and structured properties is unchanged (it rewrites such an aspect at most ENTITY_SERVICE_DELETE_REFERENCE_WRITE_LIMIT times, entityService.deleteReferenceWriteLimit, default 3). On Cassandra the bound is best-effort (rows are not locked); elsewhere an aspect created for the first time while the delete runs is kept, and the delete then fails as above. The typed OpenAPI v2 aspect delete (.../{urn}/{aspect}) first deletes that aspect as before, unbounded; only the whole-entity delete that follows is bounded. Time-series data is deleted as before (the Rest.li delete action still removes it up to endTimeMillis, or all of it without one). Set RELIABLE_HARD_DELETE_ENABLED=false for the previous behavior. Action: none.

• (Frontend / Access logging) Every response that datahub-frontend proxies to GMS now carries an X-DH-JTI response header holding the jti (JWT ID) claim of the access token or UI session token that made the request. The value is a random UUID that identifies the token; it cannot be used to authenticate, and the caller already holds the token it was read from. A reverse proxy or service mesh in front of the frontend can log it to attribute requests to a specific token, and may strip it before responding to clients. Requests that carry no JWT bearer token do not get the header. Action: none.

• #19430 (Ingestion / BigQuery) Date-suffixed tables that sit beside an un-suffixed table of the same name — my_table next to my_table_20251120 and my_table_20251121 — are no longer treated as shards of my_table. A genuine sharded set has no un-suffixed member, so when one exists the dated tables are separate physical tables (a dated copy, a backup) rather than shards. Previously all of them collapsed onto the base table's URN, and whichever happened to be written last overwrote the real table's schema and profile, so row counts and column counts could be wildly wrong and could change between runs with no other change. With the default enable_legacy_sharded_table_support: true the dated tables are now skipped, reported as a Date-suffixed tables skipped warning naming the base table, and counted one per table in the new num_sharded_tables_shadowed_by_base_table report field; with enable_legacy_sharded_table_support: false they keep a distinct _yyyymmdd URN and are ingested as before. Genuine sharded sets are unaffected either way — each set still collapses to one dataset built from its newest shard, at proj.ds.foo with the default and at proj.ds.foo_yyyymmdd when the flag is disabled, exactly as each does today. Action: none — the affected base tables get correct stats on the next run. Rename the dated copies, or set enable_legacy_sharded_table_support: false, if you want them ingested as their own datasets.

• #19957 (System Update / Usage events) On deployments where the first usage event was written before the usage event index template existed, <prefix>datahub_usage_event is a plain index whose type field is mapped as text, which makes audit event search return 500 and leaves analytics charts that filter by event type empty. System update now moves such an index onto the layout the template defines (a data stream on Elasticsearch, a rollover alias over <prefix>datahub_usage_event-000001 on OpenSearch). It clones the index to <prefix>legacy_datahub_usage_event_<timestamp>, recreates the usage event index once the clone holds every event, copies the events back, and deletes the clone when the copy accounts for all of them. Usage events sent during the few seconds the old index is write-blocked are dropped, analytics history is incomplete until the copy finishes, and on Elasticsearch events with no timestamp at all are dropped. System update waits up to 5 minutes for the copy and checks on it again on its next run; a copy that was interrupted is started again as long as the index has not rolled over since. If the copy reports failures, system update logs the first one and keeps the clone without copying it again, since a second copy can duplicate events once the index has rolled over; copy the remaining events into <prefix>datahub_usage_event yourself (op_type: create), then delete the clone. If the old index is gone but a copy cannot be started, system update fails so its retries start it; a copy still running after the wait is checked again on the next run. Only one system update migrates at a time: it holds <prefix>legacy_datahub_usage_event_lease while it moves the old index aside and starts the copies, and takes over a lease older than 30 minutes. Cases that need you (a copy that reports failures, an index that rolled over since an unfinished copy, a write block that cannot be lifted, or credentials that may not use the lease or the tasks API) are logged and counted on the Micrometer counter datahub.system_update.legacy_usage_event_migration with outcome=manual_recovery; outcome=skipped counts runs that left the migration to another system update. System update needs to create and delete <prefix>legacy_datahub_usage_event_* indices and to use the tasks API. Deployments whose usage event index was created from the template are unaffected. Action: before upgrading an affected deployment, make sure the search cluster has free disk space of about the size of that index, which the copy needs until the clone is deleted. Set SKIP_LEGACY_USAGE_EVENT_INDEX_MIGRATION=true on the system update job to postpone the migration; if a system update was interrupted mid-migration, the old index can be left write-blocked and its lease held; the next system update waits up to 11 minutes for that run's clone or lease to go stale, then migrates or lifts the block itself, and a system update once the lease is 30 minutes old takes it over and migrates. To skip that wait, lift the block with PUT <prefix>datahub_usage_event/_settings {"index.blocks.write": false} and, once no system update is running, delete <prefix>legacy_datahub_usage_event_lease before postponing. On OpenSearch the usage event step of an affected deployment fails until its index is migrated, as it did before this change; SKIP_CREATE_USAGE_EVENT_INDICES_STEP=true skips that step entirely, and deployments that set it to get past that failure need to unset it for the migration to run.

• (Frontend / health and proxy backpressure) The Play frontend caps concurrent requests that wait on an upstream (DATAHUB_FRONTEND_PROXY_MAX_IN_FLIGHT, default 1024, the same as Pekko HTTP max-connections). GMS (/api, /openapi), login and signup, SSO, and in-flight OTEL forwards share that one budget. Browser trace export also keeps its own cap of 100, so a collector storm returns 503 before it can use the rest of that budget. Above the cap, those requests return 503 with Retry-After: 1 instead of holding a Play connection until the upstream responds (GMS proxy waits up to 120s). The management listener (MANAGEMENT_SERVER_PORT, default 4319) serves GET /health/live (200 while the process can answer) and GET /health/ready (503 while starting, shutting down, or while in-flight upstream calls are above 90% of the cap; ready again at or below 70%). Those two probes are counted on frontend_management_health_inflight. They are not part of Play's play_http_requests_inflight, which still counts /admin and /health on port 9002. Docker HEALTHCHECK uses /health/live. GET /admin and GET /health on port 9002 still respond, and are deprecated for health checks (see Deprecations). Action: none for current Helm charts, which still probe /admin on 9002. Point liveness at /health/live and readiness at /health/ready on port 4319 when the chart is updated, and move any other /admin monitor off that path before the route is removed. Clients that send large bursts through the frontend may see 503 until in-flight calls drain; raise DATAHUB_FRONTEND_PROXY_MAX_IN_FLIGHT if 1024 is too low.

• #19973 (Ingestion / dbt) When target_platform_instance is set, a target-platform sibling entity the warehouse connector has not ingested (a stub dbt creates for a table the warehouse connector skips, for example one excluded by its deny pattern) is now given the container of an ingested table in the same database and schema, in addition to its browse path. Those entities previously sat directly under the platform instance; they now appear inside the real database/schema folder, are counted in that container's contents, and are matched by container-scoped search filters. Nothing is created: where the warehouse connector has ingested nothing from a schema the entity simply stays under the platform instance, and where tables the manifest places in one schema turn out to be in different containers it stays there too and a warning is reported. Action: none required — this takes effect on the next dbt run for recipes with target_platform_instance set, and is covered by the existing emit_target_platform_instance_aspects flag if you need to turn it off.

• (Ingestion / dbt) dbt semantic models and metrics can now be described with first-class semanticModel and metric entities, controlled by the new emit_semantic_model_entities setting. When it is on, the connector emits one semanticModel per dbt project holding the join relationships MetricFlow derives from matching entity names, one metric per measure with create_metric: true and per top-level metrics: entry (dbt Core only - the dbt Cloud Discovery API does not expose the metrics: block), and adds semantic field annotations plus a link to the parent semantic model onto the datasets the connector already emits. Nothing existing changes: the dataset urns, their datasetProperties, schema, subtype, tags, owners and lineage are all exactly as before, so there is nothing to migrate and no re-ingest needed to keep what you have. The setting is three-valued: left unset it follows the server, turning on against DataHub Cloud new enough to register these entity types with the Metrics feature enabled, and staying off on OSS, older Cloud, and runs with no server connection (for example a file sink); true asks for it and is refused with a reason in the ingestion report where the server cannot accept it; false keeps the old behavior. The new entities are keyed by dbt project name, read from manifest.metadata.project_name for dbt Core and from the semantic models' package name for dbt Cloud - set semantic_model_project_name to pin it, since it is part of their identity. Action: none. To keep a DataHub Cloud run emitting only the datasets, set emit_semantic_model_entities: false.

• #20017 (Sigma ingestion) Sigma chart columns gain column lineage in three cases that previously pointed to themselves: a column computed only from other columns of the same chart and parameters (for example Sum([Amount])) now inherits their upstreams; a reference through a Data Model join ([Join Element/Joined Table/Column]) now resolves to the joined table's column when every named table is one the join element's lineage lists as a source and is an element of the same Data Model (a join against a warehouse table or another Data Model stays unresolved, and so does a relationship reference unless it shares a joined table's name); and a reference to a Data Model element the workbook loads, but that Sigma does not list as the chart's upstream, resolves when exactly one element of that name has the column, no other loaded Data Model could hold a same-named element with that column, and the chart has no source of that name (dataset, warehouse table, pivot table or input table). A join edge is column-level only: the chart's entity-level inputs already include the join element. A loaded-Data-Model edge also adds that element to the chart's entity-level inputs. The loaded-Data-Model lookup runs only for charts whose own lineage Sigma returned in full, and whose Sigma Dataset sources have known warehouse tables. Both Data Model lookups need ingest_data_models (on by default); with it off they add no edges. Action: none. Expect column-lineage counts on Sigma charts to rise on the next run.

• #20000 (Sigma ingestion) A chart column's formula reference to another chart or a Data Model element now becomes column lineage only when that upstream has the referenced column; references to upstreams whose columns are unknown are unchanged. References to columns the upstream does not have no longer produce lineage edges, and a three-part reference such as [Element/Relationship/Column] resolves only when the upstream's known columns include the rest as one column name (for example a column named Rev/Cost); a column whose formula has no other resolvable reference points to itself instead. Element names now match case-insensitively, as Sigma does, so some references that were skipped now resolve, including to warehouse tables. The chart_input_fields_case_mismatch report counter now counts only references that match several names differing only in case. Action: none. Expect column-lineage counts on Sigma charts to change on the next run, mostly dropping as incorrect edges are removed.

• (GMS / Metrics) Metrics & Semantic Models are now enabled by default (METRICS_ENABLED=true) from DataHub Core v1.8.0 and DataHub Cloud 2.3.0. The Metrics item appears in the left navigation sidebar with a Beta badge, and /metrics is available without setting an env var. Set METRICS_ENABLED=false on GMS to hide the feature. Existing Snowflake Semantic View datasets are unchanged; see the migration FAQ if you need to move them to the new model.

• (Ingestion / BigQuery linked datasets) With include_linked_dataset_lineage: true and include_schema_metadata: false, the lineage recipe now emits upstreamLineage COPY edges for linked datasets with no recipe change. A linked dataset's views and snapshots are now filtered by view_pattern and table_snapshot_pattern instead of table_pattern; table_snapshot_pattern defaults to allow all with no table_pattern fallback, so a narrow table_pattern alone no longer excludes a linked dataset's snapshots.

• #19837 (GMS / Search V3 document embeddings on OpenSearch) Enabling semantic search for document after Search V3 has already created documentindex_v3 is not an in-place mapping update. OpenSearch k-NN 2.19+ treats index.knn as final, so adding knn_vector while that setting is false is rejected. Enable semantic search for document before or together with Search V3 write so the index is created with index.knn: true. If the V3 document index already exists without it, set ELASTICSEARCH_INDEX_BUILDER_SETTINGS_REINDEX=true and run system-update (or recreate documentindex_v3). Elasticsearch uses field-level dense_vector and does not need index.knn. Action: none for new OpenSearch deploys that turn on document semantic with Search V3; existing documentindex_v3 on OpenSearch needs a settings reindex as above.

• #19715 (Ingestion / LookML remote dependencies) The LookML source now reads remote_dependency and local_dependency entries declared in manifest.lkml; a key-name bug previously left both lists empty. remote_dependency entries are cloned from a URL any LookML committer can change, so the executor now makes outbound git clone calls it did not make before. local_dependency entries are local project references and are not cloned, but those local projects are now traversed, so LookML constant: values defined in their manifests now resolve where they did not before. If your views reference such a constant in sql_table_name, the upstream dataset URN for those views changes on this upgrade (previously the constant stayed unresolved and logged a "LookML constant not found" warning), which moves the affected lineage. Each URL is validated first: only https and SSH Git URLs are allowed, and loopback, link-local, and cloud-metadata hosts are always rejected; anything else is skipped with a warning. The optional remote_dependency_domain_pattern (an AllowDenyPattern, default allow-all) restricts which hosts may be cloned. Action: if your manifest.lkml declares remote_dependency URLs, set remote_dependency_domain_pattern to the hosts you expect. The default denylist covers only loopback, link-local, and cloud-metadata addresses, so internal or private-network hosts stay reachable; and because git re-resolves the host at clone time, restricting executor network egress to your expected hosts (not the allowlist alone) is what closes DNS rebinding.

• #20219 (GMS / Search V3 hybrid search) Full-text search on Search V3 can now rerank documents with their vectors. Set ELASTICSEARCH_ENTITY_INDEX_V3_HYBRID_READ_ENABLED=true, which requires V3 keyword reads, ELASTICSEARCH_ENTITY_INDEX_V3_SEMANTIC_READ_ENABLED=true, semantic search with an embedding provider, and a mapping for the active embedding model with a positive vectorDimension and a cosine, L2 or inner-product spaceType. The Search V3 cluster needs OpenSearch 3.5+ or Elasticsearch 8.18+, and DataHub refuses to start on older OpenSearch. Among the first 100 keyword results of a search sorted by relevance, entity types that have vectors (document by default) are reordered by a combined keyword and vector score among the positions they already hold, and every other result keeps its keyword position. Totals and facets do not change. A search makes one query embedding call when its first 100 keyword results include a document, and repeats of a query within a minute reuse it. Action: none; the flag is off by default, and DataHub refuses to start with it on unless those requirements are met.

• #19934 (GMS / Search V3 semantic reads) Semantic search can now be served from the Search V3 document index. Set ELASTICSEARCH_ENTITY_INDEX_V3_SEMANTIC_READ_ENABLED=true (requires ELASTICSEARCH_ENTITY_INDEX_V3_ENABLED=true) once the V3 document index holds your document embeddings (documents embedded before V3 writes were on need a re-index into V3 or a re-embed); it is independent of ELASTICSEARCH_ENTITY_INDEX_V3_KEYWORD_READ_ENABLED. Only entity types with their own V3 index and embeddings are searched (document by default). The flag needs OpenSearch 3.5+ or Elasticsearch 8.18+ on the Search V3 cluster; DataHub services given it (GMS, and system-update or the consumers if they share the setting) refuse to start on older OpenSearch. Action: none; semantic search keeps reading the existing semantic indices until you set the flag.

• #19930 (Ingestion / MSSQL dataJob lineage) Five changes to how stored-procedure lineage is written.

  • With incremental_lineage: true, dataJobInputOutput is emitted as a patch instead of a full upsert, so lineage a user added by hand survives re-ingestion. A patch only adds, so stale auto-detected lineage is no longer pruned. It writes the *Edges fields rather than the deprecated plain arrays, and entries left in those arrays by earlier runs persist, because the server-side patch template cannot reach them. If ingestion parses the same upstream a user added, the edge loses its UI stamps and stops being reported as manually added; the lineage itself is unaffected. A SQL Agent job step's lineage aspect is always empty, so it is now dropped rather than re-stated as an empty upsert.
  • Regardless of the flag, a procedure's lineage aspect is kept when it carries lineage in any field rather than only in the dataset arrays, so a body that just calls other procedures keeps its inputDatajobs. Such a procedure emitted no lineage aspect before; see Breaking Changes for the effect on the default setting.
  • Regardless of the flag, a stored procedure that cannot be processed no longer aborts the rest of its schema. It is still reported as a failure, so stale-entity soft-deletion stays suppressed for that run. A procedure whose catalogue dependency query fails keeps its other metadata but loses its dependency properties, including values ingested by earlier runs, because dataJobInfo is a full upsert.
  • Regardless of the flag, column lineage remapped from a table alias to several real tables is emitted as one entry per downstream instead of one entry with several. The server keys a patch on a single downstream and rejects the multi-downstream shape.
  • Regardless of the flag, a call to a procedure this run did not ingest is dropped rather than emitted as lineage. That covers SQL Server built-ins such as sp_rename and msdb.dbo.sp_send_dbmail, which otherwise resolved to a dataJob URN nothing creates, and procedures excluded by procedure_pattern. A genuine call into a database excluded by database_pattern is dropped with them.

  Action: none on the default incremental_lineage: false -- procedure-to-procedure lineage appears without it.

  If the flag is already on (it is shared with view and query lineage, so it may be on for unrelated reasons), procedure lineage switches to patch mode on upgrade. A multi-downstream column-lineage entry stored by an earlier run then makes every later patch to that dataJob fail, with the sink reporting Patching not supported on fine grained lineages with not exactly one downstream. The run is marked failed and its stateful-ingestion checkpoint is not committed, so this repeats until the stored entry is rewritten. To rewrite it, run ingestion once with the flag off. That run upserts every procedure that has lineage, not only the affected ones, so it clears lineage added by hand on any of them; in practice only edges added since the last run are at risk, because MSSQL upserted procedure lineage on every run before this release, with call-only procedures the exception.

  Either way, lineage from earlier runs remains in the plain arrays; the UI's lineage removal clears both and can be used if the duplication matters.

• #19799 (GMS / Lineage timeouts) Impact-analysis lineage traversals (searchAcrossLineage / scrollAcrossLineage) that exceed ELASTICSEARCH_SEARCH_GRAPH_TIMEOUT_SECONDS now fail with a distinct timeout instead of a generic 500. GraphQL returns error extensions code: 504 / type: DEADLINE_EXCEEDED (HTTP status stays 200 per GraphQL convention); Rest.li returns HTTP 504. In strict mode (ELASTICSEARCH_SEARCH_GRAPH_IMPACT_PARTIAL_RESULTS=false) a hop that runs out of time is now a timeout rather than truncated lineage returned as complete; in partial mode the response is marked partial. Timeouts are counted on the existing datahub.lineage.graph_walk.errors meter with error_type="timeout". The lineage-visualization path (getLineage) is unchanged. Action: clients or alerts keyed on SERVER_ERROR / 500 for lineage failures should treat 504 / DEADLINE_EXCEEDED as a timeout and narrow the query (fewer hops, tighter filters) or raise the timeout.

• #19751 (GMS / Column lineage APIs) Querying downstream lineage for a column (schemaField) now also returns metrics, charts, and dashboards that consume that column, not only other columns. This applies to GraphQL searchAcrossLineage and other lineage APIs that walk via LineageGraphFilters.forEntityType (including GraphService.getLineage). The reverse is not true: lineage for a metric, chart, or dashboard still does not return columns. The UI lineage graph builds column-level edges on its own and is unchanged by this PR. POST /openapi/v3/lineage/scroll on a column also still omits metrics, charts, and dashboards — that endpoint uses lineage-annotated registry edges, not the schema-field walk used by searchAcrossLineage. Action: If an API client assumes every column-lineage result is a schemaField, update it to accept metric, chart, and dashboard (or filter those types out). No action if you only use the UI or OpenAPI lineage scroll.

• #18982 (GMS / Groups) Fixed two bugs in the inline migration of a group from legacy to native membership. The migration only read the first 500 members, so members past that were left on the legacy aspect while the group was marked native — which permanently disabled the retry. It also stamped the group native before granting native membership, so an interrupted run could leave members in no group at all and unable to re-migrate. Member reads and writes on the removal path are now batched as well. Action: none. A group left half-migrated by an earlier run is migrated on the next addGroupMembers or removeGroupMembers call only if its origin was never stamped; otherwise remove and re-add the affected members.

• (GMS / AWS IRSA) GMS owns a single process-wide DefaultCredentialsProvider constructed with builder().build() (not deprecated create()) and closed on shutdown. The AWS JDBC wrapper IAM plugin is bound to that provider (AwsCredentialsManager.setCustomHandler) so MySQL IAM token minting does not allocate a new default chain per reconnect. The wrapper is pinned to 4.4.0 (wrapperPlugins=iam only; MySQL/Postgres drivers unchanged). Iceberg warehouse credential vending reuses a shared or warehouse-scoped StsClient instead of building a new client per vend. AWS SDK Java v2 is pinned to 2.54.16. On EKS IRSA, live StsAssumeRoleWithWebIdentityCredentialsProvider counts should stay near-flat (jcmd <pid> GC.class_histogram); growth of ~1.5k/hour was a credential-chain leak. Action: none; canary is histogram count, not RSS.

• (Document embeddings) The datahub-documents source no longer shifts embeddings onto the wrong chunk when a blank chunk sits between non-blank chunks (characterOffset still counts every chunk of the original text). Documents whose earlier semanticContent contained such a blank chunk stay misaligned until re-embedded, and the source skips documents whose source text hash is unchanged in incremental mode, so operators should force a re-embed for one run by setting force_reprocess: true (or disabling incremental_mode) in the recipe.

• (Document embeddings) The datahub-documents source no longer shifts embeddings onto the wrong chunk when a blank chunk sits between non-blank chunks (characterOffset still counts every chunk of the original text). Documents whose earlier semanticContent contained such a blank chunk stay misaligned until re-embedded, and the source skips documents whose source text hash is unchanged, so operators should force a re-embed for one run by setting incremental.force_reprocess: true (or incremental.enabled: false) in the datahub-documents recipe. Sources that embed documents inline as they ingest them (Notion, Confluence) re-embed a document when their own change detection sees it as new or changed.

• (Document embeddings) The datahub-documents source no longer shifts embeddings onto the wrong chunk when a blank chunk sits between non-blank chunks (characterOffset still counts every chunk of the original text). Documents whose earlier semanticContent contained such a blank chunk stay misaligned until re-embedded, and the source skips documents whose source text hash is unchanged, so operators should force a re-embed for one run by setting incremental.force_reprocess: true (or incremental.enabled: false) in the datahub-documents recipe. Sources that embed documents inline as they ingest them (Notion, Confluence) re-embed a document when their own change detection sees it as new or changed; to re-embed their already-ingested documents without waiting for a change, run datahub-documents once with incremental.force_reprocess: true, which also covers external documents (restrict it with platform_filter: [notion, confluence] if needed).

• (Document embeddings) The new classical embedding provider is a deterministic lexical hashing provider for CI, smoke tests and quickstarts, not semantic search. GMS refuses EMBEDDING_PROVIDER_TYPE=classical unless CLASSICAL_EMBEDDING_ACKNOWLEDGE_LEXICAL_ONLY=true is also set, and logs a warning while the provider is active. Action: none unless you opt in.

• (Document embeddings) The datahub-documents source no longer applies embedding.rate_limit / documents_per_minute to in-process embedding providers (onnx, and the new classical); they have no external API to protect, and a configured limit is now logged and ignored for them. Action: none.

• (Document embeddings) A server configured with the new EMBEDDING_PROVIDER_TYPE=classical needs an ingestion CLI from the same release or later. An older CLI pointed at such a server cannot map the provider, logs a warning, and ingests documents without embeddings. Action: upgrade the CLI (including any executor) before switching a server to classical.

• (GMS / Java services) Spring Framework 7.0.9, Spring Boot 4.0.8, and Spring Security 7.0.7 (current 7.0 / 4.0 patches). The temporary JDK 25 ClassFile-reader classpath shim is removed; the OOM fix shipped upstream in 7.0.9. Action: none for operators; rebuild/redeploy picks up the new JARs. Custom GMS plugins compiled against 7.0.8 remain binary-compatible.

• (GMS / Split search clusters) You can optionally send Search V2, Search V3, semantic, graph, timeseries, system-metadata, and usage traffic to named Elasticsearch/OpenSearch clusters. Routing defaults to primary; unknown cluster names fail at startup. Each cluster's shim auto-detect / engineType is independent of primary. Existing ES health routes (/check/ready, /check/elastic, /debug/elastic, /health/detailed) now require all unique configured clusters to be non-RED; probes are still cached (HEALTH_CHECK_CACHE_DURATION_SECONDS, default 5s). /health is unchanged (bootstrap only). Index names default to INDEX_PREFIX on every cluster; a non-blank ELASTICSEARCH_CLUSTERS_*_INDEX_PREFIX is used for reads and writes of families on that cluster. Blank overlay still inherits INDEX_PREFIX (or a custom IndexPrefixResolver). Single-cluster deploys need no new config. See the cluster and ELASTICSEARCH_COMPONENT_CLUSTER_* tables in environment variables.

• (View authorization) VIEW_UNRESTRICTED_ENTITY_TYPES_ADD and VIEW_UNRESTRICTED_ENTITY_TYPES_REMOVE now mutate the complete effective default list instead of replacing it. Operators can add or remove one entity type without restating every default unrestricted type. Stock application defaults live on VIEW_UNRESTRICTED_ENTITY_TYPES (merged with the registry baseline). An explicitly empty value keeps only the registry baseline.

• (GMS / AWS clients) Shared object-storage S3Client and StsClient no longer fall through to the AWS SDK default credential chain when only a region is set. That path allocated a new IRSA StsAssumeRoleWithWebIdentityCredentialsProvider per builder. Clients are created from the shared DefaultCredentialsProvider bean, LocalStack dummy credentials (AWS_ENDPOINT_URL), or skipped. Iceberg catalog FileIO uses warehouse-vended static keys only. Action: none if GMS already has a shared AWS credentials bean or LocalStack; object storage / STS stay unavailable until credentials are explicit.

• (GMS / GraphQL thread pool) When GRAPHQL_CONCURRENCY_SEPARATE_THREAD_POOL=true, pool sizes no longer default to availableProcessors() * 5 / * 100 (which followed node vCPU on Kubernetes). Defaults are 8-core equivalents: core 40, max 800, SynchronousQueue (GRAPHQL_CONCURRENCY_QUEUE_SIZE=0). GraphQL resolver fan-out is blocking I/O, so a small bounded queue plus CallerRunsPolicy on the Jetty thread can starve nested work; set QUEUE_SIZE > 0 only if you want an ArrayBlockingQueue. Restore node-scaled sizing with GRAPHQL_CONCURRENCY_SCALE_WITH_PROCESSORS=true, or sentinels CORE_POOL_SIZE < 0, MAX_POOL_SIZE <= 0. The dedicated pool remains off by default.

• (Ingestion / Dependencies) acryl-datahub no longer caps setuptools, so it installs alongside setuptools>=83 (which fixes CVE-2026-59890). setuptools>=82 dropped pkg_resources, which the redshift/cockroachdb SQLAlchemy dialects still import at load, so ingestion installs a small pkg_resources compatibility shim at runtime before those load (temporary; removed once sqlalchemy>=2 is adopted). stopit's timeout helper is now vendored into datahub.utilities (MIT; behavior unchanged, still CPython-only), so the powerbi extra no longer depends on stopit and is fully decoupled from setuptools. Action: none. Installs under Airflow constraint files that pin an older setuptools are unaffected; the >=83 floor is applied at lock time and in the datahub-actions image; the datahub-ingestion image resolves setuptools transitively (currently >=83, though not pinned for that image).

• (GMS / GraphQL containers) Container.relationships(types: [IsPartOf], direction: INCOMING) again returns all contained entities from the live graph (datasets, charts, dashboards, nested containers, etc.), matching pre–entity-graph-cache behavior. The nesting-only container graph cache remains for parentContainers, VBAC, and search filter expansion — not for this relationships listing. Prefer Container.entities when you need search-filtered asset listing under a container.

• #19440 (Ingestion / Cube) Cube views are ingested with subtype Semantic Model instead of View (dataset URNs unchanged). Optionally set emit_semantic_model_entities: true to emit first-class semanticModel / metric / Semantic Model Dataset entities instead of a view dataset — Cube datasets themselves stay Cube. That path changes view URNs (the view dataset is no longer emitted); re-ingest with stateful ingestion to drop the old view datasets. Charts that queried a view then point at the view's logical datasets. Requires a server that registers semanticModel/metric (DataHub Cloud >= 2.1.0, or OSS with METRICS_ENABLED). Action: re-ingest Cube. Saved searches for subtype View should use Semantic Model. To opt into first-class entities, set emit_semantic_model_entities: true.

• (GMS / Search backend) OpenSearch 3.x is now a supported search backend alongside OpenSearch 2.x and Elasticsearch 8.x. The engine version is auto-detected; no configuration change is needed to move a deployment from OpenSearch 2.x to 3.x. Internally, OpenSearch I/O no longer uses the deprecated RestHighLevelClient transport (retired by the OpenSearch project as of 3.0) — both 2.x and 3.x are served by a unified client layer with unchanged wire behavior. Two OpenSearch 3.x caveats: (1) 3.x rejects new indexes using the nmslib kNN engine — deployments configuring semanticSearch.models.*.knnEngine: nmslib must switch to faiss (the default) or lucene; GMS fails startup with a clear error otherwise. (2) 3.x enforces the 512-byte document-id limit on bulk writes; very long schema-field URNs that indexed on 2.x are rejected on 3.x (logged as bulk failures). For new 3.x deployments, set ELASTICSEARCH_INDEX_DOC_IDS_SCHEMA_FIELD_HASH_ID_ENABLED=true; GMS logs a startup warning when running against 3.x with hashing disabled. Action: none for existing 2.x deployments.

• (GMS / Auth helpers) POST /auth/signUp, /auth/resetNativeUserCredentials, /auth/verifyNativeUserCredentials, and /auth/getSsoSettings now require the GMS system client credentials, matching /auth/generateSessionTokenForUser. Frontend login, signup, password reset, and SSO continue to work (they already call these helpers with the system client). A user session token calling GMS /auth/* directly receives 401. Action: none for standard deployments; if you called these GMS helpers with a user token, switch to the frontend routes or system client credentials.

• #19247 (CLI) datahub check get-kafka-consumer-offsets no longer crashes (AttributeError: 'str' object has no attribute 'items') against servers that return the newer consumer-offset response formats. The command now queries /openapi/operations/messaging/*/consumer/lag first and falls back to the deprecated kafka endpoints on older servers. The table columns changed from Topic/Consumer Group/Schema to Consumer Type/Consumer Group/Topic (the old labels did not match the row content), and a new -o/--output table|json option emits normalized rows as JSON. Action: update any scripts that parse the table output, or switch them to -o json.

• (GMS / Authorization) New platform privilege VIEW_SYSTEM_STATUS grants read access to non-sensitive system status: messaging consumer lag, messaging transport, and registered consumers. It does not include system information (/openapi/v1/system-info and related endpoints), full system configuration, raw index access, throttle/rate-limit controls, or other operational mutations. MANAGE_SYSTEM_OPERATIONS remains a superset. The privilege is not on the default Admin or root boot policies; grant it explicitly (for example to a smoke/release-test PAT).

• (GMS rate limiting) Rate-limit policy is loaded once by RateLimitEffectiveConfig: Spring binds application.yaml toggles from the environment; bundled classpath rate-limit-config.yaml is the default file; RATE_LIMITS_CONFIG_FILE replaces that document (not merged with it); RATE_LIMITS_CONFIG_JSON Jackson-overlays the result. Hazelcast bootstrap and the rate-limit engine share that instance so file/JSON endpoint.enabled / scoped.enabled still provision map config. A missing mounted file or invalid JSON fails GMS startup; MAE/MCE/upgrade evaluate Hazelcast conditions without the engine and do not fail on a missing overlay. Documents may use datahub.gms.rateLimits:, a rateLimits: wrapper, or a bare fragment. file:/etc/datahub/rate-limits.yaml and a bare path both work. RATE_LIMITS_CONFIG_FILE_ENABLED remains ignored. Action: none if Helm already sets RATE_LIMITS_CONFIG_FILE=file:/etc/datahub/rate-limits.yaml. After upgrade, confirm GET /openapi/v1/rate-limits/config shows the mounted endpoint.rules and GMS logs Registered endpoint rate limit rule.

• (Security / Dependencies) Logback (logback-classic / logback-core) bumped from 1.5.32 to 1.5.38 for CVE-2026-9828 (HardenedObjectInputStream overly broad java.lang/java.util deserialization allowlist; fixed in 1.5.33+) and CVE-2026-10532 (Proxy class deserialization; fixed in 1.5.38). Action: none for operators; rebuild/redeploy picks up the new JARs. DataHub does not expose Logback SimpleSocketServer / SimpleSSLSocketServer by default.

• (Security / Dependencies) Apache Log4j API/Core bumped from 2.25.4 to 2.25.5 for CVE-2026-49844 (MapMessage.asJson() emitting non-RFC-8259 tokens for NaN/Infinity). Action: none for operators; rebuild/redeploy picks up the new JARs.

• (Security / Dependencies) Apache Parquet stack (parquet-avro, parquet-hadoop, parquet-jackson) bumped from 1.17.1 to 1.18.0. parquet-jackson shades Jackson under shaded/parquet/; 1.18.0 refreshes that shade to jackson-databind 2.22.1, covering CVE-2026-54512/54513 and related jackson-databind issues that required 2.21.4+. Application-classpath Jackson remains separately pinned. Action: none for operators; rebuild/redeploy picks up the new JARs.

• (Security / Dependencies) Apache Parquet stack bumped from 1.18.0 to 1.18.1 for CVE-2026-73334 (org.apache.parquet.crypto.keytools: file-controlled KMS URL forwarded to a pluggable KmsClient without host validation). 1.18.1 disables file-controlled KMS URLs by default. Action: none for operators; rebuild/redeploy picks up the new JARs. DataHub does not use Parquet envelope encryption / keytools.

• (Security / Dependencies) Apache Thrift libthrift bumped from 0.23.0 to 0.24.0 for CVE-2026-48586 (TZlibTransport decompression does not consistently count inflated bytes against the max message size, allowing a zip-bomb DoS). The artifact is forced transitively via Jena ARQ. Action: none for operators; rebuild/redeploy picks up the new JARs.

• (Security / Dependencies / Frontend SSO) pac4j (pac4j-core / pac4j-oidc) bumped from 6.4.2 to 6.5.8 and play-pac4j from 13.0.2 to 13.0.3 for CVE-2026-82461 (Keycloak access-token roles trusted without signature/issuer/audience/expiry validation), CVE-2026-82462 (OIDC callback accepted an access token with no authorization code or ID token, allowing session forgery via token substitution), CVE-2026-82463 (reversed CheckProfileTypeAuthorizer check), CVE-2026-82464 (backslash-prefixed logout open redirect) and CVE-2026-82465 (unsigned SAML LogoutRequest could destroy a session); 6.5.8 also rejects logout redirect URLs containing control characters or spaces. Transitively bumps nimbus-jose-jwt to 10.9.1, oauth2-oidc-sdk to 11.38.2 and json-smart to 2.6.0. Action: none for operators; rebuild/redeploy picks up the new JARs. The OIDC callback now requires an authorization code or ID token, which is already the case for the standard authorization-code flow DataHub uses.

• (Ingestion / Snowflake) Semantic views now keep their logical tables' stored casing. Logical table names were previously uppercased before use, which mangled the name shown on each logical dataset (a table declared Orders appeared as ORDERS) and silently merged logical tables that differ only by case — legal in Snowflake over two different base tables — so one table's dataset, columns and metrics were dropped, and in the non-semantic-model path its columns were given column-level lineage pointing at the wrong physical table. Primary-key and join-key column names follow the same rule and now honour preserve_column_case. Action: none for most deployments — with the default convert_urns_to_lowercase: true the dataset URNs are unchanged, because the name was lowercased after being uppercased and still is. If you run semantic_views.enabled: true and convert_urns_to_lowercase: false and a logical table whose name is not already uppercase, that table's dataset URN changes (for example ….ORDERS becomes ….Orders) and the previously ingested entity is orphaned; soft-delete the old one or re-ingest with stateful ingestion so it is cleaned up.

• (Ingestion / Snowflake) New preserve_column_case option keeps Snowflake's column casing in field paths and column-level lineage. Snowflake's quoted identifiers let columns that differ only by case (for example "col" and "COL") coexist in one table; until now both were lowercased to the same field path, and the duplicate was then dropped before the schema was written, so one of the columns disappeared from DataHub entirely along with its lineage, tags and profile. The connector now warns whenever columns collapse onto one field path, so affected tables can be found without changing any configuration. Action: none by default — preserve_column_case is false, and output is unchanged. If the new warning appears, set preserve_column_case: true to keep the columns distinct. It is also worth enabling without any collision if you use quoted mixed-case columns at all: a column stored as "MixedCol" can only be queried by that spelling, so the lowercased field path does not name anything you can select. Note that the option applies to every column, not only quoted or mixed-case ones — with it on, an unquoted column stored as CUSTOMER_ID keeps that casing in its field path instead of being lowercased to customer_id. Treat this as a one-way door: the option is part of every column's schemaField URN, so enabling it after data has been ingested re-keys every column and orphans column-level tags, glossary terms and documentation attached in the UI. Set it once, before the first run, and set it the same way on any snowflake-queries recipe pointed at the same account, or their field paths will not line up. Check other sources that resolve Snowflake columns before enabling it. Tableau resolves Snowflake upstream column names against the ingested Snowflake schema, on both its datasource and virtual-connection lineage paths, so it follows whichever casing this option produces and needs no setting of its own. That does make ingestion order matter: ingest Snowflake before Tableau, because Tableau matches against the Snowflake schema already in DataHub, and if Tableau runs first it falls back to lowercasing and its column-level lineage stays missing until the next Tableau run. Looker, Mode and Hex already adopt the ingested casing and need no change.

• (Ingestion / SQL Profiling) The SQLAlchemy profiler no longer loses columns that differ only by case. Dialects that normalize identifiers (Snowflake, Oracle) fold reflected column names, so a table with quoted identifiers such as "col" and "COL" had one of them silently dropped during reflection and left unprofiled. The profiler now re-reads the column list and addresses each column by its as-stored, quoted name. Tables without such a collision, and dialects that do not normalize identifiers, are unaffected. Recovering the column does not always mean you will see two sets of statistics: the profile has to attach to a field the source declared, so on Oracle — whose source folds column names when it builds the schema — a case-only pair stays a single field and only one column's statistics can attach. The profiler now warns in that case, naming the table and the folded path, instead of dropping the second profile silently. Relatedly, Snowflake profile field paths now line up with schema field paths: a column whose stored name was mixed-case and quoted (for example "MixedCol") previously produced a profile that never attached to its schema field, because the driver preserved the casing while the schema lowercased it. Action: none. Profiles that were silently unattached now attach, so previously blank column statistics start appearing after the next profiling run. If the new warning about dropped column statistics appears, the two columns share one field path and only one column's statistics are kept — which one depends on column order, so treat the surviving values as belonging to either column. The warning names the table and the folded path so the affected tables can be found.

• #19005 (Spark lineage / acryl-spark-lineage) Fixed three defects that silently dropped lineage from the shaded Spark agent, all introduced when the agent began relocating its bundled dependencies. (1) Kafka structured-streaming reads produced no input datasets at all, because relocating org.apache.kafka rewrote the class-name constant OpenLineage compares against and Spark supplies the canonical TopicPartition. (2) Redshift relation lineage was never emitted, because spark-redshift-community is provided at runtime but was being relocated. (3) All of the agent's vendored OpenLineage customizations were inert in the shipped jar: our copy and the upstream copy were both packaged at the same path and the JVM resolved the upstream one. Separately, the JNI-backed compression libraries (snappy-java, zstd-jni, lz4-java) and JNA are no longer bundled — relocating them broke native symbol resolution, so enabling compression.type on the Kafka emitter failed with UnsatisfiedLinkError; Spark supplies all of them. Action: none required, and Kafka/Redshift lineage starts appearing after upgrade. Note that fix (3) means the agent's dataset path handling (PathSpec / file_partition_regexp / trimmer configuration) now genuinely takes effect in the shaded jar, so dataset names for file-based sources may change to the intended, trimmed form — review spark.datahub.platform.<platform>.path_spec_list if you tuned it against the previous behavior.

• #19103 (Ingestion / Document sources) The min_text_length filter now defaults to 0 (was 50) for the datahub_documents, notion, and confluence sources, so short (non-empty) documents are embedded for semantic search by default instead of being silently skipped. Empty documents are still skipped (skip_empty_text / skip_empty_documents) — except Confluence hierarchy parent pages, which stay exempt from the empty/short filters so they can anchor the tree. Action: none to start embedding short documents; set min_text_length (filtering.min_text_length for Notion/Confluence) above 0 to restore length-based filtering. Re-running ingestion after upgrade embeds previously-skipped short documents, which increases embedding calls and index size accordingly. Relatedly, datahub_documents now falls back to a single text element when the markdown partitioner produces none (e.g. a one-character document), so any non-empty document stays embeddable instead of being silently dropped.

• #19162 (Ingestion / Hex) The Hex source's queriedTables lineage path now builds upstream dataset URNs through the same schema resolver as the SQL-cell path. Tier-1 and Tier-2 URNs agree by construction, so column lineage is no longer dropped on case mismatches with a Snowflake warehouse ingested using convert_urns_to_lowercase: true. BigQuery date-shard suffixes (events_20240101 → events_yyyymmdd) are also folded. Known limitation and action: the resolver dialect-normalizes the tableName before probing DataHub, so on case-folding dialects (mssql, postgres, redshift) the author casing is lost before the probe runs. Snowflake and BigQuery are unaffected — Snowflake defaults to lowercase, BigQuery preserves case through the resolver. For mssql/postgres/redshift warehouses ingested with their source defaults (which preserve case), the Hex tier-1 edge is emitted lowercased (mydb.dbo.orders) while the warehouse URN kept its author casing (MyDb.dbo.Orders) — the edge dangles, and re-ingesting the warehouse does not fix it. There is no per-connection override yet; re-ingest the warehouse with convert_urns_to_lowercase: true to align them. The report exposes queried_tables_unresolved_in_datahub (count) and queried_tables_unresolved_sample (URNs) so the miss is diagnosable when datahub-api is configured (without it the counter stays at 0); Column lineage dropped warnings also fire when the SQL-cell URNs disagree with the tier-1 result.

• (Ingestion / Elasticsearch) The elasticsearch source now uses opensearch-py, supporting both Elasticsearch and OpenSearch. The api_key config option is now typed as a list [id, api_key] or a base64-encoded string (its documented forms) instead of accepting any type. Action: none if api_key is already given as a list or string.

• #18690 (Ingestion / SQL Profiling) New profiling.profiling_isolation_level option for the SQLAlchemy profiler. By default the profiling connection is unchanged: one transaction spans the whole table profile, exactly as before. On MySQL this pins an InnoDB read view and grows the undo log for the duration of each table's profile; on Postgres it holds the connection idle-in-transaction and blocks VACUUM. If you see either symptom, set profiling_isolation_level: AUTOCOMMIT so each profiling SELECT runs on its own. Note the trade-off: under AUTOCOMMIT (or READ_COMMITTED) individual metrics come from different snapshots, so a profile of a concurrently-written table can be internally inconsistent (for example uniqueCount can exceed rowCount; this is not clamped). On Postgres this costs nothing, since READ COMMITTED already took a fresh snapshot per statement. On MySQL InnoDB it gives up the single consistent snapshot that REPEATABLE READ provided — set profiling_isolation_level: REPEATABLE_READ if you need both a consistent snapshot and an explicit level.

• (Operations / Elasticsearch ZDU) Incremental (zero-downtime) reindex Phase 1 now validates the alias swap against the launch-time source document count, not a live alias count. Live writes during the copy are expected and are covered by Phase 2 catch-up after the swap. A failed swap marks the index FAILED and deletes the next index so the following system-update run reindexes from scratch instead of retrying a doomed swap. Reindex polling also waits for the ES _reindex task to finish (when status is available) before treating matching doc counts as complete, so a mid-copy sample cannot falsely finish Phase 1.

• (Operations / Elasticsearch ZDU) Stall detection no longer stacks a second _reindex into the same destination while the current ES task is still running or its status cannot be read (ELASTICSEARCH_BUILD_INDICES_WAIT_FOR_UNRESOLVED_REINDEX_TASK, default true). Previously a destination already at 100% looked like "no progress," exhausted numRetries, marked the index FAILED, and deleted the copy so the next system-update pod recopied from scratch. Action: none; set the env var to false to restore overlapping stall-retries.

• (Metadata Model / Data Products) dataProductProperties now includes an optional parentDataProduct URN so Data Products can nest in a parent-child taxonomy (mirroring Domains' parentDomain). The field is additive; existing Data Products are unchanged (null parent). No migration or reindex is required. Free-text search may match child products on the parent URN string, the same way parentDomain already behaves.

• #16319 (Auth / Query entities) Adds two distinct query-visibility privileges. VIEW_ENTITY_QUERIES (Dataset, Chart, Data Job) gates queries, view definitions, and topSqlQueries in dataset usage statistics (numeric usage data stays gated by View Dataset Usage alone); a user needs it on at least one subject dataset to read a query (QUERY_ENTITY_AUTHORIZATION_REQUIRE_ALL_SUBJECTS=true requires all of them; QUERY_ENTITY_AUTHORIZATION_ENABLED=false disables read enforcement). VIEW_ALL_QUERIES is platform-level and unconditional — it short-circuits the per-dataset check entirely, is never consulted by write checks, and is the only way to read a query with no recorded subject datasets, or to satisfy REQUIRE_ALL_SUBJECTS for topSqlQueries (whose bare SQL strings carry no per-statement dataset list to check). On upgrade, one-time system-update steps backfill both onto every policy that already grants View Entity Page (VIEW_ENTITY_QUERIES also backfills onto View Dataset Usage-only policies, so a usage-only policy doesn't silently lose its pre-upgrade topSqlQueries access); fresh installs get both via the default policies regardless of VIEW_AUTHORIZATION_ENABLED. Action: neither backfill is surgical, and nothing prunes VIEW_ALL_QUERIES automatically — if you want the new per-dataset restriction to actually take effect, revoke VIEW_ALL_QUERIES from the relevant policies yourself. Under VIEW_AUTHORIZATION_ENABLED=true, the upgrade backfill narrows VIEW_ALL_QUERIES to built-in (editable=false) policies only, but this requires VIEW_AUTHORIZATION_ENABLED to reach the system-update job's own environment (the stock compose file already passes it through); a custom Helm chart or compose stack must do the same, or the backfill silently treats view authorization as off and widens every matching policy instead — BOOTSTRAP_SYSTEM_UPDATE_VIEW_ALL_QUERIES_PRIVILEGE_ENABLED=false skips the backfill entirely (including the intended Root/built-in-role grant) rather than narrowing it, so pass the environment variable through to the system-update job first. Separately, QUERY_ENTITY_AUTHORIZATION_REQUIRE_ALL_SUBJECTS now defaults to COMPAT (was false): a no-op unless VIEW_AUTHORIZATION_ENABLED is on, in which case every Query-entity read (direct reads, listQueries, REST/OpenAPI, search/related-entity masking) requires VIEW_ENTITY_QUERIES on all of a query's subject datasets — the same require-all rule view authorization already applied to Query entities via View Entity Page, so default users see no change. It matters once you deliberately restrict: if you enable view authorization and revoke VIEW_ALL_QUERIES from a policy, users on that policy who hold VIEW_ENTITY_QUERIES on only some of a multi-dataset query's subjects stop seeing that query everywhere, including the dataset's Queries tab. topSqlQueries is the exception and stays any-subject. Set the var to false explicitly to keep any-subject reads under view authorization. Also note that a policy created after upgrading that grants only View Entity Page no longer reveals query SQL; grant View Entity Queries explicitly (the backfill covers policies that existed at upgrade time). See Policies.

• #18987 (GMS / GraphQL) GraphQL entity hydration now fetches only the aspects a query's selection set requires (schema-driven aspect mapping), instead of each entity loader's full default aspect set. Reduces primary-store reads on search cards, entity profiles, and browse. Action: none; set GRAPHQL_ASPECT_OPTIMIZATION_ENABLED=false to revert to legacy full-aspect hydration if a specific query or page regresses.

• #18924 (Ingestion / SDK) DataJobs emitted through the Python SDK now set the parent DataFlow's browse-path entry id to the DataFlow URN instead of the raw flow id. This lets Browse (V2) resolve and show the flow's display name rather than an opaque id — most visibly, Airbyte connections that previously appeared as UUID folders now show their connection name. This affects every connector that emits DataJobs via the SDK (for example Airbyte, Fivetran, Azure Data Factory, Flink, Informatica, dlt). The change is to emitted metadata only; there is no model change and no reindex is required. Action: none is strictly required, but re-ingest the affected pipelines to refresh their browse paths and pick up the improved navigation.

• (GMS / Semantic search) The GraphQL semanticSearch and semanticSearchAcrossEntities resolvers no longer escape / in the query text. The semantic query only feeds the embedding provider for kNN (it never reaches an Elasticsearch query_string clause), so the escape changed the embedded text for queries containing paths or URLs and could lower recall. The same resolvers now reject a blank query with a BAD_REQUEST error instead of embedding empty text. Keyword search resolvers are unchanged. Action: none.

• #19547 (Ingestion / dbt) When target_platform_instance is set, dbt's target-platform sibling entities (stub entities dbt creates for warehouse tables the warehouse connector hasn't itself ingested, e.g. a source node excluded by the connector's deny pattern) now get a browsePathsV2 built from the entity's real container hierarchy instead of a plain-text guess. This corrects any previously-written path in that guessed shape — including the one #18539 introduced — on the next dbt ingestion run; entities the warehouse connector has ingested are unaffected, since their container-based path is never touched. Separately, emit_target_platform_display_name (patches datasetProperties.name on those same container-less stub entities, which otherwise show the full dotted instance.database.schema.table string in place of a table name) now defaults to true. Action: none required — both changes self-correct on the next dbt run for recipes with target_platform_instance set; recipes without it are unaffected. Set emit_target_platform_display_name: false to opt back out of the display-name patch.

Environment Variables

• BOOTSTRAP_SYSTEM_UPDATE_DATA_PRODUCT_ASSETS_ENABLED (default true) — Run the one-time sweep that writes asset-side dataProducts from stored dataProductProperties. See Other Notable Changes above and Environment Variables.

• BOOTSTRAP_SYSTEM_UPDATE_DATA_PRODUCT_ASSETS_DELAY_MS (default 1000) — Delay between scan batches and patch ingest chunks during that sweep.

• BOOTSTRAP_SYSTEM_UPDATE_DATA_PRODUCT_ASSETS_LIMIT (default 0) — Max Data Products to scan (0 = no limit).

• SKIP_LEGACY_USAGE_EVENT_INDEX_MIGRATION (default false) — Skip moving a usage event index that predates its index template onto the template layout during system update. See Other Notable Changes above and Environment Variables.

• GRAPHQL_ASPECT_OPTIMIZATION_ENABLED (default true) — Schema-driven GraphQL aspect fetching. See Other Notable Changes above and Environment Variables.

• ELASTICSEARCH_BUILD_INDICES_WAIT_FOR_UNRESOLVED_REINDEX_TASK (default true) — Skip stacking another _reindex while the current ES task is unresolved. See Other Notable Changes above and Environment Variables.

v1.7.0.1

Patch release for v1.7.0 — security and authorization hardening, CVE dependency bumps, graph-cache and lineage fixes, optional optimistic locking, and the remainder of the custom-assertion stack. Full changelog: v1.7.0...v1.7.0.1.

Requirements:

• CLI / Python SDK: 1.7.0.9
• Helm Chart: 1.1.0

Breaking Changes

• #18886 (GMS / Auth) The hardcoded systemClientSecret default (JohnSnowKnowsNothing) has been removed from the server configs. Action: set DATAHUB_SYSTEM_CLIENT_SECRET on GMS, MAE/MCE/PE consumers, the frontend, and Actions before upgrading — services relying on the built-in default will fail to authenticate.

• #19508 (GMS / Timeseries read authorization) Dataset profile, usage, and operations timeseries (and dashboard usage statistics) now require the matching View Dataset Profile / Usage / Operations privileges on Rest.li and OpenAPI as well as GraphQL. Dedicated timeseries APIs (getTimeseriesAspectValues, OpenAPI timeseries scroll, getTimeseriesStats with a URN filter) also require Get Timeseries Aspect API when REST API authorization is enabled. Action: grant those privileges (or rely on the default view-dataset-sensitive policy) to API clients that previously read these aspects with only entity GET. Chart usage statistics are unchanged.

• #19360 (GMS / Role and group membership writes) Writes to aspects that grant privileges are now authorized at the aspect layer across GraphQL, OpenAPI, and Rest.li. Adding a role (roleMembership) requires Manage Policies; adding a user to a group (groupMembership / nativeGroupMembership) requires Edit Group Members; adding a corpGroup owner (ownership) requires Edit Owners — or Manage Users & Groups when the actor is adding themselves. Only additions are checked; removals, unchanged re-ingestion, and system writes pass. Previously Edit Entity on a user was enough to grant that user the Admin role. Action: grant Manage Policies to automation that writes roleMembership outside the UI; membership-only sync (LDAP, Okta, Azure AD, datahub user upsert) needs Edit Group Members or Manage Users & Groups. Toggle via metadataChangeProposal.validation.aspectAuthorization.privilegeGrant.enabled (default true).

• #19212 (GMS / View Authorization) When VIEW_AUTHORIZATION_ENABLED=true, container and schemaField are view-restricted by default (they are no longer on the stock unrestricted overlay; container is no longer viewUnrestricted in the registry). View Entity Page on a schema field inherits from the parent dataset encoded in the schemaField URN, then falls back to a direct grant on the column. GraphQL container loads use field-strip redaction when the actor lacks View Entity Page. Action: ensure View Entity Page policies cover containers (and parent datasets for columns) you still want users to open. Do not re-add document / schemaField / container via VIEW_UNRESTRICTED_ENTITY_TYPES_ADD unless you intentionally want them to bypass view checks. OSS gates entity pages / post-search masking only — it does not get Cloud query-time search pushdown.

Known Issues

Potential Downtime

Deprecations

• #18882 (Assertions / Metadata Model + Ingestion) AssertionType.DATASET and DatasetAssertionInfo are deprecated for new writes. External / self-reported assertions must use AssertionType.CUSTOM with expanded CustomAssertionInfo. On the next run after upgrade, dbt, Great Expectations, and YAML data-contract writers emit CUSTOM instead of DATASET for the same checks. Assertion URNs are unchanged and run history is preserved. Assertions that are never re-ingested stay as stored DATASET aspects until overwritten. See Custom Assertions.

Other Notable Changes

Custom assertions:

• #18882 (Assertions) Completes the custom-assertion stack on the v1.7.0 line (API/SDK, UI, assertionNote aspect, and dbt/GX/data-contract writers). User-authored assertion notes are stored in a dedicated assertionNote aspect so ingestion can fully upsert assertionInfo without overwriting notes. A resumable system-update step copies existing embedded notes to the new aspect. Operators can tune the migration with SYSTEM_UPDATE_ASSERTION_NOTE_MIGRATION_ENABLED, _BATCH_SIZE, _DELAY_MS, and _LIMIT.

Security fixes:

• #19297 (Auth) The built-in self policy granted to every actor on their own entity was flattening the full ENTITY READ privilege map, which also handed out edit privileges on your own corpuser entity. It is now limited to VIEW_ENTITY_PAGE and GET_ENTITY.
• #19316 (Auth) POST /auth/signUp, /auth/resetNativeUserCredentials, /auth/verifyNativeUserCredentials, and /auth/getSsoSettings now require GMS system client credentials, matching /auth/generateSessionTokenForUser. A user session token calling these GMS helpers directly gets 401; frontend login, signup, password reset, and SSO are unaffected.
• #19384 (Auth) Aspect authorization no longer trusts client-supplied appSource system metadata to identify system writes.
• #19405 (Auth) Incidents raised on a schema field are now authorized against the parent entity.
• #19463 (OpenAPI) Unauthorized lineage scroll endpoints are redacted as restricted rather than leaking relationship data.
• #19298, #19299, #19300 (UI) Stored XSS fixes: query and incident descriptions are sanitized (the legacy MarkdownViewer is removed), the documentation editor's PDF preview iframe src is sanitized, and dangerous renderUrl schemes on embeds are rejected.
• #19515, #19526, #19561 (Ingestion) CLI report secret masking is unified, idempotent, and fail-closed (mask before truncation; install once per process so concurrent runs cannot unmask). #19332 scrubs secrets from recorded HTTP cassettes.
• #19483 (GMS / AWS) Share a process-wide AWS credential provider and S3/STS client lifecycle so OpenSearch IAM signing, Bedrock embeddings, STS, and object storage do not leak IRSA refresh tasks.

GMS, graph cache, and lineage:

• #19489 (GraphQL) Container.relationships(types: [IsPartOf], direction: INCOMING) again returns all contained entities from the live graph.
• #19498, #19288, #19301, #19412 (GMS) Entity-graph cache no longer silently drops relationships; indexed search fields are fetched for the cache; null relationship endpoints are skipped; hierarchy graph fallback batches frontier scrolls.
• #19268 (Graph) relatedEntityTypes matching is case-insensitive.
• #19413 (Lineage) scrollAcrossLineage traversal is bounded to prevent GMS OOM.
• #19404, #19476, #18266 (Perf) Cache corpGroup roleMembership via SystemEntityClient; use a cached aspect for per-request CorpUserFlags; batch-load Dashboard usageStats buckets and metrics.
• #19249 (GMS / writes) Optional optimistic locking (OPTIMISTIC_LOCKING_ENABLED, default false) replaces SELECT FOR UPDATE with compare-and-set on SystemMetadata.version for Ebean storage. Optional SCOPED_RETRY_ENABLED and ENTITY_WRITE_LOCK_BACKEND=hazelcast mitigate write thundering-herd; PostgreSQL aspect writes also acquire row locks in consistent primary-key order. All of these default off. See Environment Variables.

Ingestion:

• #18889 (Ingestion) Exclude nltk 3.10.1, which silently breaks document chunking.
• #19506 (Ingestion) Raise the GitPython floor to 3.1.58 for upstream security fixes.
• #19605 (CLI) Bundled default CLI / Python SDK is 1.7.0.9.

Dependency CVE bumps:

• Logback 1.5.38 (CVE-2026-9828, CVE-2026-10532), Log4j 2.25.5 (CVE-2026-49844), Netty 4.2.17.Final (CVE-2026-59902), libthrift 0.23.0 (CVE-2026-43869), Jetty 12.1.10 (CVE-2026-10050), httpcore5 5.4.3 (CVE-2026-54399), micrometer-core 1.16.6 (CVE-2026-40983, CVE-2026-40984), reactor-netty-core 1.3.6, netty-reactive-streams 3.0.9, wire-runtime 6.3.0 (CVE-2026-45799).
• Apache Parquet 1.18.0, which refreshes its shaded Jackson to jackson-databind 2.22.1 (CVE-2026-54512, CVE-2026-54513).
• httpclient5 5.6.3 (CVE-2026-64607), Spring Boot / Spring Kafka 4.0.7 (CVE-2026-41001), OpenTelemetry Java agent 2.28.0 (CVE-2026-54704), mariadb-java-client 2.7.14 (CVE-2026-55856 / 55857 / 55858).
• Python ingestion: nltk 3.10.3 (CVE-2026-12075); datahub-actions image cache-bust for setuptools CVE-2025-47273.

Environment Variables

• OPTIMISTIC_LOCKING_ENABLED (default false) — Use CAS aspect writes instead of SELECT FOR UPDATE with Ebean storage.
• SCOPED_RETRY_ENABLED (default false) — Requires optimistic locking. Retry only the conflicted URN's branch on conflict instead of the whole batch.
• ENTITY_WRITE_LOCK_BACKEND (default none) — none | hazelcast: per-(urn, aspect) write gate. Requires optimistic locking.
• ENTITY_WRITE_LOCK_ACQUIRE_TIMEOUT_SECONDS (default 10) — Bounds the write-lock acquire wait before proceeding lockless (CAS still guards).
• ENTITY_WRITE_LOCK_LEASE_SECONDS (default 300) — Maximum gate hold before auto-release.
• SYSTEM_UPDATE_ASSERTION_NOTE_MIGRATION_ENABLED / _BATCH_SIZE / _DELAY_MS / _LIMIT — Tune the assertion-note aspect backfill.

v1.7.0

Requirements:

• CLI / Python SDK: 1.7.0
• Helm Chart: 1.1.0

Upgrade path / ZDU (zero-downtime upgrade): You must upgrade to v1.6.0 before upgrading to v1.7.0 — do not skip 1.6.0. Deploy v1.6.0 with Helm chart 1.0.3, let system-update complete, then upgrade to v1.7.0 with Helm chart 1.1.0. Enable Elasticsearch/OpenSearch ZDU (global.datahub.systemUpdate.zdu) with the 1.1.0 chart on a subsequent OpenSearch/Elasticsearch version bump — not during the v1.6.0 install. See the v1.6.0 ZDU notes for the server-side prerequisite detail.

Breaking Changes

• (GMS / object storage) File upload/download configuration moved from datahub.s3 to datahub.objectStorage in application.yaml. Environment variables are unchanged (DATAHUB_BUCKET_NAME, DATAHUB_ROLE_ARN, DATAHUB_PRESIGNED_*_URL_EXPIRATION_SECONDS, DATAHUB_S3_ASSET_PATH_PREFIX) and now bind under datahub.objectStorage. Deployments that override only the YAML path datahub.s3.* must switch to datahub.objectStorage.*. Quickstart defaults to local file:// storage (DATAHUB_OBJECT_STORAGE_URI), which does not support presigned URLs; documentation file upload remains disabled unless S3 storage with credentials is configured. Action: update custom YAML overrides to datahub.objectStorage; for production file uploads set DATAHUB_OBJECT_STORAGE_URI=s3://your-bucket (or legacy DATAHUB_BUCKET_NAME) plus DATAHUB_ROLE_ARN or AWS_REGION/AWS_ENDPOINT_URL.

• (GMS / Ebean transaction retries) When a metadata write exhausts retries on a retryable database transaction conflict (SQLState 40001 / 40P01, or MySQL vendor code 1213), GMS now surfaces a stable conflict error instead of a generic 500. OpenAPI and Rest.li return HTTP 503 with body fields code: DATABASE_TRANSACTION_CONFLICT, retryable: true, and a Retry-After header. GraphQL returns error extensions code: 503 / type: SERVICE_UNAVAILABLE (same transient semantics; GraphQL HTTP status remains the usual GraphQL response). Non-deadlock retry exhaustion is unchanged (generic failure). Default EBEAN_MAX_TRANSACTION_RETRY is unset (null), which resolves to 3 retries → 4 total attempts (TransactionContext.DEFAULT_MAX_TRANSACTION_RETRY). Action: Clients that previously retried or alerted on 500 for write failures should treat DATABASE_TRANSACTION_CONFLICT / GraphQL SERVICE_UNAVAILABLE as retryable (honor Retry-After when present). Optional knobs: EBEAN_RETRY_BACKOFF_SQL_STATES, EBEAN_RETRY_BACKOFF_VENDOR_CODES, EBEAN_RETRY_INITIAL_BACKOFF_MS, EBEAN_RETRY_MAX_BACKOFF_MS, EBEAN_RETRY_AFTER_SECONDS (OpenAPI/Rest.li Retry-After header). Micrometer counters ebean.tx.transient_backoff and ebean.tx.transient_exhausted are tagged with path (ingest / delete).

• (Build / Python Docker) The Gradle property datahub.dependencies.python.uvProfile has been replaced by two independent properties: datahub.dependencies.python.uvDockerProfile (Python Docker image builds) and datahub.dependencies.python.uvInstallProfile (runner-side Gradle uv installs). Both default to default (PyPI). Setting pipExtraIndexUrl no longer auto-switches the profile to custom — use uvDockerProfile=custom explicitly (and pass extras). Override locally with UV_DOCKER_PROFILE / UV_INSTALL_PROFILE or -P. See developers.md. Action: replace any uvProfile / ORG_GRADLE_PROJECT_datahub.dependencies.python.uvProfile overrides with the new property names.

• (Metadata Model / AI Context) aiContext is now a first-class aspect on metric, semanticModel, and schemaField (PDL record com.linkedin.common.AiContext) instead of a nested field on metricInfo / semanticModelInfo / semanticFieldAnnotation. GraphQL clients must read aiContext from the entity root (and from schemaFieldEntity), not from nested info/annotation objects. Relationship-level aiContext on SemanticModelRelationship remains inlined. No external release has shipped the nested shape yet. Action: Re-ingest affected entities so aiContext is emitted as its own aspect; update any custom GraphQL queries/fragments that selected nested info.aiContext or semanticFieldAnnotation.aiContext.

• (GMS / Structured Properties) Creating a structured property is now rejected when its qualifiedName would collide with another property's Elasticsearch field name. Field names are derived by replacing . with _, so certification.status and certification_status both map to the field certification_status. Previously both could be created and would silently share (and clobber) the same search field; now PropertyDefinitionValidator fails the write both within a single batch and when the normalized field already exists in active entity-index mappings. The check looks at index mappings (not the primary-store definition catalog), so a definition that exists but is not yet mapped — for example during UpdateIndices/MAE lag, or when ENABLE_STRUCTURED_PROPERTIES_HOOK=false — may not be detected until the mapping appears; soft-deleted properties whose fields remain mapped still block colliding creates. Existing colliding properties are not retroactively rejected — at index-mapping time, colliding fields with identical mappings keep the property with the lexicographically smallest URN (logged as a warning), while colliding fields with different mappings are omitted entirely (logged as an error) to avoid an incorrect index type. qualifiedName remains immutable, so upserts to an existing property skip this check. Action: Choose qualifiedName values that remain distinct after .→_ normalization. If you already have colliding properties, rename or remove all but one so the intended property is indexed.

• (GMS / Structured Properties) String-backed structured property values (string, rich_text, date, urn) are now rejected when their UTF-8 encoding exceeds the configured keyword max length (default 32,766 bytes, Lucene's keyword term limit). Previously oversized values could be written to the primary store and then fail Elasticsearch / OpenSearch indexing with max_bytes_length_exceeded_exception. StructuredPropertiesValidator rejects those writes for GraphQL, OpenAPI, and MCP. STRING (and V3 RICH_TEXT) keyword mappings also set a byte-safe character ignore_above derived from that limit (keywordMaxLength / 4, default 8191) so already-stored oversized values skip inverted-index terms on reindex instead of failing the document. Configure via STRUCTURED_PROPERTIES_KEYWORD_MAX_LENGTH / structuredProperties.keywordMaxLength. Action: Shorten Text / Rich Text property values under the limit, or store large content as asset documentation. See the feature guide limitations.

• (GMS / Structured Properties) Optional STRUCTURED_PROPERTIES_DROP_OVERSIZED_KEYWORD_VALUES_FROM_INDEX / structuredProperties.dropOversizedKeywordValuesFromIndex (default false) lets oversized string-backed structured property values be written to primary storage while omitting them from Elasticsearch / OpenSearch documents. With the default, writes still 422 as above. GraphQL and OpenAPI reads still return the stored value; search and filters will not. Set this on GMS, MAE, and any standalone MCE consumer (proposed validation can run there when ALTERNATE_MCP_VALIDATION is enabled), then restart. Action: Enable only if you must ingest values larger than the Lucene keyword limit (for example internal payloads that are not meant to be searched). Prefer moving large content off structured properties.

• (GMS / View Authorization) When VIEW_AUTHORIZATION_ENABLED=true, entity types are now restricted by default. The previous VIEW_RESTRICTED_ENTITY_TYPES allowlist is replaced by a registry baseline (viewUnrestricted: true on entities in entity-registry.yml) plus optional VIEW_UNRESTRICTED_ENTITY_TYPES / _ADD / _REMOVE overlays. Empty VIEW_UNRESTRICTED_ENTITY_TYPES keeps the registry baseline; a non-empty value replaces it. Stock _ADD covers the previous unrestricted set minus types already flagged in the registry (so the effective list stays the same; types already on the baseline are not re-listed). document, schemaField, and container are not on stock unrestricted overlays — they are view-restricted by default when view authorization is enabled (container is no longer viewUnrestricted in the registry). GraphQL container loads use field-strip redaction (AuthorizationUtils.restrictEntity) when the actor lacks View Entity Page; the hierarchy cache continues to store real URNs only. Shared authorization utilities enforce these rules across GraphQL, OpenAPI, and RestLi. The former restricted catalog types (dataset, dashboard, chart, ML entities, pipelines, glossary, domain, dataProduct, notebook, dataProcessInstance, dataHubView, query, tag) remain restricted. Action: Ensure domain-scoped (or other) View Entity Page policies cover containers you still want users to open. Do not re-add document / schemaField / container via VIEW_UNRESTRICTED_ENTITY_TYPES_ADD unless you intentionally want them to bypass view checks. On self-hosted OSS this gates entity pages / post-search masking only — OSS does not get Cloud query-time search pushdown.

• (GMS / Documents) Document view authorization and DocumentService privilege checks now apply across GraphQL, OpenAPI, and RestLi. Document privileges follow standard entity CRUD with MANAGE_DOCUMENTS as an additional allowance: CREATE accepts CREATE_ENTITY / EDIT_ENTITY / MANAGE_DOCUMENTS; UPDATE accepts EDIT_ENTITY / MANAGE_DOCUMENTS; DELETE accepts DELETE_ENTITY / MANAGE_DOCUMENTS. UPSERT/UPDATE-like writes against a nonexistent document URN are authorized as CREATE; writes against an existing document URN are authorized as UPDATE. Bridge documents additionally inherit VIEW (READ only) from the source entity when they have the Bridge Document subtype, a bridge-* ID, and resolvable source metadata; an unresolvable source fails closed unless direct document access already allowed the request. UPDATE and DELETE never inherit from the bridge source. Action: Grant document CREATE/EDIT/DELETE entity privileges or MANAGE_DOCUMENTS as needed; ensure readers have View Entity / Get Entity (or MANAGE_DOCUMENTS); do not rely on document unrestricted fail-open.

• (UI / Lineage) The LINEAGE_GRAPH_V2 and LINEAGE_GRAPH_V3 feature flags (GMS environment variables) have been removed, along with the older lineage graph visualizations they gated. DataHub now always renders the latest lineage graph experience. Action: if you set LINEAGE_GRAPH_V2 or LINEAGE_GRAPH_V3 in your GMS environment or Helm values, remove them — they are no longer recognized. There is no longer a way to switch back to the previous lineage visualizations.

• (Usage metrics) The report-driven MCP usage metric was renamed from mcp_queries to mcp_query to match the mcp_query usage operation. Aggregated usage rows and the Micrometer counter (datahub.usage.mcp_query) use the new name. Action: update any dashboards, alerts, or consumers that filter on metric_name=mcp_queries or scrape datahub.usage.mcp_queries.

• #18693 (GraphQL / Forms) The batchAssignForm and batchRemoveForm mutations now read and write in batches rather than once per entity, which changes how they behave when part of a batch fails. Previously, if an entity's form state could not be read, that entity was silently skipped, the rest of the batch was applied, and the mutation still returned true — callers had no way to tell anything had been dropped. Now such a failure is reported as an error and no entity in the batch is modified. Likewise, a failed write no longer leaves a partially-applied batch: proposals are ingested together, so a failure leaves them untouched instead of updating some entities and not others. When one of the supplied entity urns does not exist, the error now lists every missing urn instead of only the first one found. Successful calls are unchanged, and both mutations remain idempotent. Action: none if you already treat a GraphQL error as "retry the whole call". If you relied on a true response meaning every entity was updated, note that this was never guaranteed — the mutation still returns true for entities that were no-ops (already assigned, or form not present on remove) — but errors are now raised in cases that previously passed silently, so a call that appeared to succeed may now surface a real failure that needs fixing before retrying.

• #14911 (Spark lineage) The acryl-spark-lineage agent now bundles OpenLineage 1.50, which enables OpenLineage's built-in dataset-name trimmers by default. These automatically strip common partition directories (key=value, date dirs like year=2024/, yyyy/MM/dd, yyyyMM) from filesystem/object-store dataset names. The plugin disables them automatically when you already configure DataHub path trimming (path_spec_list or file_partition_regexp), so those users are unaffected. But recipes with neither configured now pick up the upstream auto-stripping, so dataset URNs for partitioned file/object-store paths change (the partition segment is dropped) and lineage edges to the old, un-stripped URNs can orphan. Action: if you rely on the previous partition-inclusive URNs, either configure path_spec_list / file_partition_regexp for the affected platforms, or set spark.datahub.metadata.dataset.openLineageTrimmersEnabled=false to restore the pre-1.50 naming.

• (ingest/doris) A database written as catalog.database (for example iceberg_catalog.db_ods) is now read as a catalog plus a database rather than as one database name. Previously the whole string was passed through as the database, so datasets were emitted as iceberg_catalog.db_ods.<table>; they are now emitted as db_ods.<table>, and database_pattern matches the short database name (db_ods) instead of the qualified one. Action: Only recipes that set database (or catalog) to a qualified catalog.database value are affected — plain database names are unchanged. On the first run after upgrading, those datasets move to new URNs: with stateful ingestion enabled the old catalog.database.<table> entities are soft-deleted and re-created under the short name; without it they remain as orphans and need removing manually (for example datahub delete --platform doris --env PROD scoped to the affected datasets, or the DataHub UI). If you ingest the same database names from more than one catalog, set platform_instance to the catalog name so the short URNs do not collide.

• (ingest/teradata) When convert_urns_to_lowercase is enabled, the Teradata connector now also lower-cases database container URNs, not just dataset URNs. Previously datasets were lower-cased while their parent database containers kept the source casing, producing datasets parented under a container URN that did not match their own name. Action: This only affects deployments that explicitly set convert_urns_to_lowercase: true (the default is false). On the first run after upgrading, the database container URNs change from source-case to lower-case; with stateful ingestion enabled the old source-cased containers are soft-deleted and re-created lower-cased. No action is required for the default (false) configuration.

• #18181 (Ingestion / Power BI, Mode) Column-level lineage now preserves the original casing of upstream column names instead of lowercasing them. Previously the Power BI and Mode connectors emitted the upstream schemaField field path in lowercase (e.g. orderid) while the warehouse's schema kept its original casing (e.g. OrderId); because DataHub matches column-level edges case-sensitively, no column-level lineage rendered for any warehouse that stores columns in non-lowercase casing. The schemaField URNs emitted for Power BI and Mode column-level lineage will therefore change on the next ingestion. Action: none for most users — re-run ingestion and the previously-missing column-level edges will render. If you set convert_column_urns_to_lowercase: true on an upstream SQL source (e.g. MSSQL) as a workaround for the missing lineage, remove it so both sides preserve casing; leaving it enabled re-introduces the mismatch in the opposite direction. Note that convert_lineage_urns_to_lowercase no longer lowercases the column portion of the upstream URN (it still governs the dataset portion).

• #18367 (Ingestion / BigQuery) BigQuery now sources row count, size, and last-altered time from the supported INFORMATION_SCHEMA.PARTITIONS view instead of the undocumented __TABLES__ Legacy SQL construct, which is unsupported by Google and may change or stop working without notice. PARTITIONS is dataset-scoped and free but covers base tables only, so the default stats change per entity type:

  • Base tables: unchanged, except empty tables (zero rows), which are absent from PARTITIONS and so lose their lastModified timestamp.
  • External tables: lose their lastModified timestamp (they are absent from PARTITIONS). Row count/size were never populated for external tables regardless, since BigQuery does not store their data.
  • Views: lose their lastModified timestamp (row count/size were never emitted for views).
  • Snapshots: lose their lastModified timestamp and their rows_count / size_in_bytes custom properties.

  Note that views and snapshots previously received these stats whenever profiling was enabled; that no longer happens by default, even with profiling on.

  Action: to restore the previous behavior for all entity types, set use_legacy_table_stats: true on the BigQuery source config. For base and external tables, this only takes effect when table stats are already being fetched (profiling enabled or use_tables_list_query_v2: true); for views and snapshots it applies regardless.

• (GMS / Micrometer) When USAGE_AGGREGATION_ENABLED=true, datahub_request_count is emitted from the in-memory usage aggregation flush (default every 60s), not per-request. The actor tag is actor_class (regular, system, support) instead of the former per-request user_category tag. agent_class and request_api label formats are unchanged. New tags: usage_operation on request and byte counters. New metrics: datahub.usage.input_bytes, datahub.usage.output_bytes, datahub.usage.active_identities (gauge: unique identities per actor_class in the flush window). Legacy JMX requestContext_* metrics are unaffected. Aggregation is opt-in (default off).

• (Ingestion) The built-in datahub column-level classifier (DataHubClassifier) has been removed from acryl-datahub, along with its acryl-datahub-classify dependency, which pinned numpy<2 and an outdated spaCy stack and blocked dependency upgrades. The classification framework (the Classifier interface, the classifier registry, and the per-source orchestration) is retained, but no classifier is registered by default. Recipes that set classification.enabled: true (on Snowflake, BigQuery, Redshift, dynamodb, or any SQL source) without registering a replacement classifier now fail fast at startup with a message pointing here. Recipes that leave classification disabled (the default) are unaffected. Action: Remove the classification block from your recipe, or register a custom classifier. If you still need the built-in classifier, pin the last release that ships it: pip install 'acryl-datahub==1.6.0.5'.

• #18014 (Airflow / Dagster / Prefect / Great Expectations plugins) The default emit mode changed from synchronous to ASYNC to reduce GMS load at high volume. With async, emit() no longer raises on a rejected write and there's no read-after-write guarantee. The Python SDK, CLI, and DataHub Actions defaults are unchanged. Action: to restore the previous behavior (SYNC_PRIMARY), set [datahub] emit_mode = SYNC_PRIMARY in airflow.cfg (Airflow), emit_mode: SYNC_PRIMARY on the Dagster source config, emit_mode=EmitMode.SYNC_PRIMARY on the Prefect DatahubEmitter block, or emit_mode: SYNC_PRIMARY on the Great Expectations DataHubValidationAction config.

• (GMS / Logical Models) Linking a physical dataset to a logical parent now requires Edit Entity on both the child dataset (whose logicalParent aspect is written) and the proposed parent dataset. Clearing a logical parent requires Edit Entity on the child only. The logical models feature guide previously stated only the child was checked; enforcement now matches metadata policies. Action: Grant Edit Entity on logical parent datasets to principals who create or change logical-parent links, or update policies so those operations succeed.

• #17733 (Ingestion / Omni) The max_requests_per_minute config field has been removed. Rate limiting is now handled server-side via 429 responses with automatic exponential-backoff retry. Concurrency is controlled by the new max_workers field (default: 4). Action: Remove max_requests_per_minute from your Omni recipe if present; adjust max_workers to control parallelism.

• #17852: (Ingestion framework) Workunit processor helper functions have been removed and replaced with WorkunitProcessor classes. If you were directly calling these functions in custom code, you must update to use the processor class API. Migration: Import the processor class, create a WorkunitProcessorContext, instantiate the processor via Processor.create(ctx), and call .process(stream). Additionally, some processors have been renamed to follow a consistent naming convention:

  Renamed processors (old → new):

  • AutoFixDuplicateSchemaFieldPathsProcessor → ValidateDuplicateSchemaFieldPathsProcessor
  • AutoFixEmptyFieldPathsProcessor → ValidateEmptySchemaFieldPathsProcessor
  • StaleEntityRemovalProcessor → AutoStaleEntityRemovalProcessor

  Removed standalone functions (use processor classes instead):

  • auto_status_aspect() → Use AutoStatusAspectProcessor from datahub.ingestion.workunit_processors.auto_status_aspect
  • auto_workunit_reporter() → Use AutoWorkunitsReporterProcessor from datahub.ingestion.workunit_processors.auto_workunits_reporter
  • auto_lowercase_urns() → Use AutoLowercaseUrnsProcessor from datahub.ingestion.workunit_processors.auto_lowercase_urns
  • auto_materialize_referenced_tags_terms() → Use AutoMaterializeReferencedTagsTermsProcessor from datahub.ingestion.workunit_processors.auto_materialize_referenced_tags_terms
  • auto_fix_duplicate_schema_field_paths() → Use ValidateDuplicateSchemaFieldPathsProcessor from datahub.ingestion.workunit_processors.validate_duplicate_schema_field_paths
  • auto_fix_empty_field_paths() → Use ValidateEmptySchemaFieldPathsProcessor from datahub.ingestion.workunit_processors.validate_empty_schema_field_paths
  • auto_browse_path_v2() → Use AutoBrowsePathV2Processor from datahub.ingestion.workunit_processors.auto_browse_path_v2
  • auto_incremental_lineage() → Use AutoIncrementalLineageProcessor from datahub.ingestion.workunit_processors.auto_incremental_lineage
  • auto_incremental_ownership() → Use AutoIncrementalOwnershipProcessor from datahub.ingestion.workunit_processors.auto_incremental_ownership
  • auto_incremental_properties() → Use AutoIncrementalPropertiesProcessor from datahub.ingestion.workunit_processors.auto_incremental_properties
  • auto_patch_last_modified() → Use AutoPatchLastModifiedProcessor from datahub.ingestion.workunit_processors.auto_patch_last_modified

  Example migration:

  # OLD (removed):
  from datahub.ingestion.workunit_processors.auto_status_aspect import auto_status_aspect
  result = list(auto_status_aspect(stream))
  
  # NEW (required):
  from unittest.mock import MagicMock
  from datahub.ingestion.api.workunit_processor import WorkunitProcessorContext
  from datahub.ingestion.workunit_processors.auto_status_aspect import AutoStatusAspectProcessor
  
  ctx = WorkunitProcessorContext(
      source_report=report,
      pipeline_context=MagicMock(),
      source_config=config,
      platform="myplatform"
  )
  processor = AutoStatusAspectProcessor.create(ctx)
  result = list(processor.process(stream))

  Naming convention: Processors follow a consistent pattern: Auto*Processor (enrichment), Validate*Processor (validation/cleanup), Ensure*Processor (constraint enforcement). See WorkunitProcessor base class documentation for details.

• (GMS rate limiting) Renamed rateLimits.defaultRetryAfterSeconds / RATE_LIMITS_DEFAULT_RETRY_AFTER to minRetryAfterSeconds / RATE_LIMITS_MIN_RETRY_AFTER. The value is now the minimum Retry-After floor; endpoint (token-bucket) denials may return a longer wait derived from Bucket4j refill timing. Added retryAfterJitterPercent / RATE_LIMITS_RETRY_AFTER_JITTER_PERCENT (default 10) to spread endpoint retry timing. Action: update env vars and external rate-limit YAML if you set the old names; capacity denials still use the flat minimum.

• (GMS rate limiting) The rate-limit policy file is now loaded by Spring as a property source (@PropertySource) rather than a custom loader, so its ${ENV:default} placeholders resolve and it binds under datahub.gms.rateLimits.*. Breaking: RATE_LIMITS_CONFIG_FILE_ENABLED and RATE_LIMITS_CONFIG_JSON are removed; RATE_LIMITS_CONFIG_FILE is now a Spring resource URI and must include a scheme — use file:/etc/datahub/rate-limits.yaml, not a bare path, or the override is silently ignored and the bundled defaults load; and an external policy file must use the full datahub.gms.rateLimits: key path, not a bare rateLimits: fragment. Action: if you set any of these, migrate before upgrading — drop RATE_LIMITS_CONFIG_JSON and RATE_LIMITS_CONFIG_FILE_ENABLED (no longer read), add the file: prefix to RATE_LIMITS_CONFIG_FILE, and re-root any mounted policy file under datahub.gms.rateLimits:. A RATE_LIMITS_CONFIG_FILE value without a scheme now logs a WARN at startup.

• (Ingestion / Airflow plugin) Dropped support for Airflow 2.x. acryl-datahub-airflow-plugin now requires Airflow 3.0+. The package's apache-airflow floor is bumped to >=3.0.0. The legacy standalone openlineage-airflow package is no longer used at all — the plugin always uses apache-airflow-providers-openlineage, which is now an unconditional dependency (>=2.1.0 — the version that added Airflow 3 listener support). Drop openlineage-airflow from your constraints if it was pinned. The [airflow2] install extra is gone; [airflow3] is retained as a backward-compatible no-op (it installs the same as the bare package), so existing pip install 'acryl-datahub-airflow-plugin[airflow3]' commands keep working. The taskinstance value for datajob_url_link (an Airflow 2-only URL format) is no longer accepted; use tasks (default) or grid. The patch_snowflake_schema config option has been removed — it patched a Snowflake schema bug in the deprecated openlineage-airflow SnowflakeExtractor and has no effect under the OpenLineage provider, which handles Snowflake schema resolution natively; remove the setting from airflow.cfg if present (it is silently ignored). If you are still on Airflow 2.7–2.10, pin acryl-datahub-airflow-plugin <= 1.6.0 (the last release with Airflow 2 support) — see the compatibility section in the Airflow integration docs for the full version ladder.

• #17536: (prefect-plugin) The datahub-prefect package now requires Prefect 3.x (>=3.0.0,<4.0.0); Prefect 2.x is no longer supported. The Prefect entry point group also changed from prefect.block to prefect.collections to match Prefect 3's plugin discovery. Action: Upgrade Prefect to 3.x and re-register the DataHub block (DatahubEmitter) before upgrading the plugin.

• (Ingestion / PowerBI Report Server) The chart_pattern config field has been removed from the powerbi-report-server source. If your recipe sets chart_pattern, ingestion will log a deprecation warning. Migration: remove chart_pattern from your recipe — chart-level filtering is not yet implemented for this connector.

• (Docker / local development) Removed legacy root-level Docker Compose files (docker/docker-compose*.yml), shell scripts (docker/quickstart.sh, docker/dev*.sh, docker/nuke.sh), and old generated quickstart bundles under docker/quickstart/ (except docker-compose.quickstart-profile.yml). Migration: use datahub docker quickstart for CLI installs; use ./gradlew quickstartDebug or scripts/dev/datahub-dev.sh start for contributors; use datahub docker nuke, ./gradlew quickstartDebugNuke, or scripts/dev/datahub-dev.sh nuke for teardown; customize installs from docker/quickstart/docker-compose.quickstart-profile.yml or profiles in docker/profiles/.

• Docker image tags (:head removed, :quickstart and :sha-* added): The floating :head tag is no longer published to Docker Hub because images were tagged per-service at different times, so :head often referred to a mixed set of commits. Action for Docker Compose / local quickstart: use DATAHUB_VERSION=quickstart (or datahub docker quickstart --version head, which resolves to the quickstart tag via the version manifest). Immutable sha-<short_sha> tags are published during the coordinated depot bake on master; the :quickstart tag is updated only after the full image matrix passes smoke tests, so all services move together. Action for Kubernetes / Helm / production: pin an immutable tag — a release (v*) or a commit tag sha-<7-char short SHA> (e.g. sha-38960ae). Do not use quickstart in cluster deployments; it can change on the next green master build. Bare short SHA tags (:abc1234) are now prefixed: use :sha-abc1234. Old CLIs using --version head continue to work when they load an updated quickstart_version_mapping.yaml with docker_tag: quickstart under the head key.

• #17376: (Ingestion / Hex) Hex Components are now ingested as Chart entities instead of Dashboard entities. A Hex Component defines its own visualisation that importing projects cannot override, so it maps to a Chart (analogous to a Looker Look or a PowerBI Tile), with the project remaining a Dashboard that links to its Components via DashboardInfo.charts. Migration: if you ran the old version with stateful ingestion enabled, stale-entity removal soft-deletes the pre-existing legacy Dashboard-typed Components on the next run in most cases. Because every Component changes URN type, a component-heavy workspace can exceed the stale-removal fail-safe (stateful_ingestion.fail_safe_threshold, default 75%), in which case raise the threshold or perform a one-time bulk cleanup via the DataHub UI or CLI. Either way, Component URNs change entity type, so any saved views, glossary/tag/ownership assignments, and policies that targeted the old Dashboard-typed Component URNs will be lost and must be manually reapplied to the new Chart URNs.

• #17443 (Ingestion / MongoDB) AWS DocumentDB can now be ingested as its own data platform. The MongoDB source has a new opt-in config field platform (default mongodb). Set platform: documentdb together with hostingEnvironment: AWS_DOCUMENTDB to emit entities under the new documentdb data platform URN instead of mongodb; setting platform: documentdb with any other hosting environment is rejected at config validation time. The default is unchanged, so existing recipes continue to emit mongodb URNs even when pointed at AWS DocumentDB. If enabled on an existing recipe, both the dataset URNs and the database-level container URNs will change, and the stale mongodb entities will need to be cleaned up via stateful ingestion (if enabled) or manual soft-delete.

• (GMS / Secrets) SECRET_SERVICE_CALLER_GUARD_MODE now defaults to ENFORCE (was effectively unrestricted for human callers). Browser sessions and user Personal Access Tokens can no longer call getSecretValues or otherwise decrypt UI secrets through GraphQL. Rollout (v1.7.0): new deployments get ENFORCE by default; existing deployments should migrate before upgrading or set AUDIT during transition. Action: If you relied on admins or automation using a user PAT to read plaintext secret values via getSecretValues, migrate to datahub-actions with system client credentials (DATAHUB_SYSTEM_CLIENT_ID / DATAHUB_SYSTEM_CLIENT_SECRET) or set SECRET_SERVICE_CALLER_GUARD_MODE=AUDIT temporarily during rollout. Scheduled UI ingestion via datahub-actions is unaffected. See Environment Variables and Ingestion executor security.

• (Java / Plugin API) SecretService.encrypt and SecretService.decrypt no-context overloads are removed; use encrypt(OperationContext, String) and decrypt(OperationContext, String). Custom GMS plugins or extensions that call SecretService directly must pass OperationContext from the request, or null for background jobs (allowed by the guard). With ENFORCE, decrypt throws SecurityException for human browser/mobile callers and callers other than trusted ingestion workers (datahub-actions in OSS).

• #17721: (Ingestion / Snowplow) The connector now fetches pipeline enrichments from the supported pipelines/v1/{pipelineId}/enrichments endpoint instead of the deprecated resources/v1/.../configuration/enrichments endpoint Snowplow is retiring. Because the new payload no longer returns the per-enrichment UUID, enrichment DataJob URNs are now derived from the enrichment name (e.g. ...,campaign-attribution) instead of ...,07409eac-...). Existing enrichment DataJobs ingested under the old UUID-based URNs are re-created under the new name-based URNs on the next run. If stateful ingestion is enabled, the old UUID-based DataJobs are soft-deleted by stale-entity removal; otherwise they persist as stale DataJobs and must be cleaned up manually. No recipe changes are required.

• #18266 (GraphQL / Dashboard.usageStats) When Dashboard.usageStats is queried with the metrics field and no limit argument, the returned list of absolute usage-metric snapshots is now capped at 100 (previously up to 5000). This lets the absolute-metrics fetch batch across a page of dashboards into a single query instead of one per dashboard. Dashboards with 100 or fewer snapshots in the queried window are unaffected; those with more have their metrics list truncated to the 100 most recent. The daily buckets and the rolled-up aggregations are unchanged. Action: callers that need more than 100 absolute snapshots must pass an explicit limit to usageStats(limit: ...).

• #18845: (Metadata Model / Entity Registry) Each directed relationship edge signature — (source entity type, destination entity type, relationship name) — may now be produced by exactly one aspect on that source entity. Multiple fields within the same aspect may still share a signature. The check runs when entity specs are built and again when registries are merged at GMS startup, including custom / plugin entity registries. Plugin relationship-edge uniqueness failures are never soft-ignored, even when IGNORE_FAILURE_WHEN_LOADING_ENTITY_REGISTRY_PLUGIN is true. Action: If you ship a custom or plugin entity registry that declares the same edge signature from two different aspects on one source type, consolidate those fields onto a single aspect or rename the relationship before upgrading — otherwise GMS (or registry load) fails. See Aspect ownership uniqueness.

• #17982: (Ingestion framework) acryl-datahub now requires SQLAlchemy 2.0 (sqlalchemy>=2.0,<2.1). The migration is all-or-nothing: SQLAlchemy resolves to a single version per environment, and some dialect drivers pin SQLAlchemy<2, so a mixed 1.4/2.0 install is not possible. Action: if you install acryl-datahub into a shared environment, make sure nothing else pins SQLAlchemy<2; if you maintain a custom ingestion source or transformer that uses SQLAlchemy directly, migrate it to the 2.0 API (wrap raw SQL in text(), pass bind parameters as a dict rather than keyword arguments, replace Engine.execute() with engine.connect(), and access result rows via .mappings() instead of row["col"]). The following sub-items are the user-visible consequences:

  • Vertica driver changed. The vertica source now uses the maintained sqlalchemy-vertica-python dialect (the previous dialect pinned SQLAlchemy<=1.4.44). Action: reinstall the vertica extra (pip install 'acryl-datahub[vertica]') so the new driver is present.

  • Postgres-family connections use AUTOCOMMIT by default. The Postgres, TimescaleDB and CockroachDB sources now open their reflection, lineage, sampling and profiling connections with isolation_level: AUTOCOMMIT. Without it, SQLAlchemy 2.0 keeps a transaction open, so one failed statement (for example a permission error on one table) made every later query in the run fail with "current transaction is aborted". This also stops long runs from holding connections idle-in-transaction. To keep the previous behaviour, set options: {isolation_level: "READ COMMITTED"} in the recipe; an explicit profiling.profiling_isolation_level still applies to profiling.

  • Postgres materialized views: no change. They are still ingested as views. (SQLAlchemy 2.0 no longer includes them in its regular view listing, so the Postgres-family sources now list them separately.)

  • Great Expectations plugin. acryl-datahub-gx-plugin now allows SQLAlchemy 2.0 (sqlalchemy>=1.4.39,<2.1, matching acryl-datahub), so it can be installed alongside the SQL source extras. If you use Great Expectations 0.x dialect extras that still require sqlalchemy<2 (for example GX's Snowflake, BigQuery or Redshift extras), install those in a separate environment from DataHub's SQL sources.

  • Oracle native type labels. RAW columns now report their length (for example RAW(16) or RAW(2000)), and ROWID columns report ROWID instead of an empty native type, so the nativeDataType shown for those columns changes on the next run. The DataHub column types, field paths and URNs are unchanged. No action is required, but the change may appear in schema history.

  • DB2 native type labels. The upgraded ibm-db-sa driver reports integer columns as INT instead of INTEGER, so the nativeDataType shown for those columns changes on the next run. The DataHub column type (NumberType), field paths and URNs are unchanged. No action is required, but the change may appear in schema history.

  • ClickHouse column metadata. With the upgraded clickhouse-sqlalchemy driver, timezone-aware DateTime columns are now typed as time columns (previously unknown), and primary-key columns are now marked as part of the key. Expect these to appear as schema changes on the next run.

  • Unity Catalog / Databricks profiling now requires the databricks-sqlalchemy dialect, which is pulled in automatically by the databricks/unity-catalog extras. No recipe change is required; just reinstall the extra.

Known Issues

• (GMS / schemaField domain & ownership mirroring) Opt-in MCP side effects can copy a dataset's domains / ownership onto each of its schemaField entities (MCP_SIDE_EFFECTS_SCHEMA_FIELD_ENABLED plus MCP_SIDE_EFFECTS_SCHEMA_FIELD_DOMAIN_ENABLED / MCP_SIDE_EFFECTS_SCHEMA_FIELD_OWNERSHIP_ENABLED). While a sub-flag is on, dataset domain/ownership deletes cascade to the corresponding field aspects. Changing a sub-flag (or the master flag) does not immediately rewrite historical fields — ongoing writes use the new flag state, but existing inventory is reconciled by the schemaFieldsFromSchemaMetadata system-update step when a mirror flag is enabled (backfill). Turning a flag off stops further mirroring and does not delete leftover field aspects. Note: toggling flags back to a previously SUCCEEDED fingerprint does not re-run the job unless you use manual reprocess (SYSTEM_UPDATE_SCHEMA_FIELDS_FROM_SCHEMA_METADATA_REPROCESS=true) or clear/modify that fingerprint's upgrade result — either is valid.

Potential Downtime

• (GMS / Structured Properties / search indices) With both ENABLE_STRUCTURED_PROPERTIES_SYSTEM_UPDATE=true and ENABLE_STRUCTURED_PROPERTIES_TYPE_MISMATCH_REINDEX=true (the latter defaults to true), system-update BuildIndices reindexes entity search indices when an existing structured-property field's Elasticsearch type differs from the definition-driven target (for example float or long vs intended double for urn:li:dataType:datahub.number). Previously those mismatches were ignored because structuredProperties is dynamic: true and excluded from the normal mapping diff. Both flags are required. Action: Expect longer system-update runtime / search reindex on the first upgrade if any NUMBER (or other) structured-property fields were dynamically mapped to the wrong type. Set ENABLE_STRUCTURED_PROPERTIES_TYPE_MISMATCH_REINDEX=false to skip type-mismatch reindex while leaving other structured-property system-update behavior intact. Ensure mappings reindex flags remain enabled as for other BuildIndices reindexes.

Deprecations

• (Assertions / Metadata Model + Ingestion) AssertionType.DATASET and DatasetAssertionInfo are deprecated for new writes. External / self-reported assertions must use AssertionType.CUSTOM with expanded CustomAssertionInfo (optional structured fields: scope, aggregation, operator, parameters, fields, nativeType, nativeParameters). Native typed models (FIELD, VOLUME, FRESHNESS, DATA_SCHEMA, SQL) remain for assertions DataHub evaluates or schedules natively — do not use them for external self-reporting. DatasetAssertionScope remains the shared GraphQL / PDL enum name (same values); it is also used on CustomAssertionInfo.scope.

  Ingestion sources (dbt, Great Expectations, YAML data contracts): on the next run after upgrade, these writers emit CUSTOM instead of DATASET for the same checks. Assertion URNs are unchanged — GUID inputs do not include assertion type (dbt uses platform/name/instance[/env]; GX uses platform/nativeType/nativeParameters/dataset/fields; data contracts use contract/entity/check id). The writers fully upsert assertionInfo, replacing the legacy datasetAssertion payload with customAssertion on the existing assertion entity, while run history (assertionRunEvent) is preserved. No migration / rewrite job is required for identity or history. Assertions that are never re-ingested stay as stored DATASET aspects and continue to render via read-path compatibility until removed or overwritten.

  Action: No recipe changes required for dbt / GX. Switch custom writers to upsertCustomAssertion / SDK client.assertions.sync_custom_assertion (+ reportAssertionResult).

• (Ingestion / Executor) The acryl-executor package is retired. The execution engine it provided now ships inside acryl-datahub as datahub.executor, so UI-triggered ingestion and connection testing no longer need a separate package. Existing published acryl-executor versions remain on PyPI but will not receive further updates. If you set custom task types in the executor action's task_configs, the old acryl.executor.execution.* dotted paths still work but log a warning on every run and will stop working in a future release. Action: for most deployments, none — acryl-datahub-actions[executor] continues to install what it needs. If your executor.yaml names task types explicitly, change the acryl.executor. prefix to datahub.executor. (for example acryl.executor.execution.sub_process_ingestion_task.SubProcessIngestionTask becomes datahub.executor.execution.sub_process_ingestion_task.SubProcessIngestionTask).

• #17376: (Ingestion / Hex) Three recipe fields are removed and now emit a deprecation warning when set: lineage_start_time, lineage_end_time, and datahub_page_size. These belonged to the old lineage path that searched DataHub for Hex-tagged Query entities. Lineage now comes directly from the Hex REST API (the queriedTables API on Hex Enterprise workspaces, or SQL parsing of project/component cells on all workspaces), so these fields no longer have any effect. Migration: remove them from your recipe.

• #18133 (Ingestion / BigQuery) The usage.start_time, usage.end_time, and usage.bucket_duration recipe fields are deprecated in favor of the top-level start_time, end_time, and bucket_duration fields, which control the ingestion window for lineage, usage, and operations together. Previously, setting these fields only under usage had no effect at all — they were silently ignored, and only the default "last full day" window was ingested. Behavior change: if your recipe sets one of these fields only under usage, it will now actually take effect and be applied at the top level (with a deprecation warning logged); this may widen the range of data ingested. Since BigQuery usage ingestion scans audit logs, a wider window can increase query cost. Action: move start_time, end_time, and bucket_duration out of the usage block and set them at the top level of your recipe instead. Setting the same field in both places is now a configuration error.

• #18133 (Ingestion / BigQuery) The usage.max_query_duration recipe field is deprecated in favor of the top-level max_query_duration field. Unlike the time-window fields above, max_query_duration only takes effect with the older, legacy extraction path (use_queries_v2: False) — it has no effect on the default queries-v2 path either way. Action: move max_query_duration out of the usage block and set it at the top level of your recipe instead. Setting the same field in both places is now a configuration error.

• #18133 (Ingestion / BigQuery) usage.format_sql_queries, usage.include_top_n_queries, and usage.queries_character_limit now take effect when using the default queries-v2 extraction path (use_queries_v2: True); previously they were silently ignored on that path and only worked with the older, legacy extraction path. Behavior change: if your recipe already sets usage.format_sql_queries: true, DataHub will now actually reformat SQL queries before displaying them (this was silently skipped before). If you don't set these fields, nothing changes.

• #18133 (Ingestion / BigQuery) usage.include_read_operational_stats and usage.apply_view_usage_to_tables only work with the older, legacy extraction path (use_queries_v2: False) — the default queries-v2 path has no equivalent setting. Setting either to a non-default value while using queries-v2 now logs a warning explaining that it will be ignored.

Other Notable Changes

• (Docker / Python deps) datahub-actions default app venv now applies docker/snippets/ingestion/constraints.txt via UV_CONSTRAINT. Floors include setuptools>=80.10.1,<82 (CVE-2025-47273), cryptography>=49.0.0 (CVE-2026-69249), and msgpack>=1.2.1 (GHSA-6v7p-g79w-8964). Ingestion SDK cryptography raised to >=49.0.0,<51.0.0. Images also drop pip/_vendor/bom.cdx.json so scanners do not treat pip's vendored build-time SBOM pins as installed packages.

• (GMS / schemaField MCP side effects + system update) Optional nested flags under metadataChangeProposal.sideEffects.schemaField can mirror dataset domains and ownership onto schemaField entities (MCP_SIDE_EFFECTS_SCHEMA_FIELD_DOMAIN_ENABLED / MCP_SIDE_EFFECTS_SCHEMA_FIELD_OWNERSHIP_ENABLED, both requiring MCP_SIDE_EFFECTS_SCHEMA_FIELD_ENABLED). Defaults remain off. Enabling increases write fan-out (one MCP per column per mirrored aspect change) and can change DOMAIN / OWNER policy matches for schemaField URNs once those aspects are present.

  • Backfill: The non-blocking SystemUpdate step GenerateSchemaFieldsFromSchemaMetadata (SYSTEM_UPDATE_SCHEMA_FIELDS_FROM_SCHEMA_METADATA_ENABLED) scans dataset schemaMetadata/status, builds RESTATE MCLItems stamped with APP_SOURCE=SYSTEM_UPDATE, and invokes post-MCP schemaField side effects (then async ingestProposal) — it does not rely on a no-op parent upsert. Enabling a mirror flag backfills mirrored aspects; disabling stops further mirroring and does not delete leftover field domains/ownership. Upgrade id is fingerprinted by the effective domain/ownership mirror flags (schema-field-from-schema-metadata-v2-d{0|1}-o{0|1}), so a first-time change to a new fingerprint schedules a fresh run on the next SystemUpdateNonBlocking. Keep the MCP flags consistent on GMS, MCE, and the upgrade job.
  • Cycles / re-runs: Toggling flags back to a combination that already SUCCEEDED (e.g. off → on → off, or on → off → on) does not re-run that fingerprint. To re-run the same fingerprint, either use manual reprocess (SYSTEM_UPDATE_SCHEMA_FIELDS_FROM_SCHEMA_METADATA_REPROCESS=true) or clear/modify the stored dataHubUpgradeResult on the corresponding urn:li:dataHubUpgrade:schema-field-from-schema-metadata-v2-… entity so the step is no longer treated as SUCCEEDED/ABORTED. Both approaches are valid. See Environment Variables.

• (Kafka / OAUTHBEARER) Java service images (datahub-upgrade / system-update, GMS, MAE/MCE consumers, and the frontend) now ship jose4j 0.9.6 on the runtime classpath. Kafka's client library needs this for SASL/OAUTHBEARER JWT validation but does not package it (KAFKA-20184). After the Kafka client upgrade in #13667 removed a transitive kafka_2.13 dependency that previously pulled jose4j in, secured Kafka setups using OAuthBearerLoginCallbackHandler could fail system-update (and other producers) with NoClassDefFoundError: org/jose4j/keys/resolvers/VerificationKeyResolver. Action: none — existing springKafkaConfigurationOverrides for SASL_SSL + OAUTHBEARER keep working; no Helm config change is required.

• (Assertions) User-authored assertion notes are now stored in a dedicated assertionNote aspect, so ingestion sources can fully upsert assertionInfo without overwriting notes. A resumable system-update step copies existing embedded notes to the new aspect and leaves the deprecated AssertionInfo.note field readable as a fallback during migration. No action is required. Operators can tune the migration with SYSTEM_UPDATE_ASSERTION_NOTE_MIGRATION_ENABLED, _BATCH_SIZE, _DELAY_MS, and _LIMIT.

• #18395 (Ingestion / Snowflake) Snowflake Semantic Views can now be ingested as first-class semanticModel + metric + logical-dataset entities instead of a single dataset (subtype "Semantic View"), gated by semantic_views.emit_semantic_model_entities. Unset (default) auto-resolves — enabled on newer DataHub Cloud, off on OSS, older Cloud, and connectionless runs (e.g. file sink) unless the recipe sets it true; false always forces the legacy dataset. Servers that don't support the new entities, an unreadable Metrics probe, or an unparseable server version also fall back to the legacy dataset. Action: URNs change, so repoint links, tags, and automations from the old urn:li:dataset:(...) URNs to the new ones (old datasets are soft-deleted with stateful ingestion enabled, else cleaned up manually); include_usage is ignored in the new mode; and semanticModel/metric URNs carry no env (use a distinct platform_instance per environment), while the logical datasets remain ordinary dataset URNs that keep env. Separately, include_queries: true now emits query entities even when include_usage: false — in both modes.

• (GMS / PostgreSQL write locking) Aspect writes and hard-deletes on PostgreSQL now acquire row locks in a consistent primary-key order — ORDER BY urn, aspect, version on the FOR UPDATE reads, plus an up-front ordered lock in deleteUrn. This prevents lock-order deadlocks between a multi-row FOR UPDATE write (e.g. logical-model linking) and a concurrent hard-delete on overlapping rows, which previously surfaced as RetryLimitReached: Failed to add after N retries. PostgreSQL-only: on PostgreSQL the default plan can acquire FOR UPDATE locks in physical scan order, whereas MySQL/InnoDB already locks the primary-key IN-list in index order, so MySQL behavior is unchanged. A new opt-in EBEAN_ENTITY_WRITE_ADVISORY_LOCK_ENABLED (PostgreSQL-only, default false) additionally serializes concurrent writers to the same entity via a transaction-scoped pg_advisory_xact_lock (auto-released on commit). This is deadlock-ordering only, and it blocks until the lock is released against a concurrent hard-delete of the same entity — there is no Java-side timeout (ENTITY_WRITE_LOCK_ACQUIRE_TIMEOUT_SECONDS bounds only the Hazelcast gate, not this lock); set a Postgres lock_timeout if you need to bound it. It is not a herd/contention guard — use ENTITY_WRITE_LOCK_BACKEND=hazelcast for bounded, off-connection waits under a write thundering-herd. Action: none; the ordering fix is automatic on PostgreSQL and the advisory lock is opt-in.

• (GMS / PostgreSQL write locking) Aspect writes and hard-deletes on PostgreSQL now acquire row locks in a consistent primary-key order — ORDER BY urn, aspect, version on the FOR UPDATE reads, plus an up-front ordered lock in deleteUrn. This prevents lock-order deadlocks between a multi-row FOR UPDATE write (e.g. logical-model linking) and a concurrent hard-delete on overlapping rows, which previously surfaced as RetryLimitReached: Failed to add after N retries. PostgreSQL-only: on PostgreSQL the default plan can acquire FOR UPDATE locks in physical scan order, whereas MySQL/InnoDB already locks the primary-key IN-list in index order, so MySQL behavior is unchanged. A new opt-in EBEAN_ENTITY_WRITE_ADVISORY_LOCK_ENABLED (PostgreSQL only, default false) additionally serializes concurrent writers on the same (urn, aspect) pair via a transaction-scoped pg_advisory_xact_lock (auto-released on commit). Keying on (urn, aspect) — the actual CAS / FOR UPDATE conflict unit — means cross-aspect writers on the same URN do not serialize (no over-serialization of the common case); whole-entity ops like deleteUrn lock the entity's full aspect key-set so delete↔upsert safety is key-set overlap. pg_advisory_xact_lock blocks until the holder's transaction commits/rolls back — there is no Java-side acquire timeout for this lock (ENTITY_WRITE_LOCK_ACQUIRE_TIMEOUT_SECONDS bounds only the Hazelcast write gate, not this advisory lock). To bound the wait, set a PostgreSQL lock_timeout (e.g. SET lock_timeout = '10s' at the session/role level); on timeout Postgres aborts the waiting statement with ERROR: canceling statement due to lock timeout (SQLSTATE 55P03). Whether that surfaces as a retryable conflict depends on the Ebean retry conflict classification (EBEAN_RETRY_BACKOFF_SQL_STATES) — the default classification covers deadlock states (40001/40P01) but not 55P03, so a lock_timeout abort may end as RetryLimitReached unless 55P03 is added to EBEAN_RETRY_BACKOFF_SQL_STATES. Action: none; the ordering fix is automatic on PostgreSQL and the advisory lock is opt-in.

• (GMS / MCE / System Update) Optional optimistic locking for aspect writes. Set OPTIMISTIC_LOCKING_ENABLED=true independently on each process to replace SELECT FOR UPDATE with compare-and-set updates on SystemMetadata.version for Ebean storage. Legacy rows without a version use plain updates until a write stamps one; Cassandra ignores the flag. Optimistic conflicts use the existing configurable Ebean retry/backoff policy (EBEAN_RETRY_*, including Retry-After). Action: none; the flag defaults to false.

• (GMS / MCE / System Update) Optional scoped retry + Hazelcast write gate for write thundering-herd mitigation. Both require optimistic locking. SCOPED_RETRY_ENABLED=true retries only the conflicted URN's branch within the transaction on an optimistic-lock conflict, instead of re-running the whole batch (up to 3 in-transaction attempts per conflict, bounded to avoid compounding with the outer transaction retry; not separately configurable). ENTITY_WRITE_LOCK_BACKEND=hazelcast (none | hazelcast, default none) serializes concurrent writers of the same (urn, aspect) — the actual CAS conflict unit, so cross-aspect writers on the same URN don't serialize — off the DB connection pool before they contend — the intended fix when DB connections are the bottleneck. It queues waiters in Hazelcast before they grab a DB connection (waiters hold zero pooled connections), whereas an in-transaction DB lock (the advisory, or MySQL GET_LOCK) pins a connection while blocked and so can't relieve pool exhaustion under a herd. It reuses the existing Hazelcast instance and activates whenever optimistic locking is on (independent of scoped retry; MySQL uses the gate only). On acquire timeout or a Hazelcast outage it degrades to lockless (CAS still guards). ENTITY_WRITE_LOCK_ACQUIRE_TIMEOUT_SECONDS (default 10) bounds the acquire wait; ENTITY_WRITE_LOCK_LEASE_SECONDS (default 300) auto-releases a lock held by a dead/hung holder. The Postgres pg_advisory_xact_lock above (EBEAN_ENTITY_WRITE_ADVISORY_LOCK_ENABLED) is a separate, independent option; when the Hazelcast gate is engaged the advisory lock is skipped to avoid double-locking (the gate degrades to lockless CAS on timeout/outage, not to the advisory — so the advisory is not a gate fallback). Action: none; all default off/none.

• (GMS / MCE / System Update) Ebean transaction retry state clearing. After a later retry attempt successfully commits or intentionally rolls back, transaction retry state is now cleared (transactionContext.success()), so a recovered conflict is not reported as retry exhaustion (RetryLimitReached). This is independent of optimistic locking. Action: none.

• (Great Expectations plugin) acryl-datahub-gx-plugin now allows great-expectations>=0.17.15 (upper bound <1.0.0 removed). GX 0.17/0.18 continues to use datahub_gx_plugin.action.DataHubValidationAction. GX Core 1.x uses the additive datahub_gx_plugin.action_v1.DataHubValidationAction. Action: Existing 0.x checkpoint configs are unchanged. For GX 1.x, instantiate the V1 action in Fluent Checkpoint actions=[...] (see the Great Expectations integration docs).

• (GMS / System Update) A new default-on, non-blocking system update, BackfillDatasetAliases, populates the system-owned aliases aspect for datasets created before that aspect existed. It scans Elasticsearch for datasets missing the alias, including soft-deleted ones, and emits metadata change proposals that the MCE consumer writes in batches; it never writes to the primary store directly. Because it is non-blocking it runs after startup and never delays GMS. It pages through the index without pausing by default; set ..._DELAY_MS above zero to throttle it if the emit rate competes with normal write traffic. Configure with SYSTEM_UPDATE_DATASET_ALIASES_ENABLED (default true), ..._BATCH_SIZE (1000), and ..._DELAY_MS (0). Set SYSTEM_UPDATE_DATASET_ALIASES_REPROCESS=true to re-emit for every dataset after the job has already completed. Action: none required. If the system update job fails while this step is running, re-run it — no completion marker is written on failure, and the next run resumes from the last page it reached rather than starting over.

• (GMS / Structured Properties) System-update reindex detection now compares Elasticsearch field type for structured properties that already exist in both current and target mappings. This repairs indices where NUMBER properties were locked as float/long by dynamic mapping before an explicit double put-mapping landed. Requires both ENABLE_STRUCTURED_PROPERTIES_SYSTEM_UPDATE=true and ENABLE_STRUCTURED_PROPERTIES_TYPE_MISMATCH_REINDEX=true / structuredProperties.typeMismatchReindexEnabled (default true).

• (GMS / View Authorization) schemaField and container are view-restricted by default (not in the effective unrestricted defaults; container no longer has viewUnrestricted: true in the registry). View Entity Page on a schema field inherits from the parent dataset encoded in the schemaField URN (urn:li:schemaField:(<datasetUrn>,<fieldPath>)), then falls back to a direct grant on the column URN. This matches the existing logical-parent write candidate order. OpenAPI and RestLi READ-by-URN use the same parent-inheritance rules when REST API authorization is enabled, so GraphQL entity reads and OpenAPI/RestLi GET agree for the same actor and schemaField URN. GraphQL container entity loads and ParentContainers breadcrumbs use field-strip redaction (real urn/name may still appear). The entity-graph hierarchy cache remains unauthenticated (real URNs only). OSS has no Cloud search pushdown. Resource-scoped grants authorize GET-by-URN / entity-page VIEW; they do not by themselves unlock OpenAPI/RestLi list or scroll discovery on OSS.

• (GMS / search defaults) Default entity-type lists for GraphQL search, autocomplete, browse V2, and quick-filter priority are now configurable via elasticsearch.search.*EntityTypes (value / add / remove) and the matching SEARCH_*_ENTITY_TYPES{,_ADD,_REMOVE} environment variables. Stock YAML defaults match the former hardcoded GraphQL lists. An explicitly empty resolved list means GraphQL searches no entity types (it does not expand to all indices). Unknown registry names in these lists are soft-dropped with a warn at startup. See Environment Variables and Customizing Search.

• (Metadata Model / Semantic Model) Logical datasets exposed by a semanticModel are now first-class dataset entities (with the Semantic Model Dataset subtype and a semanticModelProperties aspect) rather than an embedded ModelDataset/SemanticField record; per-field semantic metadata lives on semanticFieldAnnotation. The SemanticModel is a container (bounding box), not a lineage hop — see Deprecations for semanticModelInfo.datasets and the container-model migration.

• #18542 (Ingestion / Redshift) Multi-line queries are no longer silently dropped from lineage, usage, and query history. Redshift's system tables store query newlines as literal \n characters; the unified query-history feed passed this text to the SQL parser unchanged, so any formatted, multi-line query (most BI-tool, hand-written, and dbt/ELT SQL) failed to parse and was discarded without error. These queries are now parsed correctly. No action required; after upgrade, affected Redshift sources will capture substantially more lineage, usage, and query entities than before.

• #18727 (Ingestion / Airbyte) Stream namespaces are now recovered for connections read through Airbyte's Public API, which omits them from the connection payload. Previously every stream in such a connection fell back to the source connector's default schema, so datasets for streams in other schemas pointed at the wrong table. DataHub now reads the namespace from Airbyte's /streams endpoint (which reports namespaces from Airbyte 1.7.0 onwards) and only uses it when it is unambiguous; where several schemas expose a same-named stream and Airbyte does not say which is which, the namespace is left alone and a warning names the candidates. Action: none required, but affected datasets change URN — they move from <database>.<default_schema>.<table> to <database>.<actual_schema>.<table>. Expect new dataset entities on the next run, and soft-delete the stale ones (or enable stateful ingestion so they are marked removed automatically). On Airbyte versions with no /streams endpoint behaviour is unchanged, and the run now warns that stream metadata is unavailable.

• #18856 (Ingestion / Airbyte) A schema declared per table in the source connector's configuration (tables[].schema, used by SQL Server among others) now takes precedence over the connector's single schema-shaped configuration key. That key sometimes holds something that is not a schema at all — a database name, for instance — and it was overriding the per-table value, which then dropped the schema tier from the dataset URN entirely when it matched the database name. A new per-source default_schema under sources_to_platform_instance supplies a schema for deployments where neither Airbyte nor the connector configuration reveals one; it is only consulted as a last resort, so it can be left in place after an Airbyte upgrade. Runs also warn (Stream Namespaces Not Reported) when Airbyte describes a source's streams but reports no namespace for any of them, which is what pre-1.7.0 Airbyte does. Action: none required, but datasets that were missing a schema tier change URN — they move from <database>.<table> to <database>.<schema>.<table>. Expect new dataset entities on the next run, and soft-delete the stale ones (or enable stateful ingestion so they are marked removed automatically).

• (GMS / search) Data Products are now a first-class search filter and facet on their member assets. Data Product membership remains authoritative on the Data Product side (dataProductProperties.assets), but member assets previously carried no data-product field: a dataProduct search filter matched nothing (even combined with other filters), and the Data Product facet never appeared in search. A new asset-side dataProducts aspect (array[DataProductAssociation], searchable fields dataProduct and isDataProductOutputPort, addToFilters) is kept in sync by DataProductAssetsSideEffect whenever a Data Product's asset list changes, so filtering by Data Product composes with any other filter and the facet is returned automatically. Like domains, the Data Product URN is included in default free-text search (queryByDefault) on member assets. Existing memberships are populated by the ZDU / MigrateAspects sweep when dataProductProperties schemaVersion advances (v1→v2 via DataProductPropertiesMigrationMutator); set REPROCESS_DATA_PRODUCT_ASSETS=true to force a re-upsert resync if drift is observed (heals missing ADDs and stale REMOVEs via search). Hard-delete of a Data Product (or DELETE of dataProductProperties) scrubs the asset-side aspect. MCP_SIDE_EFFECTS_DATA_PRODUCT_ASSETS_MAX_FANOUT (default 500) controls the batch size for reading existing asset-side dataProducts during sync and for search scroll pages during reprocess; all unsynced ADD patches for a commit are emitted in one side-effect pass (apply is Kafka-batched at 500). Very large memberships (tens of thousands of assets) mean a larger one-shot memory/Kafka wave on migrate or reprocess. REMOVE patches from delete are not capped. Action: for pre-existing memberships, run system-update with MigrateAspects enabled (SYSTEM_UPDATE_MIGRATE_ASPECTS_ENABLED=true or ZDU_STAGE_20=true, as in quickstart); newly created or edited memberships sync live with no action. New env vars: MCP_SIDE_EFFECTS_DATA_PRODUCT_ASSETS_ENABLED (default true), MCP_SIDE_EFFECTS_DATA_PRODUCT_ASSETS_MAX_FANOUT (default 500), REPROCESS_DATA_PRODUCT_ASSETS (default false), and BOOTSTRAP_SYSTEM_UPDATE_DATA_PRODUCT_ASSETS_BATCH_SIZE (default 1000).

• (GMS / search indexing) Analyzed-text search fields (@Searchable(fieldType=TEXT / TEXT_PARTIAL / WORD_GRAM)) now guard their keyword surfaces with a byte-safe ignore_above (8191 characters, i.e. 32766 ÷ 4) instead of 32766. The Elasticsearch/OpenSearch ignore_above limit is counted in characters while Lucene's hard 32,766-byte per-term limit is counted in bytes, so a multi-byte value could previously slip past ignore_above and still be rejected — dropping the entire document from the search index (its tokenized .delimited sub-field, which has no per-term limit and is what full-text queries target, indexes the value in full regardless). This is applied as an in-place mapping update, so no reindex is required; documents previously dropped by this limit are re-indexed on their next write. The only behavioral change is that exact-match/sort/aggregation on the keyword parent of a TEXT field is now skipped above 8191 characters rather than 32766 — irrelevant for the long-form content these fields hold.

• #14911 (Spark lineage) Apache Spark 4.x is now supported. The agent is verified to attach and emit lineage on Apache Spark 4.0.0 via a dedicated compatibility smoke test (sparkSmoke4Test). Spark 4 requires Scala 2.13 + Java 17, so use the Scala 2.13 build of the agent (io.acryl:acryl-spark-lineage_2.13); the Scala 2.12 build continues to target Spark 3.x. No recipe changes are required.

• #18102 (Ingestion framework) Oversized queryProperties aspects no longer fail ingestion runs. The EnsureAspectSizeProcessor size guard for queryProperties was fixed to measure serialized (JSON) size correctly instead of comparing a raw character count against a byte-based target — the previous unit mismatch caused the guard to give up and emit oversized aspects, which GMS then rejected with a 400 Cannot parse request entity, failing the run after enough rejections. This most often affected Snowflake sources with include_queries enabled against accounts with heavy temp-table query flows (e.g. Hightouch), but applies to any SQL connector that emits query entities. No action required; oversized statements are now truncated (with a warning) rather than failing the run.

• #17963 (Ingestion / Glue) Lake Formation resource links (which are schemaless pointers in the catalog) now have their schema populated from the owning table so their columns are visible on the ingested dataset. This is controlled by a new resolve_resource_link_schema flag, which defaults to true. The owning table's schema is read from DataHub first (no AWS call when that account is already ingested) and otherwise via a cross-account glue:GetTable on the target. If you ingest a catalog that contains resource links, after upgrade the connector may issue new cross-account glue:GetTable calls and, for any shared table you lack glue:GetTable on, emit a warning (the link still ingests, schemaless, and remains reachable through its upstream lineage edge — the run does not fail). Action: grant glue:GetTable on the shared tables to populate their schemas, or set resolve_resource_link_schema: false to keep resource links schemaless as before.

• #17962 (Spark lineage) spark.datahub.metadata.dataset.file_partition_regexp now also applies to bare-namespace filesystem datasets (file, dbfs) that previously bypassed partition stripping. This makes behavior consistent with scheme-based FS paths (s3://, hdfs://, …) and is arguably a bugfix, but if you have set file_partition_regexp, the emitted dataset URNs for file/dbfs sources will change (the partition segment is now stripped), so lineage edges to the old, un-stripped URNs can orphan. Action: only affects opted-in users — if you rely on the previous file/dbfs URNs, unset file_partition_regexp or re-point downstream lineage to the new (partition-stripped) URNs. Recipes that leave file_partition_regexp unset are unaffected (byte-identical output).

• #17617 (Ingestion / Tableau) lineage_overrides.platform_override_map is now applied to the body of the generated URN, not just to its platform stamp. Previously, an override that crossed naming tiers — most commonly presto → athena, since Tableau models Athena connections as presto with a hive catalog — produced a URN stamped with the target platform but shaped like the source platform (catalog.schema.table rather than schema.table). Those URNs never matched what the target platform's own ingestion source emits, so the lineage edges resolved to nothing and no error was raised. If you use lineage_overrides to remap a three-tier platform to a two-tier one, the emitted upstream URNs change after upgrade and lineage starts resolving. (Overrides in the other direction are unchanged: a two-tier connection has no catalog to fill a three-tier target's first segment, so those URNs stay two-part and the catalog has to come from platform_instance_map.) The old URNs were already dangling, but DataHub will have auto-created stub entities for them and lineage now points away from those stubs. Action: none for lineage itself. If anyone hand-applied tags, documentation, or ownership to one of those stub datasets, that metadata stays behind on the now-orphaned stub — move it to the real dataset and soft-delete the stub. Recipes without lineage_overrides, and overrides between platforms of the same tier, are unaffected.

• #18001 (Ingestion / Redshift) table_pattern is now applied to SQL-parsing-derived lineage and usage, not just to the catalog walk. Previously, lineage/usage/query entities produced by the SQL parsing aggregator were emitted for all tables regardless of table_pattern; they are now filtered consistently with the other SQL connectors (Snowflake, BigQuery, Databricks). The default table_pattern allows everything, so most setups are unaffected. If you set table_pattern and relied on lineage/usage edges to excluded tables, widen the pattern to keep them.

• #18001 (Ingestion / Redshift) The per-user userEmail field in Redshift dataset usage statistics (DatasetUserUsageCounts) is no longer populated. Redshift's catalog only stores usernames, so this email was previously synthesized as <username>@<email_domain> and carried no information beyond the user's CorpUser URN. That URN is unchanged, so user attribution and ownership links are unaffected. No action required; if you relied on the synthetic email, derive it from the user URN plus your email_domain setting. (This is a side effect of Redshift usage now flowing through the standard SQL-parsing usage aggregator. A new opt-in include_column_usage_stats config additionally enables column-level usage statistics.)

• #18065 (Ingestion / Redshift) The Redshift connector now emits per-query popularity statistics (queryUsageStatistics) on Query entities, matching Snowflake, BigQuery, and Databricks. This is controlled by a new include_query_usage_statistics flag, which defaults to true but, like include_column_usage_stats, only takes effect when include_usage_statistics is enabled (off by default) and requires lineage_generate_queries (default true). If you already run Redshift usage ingestion (include_usage_statistics: true), query popularity stats will start being emitted after upgrade with no config change; set include_query_usage_statistics: false to opt out. Enabling query usage causes read queries to be parsed by the SQL lineage aggregator, which is heavier than the default stl_scan table-usage path.

• #18285 (GMS / UI) Multi-language support (I18N_ENABLED) now defaults to on in OSS (previously off). Combined with the UI defaulting to the user's browser locale, users whose browser is set to a non-English language will now see DataHub in that language by default where a translation exists. Several languages are still Beta and may contain untranslated strings. Users can always pick their language under Settings → Preferences. Action: to restore English-only behavior, set I18N_ENABLED=false on GMS.

• #18130 (Ingestion / SQL parsing) Fixed per-query popularity statistics (queryUsageStatistics) being dropped or misattributed for queries that load data through temporary/staging tables — the common ELT/dbt pattern — affecting every connector that uses the shared SQL parsing aggregator (Snowflake, BigQuery, Redshift, Databricks Unity Catalog). Such a query is represented by a single merged ("composite") Query entity that lineage points to. Previously its usage stats either went missing entirely (when the temporary tables were excluded by table filters) or were emitted on separate per-statement Query entities that lineage never referenced. Usage now lands on the composite Query, the redundant per-statement Query entities are no longer emitted, and operations (operationType) on these tables reference the same composite Query as lineage. No action required — query popularity for temp-table-fed tables becomes correct after upgrade; if you previously saw usage on stray Query entities, it now consolidates onto the Query that lineage actually links to.

• (GMS / search indexing) Inline Elasticsearch indexing and entity-graph cache invalidation on ingest now require explicit systemMetadata.properties.appSource = ui (or the sync-index header), not merely a GraphQL request context. GroupService and RoleService stamp UI source via AspectUtils.buildSynchronousMetadataChangeProposal; other GraphQL resolvers that bypass MutationUtils revert to async MAE indexing. See GMS Entity Graph Cache — Invalidation (sync writes).

• (GMS / primary storage read pool) Optional second connection pool (Ebean) or Cassandra session for entity-aspect reads, while writes and FOR UPDATE reads stay on PRIMARY. Enable with EBEAN_READ_POOL_ENABLED / CASSANDRA_READ_POOL_ENABLED (default off). Use split-pool (same URL/hosts) to isolate connection limits, or set EBEAN_READ_POOL_URL / CASSANDRA_READ_POOL_HOSTS to offload to a replica. Separate from DATAHUB_READ_ONLY=true, which disables writes and does not register the read pool. See Primary storage read pool.

• #18393 / #18421: (Operations / logging) Core services and the frontend can optionally ship logs to a Loki-compatible aggregator. Set LOG_AGGREGATOR_ENDPOINT to enable shipping (also used for support-bundle log collection without relying on the Kubernetes API). Opt-in; unset leaves local logging unchanged.

• (GMS / Groups) Fixed native group membership becoming permanently unrecoverable after a group was deleted and recreated under the same identifier. Deleting a group removed its membership edges from the graph but left each member's nativeGroupMembership aspect still referencing it; re-adding those members then wrote unchanged aspect content, which is suppressed both at change-log emission and by graph diff mode, so the membership never took effect — addGroupMembers reported success while the group stayed empty. Group deletion now clears the member-side references it is responsible for, and re-adding a member repairs a membership whose graph edge is missing. Because policies evaluate membership from the aspect while the UI and notifications read the graph, affected users could retain a deleted group's privileges while appearing to belong to no group. Action: none for new deletions. Members stranded by an earlier delete are repaired by re-adding them to the group; a restore-indices run over corpuser / nativeGroupMembership also rebuilds the missing edges, but it rebuilds references to groups that no longer exist too, so clear those aspect entries first where the group was intentionally deleted.

• (GMS / Groups) Adding members to a group is now all-or-nothing with respect to unknown users. Previously members were processed one at a time, so a request naming a user that does not exist added the users listed before it and then failed, leaving a partial application. The request now fails without adding anyone, and names the users it could not find. Action: none, unless you relied on the partial-application behavior — split such requests per user to get the old semantics.

• (GMS / Groups) Adding members to a group is now written in a single batched request rather than one write per member, so a large addGroupMembers call is substantially faster. The writes stay synchronous, so the member list is readable immediately afterwards. Note that a very large request is still partitioned by the entity client's configured batch size, so a failure part-way through can leave earlier members added — the all-or-nothing guarantee above covers unknown users, which are rejected before anything is written. Action: none.

Environment Variables

• OPTIMISTIC_LOCKING_ENABLED (default false) — Use CAS aspect writes instead of SELECT FOR UPDATE with Ebean storage. See Other Notable Changes above and Environment Variables.

• SCOPED_RETRY_ENABLED (default false) — Requires optimistic locking. Retry only the conflicted URN's branch on conflict instead of the whole batch.

• ENTITY_WRITE_LOCK_BACKEND (default none) — none | hazelcast: per-(urn, aspect) write gate that serializes concurrent writers of the same aspect off the DB connection pool (the actual CAS conflict unit — cross-aspect writers on the same URN don't serialize; whole-entity ops like hard-delete lock the entity's full aspect set). Requires optimistic locking; independent of scoped retry.

• ENTITY_WRITE_LOCK_ACQUIRE_TIMEOUT_SECONDS (default 10) — Bounds the write-lock acquire wait before proceeding lockless (CAS still guards).

• ENTITY_WRITE_LOCK_LEASE_SECONDS (default 300) — Maximum gate hold: auto-releases a write lock after this long so a dead/hung holder can't wedge a URN. It also fires for a healthy holder whose gated write runs longer than the lease — after which the gate no longer serializes that URN (CAS still guards). Set it above your longest expected gated write.

• VIEW_UNRESTRICTED_ENTITY_TYPES / _ADD / _REMOVE — Replaces VIEW_RESTRICTED_ENTITY_TYPES. Overlays on the entity-registry.yml viewUnrestricted baseline when VIEW_AUTHORIZATION_ENABLED=true (shared GMS config for Cloud Search Access Controls and OSS entity-page VBAC). Query-time search filtering remains DataHub Cloud–only. See Breaking Changes above and Environment Variables.

• SEARCH_DEFAULT_ENTITY_TYPES / SEARCH_AUTOCOMPLETE_ENTITY_TYPES / SEARCH_BROWSE_ENTITY_TYPES / SEARCH_PRIORITIZED_*_ENTITY_TYPES (each with optional _ADD / _REMOVE) — Configure GraphQL default entity-type lists when callers omit types. Unset keeps application.yaml defaults. An explicitly empty value searches no entity types. See Environment Variables.

• EBEAN_READ_POOL_ENABLED / EBEAN_READ_POOL_URL / CASSANDRA_READ_POOL_ENABLED / CASSANDRA_READ_POOL_HOSTS — Optional primary-storage read pool for entity-aspect reads. See Notable Changes above and Primary storage read pool.

• LOG_AGGREGATOR_ENDPOINT — Opt-in endpoint for shipping service logs to a Loki-compatible aggregator (also enables support-bundle log collection). See Notable Changes above.

• #18334 (GMS / patch) Fixed keyed-array patch rebase (ArrayMergingTemplate) dropping the map key when a patch creates a new element through a deeper path — e.g. adding a field-level tag or term to a schema field with no existing editableSchemaMetadata entry produced an EditableSchemaFieldInfo with no fieldPath and was rejected with HTTP 422 - fieldPath is required. The fix applies to every keyed array (tags, terms, ownership, upstreams, etc.). This previously required an SDK-side workaround, now removed: the Python SDK's Dataset.patch_builder().add_tag()/add_term() on a new schema field, and any client that patches a keyed array through a deep path, now depend on this GMS fix. A new SDK against an older GMS (DataHub Core < 1.8.0 / DataHub Cloud < 2.1.0) regresses to the 422 on those field-level adds. Action: upgrade GMS to a version containing this fix before upgrading clients that rely on it.

• #18220 (Ingestion / Unity Catalog) ML model ingestion controls were fixed. Two things changed:

  • There is now an include_ml_models flag (default true). Set it to false to skip ML model ingestion entirely — no calls are made to the Databricks registered-models API. Previously there was no way to turn ML models off short of catalog/schema deny patterns.
  • ml_model_max_results is now enforced as a real per-schema cap on the number of ML models ingested, matching its description. 0 now ingests none. Previously the value was passed through to the API only as a page-size hint and did not bound the total, so it silently returned every model (and 0 did not disable anything). If you have more than ml_model_max_results (default 1000) ML models in a single schema and were relying on all of them being ingested, raise the limit or set include_ml_models: false to opt out.

  Separately, MLModel entities were being emitted with a hardcoded PROD environment regardless of the recipe's env, while their MLModelGroup used the configured env. MLModel URNs now honor env, so groups and models land in the same environment. Models ingested before the fix keep their old PROD URNs until re-ingested.

• #17894 (GMS / metadata service) Raised Jackson's JSON property-name length limit (maxNameLength) on the metadata deserialization paths. Previously only the string-value length limit was raised, leaving maxNameLength at Jackson's default of 50,000 characters. Deeply-nested struct field paths (e.g. dbt column-level lineage) can be carried as JSON property names in upstreamLineage patches and exceed that, causing GMS to reject the change with StreamConstraintsException: Name length (N) exceeds the maximum allowed (50000) when applying the patch. The limit now defaults to 16 MB (matching the existing value-length limit) and is tunable via the new INGESTION_MAX_SERIALIZED_NAME_LENGTH environment variable. GMS additionally logs a WARN identifying the entity/aspect when a patch carries an over-long property name or string value (the exception itself omits the offending content); these advisory thresholds are tunable via INGESTION_OVERSIZED_NAME_WARN_THRESHOLD (default 50,000) and INGESTION_OVERSIZED_VALUE_WARN_THRESHOLD (default 10,000,000). Note the string-value limit is unchanged (still 16 MB) — only the name limit was raised.

• #17376: (Ingestion / Hex) Major in-place upgrade of the hex connector: upstream lineage (table-level and column-level), Project → Component links, run history (lastRefreshed), and optional AI context documents are now extracted directly from Hex REST APIs — no external CLI, warehouse-side ingestion dependency, or query-tag scraping required. See the Hex connector docs for the new include_lineage, use_queried_tables_lineage, connection_platform_map, and include_context_documents options.

• (Ingestion / dbt) dbt test assertion entities now emit an ownership aspect when the dbt test node has explicit owner metadata (meta.owner / config.meta.owner).

• #17646 (Ingestion / Dremio) Query lineage and view-parent lineage emission now respect schema_pattern and dataset_pattern (and skip the _accelerator_ reflection schema), matching the behavior of Snowflake and Redshift. Previously these filters only applied to the catalog walk: URNs discovered through sys.jobs_recent or view-parent references were emitted unconditionally, so DataHub materialized ghost dataset entities for filtered schemas with no parent container or browse path. A new report counter lineage_dropped_filtered records how many lineage references were skipped. If you relied on lineage edges to filtered upstreams (e.g. to keep references to system schemas that you excluded from the main catalog), widen schema_pattern / dataset_pattern to admit them — or run with the defaults, which allow everything.

• #17648 (Ingestion / Dremio) Added DataHub platform mappings for several Dremio source types that previously fell through to a lowercased default: BIGQUERY → bigquery, RESTCATALOG → iceberg (Apache Polaris OSS, Nessie with Iceberg REST, AWS Glue Iceberg REST, S3 Tables, Confluent Tableflow, Microsoft OneLake), SAPHANA → hana, SNOWFLAKEOPENCATALOG → iceberg (Snowflake's hosted Polaris), and UNITY → databricks (Databricks Unity Catalog). Existing Dremio recipes that previously produced lineage URNs like urn:li:dataPlatform:restcatalog will now produce urn:li:dataPlatform:iceberg (etc.); update any URN-based lineage / dashboards that depend on the old values.

• #17649 (Ingestion / Dremio) The domain recipe field now actually emits a Domains aspect for every Dremio container and dataset. Previously, the config field was documented and plumbed through to the aspect builder, but the aspect was never yielded, so the setting silently did nothing. Recipes that already set domain will now start attaching the configured domain to ingested entities — verify the domain URN/name is what you intend before upgrading. Bare names (e.g. domain: marketing) are now resolved against the live DataHub graph via DomainRegistry (same pattern as Snowflake/BigQuery), so the recipe must have a datahub_api/sink with a graph configured; otherwise pass a full urn:li:domain:<id>.

• #17651 (Ingestion / Dremio) Dataset properties no longer emit a synthetic created timestamp when Dremio does not report one. Previously, missing creation timestamps surfaced as epoch 0 (1970-01-01) on the dataset properties aspect; the field is now omitted entirely. Dashboards or alerting that treated epoch 0 as a valid created time should treat it as missing instead.

• #17812 (Ingestion / Glue) When extract_lakeformation_tags is enabled, the Glue connector now also extracts column-level Lake Formation tags by default (new extract_lakeformation_column_tags, default true) and can propagate a database's Lake Formation tags down to its tables and columns (new propagate_lakeformation_tags, default false). Existing recipes with extract_lakeformation_tags: true will start emitting directly-assigned column tags after upgrade with no config change; set extract_lakeformation_column_tags: false to retain the previous (table/database-only) behavior. Inherited (propagated) tags are marked with propagation attribution and a propagated context so they can be distinguished from directly-assigned tags. No additional AWS API calls are made — all tag levels come from the existing per-table GetResourceLFTags response.

• #17860 (CLI / Python SDK) Mutual TLS (mTLS) client authentication is now supported for every outbound HTTPS path in the CLI / SDK and any caller that builds a session via RequestsSessionConfig. Set DATAHUB_CLIENT_CERT_PATH (and optionally DATAHUB_CLIENT_KEY_PATH for the kubernetes tls-secret layout where cert and key live in separate files) in the process environment and the SDK presents the client certificate during the TLS handshake. No client certificate is sent when neither is configured — fully backwards compatible.

• #17927 (Ingestion / Matillion) Several improvements to the matillion-dpc connector:

  • Folder hierarchy in containers: pipelines and components are now organized into a browsable container tree that mirrors their path in Matillion (Project › Environment › <folder> … › Pipeline › Component), instead of all pipelines sitting flat under the environment. Folder levels are derived from the pipeline path, so the browse tree lines up with the paths used in pipeline_patterns. Components (DataJobs) now also carry a container aspect and browse directly under their pipeline. Browse paths no longer include the entity itself as a trailing node.
  • Environment-scoped lineage: lineage jobs whose environment cannot be resolved (not published and not seen in execution history) are now skipped rather than emitted as flows floating under the bare project. Enabling extract_run_history (or include_unpublished_pipelines) helps resolve environments for unpublished child orchestrations. A lineage_jobs_skipped_no_environment report counter records skips.
  • Run history at both levels: per-execution DataProcessInstance entities are now emitted for the pipeline (DataFlow) as well as each component (DataJob), populating the "Runs" tab at both levels.
  • New extract_run_history flag (default false): emit run history for published pipelines without also discovering unpublished ones (implied when include_unpublished_pipelines is enabled).
  • Fixed external links: pipeline and run links now point at the Data Productivity Cloud (Maia) console (app.matillion.com) with a valid query, instead of an invalid timeFrame filter. Pipeline links search the pipeline file name only (the console search does not match the full folder path). Streaming pipelines, which have no dedicated console deep-link, no longer emit a link built from the API host that did not resolve.
  • New include_dependent_pipelines flag (default true, preserves prior behavior): when disabled, child pipelines that appear only in lineage events (and were not discovered as project pipelines) are no longer created as their own DataFlows/DataJobs; lineage for discovered pipelines is unaffected.
  • include_external_urls now defaults to false: Matillion console links (project, run, and pipeline) are no longer emitted unless you opt in with include_external_urls: true. On re-ingest, console links emitted by earlier versions are removed. The default was flipped because the pipeline link searches the observability dashboard by file name and only resolves for pipelines that ran recently and whose editor name matches their file name.
  • Environment and folder containers are always emitted: extract_projects_to_containers now only controls whether the environment and folder containers nest under a top-level project container. Setting it to false no longer suppresses those containers — they hang at the root instead.
  • Multi-environment lineage: a pipeline path that is published or executed under more than one environment now has its lineage flow and component nested under each of those environments, rather than being attributed to whichever environment was discovered last.
  • Note on toggling include_unpublished_pipelines: published-only ingestion keys DataFlows on the full pipeline path, while unpublished discovery keys on the base pipeline name, so flipping this flag can change a pipeline's DataFlow/DataProcessInstance URNs and orphan entities emitted under the previous setting. Pick a value per source and keep it stable.

• #17891 (Ingestion / Teradata) Teradata profiling now respects profile_table_size_limit (default 5 GB); tables larger than the limit are skipped by default, bringing Teradata in line with Snowflake, BigQuery, Databricks, and Oracle. Table size is derived from DBC space accounting (SUM(CurrentPerm) in DBC.TableSizeV). Tables whose size cannot be determined (e.g. the ingestion user lacks SELECT on DBC.TableSizeV) are still profiled (fail-open). Action: to restore the previous behavior of profiling all tables regardless of size, set profile_table_size_limit: null under profiling.

• (Ingestion / Databricks Unity Catalog) When usage_data_source is set to SYSTEM_TABLES (or AUTO with a SQL warehouse configured), the connector now reads query history from system.query.history and derives usage statistics, operational stats, and query entities by parsing those statements through the shared SQL parsing aggregator (the same component used by the Snowflake, BigQuery, and Redshift connectors), which deduplicates identical statements by fingerprint so each distinct query is parsed only once. Table and column lineage continue to be sourced natively from system.access.table_lineage and system.access.column_lineage and are unaffected. Query entities are now emitted for warehouse statements on this path (include_queries, default true; include_query_usage_statistics, default true). The usage and lineage history window on the system-tables path may extend up to 365 days (the previous cap was 30 days, which applied only to the REST API path). Existing Databricks Unity Catalog ingestions that set a warehouse_id or use usage_data_source: SYSTEM_TABLES are affected; no recipe changes are required unless you want to opt out (usage_data_source: API). Known limitation: Databricks query history does not record the catalog or schema that was active when each query ran. As a result, queries that reference tables by a short (one-part) name only — typical of ad-hoc notebook queries — cannot be attributed to a specific table, and usage or operation counts for those queries may be lower than in previous releases.

• #17971 (Ingestion / Databricks Unity Catalog) On the system-tables usage path, a time window that returns no queries (an idle workspace) now emits zero-value datasetUsageStatistics for the ingested tables, where it previously emitted none. This aligns Unity with the Snowflake, BigQuery, and Redshift connectors, which stamp a zero datapoint whenever the usage read succeeds. Because datasetUsageStatistics is a timeseries aspect written via UPSERT, this records a current-bucket zero datapoint and does not delete prior usage history. A genuine fetch or permission failure still emits nothing (it is surfaced as a run failure instead), so existing usage history is never overwritten with zeros on error.

• #18017 (Ingestion / Power BI) Hive ODBC sources are now resolved to the hive data platform instead of hadoop. Previously both folded into a shared (hadoop|hive) mapping, which emitted upstream URNs under urn:li:dataPlatform:hadoop (never matching the Hive connector's hive URNs) and broke native SQL parsing because sqlglot has no hadoop dialect, so Hive lineage was silently dropped. After upgrade, new runs emit hive upstreams that parse correctly and align with the Hive connector; any previously-emitted hadoop upstreams were already broken/dangling and are retired by stateful stale-entity removal (if enabled) or can be cleaned up manually. Hadoop (non-Hive) resolution is unchanged and no recipe changes are required.

• #18289 (Ingestion / S3, GCS, ABS) max_rows (default 100) now also bounds schema inference for .json files, where it was previously ignored. A .json file holding a single object is no longer read end to end: it is streamed and every array inside it is truncated to max_rows elements. This removes an out-of-memory risk on large files such as {"data": [ ...millions... ]}, but the inferred schema is now a sample — for {"rows": [ ...5000 records... ]}, a field that first appears in record 3000 is no longer reported, and nullability is decided from the first max_rows records instead of all of them. Top-level JSON arrays and .jsonl files were already sampled this way, so only single-object .json files change. Action: if you need the previous exhaustive behavior for .json files, raise max_rows above the largest array you care about (note this raises memory use for CSV/TSV inference too).

• (Spark lineage) The acryl-spark-lineage agent now shades its bundled OpenLineage under io.acryl.shaded.io.openlineage instead of exposing it at io.openlineage. This lets the agent run alongside environments that ship their own OpenLineage on the Spark classpath — notably Amazon EMR 7.12+ / SageMaker Unified Studio (DataZone), whose built-in /usr/share/aws/datazone-openlineage-spark/lib/ jars previously collided with the agent's copy and failed the Spark job. The destructive workaround (rm -rf /usr/share/aws/datazone-openlineage-spark/lib/) is no longer needed; both DataHub and DataZone lineage can be captured on the same cluster. The user-facing listener (datahub.spark.DatahubSparkListener) and all spark.datahub.* / spark.openlineage.* config are unchanged — no recipe changes required. The OpenLineage extension SPI (io.openlineage.spark.extension.*), which data-source connectors implement at its canonical name, is intentionally left unrelocated. Action: only required if you depended on the agent transitively exposing io.openlineage.* classes to your own code (rare) — reference the shaded coordinates instead.

v1.6.0.3

LTS hotfix on v1.6.0: authorization tightening, SecretService caller guard, Python 3.11+ on this line, and CVE dependency bumps. Changelog: v1.6.0.2...v1.6.0.3.

Requirements:

• CLI / Python SDK: 1.7.0.13 (default; security patches). Older 1.6.0.x / 1.7.0.x CLIs still work.
• Helm Chart: 1.1.2

From v1.6.0 / v1.6.0.1 / v1.6.0.2: drop-in image bump. Upgrade the CLI to 1.7.0.13 for the security patches; older CLIs remain compatible. No ZDU or Helm chart change.

Breaking Changes

• (CLI / Python, v1.6.0.x packages only) The v1.6.0 LTS hotfix CLI/SDK line requires Python 3.11+ (master / 1.7.0.x still supports 3.10). If you stay on 1.6.0.3 packages, rebuild venvs. Notable floors on that line: urllib3 2.x, confluent-kafka ≥2.15.1, snowflake-connector-python ≥4.7.3, patched unstructured extras. Airflow 2.x urllib3 1.26 constraint files are unsatisfiable.

• (GMS / Secrets) SECRET_SERVICE_CALLER_GUARD_MODE defaults to ENFORCE. Browser sessions and user PATs can no longer decrypt UI secrets via getSecretValues. Use datahub-actions with system client credentials, or AUDIT during rollout. Custom plugins must call SecretService.encrypt/decrypt(OperationContext, ...).

• (GMS / Authorization) Stricter checks across GraphQL, OpenAPI, and Rest.li: patchEntity authorizes by URN type; asset-settings writes need Edit Entity or Manage Asset Summary; form assign/unassign needs Manage Compliance Forms; analytics _search needs Analytics API access; corpUserCredentials reads need Manage User Credentials.

Other Notable Changes

• Default bundled CLI / Python SDK is 1.7.0.13 for security patches; older CLIs still work.
• Opt-in Play security headers (DATAHUB_SECURITY_HEADERS_*). getSecretValues isolates per-secret decrypt failures.
• Additive document semanticText embedding-source aspect.
• Java and Python CVE bumps (Spring 7.0/4.0 patches, Jackson 2.22.2 / Parquet 1.18.1, mariadb-java-client 2.7.14, aiohttp 3.14.3, and related lockfile pins). Rebuild/redeploy picks them up.

v1.6.0.2

Patch release for v1.6.0 — security and authorization hardening plus CVE dependency bumps. No new features; no schema or model changes.

Requirements:

• CLI / Python SDK: 1.6.0.17, 1.7.0.3
• Helm Chart: 1.1.2

Breaking Changes

• #18886 (GMS / Auth) The hardcoded systemClientSecret default (JohnSnowKnowsNothing) has been removed from the server configs. Action: set DATAHUB_SYSTEM_CLIENT_SECRET on GMS, MAE/MCE/PE consumers, the frontend, and Actions before upgrading — services relying on the built-in default will fail to authenticate.

• #19508 (GMS / Timeseries read authorization) Dataset profile, usage, and operations timeseries (and dashboard usage statistics) now require the matching View Dataset Profile / Usage / Operations privileges on Rest.li and OpenAPI as well as GraphQL. Dedicated timeseries APIs (getTimeseriesAspectValues, OpenAPI timeseries scroll, getTimeseriesStats with a URN filter) also require Get Timeseries Aspect API when REST API authorization is enabled. Action: grant those privileges (or rely on the default view-dataset-sensitive policy) to API clients that previously read these aspects with only entity GET. Chart usage statistics are unchanged.

• #19360 (GMS / Role and group membership writes) Writes to aspects that grant privileges are now authorized at the aspect layer across GraphQL, OpenAPI, and Rest.li. Adding a role (roleMembership) requires Manage Policies; adding a user to a group (groupMembership / nativeGroupMembership) requires Edit Group Members; adding a corpGroup owner (ownership) requires Edit Owners — or Manage Users & Groups when the actor is adding themselves. Only additions are checked; removals, unchanged re-ingestion, and system writes pass. Previously Edit Entity on a user was enough to grant that user the Admin role. Action: grant Manage Policies to automation that writes roleMembership outside the UI; membership-only sync (LDAP, Okta, Azure AD, datahub user upsert) needs Edit Group Members or Manage Users & Groups. Toggle via metadataChangeProposal.validation.aspectAuthorization.privilegeGrant.enabled (default true).

• #18740 (GMS / Documents) Document authorization is now enforced across GraphQL, OpenAPI, and Rest.li: CREATE accepts CREATE_ENTITY / EDIT_ENTITY / MANAGE_DOCUMENTS, UPDATE accepts EDIT_ENTITY / MANAGE_DOCUMENTS, DELETE accepts DELETE_ENTITY / MANAGE_DOCUMENTS. document is also added to VIEW_RESTRICTED_ENTITY_TYPES, so with VIEW_AUTHORIZATION_ENABLED=true documents are gated like other catalog entities. Action: grant the document privileges (or MANAGE_DOCUMENTS) as needed and ensure readers have View Entity / Get Entity.

Known Issues

Potential Downtime

Other Notable Changes

Security fixes:

• #19297 (Auth) The built-in self policy granted to every actor on their own entity was flattening the full ENTITY READ privilege map, which also handed out edit privileges on your own corpuser entity. It is now limited to VIEW_ENTITY_PAGE and GET_ENTITY.
• #19316 (Auth) POST /auth/signUp, /auth/resetNativeUserCredentials, /auth/verifyNativeUserCredentials, and /auth/getSsoSettings now require GMS system client credentials, matching /auth/generateSessionTokenForUser. A user session token calling these GMS helpers directly gets 401; frontend login, signup, password reset, and SSO are unaffected.
• #19384 (Auth) Aspect authorization no longer trusts client-supplied appSource system metadata to identify system writes.
• #19405 (Auth) Incidents raised on a schema field are now authorized against the parent entity.
• #19298, #19299, #19300 (UI) Stored XSS fixes: query and incident descriptions are sanitized (the legacy MarkdownViewer is removed), the documentation editor's PDF preview iframe src is sanitized, and dangerous renderUrl schemes on embeds are rejected.

Dependency CVE bumps:

• Logback 1.5.38 (CVE-2026-9828, CVE-2026-10532), Log4j 2.25.5 (CVE-2026-49844), Netty 4.2.17.Final (CVE-2026-59902), libthrift 0.23.0 (CVE-2026-43869), Jetty 12.1.10 (CVE-2026-10050), httpcore5 5.4.3 (CVE-2026-54399), micrometer-core 1.16.6 (CVE-2026-40983, CVE-2026-40984), reactor-netty-core 1.3.6, netty-reactive-streams 3.0.9.
• Apache Parquet 1.18.0, which refreshes its shaded Jackson to jackson-databind 2.22.1 (CVE-2026-54512, CVE-2026-54513).
• Python ingestion: nltk 3.10.3 (CVE-2026-12075).

v1.6.0.1

Patch release for v1.6.0 — security patches, bug fixes, and stability improvements. No new features; no schema or model changes.

Requirements:

• CLI / Python SDK: 1.6.0.17, 1.7.0.3
• Helm Chart: 1.1.2

Breaking Changes

• #17904 (Auth / Logical Parent) Linking a physical dataset to a logical parent now requires Edit Entity on both the child dataset and the proposed parent dataset. Clearing a logical parent requires Edit Entity on the child only. This runs as an AspectPayloadValidator across all APIs (GraphQL, OpenAPI, RestLI, ingestion MCPs). System-sourced writes are exempt. Action: Grant Edit Entity on logical-parent datasets to principals that create or change logical-parent links, or disable via metadataChangeProposal.validation.aspectAuthorization.logicalParent.enabled=false in Spring config.

• #17904 (Auth / Data Products) Modifying dataProductProperties.assets (adding or removing members from a data product) now requires authorization. The actor must satisfy either: MANAGE_DATA_PRODUCTS on at least one domain associated with the data product, or EDIT_ENTITY_DATA_PRODUCTS on every changed asset entity. EDIT_ENTITY satisfies both paths. This applies to UPSERT and PATCH operations. Action: Ensure service accounts have the required privileges, or disable via metadataChangeProposal.validation.aspectAuthorization.dataProductMembership.enabled=false in Spring config.

• #17904 (Auth / Query entities) When view authorization is enabled (VIEW_AUTHORIZATION_ENABLED=true, default false), reading a Query entity now requires VIEW_ENTITY_PAGE or EDIT_QUERIES (or EDIT_ENTITY) on every subject dataset of that query. Queries with no subjects are denied. Action: Only affects deployments with VIEW_AUTHORIZATION_ENABLED=true. Grant view or edit-queries privileges on subject datasets as needed.

• #18533 (GMS / Structured Properties) Creating a structured property is now rejected when its qualifiedName would collide with another property's Elasticsearch field name after . → _ normalization (e.g. certification.status vs certification_status). Additionally, string-backed structured property values (string, rich_text, date, urn) are now rejected when their UTF-8 encoding exceeds the keyword max length (default 32,766 bytes, configurable via STRUCTURED_PROPERTIES_KEYWORD_MAX_LENGTH). Previously oversized values were accepted by the primary store and failed silently at ES indexing time. Action: Choose qualifiedName values that remain distinct after . → _ normalization. Shorten text property values under the limit, or store large content as asset documentation.

• #18579 (GMS / PATCH validation) Six aspect validators now run on PATCH operations in addition to CREATE/UPSERT: StructuredPropertiesValidator, FieldPathValidator, FormPromptValidator, LifecycleStageValidator, PolicyFieldTypeValidator, and LogicalParentFieldPathValidator. PATCH validation only checks add/replace values — remove operations and already-stored values are not re-validated. There is no env var to disable PATCH validation specifically; individual validators can be disabled entirely via their @ConditionalOnProperty where available, but StructuredPropertiesValidator, FieldPathValidator, FormPromptValidator, LifecycleStageValidator, and PolicyFieldTypeValidator have no config toggle. Action: None for valid data; a PATCH that now fails was writing a value an equivalent upsert would already have rejected.

• #18798 (GMS / Ebean transaction retries) When a metadata write exhausts retries on a retryable database transaction conflict (SQLState 40001 / 40P01, or MySQL vendor code 1213), GMS now surfaces a structured conflict error instead of a generic 500. OpenAPI and Rest.li return HTTP 503 with body {"error": "...", "code": "DATABASE_TRANSACTION_CONFLICT", "retryable": true} and a Retry-After header (default 1 second). GraphQL returns error extensions {"code": 503, "type": "SERVICE_UNAVAILABLE"} (no Retry-After in GraphQL; HTTP status remains 200 per GraphQL convention). Non-deadlock retry exhaustion is unchanged (generic 500). PostgreSQL writes also acquire row locks in consistent primary-key order to prevent lock-order deadlocks. Action: Clients that previously retried or alerted on 500 for write failures should treat DATABASE_TRANSACTION_CONFLICT / GraphQL SERVICE_UNAVAILABLE as retryable. Configuration: EBEAN_MAX_TRANSACTION_RETRY (default null → 3 retries / 4 total attempts), EBEAN_RETRY_INITIAL_BACKOFF_MS (default 50), EBEAN_RETRY_MAX_BACKOFF_MS (default 1000), EBEAN_RETRY_AFTER_SECONDS (default 1), EBEAN_RETRY_BACKOFF_SQL_STATES (default 40001,40P01), EBEAN_RETRY_BACKOFF_VENDOR_CODES (default 1213). Opt-in EBEAN_ENTITY_WRITE_ADVISORY_LOCK_ENABLED (default false, PostgreSQL only) serializes concurrent writers to the same entity via pg_advisory_xact_lock.

• #18115 (System-update) RestoreColumnLineageIndices and RestoreDbtSiblingsIndices bootstrap steps are now disabled by default (previously always-on). These index-repair steps ran on every GMS startup; they now require RESTORE_COLUMN_LINEAGE_INDICES_ENABLED=true or SYSTEM_UPDATE_RESTORE_DBT_SIBLINGS_INDICES_ENABLED=true to run. Action: If you relied on these steps to repair lineage or dbt sibling index data on startup, set the corresponding env var to true.

Known Issues

Potential Downtime

• (Search / Structured Properties) With both ENABLE_STRUCTURED_PROPERTIES_SYSTEM_UPDATE=true and ENABLE_STRUCTURED_PROPERTIES_TYPE_MISMATCH_REINDEX=true (the latter defaults to true), system-update reindexes entity search indices when an existing structured-property field's Elasticsearch type differs from the definition-driven target (e.g. float vs intended double). Expect longer system-update runtime on the first upgrade if mismatched fields exist. Set ENABLE_STRUCTURED_PROPERTIES_TYPE_MISMATCH_REINDEX=false to skip. (#18698, #18768)

• #18531 (GMS / MySQL collation repair) SqlSetup now automatically detects and repairs non-binary collation on metadata_aspect_v2 urn and aspect columns. If your MySQL table uses a case-insensitive collation (e.g. utf8mb4_general_ci, utf8mb4_0900_ai_ci), startup runs ALTER TABLE metadata_aspect_v2 CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin, which rebuilds and locks the entire table. On large tables (millions of rows) this can take minutes to hours. There is no env var to skip this check. PostgreSQL is unaffected. Action: Before upgrading, check collation with SELECT COLUMN_NAME, COLLATION_NAME FROM information_schema.columns WHERE table_name='metadata_aspect_v2' AND column_name IN ('urn','aspect'). If already utf8mb4_bin, no action needed. If not, schedule the upgrade during a maintenance window.

Other Notable Changes

Security fixes:

• #18140 (Auth) DataHubOAuthAuthenticator no longer logs bearer tokens at DEBUG level. Tokens are now redacted in all log output.
• #17252 (Security) Profile image URLs (pictureLink) are now validated server-side. Only HTTPS URLs pointing to public hosts are accepted; javascript:, data:, file:, and private-network URLs are rejected. Configure via MCP_VALIDATION_URL_ENABLED (default true), MCP_VALIDATION_URL_ALLOW_HTTP (default false), and MCP_VALIDATION_URL_EXTRA_DENY_HOSTS.
• #18023 (GraphQL) Typed LoadableType resolvers now reject Restricted placeholder entities, preventing authorization bypass on restricted content.
• (Deps) wire-runtime 5.2.0 → 6.3.0 (CVE-2026-45799), cryptography >= 49.0.0 (CVE-2026-69249), lz4-java → 1.11.1 (CVE-2025-12183), jackson-databind → 2.21.5 (CVE-2026-54512/54513), PostgreSQL JDBC → 42.7.12 (CVE-2026-54291), Netty → 4.2.16.Final, Jetty → 12.1.10, spring-security (patch bump), commons-configuration2 (patch bump), Shiro → 2.2.1, PyJWT → 2.13.0 (GHSA-xgmm-8j9v-c9wx), redshift-connector >= 2.1.14 (CVE-2026-8838), idna >= 3.15 (CVE-2026-45409), pyasn1 and gitpython (patch bumps).

Search & indexing:

• #17716, #18842 (Search) Structured property filters now honor valueType and return complete result sets for URN-typed and number-typed properties.
• #18549, #18552, #18708, #18768 (Search) String-backed structured property keyword mappings now set a byte-safe ignore_above limit (default 8191 chars). Applied in-place without forcing a full reindex. Reverted on TEXT fields to avoid breaking text search. Values exceeding the limit remain in _source but are excluded from the keyword inverted index (not searchable/filterable) — this is graceful degradation replacing the previous behavior where oversized values caused the entire document to be dropped from the index.
• #18725 (Search) Invalid URN values are skipped with a warning during structured property string mapping instead of failing the entire mapping.
• #18276 (Search) allowedValues ordering on structured properties is preserved in search filter dropdowns.
• #17641, #17644 (Search) Fixed ES8 spurious reindex loops caused by mapping normalization differences, and legacy range queries in aggregations that failed under ES8's stricter parser.
• #17668 (Search) Search no longer truncates result pages when individual hits contain unparseable URNs.

GMS & backend:

• #18334, #18346 (Patch) Fixed ArrayMergingTemplate dropping the map key when a patch creates a new element through a deeper path — e.g. adding a field-level tag to a schema field with no existing editableSchemaMetadata entry produced an entry with no fieldPath (HTTP 422). Applies to tags, terms, ownership, and all keyed arrays.
• #18401 (GMS) Aspect keys are now sorted before FOR UPDATE to prevent lock-order deadlocks between concurrent multi-row writes and hard-deletes on MySQL.
• #17415, #17604 (GMS) ownerTypes are now materialized from the owners list on both CREATE_ENTITY and regular upsert writes.
• #17959 (GMS) Hard-deleting an entity no longer throws a NullPointerException when the version range for an aspect is empty.
• #17732 (GMS) hasChildDomains() now cross-checks OpenSearch results against MySQL before blocking parent deletion, preventing stale search index results from blocking valid deletes.
• #17634 (GMS) getMergedEdges now keys by the full Edge object instead of hashCode, preventing "Duplicate key" exceptions on hash collisions.
• #17449, #17905 (Patch) PATCH operations with JSON Pointer paths ending in / no longer throw JsonException when the targeted aspect doesn't exist.

Auth & policies:

• #18011 (Auth) REST API tag writes now accept the EDIT_ENTITY_TAGS privilege as intended.
• #17977 (Auth) Authorization checks for data product mutations now read domain info directly instead of through the entity client cache.
• #18117 (Auth) The policy list endpoint no longer throws on policies with unrecognized types or states.
• #18119 (Auth) The role policies list API now filters out non-policy entities.
• #18446 (GraphQL) Data products with no associated domain can now be deleted.

Frontend:

• #18481 (UI) The Data Contract tab now displays custom assertion names when set.
• #18012 (UI) Field-level profile data now matches correctly when BigQuery returns lowercased column paths.
• #17757 (UI) Fixed a regression that caused timezone data to be unavailable in date/time pickers.
• #18394 (UI) Documents marked as hidden no longer inflate search result counts.

Upgrade & system-update:

• #18598 (Upgrade) The reindex alias swap now retries on doc-count mismatch instead of failing immediately.
• #18308 (Upgrade) Consistency check IDs are now filtered per entity type before checkBatch.
• #18384, #18535 (ZDU) Zero-downtime upgrade orphan cleanup no longer deletes the semantic search index; missing-index case handled gracefully.
• #18204 (GMS) Dashboard usage statistics now correctly set per-user view counts.
• #18307, #18043 (Consumer) MCE consumer commitAndContinue catches RejectedExecutionException during shutdown; OpenTelemetry span export switched to non-blocking.

v1.6.0

Requirements:

• CLI / Python SDK: 1.6.0
• Helm Chart: 1.0.3

ZDU (zero-downtime upgrade): v1.6.0 is a must-install release before enabling Elasticsearch/OpenSearch ZDU. Deploy this version with Helm chart 1.0.3 and let system-update complete. Enable global.datahub.systemUpdate.zdu later with Helm chart 1.1.0 (for example when upgrading to v1.7.0) for a subsequent OpenSearch/Elasticsearch version bump. This release ships the server-side ZDU infrastructure, schema version index, aspect migration sweep safeguards, and the Helm values foundation required for staged index upgrades.

Breaking Changes

• #16816 / #17351: (GMS / Java services) Spring Boot upgraded from 3.5.6 to 4.0.5 (later patch to 4.0.6 for CVE fixes), including Spring Framework 7.0 and Spring Kafka 4.0. Official Docker images and Helm deployments are updated together. Action: If you ship custom GMS plugins, Spring @Configuration extensions, or Kafka listener customizations against DataHub's classpath, recompile and retest against Spring Boot 4 before upgrading production.

• #16625 / #16680: (Frontend) The V1 UI has been fully removed from the codebase (not merely deprecated). THEME_V2_ENABLED, THEME_V2_DEFAULT, and related env vars are required; there is no fallback to V1. Helm users should set datahub-gms.theme_v2.enabled and datahub-gms.theme_v2.default to true and datahub-gms.theme_v2.toggleable to false.

• #17214: (Frontend / Play 3 + Apache Pekko) datahub-frontend is upgraded from Play 2 / Akka to Play 3 and Apache Pekko. Session and flash cookies use Play 3's JWT-based signing; error responses from the frontend proxy use a custom JSON HttpErrorHandler instead of Play's default HTML pages. See the DATAHUB_SECRET requirement below.

• #16982: (Auth) The deprecated corpUserInfo.active field is no longer considered for session eligibility. Users gated on active for login must migrate to supported CorpUser status mechanisms.

• #16887: (Operations / Elasticsearch) Optional Elasticsearch side zero-downtime upgrade (ZDU) for OpenSearch/Elasticsearch version bumps. v1.6.0 is a must-install prerequisite — deploy it with Helm chart 1.0.3 and complete system-update before enabling ZDU on a later upgrade. When enabled via Helm (global.datahub.systemUpdate.zdu), system-update runs staged index upgrades with reduced downtime; scale-down during system-update is disabled automatically when ZDU is enabled. Enable ZDU with Helm chart 1.1.0 (for example on the v1.7.0 upgrade path). See chart values for preEnable / enable and index upgrade tuning (global.elasticsearch.index.upgrade, global.elasticsearch.buildIndices).

• #16930: (Operations / Upgrades) Background aspect schema version migration sweep runs during upgrades on large deployments. The sweep migrates stale aspect schema versions and avoids overwriting aspects that were updated concurrently during the sweep window.

• #17493: (Operations / Retention) Retention policies from boot/retention.yaml are re-applied during system-update when retention is configured, so policy changes shipped with an upgrade take effect without a separate manual ingest step.

• Structured properties write validation (GMS): Upserts, patches, and MCP writes to the structuredProperties aspect are now handled in GMS by StructuredPropertiesValidator (at validateProposed) and StructuredPropertiesAssignmentMutator (before commit), for all APIs (GraphQL, OpenAPI, ingestion). With the default STRUCTURED_PROPERTIES_DROP_MISSING_PROPERTY_VALUES_WITH_WARNING=true, assignments whose property definition entity is missing or has no propertyDefinition aspect are removed from the proposal (server WARN per dropped URN) instead of failing the entire write at definition lookup. If the proposal had assignments but none remain valid after filtering, the write is rejected. Clearing structured properties (empty aspect) is unchanged. Action: Pipelines that must fail when any orphaned assignment is present should set STRUCTURED_PROPERTIES_DROP_MISSING_PROPERTY_VALUES_WITH_WARNING=false on GMS and restart; otherwise expect stale property URNs to be stripped on the next write. See Structured Properties — orphaned assignments and STRUCTURED_PROPERTIES_DROP_MISSING_PROPERTY_VALUES_WITH_WARNING.

• #17465 The default ingestion profiler for SQL connectors is now sqlalchemy instead of ge (Great Expectations). The SQLAlchemy profiler provides equivalent functionality without requiring the Great Expectations library. To keep using the Great Expectations profiler, install the new optional extra and set the method explicitly in your recipe:

  pip install 'acryl-datahub[profiling-ge]'

  source:
    config:
      profiling:
        enabled: true
        method: ge

  Note: acryl-datahub[snowflake] (and other SQL connector extras) no longer install acryl-great-expectations by default. Recipes that previously relied on the GE default profiler will now use the SQLAlchemy profiler automatically (no action needed in most cases). Recipes that explicitly set profiling.method: ge without installing [profiling-ge] will fail with a ConfigurationError pointing at the fix.

  Deprecation notice: The Great Expectations profiler is now considered legacy and is planned for removal in a future release. New deployments should rely on the default SQLAlchemy profiler. Existing users that still depend on method: ge should plan to migrate.

• #17563 Unity Catalog profiling now defaults to method: sqlalchemy instead of method: ge. This aligns Unity Catalog with all other SQL connectors after #17465 flipped the global default. If your Unity Catalog recipe enables profiling and relied on the Great Expectations profiler, add the following to your recipe and install the extra:

  pip install 'acryl-datahub[unity-catalog,profiling-ge]'

  source:
    config:
      profiling:
        enabled: true
        method: ge

• Search filters (REST, GraphQL, SDK): The Pegasus Criterion record and GraphQL FacetFilterInput no longer expose a singular value field. Send match values only via values (a string array; use a one-element array where you previously passed a single string). Clients that still serialize value must migrate; server-side auto-conversion and related logging have been removed.

• #16842: (Ingestion / Athena) Upstream lineage URNs changed for Glue-backed catalogs. Athena tables backed by the AWS Glue Data Catalog (i.e. using awsdatacatalog or any catalog whose type resolves to GLUE) now emit upstream lineage to Glue dataset URNs (urn:li:dataPlatform:glue) instead of S3 URNs (urn:li:dataPlatform:s3). Iceberg tables in non-Glue catalogs (e.g. Hive Metastore, Lambda, Federated) now emit upstream lineage to Iceberg dataset URNs (urn:li:dataPlatform:iceberg). If your upstream Glue or Iceberg connector is configured with a non-default platform_instance, set the new glue_platform_instance / iceberg_platform_instance options on the Athena source so the upstream URNs match. If you have downstream dashboards, saved searches, or lineage queries that reference the old S3 upstream URNs for Athena tables, update them after re-running ingestion.

• (Docker / Gradle) Java service images (datahub-gms, datahub-mce-consumer, datahub-mae-consumer, datahub-upgrade, datahub-frontend-react) use a default base image and apk-style packages at build time. Override the base with Docker BASE_IMAGE / Gradle -PdockerBaseImage=..., and the apk repository line with APK_REPOSITORY_URL / -PapkRepositoryUrl=.... If you previously passed a legacy Gradle property for a custom package mirror, migrate to apkRepositoryUrl (and ensure the mirror layout matches your chosen base image). datahub-actions uses the same BASE_IMAGE build arg.

• #17407: BigQuery extract_policy_tags_from_catalog: The policy tag extraction feature has been rewritten to use INFORMATION_SCHEMA.COLUMN_FIELD_PATHS combined with batch Data Catalog API calls instead of per-column get_table() and get_policy_tag() calls. This eliminates millions of redundant API calls (e.g. 434k → ~10 for a 29k-table warehouse), reducing extraction time from hours to seconds. The old code path has been removed with no fallback. Action required if you use extract_policy_tags_from_catalog: true: (1) Verify BigQuery API v2 is available in your environment (available since ~2020). (2) If the Data Catalog API is blocked by VPC Service Controls, policy tag resource names will be stored instead of display names — this is graceful degradation, not a failure. (3) Review ingestion logs for any new warnings about missing policy_tags column in INFORMATION_SCHEMA results.

• Bootstrap Steps Moved to System-Update Job. The following metadata ingestion steps now run during system-update instead of GMS startup:

  • Entity Types ingestion

  On the first deploy after this change, these entities will be re-ingested across the cluster. Per-entity existence checks prevent duplicate data writes.

• #16473: Vertex AI Source - ML Model version set URNs now scoped per project: Previously, all Vertex AI model versions shared a version set based solely on the numeric model ID, which caused conflicts when multiple GCP projects contained models with the same numeric ID (GCP model IDs are project-local, not globally unique). Version sets are now scoped per project, preventing 422 ValidationException errors during ingestion. After upgrading, existing model version entities will be orphaned in their old version sets; new ingestion runs will create correctly scoped version sets. Old orphaned entities can be cleaned up by enabling stateful ingestion with stale entity removal.

• #17196: (Ingestion / Sigma) Chart InputFields now emit resolved upstream fields only. Previously, Sigma charts emitted self-referential InputFields entries for every chart column, which the DataHub UI ignored as dead edges. Sigma now derives chart field lineage from column formulas and emits only fields that resolve to an upstream workbook element or warehouse table. Downstream consumers that read raw InputFields aspects may see self-referential entries disappear, especially when extract_lineage is disabled, formulas are unavailable, or chart columns are formula-less direct passthroughs; those charts can now emit fields: [] instead of one self-reference per chart column.

• #17296: (Ingestion / Sigma) New connection_to_platform_map config for per-connection warehouse overrides. Sigma recipes now support a connection_to_platform_map field keyed by Sigma connectionId (UUID from /v2/connections). Use it to set env, platform_instance, convert_urns_to_lowercase, default_database, and default_schema per connection so emitted lineage URNs match your warehouse connector. This is required for Redshift (and other warehouses where Sigma's /v2/connections API returns null for database/schema): without default_database/default_schema overrides, the SQL parser produces under-qualified URNs that will not match your warehouse connector's datasets. These overrides apply to all lineage paths that have a connectionId on hand — DM element → warehouse table, DM customSQL, and workbook customSQL. No action required if you use Snowflake or BigQuery with default env/platform_instance settings.

• #17347: (Ingestion / Sigma) Workbook customSQL chart lineage is now emitted automatically. When extract_lineage: true (default), charts whose data source is a customSQL definition will emit warehouse UpstreamLineage and column-level FineGrainedLineage on the next ingestion run. No configuration change is required for Snowflake and BigQuery. For Redshift, set default_database/default_schema in connection_to_platform_map (see #17296 above).

• #17370: (Ingestion / Sigma) Two bug fixes with operator-visible side effects for warehouse-backed chart column lineage and DM element schema. When extract_lineage: true (default): (1) Charts whose column IDs carry warehouse-native names were incorrectly emitting Sigma display names as InputFields.fields[].schemaField.fieldPath (e.g. Customer Id instead of customer_id). This is now fixed. Downstream consumers that matched on the display-name spelling will need to update those references after the next ingestion run. (2) A bug in DM element schema extraction was silently suppressing valid columns when sibling DM elements shared the same columnId. This is now fixed. Affected DM element Datasets may gain previously suppressed schema columns on the next ingestion run; column-count alerts and soft-delete policies scoped to those Datasets may fire. No configuration change is required.

• #17369: (Ingestion / Sigma) Workbook charts that pull directly from warehouse tables now emit entity-level ChartInfo.inputs upstreams. When extract_lineage: true (default), Sigma's BFS node type=table upstreams (direct warehouse tables, not Sigma Datasets) are now resolved to warehouse Dataset URNs and added to chartInfo.inputs. This is a visible behavior change: on the next ingestion run, charts that previously had empty inputs will now show warehouse table lineage edges in the DataHub UI if the workbook contains direct table references. For Snowflake and BigQuery, no configuration change is required. For Redshift, set default_database and default_schema in connection_to_platform_map (see #17296) to ensure URNs match your warehouse connector. Additionally, the 2-segment /files path (e.g. Redshift Connection Root/<SCHEMA>) is now accepted in the workbook warehouse table index; previously only 3-segment paths (e.g. Snowflake Connection Root/<DB>/<SCHEMA>) were accepted. The counter chart_warehouse_unknown_connection has been renamed to chart_warehouse_table_name_unmatched to reflect its actual semantics (workbook table short name not found in the index, not a connection error).

• #17276: (Ingestion / Sigma) Sigma Data Model ingestion is now on by default. ingest_data_models defaults to true. On the next ingestion run, new Data Model Containers and element Datasets will appear in DataHub for any Sigma tenant that has Data Models. Workbook charts that reference a DM element will also gain new ChartInfo.inputs entries. Action required to opt out: set ingest_data_models: false in your Sigma recipe. If you keep it enabled, update any soft-delete policy to account for the new Container and Dataset entities, and silence or raise fan-in cardinality alerts before the first run.

• (Ingestion / Sigma — Action required) Null DatasetUpstream.name now tolerated. Previously raised a Pydantic ValidationError warning; now counted under chart_dataset_upstream_name_missing with a SourceReport.warning titled Sigma workbook dataset upstream dropped (name missing). Lineage semantics unchanged. Action: migrate any alert keyed on the old Pydantic warning text to the new counter or warning title.

• (Actions / Kafka) The default Kafka offset commit strategy for Actions has changed from synchronous (async_commit_enabled: false) to asynchronous (async_commit_enabled: true). With asynchronous commits, offsets are stored locally after each event and committed periodically by a background thread (default interval: 10 seconds via async_commit_interval). This provides up to 25x throughput improvement for high-volume action pipelines. The tradeoff: on consumer crash, up to async_commit_interval milliseconds of events may be redelivered. All built-in actions are idempotent or tolerate redelivery, so no behavior change is expected for standard deployments. To restore the previous synchronous default for any action, add async_commit_enabled: false to the source.config section of your action YAML.

• Docker / Prometheus (Micrometer): When using the stock DataHub Docker start.sh entrypoints, GMS, MAE consumer, MCE consumer, and the frontend expose Micrometer Actuator (including GET /actuator/prometheus and /actuator/health) on container port 4319 by default (MANAGEMENT_SERVER_PORT), separate from the main API/UI port. Profile compose (docker/profiles/docker-compose.gms.yml) exposes 4319 for on-network scraping and maps ${DATAHUB_MAPPED_GMS_MANAGEMENT_PORT:-4319}:4319 on the host for GMS (same pattern as ${DATAHUB_MAPPED_GMS_PORT:-8080}:8080 for HTTP). Override DATAHUB_GMS_MANAGEMENT_URL in smoke tests if needed, or change the host port via DATAHUB_MAPPED_GMS_MANAGEMENT_PORT. Scrape from another container on the same compose network (e.g. http://datahub-gms:4319/actuator/prometheus as in docker/monitoring/prometheus.yaml). MAE and MCE image HEALTHCHECK probes the management port first, then falls back to the main consumer port for older layouts. The JMX Prometheus Java agent remains on 4318. If you previously scraped Micrometer from the GMS HTTP port (e.g. 8080), update scrapes to the management listener. To keep Actuator on the main port, unset MANAGEMENT_SERVER_PORT in the container or set it equal to the main server port.

• (Operations / monitoring) Docker images that bundle the JMX Prometheus Java agent (datahub-gms, datahub-frontend-react, datahub-mae-consumer, datahub-mce-consumer, datahub-upgrade) now ship jmx_prometheus_javaagent 1.0.1 (previously 0.20.0). The agent uses Prometheus client_java 1.x; upgrade scrapers and dashboards accordingly.

  • HTTP scrape path: Metrics are exposed at /metrics, not at /. If you scrape the JMX port directly (often 4318 when Prometheus export is enabled), set your Prometheus metrics_path (or Kubernetes ServiceMonitor / PodMonitor path) to /metrics. Anything still requesting / will receive the default HTML page, not the metrics exposition.
  • JVM metrics: Some built-in JVM metric names changed to align with OpenMetrics (for example, memory-related series). Update Grafana panels, recording rules, and alerts that referenced the old names. See the client_java JVM migration notes.
  • Labels: When multiple MBeans map to the same metric name, series may include an _objectname label; adjust queries or aggregations if you previously assumed a single series per name.

• #16723 (Ingestion) Dataplex source configuration cleanup: filter_config.entries.dataset_pattern was removed, use filter_config.entries.pattern instead; entries_location was removed, use entries_locations (list) instead.

• #16889 (Ingestion) SQL parsing: Fixed a bug where table references with 5 or more path components (e.g. catalog.namespace.subspace.schema.table) silently dropped middle components when resolved through the sqlglot-based parser. The previous (buggy) behavior produced truncated dataset URNs like urn:li:dataset:(urn:li:dataPlatform:dremio,catalog.namespace.schema.table,PROD); after this fix, the correct URN urn:li:dataset:(urn:li:dataPlatform:dremio,catalog.namespace.subspace.schema.table,PROD) is produced. This primarily affects connectors that ingest deep folder hierarchies (most commonly Dremio, but also any source resolving SQL through the shared parser). After upgrading, re-ingestion will create new entities under the corrected URNs, and the previously-ingested truncated entities will be orphaned along with any tags, owners, glossary terms, or lineage attached to them. To migrate, either (a) enable stateful ingestion with stale entity removal so the next run cleans up the old URNs, or (b) manually delete the old URNs (e.g. via datahub delete --urn ...) before or after re-ingestion.

• (Ingestion / dbt) Assertion result mapping for dbt tests is more faithful to dbt's status semantics:

  • dbt test results with status error or runtime error (the test itself could not run due to a compilation, SQL, or infrastructure problem) are now emitted with AssertionResult.type = ERROR instead of FAILURE. Previously these were conflated with real data-quality failures.
  • dbt source freshness results with status runtime error are similarly emitted with AssertionResult.type = ERROR. Freshness status error (meaning the error_after threshold was exceeded) continues to map to FAILURE since it represents a legitimate verdict.
  • A new optional AssertionResult.severity field is populated on failure results: LOW for dbt warn (only emitted as FAILURE when test_warnings_are_errors: true), HIGH for fail / freshness error.
  • Downstream consumers (alerting, dashboards, saved searches) that filter assertion runs by result.type == FAILURE will no longer see infrastructure-level test failures in that set; include type == ERROR if you want to continue capturing them.

• #16912: (Build / java 25) DataHub now compiles with java 25 JDK and produces Java 25 bytecode for the main artifacts; the spark-lineage agent is the exception, targeting Java 17 bytecode for broader Spark runtime compatibility. Self-hosted builds require java 25+.

  • Spark integration upgraded from 3.3.4 to 3.5.0 (enables Spark 4.x forward compatibility and improves lineage handling for complex query plans)
  • Hadoop upgraded from 2.7.2 to 3.3.6 (addresses Hadoop CVEs, bundled with Spark 3.5+)
  • For self-hosted deployments: Build/compile requires java 25 JDK. Runtime environments can use Java 8+ (DataHub produces Java 8 bytecode for compatibility). Spark lineage users must upgrade to Spark 3.5.0+ if using DataHub's Spark integration.
  • java 25 runtime configuration: When running DataHub services with java 25, add --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED to JAVA_OPTS or your JVM startup command to allow Spark lineage listener reflection-based operations. DataHub Docker images include this automatically.

• #17340: (Docker Runtime / Java 25 LTS) Docker images (datahub-gms, datahub-mce-consumer, datahub-mae-consumer, datahub-upgrade, datahub-frontend-react, datahub-actions, and ingestion images) now use Java 25 LTS runtime while maintaining java 25 bytecode compilation for maximum compatibility. This enables JEP 491 (virtual thread pinning fix in synchronized blocks) and other Java 25 runtime benefits without breaking build toolchain compatibility. Self-hosted users: no action required if using official Docker images. Build system remains on java 25 (Gradle 9.5.0 compatibility); custom deployments compiling from source should follow AGENTS.md for Java version strategy.

• (Frontend / Play 3 — DATAHUB_SECRET) With the Play 3 upgrade (#17214), DATAHUB_SECRET is wired to play.http.secret.key. Play 3 rejects application secrets that are unset, left as the default changeme, or shorter than the minimum required for session/flash JWT signing (default algorithm HS256 needs at least 32 bytes of key material; see Play Application secret). If DATAHUB_SECRET is too short, datahub-frontend will fail on startup. Before upgrading, set DATAHUB_SECRET to a random value of at least 32 characters (for example openssl rand -base64 32 or head -c 32 /dev/urandom | base64). Helm and other production layouts typically already generate a long secret; this mainly affects ad hoc Docker Compose, copied env files, or hand-crafted short values.

• (Ingestion / Fivetran — Lineage URN shift for multi-destination accounts). Per-destination URN routing now reflects each destination's actual platform instead of defaulting every destination to fivetran_log_config.destination_platform. Previous behavior: in hybrid mode (when both fivetran_log_config and api_config are configured), every connector's destination URN was emitted with the log warehouse's platform — e.g. if the Fivetran log lives in BigQuery, all destinations were urn:li:dataset:(bigquery, ...) regardless of where the data actually went. New behavior: the connector calls GET /v1/destinations/{id} per destination and emits a URN matching the actual service. Action required if you are running Fivetran in hybrid mode (fivetran_log_config + api_config) AND your account has connectors writing to multiple destination types (e.g. some to Snowflake, some to BigQuery, some to a Managed Data Lake): existing lineage URNs will shift to the correct platform/database on the next ingest, which may break dashboards, queries, or saved searches keyed on the old (incorrectly-conflated) URNs. Audit affected URNs against your Fivetran destinations console before upgrading. To opt out and preserve the old behavior, pin each non-default destination explicitly via destination_to_platform_instance.<destination_id>.platform (and database for relational platforms) — declarative overrides always win over discovery. Single-destination accounts and recipes that already pin every destination explicitly are unaffected.

• #17466: (Ingestion / two-tier SQL sources — stored procedure ingestion overhaul.) Two changes to two-tier SQL connectors (MySQL, MariaDB, Hive, Clickhouse, Teradata, and any other source that extends TwoTierSQLAlchemySource):

  1. URN shape changed (breaking). Stored-procedure DataFlow and DataJob URNs no longer duplicate the database name. The flow name now resolves to {database}.stored_procedures instead of {database}.{database}.stored_procedures. For example, a MariaDB procedure in test_db previously emitted urn:li:dataFlow:(mariadb,test_db.test_db.stored_procedures,PROD) and now emits urn:li:dataFlow:(mariadb,test_db.stored_procedures,PROD); the corresponding dataJob URNs shift accordingly.
  2. Stored-procedure lineage now actually runs for MySQL/MariaDB (additive). Native SQL procedures in MySQL/MariaDB store NULL in information_schema.ROUTINES.EXTERNAL_LANGUAGE, which silently skipped lineage extraction. With this release, language defaults to SQL when unset and the parser runs over the procedure body. Existing MySQL/MariaDB procedures will start emitting dataJobInputOutput aspects with inputDatasets, outputDatasets, and fineGrainedLineages on the next ingestion run. No config change required.

  What you need to do: after upgrading, re-run ingestion. Existing DataFlow / DataJob entities under the old (duplicated) URNs will be orphaned; if you have stateful ingestion with stale-entity removal enabled, the next run will soft-delete them. Otherwise, clean them up with the GraphQL removeEntities mutation or the CLI. Any dashboards, saved searches, alerts, or downstream tooling that reference the old *.{database}.{database}.stored_procedures.* URNs need to be updated to the new format. Three-tier sources (Postgres, Oracle, MSSQL, DB2, Snowflake, …) are unaffected by the URN change; they already gated procedure language correctly so the lineage change is also a no-op for them.

• #16873 (Ingestion) Unity Catalog (Databricks) connector now emits ownership and datasetProperties as UPSERT (full replace) by default, instead of incremental PATCH. This means each ingestion run will re-state the complete ownership and properties for each dataset. If you rely on the previous additive/merge behavior (e.g., manually added owners surviving ingestion runs), set incremental_ownership: true and/or incremental_properties: true in your recipe config.

Known Issues

• (Operations / Elasticsearch 8) Spurious mappings reindex on every Helm upgrade. Deployments backed by Elasticsearch 8.x may trigger a full index mappings reindex on every system-update run (including each Helm upgrade), even when no authored mapping change was shipped. ES8 persists mappings differently than DataHub's authored definitions (for example injecting type: object on nested object fields), so the index builder detects spurious drift and schedules reindex work repeatedly. This is most likely when ELASTICSEARCH_INDEX_BUILDER_MAPPINGS_REINDEX=true (including Docker quickstart, which enables it for initial index builds). Workaround: Let the first system-update after install or upgrade finish so indices are created, then set ELASTICSEARCH_INDEX_BUILDER_MAPPINGS_REINDEX=false on datahub-gms and datahub-upgrade before later Helm upgrades until a fixed release is available.

Potential Downtime

• System-update / Elasticsearch: Upgrades that trigger reindexing, incremental index migration, or optional Elasticsearch ZDU (#16887) can run for an extended period depending on catalog size. Reindex duration was improved (#16949). Elasticsearch 8 deployments with mappings reindex enabled may repeat unnecessary reindex work on every Helm upgrade — see Known Issues above. Helm upgrades without --atomic may report failure while background reindex jobs continue — allow system-update to finish before retrying. See Helm Notes under the 0.10.0 section below for timeout guidance patterns.
• First deploy after bootstrap moves: Entity Types ingestion moved to system-update (see Breaking Changes). Expect additional system-update work on the first upgrade after this release.
• Aspect schema version sweep (#16930): Large deployments may observe background migration activity during upgrade; plan maintenance windows accordingly.

Deprecations

• (Build / Gradle) Legacy Gradle properties for overriding artifact download URLs are deprecated in favour of the datahub.dependencies.* namespace. Old properties are still honoured but will be removed in a future release. See Custom Repositories.

  Legacy property	Replacement
  apacheMavenRepositoryUrl, mavenCentralRepositoryUrl	datahub.dependencies.maven.central
  confluentMavenRepositoryUrl	datahub.dependencies.maven.confluent
  linkedinOpenSourceRepositoryUrl	datahub.dependencies.maven.linkedinOpenSource
  nodeDistBaseUrl	datahub.dependencies.node.distBaseUrl
  githubMirrorUrl	datahub.dependencies.github.baseURL
  pipMirrorUrl	datahub.dependencies.python.pipMirrorUrl
  pipExtraIndexUrls	datahub.dependencies.python.pipExtraIndexUrl

• (Operations / Helm) Per-workload monitoring configuration is deprecated in favor of cluster-wide settings (global.datahub.monitoring).

• (Operations / Helm) Enable global.datahub.systemUpdate.consolidatedUpgrade so upgrades run through the consolidated system-update path and the chart no longer relies on separate one-off setup jobs (e.g. mysql/ES setup jobs) for that flow.

• (Ingestion / dbt) The test_warnings_are_errors default will change from false to true in a future release. Once assertion result consumers can filter by severity, warn test statuses will always emit as FAILURE with severity: LOW, and the flag itself will be deprecated. To adopt the forthcoming behavior today, set test_warnings_are_errors: true in your dbt recipe.

• (Automations / Glossary Term AI) The Glossary Term AI automation has been deprecated. This capability will be replaced in an upcoming release.

Other Notable Changes

• (Ingestion / Snowflake) Internal Marketplace support: The Snowflake connector now ingests Snowflake Internal Marketplace (private data exchange) listings as DataHub Data Products. Enable with marketplace.enabled: true; choose direction via marketplace.marketplace_mode: consumer | provider | both (default consumer). Each listing becomes a Data Product entity with assets, ownership, structured properties, and optional usage statistics from SNOWFLAKE.DATA_SHARING_USAGE.LISTING_TELEMETRY_DAILY. Requires the SHOW AVAILABLE LISTINGS privilege (and IMPORTED PRIVILEGES ON DATABASE snowflake granted to the user for provider/both).
• #17450 (Ingestion / Snowflake) Opt-in structured_properties_write_mode: patch on the Snowflake source preserves UI- and pipeline-added structured properties across ingestion runs. Default remains upsert (recipe-as-source-of-truth, current behaviour). When set to patch, properties removed from the recipe no longer propagate to DataHub — clean those up via the UI or API. The helper add_structured_properties_to_entity_wu gains an opt-in write_mode parameter; all existing callers (third-party connectors included) keep their current UPSERT behaviour. Requires the server-side TemplateUtil fix in #17449 to be deployed for patch mode to work end-to-end.
• #17449 (GMS) Fix server-side handling of PATCH ops whose JSON Pointer path ends in / (e.g. /tags/<urn>/). Parsson previously threw JsonException on these paths when the targeted aspect did not yet exist, breaking unattributed addTag/addTerm/addOwner/setNumberProperty on fresh entities. No wire-format change and no client-side action required.
• (Security / CWE-200) Error responses from GMS (GraphQL, OpenAPI) and the Play frontend proxy no longer leak internal Java class names, Jackson source locations, or stack traces. Previously, a malformed JSON request to /api/v2/graphql could return detailed internal information such as com.fasterxml.jackson.core.JsonParseException and StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION in the HTTP response body. All error responses now return sanitized messages (e.g. "Invalid JSON format", "Malformed request body", "Bad Request"). The frontend now uses a custom HttpErrorHandler that returns JSON instead of Play's default HTML error pages. Configuration: The frontend error handler is enabled automatically via play.http.errorHandler in application.conf. GMS error sanitization is applied in GlobalControllerExceptionHandler and the global ObjectMapper explicitly disables StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION for defense in depth. No operator action is required; internal exception details remain in server-side logs for debugging.
• #17086 (Ingestion / Sigma) Intra-workbook element-to-element lineage is now emitted via ChartInfo.inputEdges. On the next ingestion run, new chart→chart edges will appear in the lineage graph for any Sigma workbook where one element feeds another. This is additive — no existing dataset or warehouse lineage edges are removed.
• #16358 (Ingestion / dbt) dbt URN lowercasing is now configurable via convert_urns_to_lowercase. Previously, dbt unconditionally lowercased all target-platform URNs with no way to disable it. This caused lineage breaks with case-sensitive platforms like BigQuery when table names contained uppercase characters (e.g. AM100 lowercased to am100). The default is true (preserving existing behavior). Users on case-sensitive platforms like BigQuery should set convert_urns_to_lowercase: false in their dbt recipe to preserve original identifier casing.
• (Frontend) The OIDC post-login redirect cookie (REDIRECT_URL) format has changed. It now stores a plain Base64-encoded URL string instead of the previous serialized format. Any in-flight REDIRECT_URL cookies set before the upgrade will fail to decode; affected users will be redirected to the DataHub home page (/) after their next SSO login instead of the page they originally requested. No action is required — users simply log in again and the new cookie format takes effect automatically.
• (Operations / Helm) Added global.datahub.monitoring.metricsMode with three modes: legacy (default), jmx_and_actuator, and actuator_only, so JMX vs Spring Boot Actuator scraping can be chosen cluster-wide. See the Micrometer transition plan.
• #16619 (Operations / Helm) Added a Cleanup upgrade that tears down all DataHub-owned infrastructure resources (Elasticsearch indices, Kafka topics, SQL database and users). It is designed to run as a Helm pre-delete hook so that helm uninstall leaves no DataHub-specific state on shared infrastructure. Each component (ES, Kafka, SQL) can be disabled independently via environment variables (CLEANUP_ELASTICSEARCH_ENABLED, CLEANUP_KAFKA_ENABLED, CLEANUP_SQL_ENABLED; all default to true). Elasticsearch cleanup uses scoped IndexConvention patterns to enumerate only DataHub-owned indices — it does not issue a wildcard DELETE /* that would be dangerous on shared clusters without an index prefix configured.
• #16879 (Ingestion) PowerBI: When convert_lineage_urns_to_lowercase is enabled, column-level lineage and upstream dataset URNs are now consistently lowercased. Previously, only parts of the URN were lowercased, which could cause lineage mismatches. After upgrading, re-running ingestion will emit corrected URNs.
• #17033 (Ingestion / MongoDB) system.* collections (e.g. system.profile, system.views) are now excluded from ingestion by default via the new excludeSystemCollections option (default: true). If you were relying on ingesting system collections, set excludeSystemCollections: false in your recipe. The read role is now sufficient for all standard ingestion use cases; dbAdmin is only required if excludeSystemCollections is disabled and MongoDB profiling is enabled.
• #16871 (Ingestion) Postgres stored procedures: The dataTransformLogic.queryStatement.value for Postgres stored procedures now contains only the SQL body (from pg_proc.prosrc) instead of the full CREATE OR REPLACE PROCEDURE ... AS $$ ... $$ wrapper previously returned by pg_get_functiondef(). This change enables lineage extraction (the parser previously skipped the CREATE wrapper), but may affect custom tooling that parsed the full DDL from this field. Additionally, the language field is now normalized to uppercase (e.g. "SQL" instead of "sql").
• (Security / GMS) Profile image URLs (pictureLink) on user and group profiles are now validated server-side. Only HTTPS URLs pointing to public (non-internal) hosts are accepted. Existing stored URLs are unaffected, but any new profile image update via the UI, GraphQL, or REST API that uses http://, javascript:, data:, file:, or points to a private/internal network address (localhost, 127.0.0.1, 169.254.169.254, RFC 1918 ranges) will be rejected. If you rely on HTTP image URLs for profile pictures (e.g. from an internal image server), those URLs must be migrated to HTTPS before upgrading. The default avatar relative path (assets/…) continues to work.
• #17271 (Ingestion / ClickHouse) New opt-in column profiling backend (profiling.method: sqlalchemy). The default Great Expectations profiler issues CREATE TEMPORARY TABLE ... AS SELECT (when custom_sql, limit, or offset is set), but ClickHouse temporary tables are session-scoped, so the follow-up read on a different pooled connection raises UNKNOWN_TABLE. Set profiling: { method: sqlalchemy } in your ClickHouse recipe to route through a SQLAlchemy-based profiler that never creates temp tables and uses ClickHouse-native aggregates (uniq(), quantile(), stddevSamp()). The default is unchanged (ge); existing recipes are not affected. Note: custom_sql, limit, and offset are not supported on the SQLAlchemy path — a warning is emitted and full-table profiling is used instead.
• #17191 (Ingestion / Airflow plugin) OpenLineage Dataset names that contain a literal "None" segment — typically produced by Apache Airflow providers that build dataset names with f-strings like f"{database}.{self.schema}.{self.table}" without a None guard (notably S3ToRedshiftOperator when schema is unset) — are now sanitized before the DataHub URN is constructed. Lineage edges from these operators will therefore stitch to the same URN that DataHub's native ingestion sources (e.g. the Redshift source) emit for the same physical table, instead of producing orphan urn:li:dataset:(...,db.None.schema.tbl,PROD) entities. A WARNING is logged at most once per worker process from datahub_airflow_plugin._datahub_ol_adapter so the offending operator can be tracked down upstream without log spam. Orphan URNs created before upgrading are not removed automatically — see the new Orphan dataset URNs containing .None. section in the Airflow lineage docs for cleanup guidance.
• (Ingestion / Fabric OneLake) The queryinsights schema is now excluded from table and view discovery alongside INFORMATION_SCHEMA and sys. Fabric Warehouse's Microsoft-managed Query Insights views (exec_requests_history, long_running_queries, etc.) are no longer ingested as user datasets. This is additive cleanup for most deployments.
• #17380 (Ingestion / Databricks Unity Catalog) New opt-in flag include_metric_views (default false) for Unity Catalog Metric Views. When enabled, metric views are emitted with subtype Metric View (instead of Table), their YAML body is preserved as a ViewProperties aspect with viewLanguage: YAML, and upstream lineage is derived from the YAML source and joins[].source fields. Lineage resolution supports 3-part (catalog.schema.table) and 2-part (schema.table, resolved against the metric view's own catalog) identifiers as well as backtick-quoted parts; 1-part names and SQL subqueries fall back to the existing table-lineage REST API. Column-level lineage is emitted from each dimensions[].expr / measures[].expr using the Databricks SQL dialect (controlled by the existing include_column_lineage flag, which is on by default). Schema fields are tagged Dimension or Measure when their names match the YAML; per-column descriptions are taken from dimensions[].description / measures[].description when present and otherwise fall back to the Unity Catalog column comment; the top-level filter field surfaces as a metric_view_filter custom property; and materialization: materialized drives ViewProperties.materialized = true. On the opt-in path, the noisy Spark engine config snapshot Unity Catalog injects as view.sqlConfig.spark.* custom properties (~150 entries per view) is filtered out — other view.* properties and Databricks' own metric_view.* properties pass through unchanged. The flag remains false by default so the upgrade is byte-for-byte identical to the previous release. Requires a databricks-sdk recent enough to expose TableType.METRIC_VIEW (older SDKs short-circuit silently and the flag has no effect; a startup warning is emitted in that case).

New ingestion sources

• #11838 Aerospike
• #13217 Airbyte
• #15913 StarRocks
• #15966 Matillion
• #16218 Apache Flink (metadata and lineage)
• #16426 dlt (data load tool)
• #16472 Pinecone Vector DB
• #16564 Omni BI platform (INCUBATING)
• #16646 Microsoft Fabric Data Factory
• #17467 pg_queue metadata-ingestion sink and Actions event source

Connector and platform enhancements

• #16505 (Ingestion / Glue) JDBC upstream lineage for Glue jobs
• #16562 (Ingestion / Glue) Iceberg lineage
• #16636 (Ingestion / Glue) Job subtype on Glue jobs
• #16515 (Ingestion / Kafka Connect) Column-level lineage for sink connectors and field-level transform support
• #16348 (Ingestion / BigQuery) External table metadata enrichment (format, URIs, compression, max bad records)
• #16621 (Ingestion / Power BI) browsePathsV2 hierarchy for tables, tiles, and pages under parent dashboards
• #16616 (Ingestion / Power BI) Sql.Databases M-Query data access function support
• #16486 (SDK) Configurable connection pool settings on REST emitter
• #16798 (Ingestion) Multi-entity domain and ownership transformers
• #16044 (Ingestion / dbt) Extract and emit stats from catalog.json
• #17284 (Ingestion / Fabric OneLake) Query usage extraction from queryinsights
• #17475 (Ingestion / Confluence) Page body HTML converted to Markdown via markdownify

Frontend and security

• #17277 (Frontend) Content-Security-Policy via Play CSPFilter
• #17468 (Frontend) Collapsible nav redesign and home hero toggler
• (Frontend) Semantic color tokens migration across entity, lineage, home, and settings surfaces (use theme.colors.* tokens per theming guidelines)
• #17487 (Security) Home page templates/module scope validation
• #17489 (Security) URL validation before rendering external links in the app

Operations

• #17393 (Kubernetes) KEDA-aware scaling support in KubernetesController
• #17431 (Search) Timeseries write throttle for Elasticsearch index updates
• #17456 (Docker) Published DataHub Postgres extensions image

v1.5.0

Requirements:

• Helm Chart: 0.9.0

Breaking Changes

• #16341 (Ingestion) SQL parsing: View query IDs are now generated using a SHA-256 hash instead of URL-encoding the view URN. This affects all connectors that use view lineage tracking (Snowflake, Oracle, BigQuery, Postgres, MySQL, Hive, Trino, ClickHouse, DB2, Dremio, SQL Server, and others). Previously, query entities had URNs like urn:li:query:view_urn%3Ali%3Adataset%3A%28...%29; they now use urn:li:query:view_<sha256hash>. After upgrading, the old URL-encoded query entities will become stale and orphaned. To clean them up, enable stateful ingestion with stale entity removal in your recipe and re-run ingestion.
• #16685:(Ingestion) PowerBI M-Query lineage extraction has been rewritten using Microsoft's official @microsoft/powerquery-parser. As part of this change, the native_query_parsing: false configuration flag now suppresses only expressions containing Value.NativeQuery. Previously it suppressed all M-Query lineage extraction. Users who set this flag to block all lineage extraction will now see lineage produced for non-NativeQuery sources (Snowflake, PostgreSQL, MSSQL, etc.). To restore the suppress-all behaviour, add a table_pattern.deny rule in your recipe.
• #16396: Oracle connector: When connecting via service_name to a multitenant Oracle database, the database name used in URNs will now reflect the Pluggable Database (PDB) name instead of the Container Database (CDB) name. In Oracle Multitenant architecture, a CDB is the top-level container (e.g. cdb) and a PDB is an individual tenant database within it (e.g. mypdb); service_name typically routes to the PDB, so the PDB name is the correct identifier for your datasets. This affects both dataset URNs (when add_database_name_to_urn: true) and database/schema container URNs (always, since containers always include the database name). If your existing metadata was ingested with the old CDB-based URNs, re-ingesting will create new entities under the corrected URNs. To preserve the old URN shape and avoid re-creating entities, set urn_db_name explicitly in your recipe to match your previous CDB name.
• #16628 (Ingestion) Fabric OneLake source: Workspace containers now use the fabric platform instead of fabric-onelake. This changes workspace container URNs and the dataPlatformInstance.platform emitted for workspace entities. Lakehouse, warehouse, schema, and dataset entities remain on fabric-onelake.
• Retention service disabled: only current version retained. When the retention service is not enabled (not configured or unavailable), the write path now retains only the current version (version 0) and does not create version-history rows. Previously, version history was still written when retention was disabled. Impact: Deployments that run without retention enabled will no longer accumulate aspect version history; only the latest aspect value is stored. Migration: Enable and configure the retention service (e.g. ingest retention policies from boot/retention.yaml) if you need version history for any entity/aspect.
• #16134: Java 17 Runtime Required - DataHub now compiles to Java 17 bytecode (previously Java 11). All Docker images already ship with Java 17 runtime. Self-hosted users must ensure their runtime environment uses Java 17+. The Maven artifact name remains datahub-client-java8 for backward compatibility, but now requires Java 17+ at runtime. This change also includes:
  • Spark integration upgraded from 3.0.3 to 3.3.4 (minimum version required for Java 17 support)
  • Hadoop upgraded from 2.7.2 to 3.3.6 (addresses Hadoop CVEs, bundled with Spark 3.3+)
  • Note: If you're a self-hosted user still running Java 11, you must upgrade to Java 17 before deploying this release. Spark lineage users must upgrade to Spark 3.3.0+.
• (Frontend) CustomHttpClientFactory now restricts TLS to 1.2 and 1.3 only when using a custom truststore; TLS 1.0 and 1.1 are disabled for security. If the frontend uses a custom truststore to reach GMS or an IdP that only supports TLS 1.0/1.1, those connections will fail until the server is upgraded to support at least TLS 1.2.
• #16176: Vertex AI Source - Two breaking changes:
  1. Pipeline URN structure changed: Previously, each pipeline run created a new DataFlow entity. Now, Vertex AI pipelines are modeled as stable DataFlow entities (one per pipeline template), with DataJobs for tasks and DataProcessInstances for individual runs. This enables proper incremental lineage aggregation. Pipeline URNs now use the compiled pipeline spec name (pipelineInfo.name, e.g. the @pipeline(name="...") argument in Kubeflow Pipelines) as the stable identifier; non-Kubeflow pipelines fall back to display_name with any timestamp suffix stripped. After upgrading, existing pipeline entities will appear as separate entities from new ingestion runs. To clean up old entities, enable stateful ingestion with stale entity removal.
  2. ML Metadata extraction now enabled by default: Enhanced ML Metadata extraction for CustomJob lineage, hyperparameters, metrics, and model evaluations is now enabled by default. This requires additional GCP permissions: aiplatform.metadataStores.get, aiplatform.metadataStores.list, aiplatform.executions.get, and aiplatform.executions.list. If these permissions are missing, the connector gracefully falls back with warnings. To disable these features, set use_ml_metadata_for_lineage: false, extract_execution_metrics: false, and include_evaluations: false.
• #16149 (Ingestion) DataHub source: now uses URN pattern filtering to exclude environment-specific entities by default. This prevents copying credentials and creating invalid entities.
  • Default behavior change: Previously exclude_aspects contained dataHubIngestionSourceInfo, dataHubSecretValue, dataHubExecutionRequestInput, globalSettingsInfo. Now urn_pattern.deny contains urn:li:dataHubIngestionSource:.*, urn:li:dataHubSecret:.*, urn:li:globalSettings:.*, urn:li:dataHubExecutionRequest:.* to exclude entire entity types instead of individual aspects.
  • Note: If you override urn_pattern or exclude_aspects in recipe, carefully configure them based on your requirements to avoid syncing sensitive data or creating invalid entities. Recommended to keep new defaults.
• #16333 (Ingestion) Sigma: Owner urns are now created from the email address of the user. Any existing references to the urns of firstName_lastName will break.
• (Ingestion) Kafka Connect: The Debezium SQL Server connector now reports its platform as mssql instead of sqlserver, matching DataHub's canonical platform name. This fixes column-level lineage when using use_schema_resolver: true, as the SchemaResolver now correctly queries for platform=mssql. If you have platform_instance_map or connect_to_platform_map entries keyed on "sqlserver" for Debezium SQL Server connectors, update them to use "mssql" instead.
• #16385 Default token signing key and salt have been removed from metadata-service/configuration/src/main/resources/application.yaml. It is recommended to set authentication.tokenService.signingKey or env var DATAHUB_TOKEN_SERVICE_SIGNING_KEY and authentication.tokenService.salt or env var DATAHUB_TOKEN_SERVICE_SALT before starting DataHub. Refer the linked pages to know this is handled for local development and CLI quickstart.
  • If you are using helm to deploy DataHub you should be unaffected as the helm charts don't use the default values in application.yaml but rather generate a random secret to use.
  • IMPACT: Due to the change in signing keys for local development and quickstart, PATs generated before this release will be invalidated and will need to be regenerated in those instances.
• #16453 (Ingestion) dbt Cloud: Auto-discovery now ingests all production jobs by default, regardless of the "Generate docs on run" setting. Previously, jobs without generate_docs=True were silently skipped. If you rely on the old behavior (only ingesting jobs with doc generation enabled), add require_generate_docs: true under auto_discovery in your recipe.
• #16342 (Timeline API) The backend ChangeCategory enum value OWNER has been renamed to OWNERSHIP to match the GraphQL ChangeCategoryType enum (which has always used OWNERSHIP). Clients calling the REST API (/timeline) with OWNER will still work (backward-compatible alias). The GraphQL schema is unchanged and only recognizes OWNERSHIP. Entity Change Events (Actions EntityChangeEvent_v1, Cloud Entity Events API, EventBridge) serialize this enum into the JSON category field, so new events emit "category": "OWNERSHIP". Filters and automations that still match OWNER will miss those events and should be updated.
• #15744: The emit_mcps() method on DataHubRestEmitter now returns List[TraceData] instead of int. Previously it returned the number of chunks/batches sent. Now it returns a list of TraceData objects (one per batch) containing trace IDs for debugging and status checking. To get the previous chunk count, use len(result) on the returned list. Additionally, emit_mcp() now returns Optional[TraceData] instead of None.
• #16680 (Frontend) The V1 UI theme is now officially sunset. All new features and patches going forward will be on the V2 UI (THEME_V2_ENABLED=true). If you are a customer of DataHub Cloud, or started using DataHub after February 2025 (or kept your clone/fork's environment variables in-sync with upstream since then), then this is already your default experience and you do not need to change anything.
  • To ensure you are experiencing the latest features of DataHub, please ensure the GMS environment variables THEME_V2_ENABLED and THEME_V2_DEFAULT are set to true and THEME_V2_TOGGLEABLE is set to false.
  • If you are using helm to deploy Datahub, ensure datahub-gms.theme_v2.enabled and datahub-gms.theme_v2.default are both set to true in values.yaml.

Known Issues

Potential Downtime

Deprecations

• #16176: Vertex AI Source - The region configuration field is deprecated in favor of regions (list type). The region field continues to work for backward compatibility.
• #16176: Vertex AI Source - Partition handling behavior will change in a future major version. Currently, normalize_external_dataset_paths defaults to false, meaning partitioned paths like gs://bucket/data/year=2024/month=01/ create separate dataset entities. In the next major version, this will default to true, normalizing paths to gs://bucket/data/ for stable dataset URNs with lineage aggregation across partitions. To opt-in to the new behavior now, set normalize_external_dataset_paths: true in your configuration.
• #16375: Vertex AI Source - The project_id (singular) configuration field is deprecated in favor of project_ids (list). The field is automatically migrated at startup — no immediate action required. Update your recipe to use project_ids: ["my-project"] to silence the deprecation warning. project_id will be removed in a future major release.

Other Notable Changes

• #16317 (Ingestion) Looker/LookML: The tag_measures_and_dimensions config is now correctly honored. Previously, setting it to false had no effect — Dimension/Measure/Temporal tags were always added to schema fields. After this fix, setting tag_measures_and_dimensions: false will remove those tags and prepend the field type to the description instead (e.g., "Dimension. My field"). If you had this config set to false but were relying on the tags being present, be aware they will now be removed as intended.
• #15270 (Ingestion) Browse paths: When platform_instance is configured, DataFlow and DataJob entities now receive a browsePathsV2 aspect with the platform instance as the root path. Previously, these entities had no browse path from ingestion and the backend would place them in a generic "Default" folder, causing entities from multiple platform instances to be mixed together. This affects sources like Fivetran, Glue, and Kafka-Connect that emit DataFlow/DataJob entities with platform_instance. Sources without platform_instance configured are unaffected.
• #16339 (Ingestion) Migrating Python dependency declarations from setup.py to pyproject.toml (PEP 621). setup.py remains the source of truth for editing dependencies for now; pyproject.toml is auto-generated from it via ./gradlew :metadata-ingestion:updateLockFile. Sync verification (./gradlew :metadata-ingestion:verifyPyprojectSync) runs automatically during check and buildWheel. setup.py will be deprecated in a future release in favor of pyproject.toml as the sole dependency source.
• #16358 (Ingestion) dbt: Added convert_urns_to_lowercase config option for dbt ingestion (both dbt-core and dbt-cloud). When enabled, dbt platform URNs are lowercased, preventing duplicate entities caused by schema name casing differences (e.g., app_sales vs APP_SALES). This is an opt-in flag (default: false for all platforms). Recommended for case-insensitive platforms like Snowflake or BigQuery where dbt manifests may contain mixed-case identifiers.
• Kubernetes: optional scale-down during system-update. When the system-update job runs in a Kubernetes cluster, an optional Java-based step can scale down deployments (by label selector, e.g. MAE/MCE) and set environment variables on other deployments (by label selector, e.g. GMS) before blocking upgrades (e.g. reindex), then restore them after. Scale-down is conditional: it runs only when a blocking upgrade (such as BuildIndices when reindex is needed) requires it. Rollout and scale-down operations run in parallel. State is stored in a ConfigMap; if retries are exceeded, the step restores state, deletes the ConfigMap, and fails. The Java implementation is disabled by default; enable via Helm (datahubSystemUpdate.scaleDown.useJavaImplementation) and the job env vars described in Environment variables. Outside Kubernetes or when disabled, the step is a no-op.
• #16176: Vertex AI Source Connector - Additional improvements beyond breaking changes:
  • Cross-platform lineage to external data sources (GCS, BigQuery, S3, Azure Blob Storage, Snowflake) referenced in training jobs, with configurable platform instances via platform_instance_map to ensure URNs match native connectors
  • UI organization improved with hierarchical folders: model versions appear under their model group folders, and pipeline tasks nest under their parent pipelines
  • Enhanced lineage relationships: direct model→dataset relationships via TrainingData aspect; experiment run→model outputs and pipeline task run→model/dataset lineage via DataProcessInstance aspects
  • For performance with large projects (1000+ training jobs), use stateful ingestion (recommended) which only processes new/changed resources on subsequent runs. Limits like max_training_jobs_per_type control which resources are processed and sent to DataHub (most recently updated N items)
  • Optional platform_instance configuration field for environments running multiple Vertex AI instances
  • Existing ingestion configurations continue to work without modification, and URNs remain unchanged unless platform_instance is explicitly configured
• #16142: Oracle ingestion: Fixed database container naming when using service_name instead of database configuration. Previously, Oracle containers had no name (only an ID) when using service_name with data_dictionary_mode: ALL (the default). Container URNs will change for affected users as the database name is now properly populated in the container GUID. If stateful ingestion is enabled, running ingestion with the latest CLI version will automatically clean up the old containers and create properly named ones. This fix also ensures database names are normalized consistently with schema and table names.
• #16300 (Ingestion) Mode connector: Performance improvements for large Mode workspaces including concurrent API fetching, response caching, and SQL parsing optimizations. Column-level lineage now emits table-level lineage as a fallback when column-level parsing times out.
• #16300 (Ingestion) SQL parsing: Join resolution for queries involving CTEs and subqueries was optimized to avoid expensive deep-copy operations that caused significant slowdowns on complex queries. This affects all connectors that use SQL-based lineage (Snowflake, BigQuery, dbt, Mode, Redshift, etc.). The new approach directly walks SQL scope sources to resolve physical tables, replacing the previous indirect column-tracing fallback. This produces more accurate join metadata — for example, unused CTEs are now correctly excluded from join tables. In rare edge cases with unusual CTE patterns, join metadata may differ from previous results, but the new output better reflects the actual query structure.
• #15741: New RDF ingestion source.
• #15735: New Snowplow ingestion source.
• #15249: New Apache Doris ingestion source.
• #16236 (Ingestion) dbt: Support for ingesting dbt semantic models.
• #16483 / #16445 (Ingestion) Kafka Connect: Added support for Debezium and Confluent JDBC sink connectors; the connector image bundles a JVM via jdk4py so a system Java installation is not required for Kafka Connect ingestion.
• #16292 (Ingestion) Trino: Column-level lineage on upstream datasets via the connector.
• #16443 (Ingestion) Iceberg: Optional ingestion-time domain assignment configuration.
• #13232 (Ingestion) BigQuery: Added convert_column_urns_to_lowercase configuration option (opt-in).
• #16568 (Ingestion) Power BI: extract_column_level_lineage is enabled by default for richer lineage; disable in the recipe if you need the previous behavior.
• #16387 / #16388: Assets can be associated with multiple data products (backend and UI).
• #16365: Policies can target resources by Glossary Terms and Groups.
• #16207: Domain-scoped policies can include assets in child domains.
• #16303 (Ingestion) Sigma: Workbook filtering in addition to workspace filtering.
• #16100 (Ingestion) Snowflake: Metadata pattern pushdown and table type filtering to reduce warehouse metadata queries.
• #16310 (Ingestion) Kafka source: Optional configuration to disable Avro schema name validation.
• #16471 (CLI) datahub search with semantic search, query projection, and agent-context integration.
• #15744 (SDK) Trace IDs are exposed for SYNC_PRIMARY and ASYNC emit modes on the REST emitter.
• #16529 (Ingestion) Great Expectations and SQLAlchemy-based profilers are brought to feature parity.
• #16355: Elasticsearch reindex and index-creation paths include additional retry behavior for resilience.
• #16489 / #16498: Ingestion Docker images and builds use pinned transitive dependencies (uv.lock, constraints.txt) for more reproducible installs.
• #16165 (Ingestion) Configurable report sample sizes and richer failure logging for ingestion reports.
• #15855 (Ingestion) dbt: Support for ingesting dbt exposures.
• #16058 (Ingestion) Snowflake: Support for ingesting external data metric function (DMF) assertions.
• #16265 (Ingestion) Azure Data Factory: Column lineage extraction for Copy activity.
• #16235 (Ingestion) Airflow plugin: Multi-statement SQL parsing support for lineage extraction.

v1.5.0.1

Patch release for the Actions image packaging.

• #16781 (Actions): The datahub-actions Docker image bundles dedicated virtualenvs for datahub-gc and datahub-documents.

v1.5.0.2

Patch release focused on dependency and security updates (Java stack, Python ingestion, observability).

• #16829 / #16828: Spring bumped to 6.2.17 and Netty to 4.1.132.Final.
• #16950: Patched Rhino, Logback, and Commons Lang3 for CVEs.
• OpenTelemetry API/SDK and instrumentation upgraded (Docker images use opentelemetry-javaagent 2.27.0; JMX Prometheus agent unchanged at 0.20.0); addresses CVE-2026-33701.
• #16822 / #16868 / #16955: Python ingestion dependency fixes (CVE-2024-27459), raised pyOpenSSL floor with expanded security constraints, and LiteLLM 1.83.0 for CVE-2026-35030.

Bug Fixes

• #15748 (Ingestion) PowerBI: Reverted create_corp_user default back to True (was changed to False in v1.5.0). Customers on v1.5.0 or v1.5.0.1 with stateful ingestion enabled and without explicitly setting ownership.create_corp_user: true in their recipe may have had PowerBI-discovered users soft-deleted.

  Remediation — if users were already soft-deleted, restore them using one of:

  • Re-ingest from your authoritative source (LDAP/Okta/SCIM) — recommended.
  • CLI: datahub delete --urn "urn:li:corpuser:<email>" --soft --undelete for each affected user.
  • Python SDK: emit StatusClass(removed=False) for each affected user URN.

  Behavior with create_corp_user: true (new default): PowerBI creates CorpUser entities with display name and email. These are marked as non-primary (is_primary_source=False), so stateful ingestion will not soft-delete them if they disappear from PowerBI. Note: PowerBI user info may overwrite richer profiles from authoritative sources like LDAP/Okta. If this is a concern, follow the migration steps below to switch to false.

  If you want create_corp_user: false (to avoid overwriting LDAP/Okta profiles): Do not switch directly to false — this will cause the same soft-deletion bug. Instead:

  1. Upgrade to this version and run at least one ingestion with create_corp_user: true (the default). This updates the stateful ingestion checkpoint to mark user URNs as non-primary.
  2. After that run completes, set ownership.create_corp_user: false in your recipe.
  3. On subsequent runs, users will no longer be emitted, but stateful ingestion will safely skip them (they were already marked non-primary in the checkpoint).

  Skipping step 1 and going directly to false means the checkpoint still has users marked as primary from the old code, and stateful ingestion will soft-delete them.

v1.5.0.7

Patch release focused on dependency and security updates.

• OpenTelemetry bumped to 1.62.0 (CVE-2026-45292).
• Jackson jackson-core 2.21.3; Parquet stack aligned to 1.17.1.
• #17504: Exclude grpc-netty-shaded; align reactor-netty.
• #17386: Vert.x 4.5.27 (CVE-2026-6860).
• #17452: Sanitize GMS/frontend API error responses (CWE-200).
• #17489: Validate URLs before rendering links in the app.
• #17487: Fix home page templates/module scope hijacking.
• #17453: Python ingestion ujson >= 5.12.1 (CVE-2026-44660).
• #17277: Content-Security-Policy via Play CSPFilter on the frontend.

v1.5.0.6

Patch release focused on image maintenance.

• Remove bundled kubectl from the datahub-upgrade Docker image.

v1.5.0.5

Patch release focused on Java dependency CVE fixes.

• #17352: Netty 4.2.13.Final (CVE-2026-41417 and related).
• #17330: Hadoop third-party, Netty, and gRPC CVE hygiene.
• Spring Boot 3.5.14 (CVE-2026-40973).
• Spring Framework 6.2.18 (CVE-2026-22740).

v1.5.0.4

Patch release focused on HTTP client CVE remediation.

• #17323: Apache httpclient5 5.6.1 (CVE-2026-40542).

v1.5.0.3

Patch release focused on security and dependency updates (Play frontend and Java stack, Python ingestion), OIDC and native sign-up fixes, and Docker/CI maintenance.

• #17030: Jetty upgrade with refreshed Gradle lockfiles.
• #17114: Reactor Netty 1.2.17.
• #17112: Apache Shiro 2.1.0 (CVE-2026-23903).
• #17110: pac4j 6.4.1 (CVE-2026-40458).
• #17113: Hibernate Validator upgrade (CVE-2025-35036).
• #17132: jakarta.mail 1.6.8 (CVE-2025-7962).
• #17140: OpenTelemetry Java agent 2.27.0.
• #17152: maven-artifact 3.9.15.
• #17092: Python cryptography raised to >= 46.0.7.
• #17116: Ingestion lockfile updates for aiohttp, pytest, and tighter declared bounds.
• #17131: Ingestion constraints for Authlib (>= 1.6.11) and pypdf (>= 6.10.2) addressing GHSA-jj8c-mmj3-mmgv and GHSA-4pxv-j86v-mhcw.
• Docker-bundled Python virtualenvs: lxml version floor for CVE-2026-41066.
• #17045: constraints.txt included in lock file materialization for more reproducible ingestion installs.
• #17182: locked datahub-actions removes pip/uv

Notable Changes

• datahub-actions default base image updated (override with BASE_IMAGE)

Bug Fixes

• #17032: OIDC post-login redirect cookie handling.
• Play Framework: add play-java-forms at runtime for native sign-up email validation; smoke tests use valid email addresses for those flows.
• Shiro/pac4j: AuthModule refactored so the runtime no longer depends on master-only build artifacts for authentication wiring.

Build, Docker, and Tooling

• #17156: Slimmer datahub-actions image build; test-only dependency moves; documentation updates for redacting certain Excel-sourced content.
• #17133: datahub-upgrade image bundles kubectl v1.35.4.
• macOS: pin JPype1 below 1.7.0 so dependency resolution keeps compatible wheels on Apple Silicon (reflected in ingestion constraints.txt).
• CI: pin remaining GitHub Actions to full commit SHAs for supply-chain hardening.

v1.4.0

Breaking Changes

• Python 3.9 support has been dropped. All Python modules now require Python 3.10 or later. This affects acryl-datahub, acryl-datahub-airflow-plugin, acryl-datahub-dagster-plugin, acryl-datahub-gx-plugin, prefect-datahub, and acryl-datahub-actions. Upgrade to Python 3.10+ before upgrading these packages.
• #15930: The Airflow plugin's kill switch for Airflow 2.x now uses environment variables instead of Airflow Variables. If you were using airflow variables set datahub_airflow_plugin_disable_listener true to disable the plugin, you must now use export AIRFLOW_VAR_DATAHUB_AIRFLOW_PLUGIN_DISABLE_LISTENER=true instead.
• #15877: The LDAP ingestion source now enforces TLS certificate verification by default (tls_verify: true). This prevents Man-in-the-Middle attacks (CWE-295) but may break existing configurations using self-signed certificates. To restore the previous behavior, explicitly set tls_verify: false in your recipe.
• #15748: PowerBI ingestion source create_corp_user default changed from True to False. Previously, PowerBI would create user entities by default, potentially overwriting existing user profiles from LDAP/Okta. Now, PowerBI emits ownership URNs only (soft references) by default. To restore previous behavior, explicitly set ownership.create_corp_user: true in your recipe. Migration note: If upgrading and users appear with incomplete profiles, re-ingest from your authoritative source (LDAP/Okta/SCIM). Additionally, when create_corp_user: true, the connector now emits both CorpUserKeyClass and CorpUserInfoClass (was only CorpUserKeyClass), providing complete user metadata. Stateful ingestion note: User entities created by PowerBI are now marked as non-primary (is_primary_source=False), so they will NOT be soft-deleted by stateful ingestion when they disappear. This prevents accidental deletion of users who may also exist in LDAP/Okta/SCIM.
• #15415: MLModel and MLModelGroup search field mapping has been updated to resolve duplicate field name conflicts. Existing entities will be automatically reindexed in the background after upgrade. New MLModel and MLModelGroup entities ingested after the upgrade will work immediately.
• #15397: Grafana ingestion source dataset granularity changed from per-datasource to per-panel (per visual). This improves lineage accuracy by ensuring each panel's query results in a unique dataset entity with precise upstream/downstream connections. Dataset URN format changed from {ds_type}.{ds_uid} to {ds_type}.{ds_uid}.{dashboard_uid}.{panel_id}. This means all existing Grafana dataset entities will have different URNs. If stateful ingestion is enabled, running ingestion with the latest CLI version will automatically clean up old entities and create new ones. Otherwise, we recommend soft deleting all Grafana datasets via the DataHub CLI: datahub delete --platform grafana --soft and then re-ingesting with the latest CLI version.
• #15005: SqlParsingBuilder is removed, use SqlParsingAggregator instead
• #14710: LookML ingestion source migrated to SDKv2 resulting in:
  • browsePaths aspect replaced with browsePathsV2
  • Only emits MCPs
• #14693: Looker ingestion source migrated to SDKv2, resulting in:
  • browsePaths aspect replaced with browsePathsV2
  • In Dashboard entity and dashboardInfo aspect, charts property (deprecated) is replaced with chartEdges
  • Only emits MCPs
• #16023: In PowerBI, Container URNs change for users with platform_instance configured, as this PR now passes platform_instance to dataset containers (affects GUID generation). The env parameter addition is harmless as it's excluded from GUID calculation. Stateful ingestion will soft-delete old containers and create new ones on the next run. Dataset entities and their lineage are unaffected.
• #16067: Oracle stored procedure URN format has been corrected to match table URN format. For most users (using service_name or database without add_database_name_to_urn: true), stored procedure DataJob URNs will change from database.schema.stored_procedures to schema.stored_procedures. This fixes a URN mismatch that prevented stored procedure lineage from working. Stateful ingestion will soft-delete old stored procedure entities and create new ones with correct lineage on the next run. Users with database config parameter and add_database_name_to_urn: true are unaffected.

Deprecations

• (CLI) The --use-password flag in datahub init command is deprecated. Token generation is now automatically detected when both --username and --password are provided together. The flag continues to work for backward compatibility but will be removed in a future release.

Other Notable Changes

• #15262: (Ingestion) The Metabase ingestion source has been improved with several robustness and performance enhancements:
  • Collection name sanitization now properly handles special characters (e.g., "Sales & Marketing" → metabase_collection_sales_marketing)
  • Added recursion depth protection to prevent stack overflow from circular card references
  • Optimized collection API calls with caching to eliminate N+1 query problems (up to 620x performance improvement for large deployments)
  • Improved error reporting to differentiate expected 404 errors from authentication/permission issues
  • Consolidated duplicate lineage extraction logic for better maintainability
  • Optimized model extraction to reduce duplicate API calls
  • Recipes must provide api_key, or both username and password; blank values are rejected. connect_uri should include a scheme (http:// or https://); a host-only value is treated as http://.
• (Ingestion) BigQuery source: Improved dataset_pattern filtering to apply earlier in the ingestion pipeline, reducing unnecessary API calls to BigQuery for datasets that will be filtered out.
• #15714: Kafka topic partition counts can now automatically be increased during upgrades if configured values exceed existing partition counts. Set DATAHUB_AUTO_INCREASE_PARTITIONS=true to enable.
• (CLI) Added --extra-env option to datahub ingest deploy command to pass environment variables as comma-separated KEY=VALUE pairs (e.g., --extra-env "VAR1=value1,VAR2=value2"). These are stored in the ingestion source configuration and made available to the executor at runtime.
• #14968: Added an ingestion source for IBM Db2 databases.
• #15824: The Databricks ingestion source now additionally supports authentication via M2M OAuth or Databricks unified authentication. The Azure authentication method now supports profiling as well as ingesting lineage from system tables.
• #16067: Oracle ingestion source improvements: Added procedure-to-procedure lineage support, improved PL/SQL parsing for control flow keywords (WHILE, LOOP, EXCEPTION, etc.), fixed overloaded procedure handling, and added distinct subtypes for functions vs procedures. Stored procedures and functions now have proper container hierarchy (Database → Schema → Flow) matching tables and views. Both functions and procedures are organized in the same {schema}.stored_procedures container (consistent with PostgreSQL, MySQL, and Snowflake), with individual subtypes to distinguish them.

1.3.0

Breaking Changes

• #14580: (Ingestion) The redshift lineage v1 implementation (RedshiftLineageExtractor) has been removed, as lineage v2 (RedshiftSqlLineageV2) implementation has been default for a while already. As an effect use_lineage_v2 config has also been removed along with all lineage v1 references and tests have been updated to v2 implementation. This should not impact most users as change is isolated in redshift ingestion source only.
• #14014: The acryl-datahub now requires pydantic v2. Support for pydantic v1 has been dropped and users must upgrade to pydantic v2 when using DataHub python package.
  • As a side effect, this upgrade in pydantic version has implicit consequences for iceberg ingestion source. If it is run from CLI and datahub CLI was installed with all extras (acryl-datahub[all]), then pyiceberg has been kept at 0.4.0 version in such environment, just to satisfy the pydantic v1 restriction. However now, pyiceberg will be installed in the newest available version. While this is a breaking change in the behaviour, versions >0.4.0 have been used for some time by Managed Ingestion.
  • Additionally, there have been changes to the catalog connection configuration details - especially for AWS-based catalogs and warehouses, the properties profile_name, region_name, aws_access_key_id, aws_secret_access_key, and aws_session_token were deprecated and removed in version 0.8.0. To check whether your configuration will work, consult https://py.iceberg.apache.org/configuration/#catalogs. Because of that, pyiceberg dependency has been restricted to be 0.8.0 at least.
  • Anyway, there are no changes needed for iceberg recipes orchestrated via the Managed Ingestion UI
• The acryl-datahub-airflow-plugin now requires specifying the appropriate installation extra based on your Airflow version due to different OpenLineage dependencies:
  • For Airflow 2.x (2.7+): Install with pip install 'acryl-datahub-airflow-plugin[airflow2]'
  • For Airflow 3.x (3.0+): Install with pip install 'acryl-datahub-airflow-plugin[airflow3]'
  • For Airflow 3.0.x specifically: You'll also need to bump pydantic: pip install 'acryl-datahub-airflow-plugin[airflow3]' 'pydantic>=2.11.8'
  • Installing without the appropriate extra will result in missing OpenLineage dependencies and lineage extraction will not work properly. The plugin automatically detects your Airflow version and uses the appropriate integration once the correct extra is installed.
• Auto-detection for SearchClientShim requires permission for the cluster info endpoint on ElasticSearch/OpenSearch. If you are using a restrictive account to access your cluster, you may need to directly configure the engine type rather than relying on auto-detection. These are the two environment variables needed to be configured for GMS as well as MCE & MAE consumers (available in helm charts as well):
  • ELASTICSEARCH_SHIM_ENGINE_TYPE & ELASTICSEARCH_SHIM_AUTO_DETECT (.Values.global.elasticsearch.engineType and .Values.global.elasticsearch.autoDetect respectively for helm)
  • Allowed values: ELASTICSEARCH_SHIM_ENGINE_TYPE["ELASTICSEARCH_7", "ELASTICSEARCH_8", "OPENSEARCH_2", "AUTO_DETECT"] , ELASTICSEARCH_SHIM_AUTO_DETECT["true","false"]

Known Issues

Potential Downtime

• #15226: Requires reindexing the query entity index on upgrade. Depending on the size of this index, upgrade times may vary.

Deprecations

Other Notable Changes

• Airflow 3.x Support: The acryl-datahub-airflow-plugin now supports Apache Airflow 3.x while maintaining backward compatibility with Airflow 2.5+. Key changes include:
  • Kill Switch Migration: The plugin's kill switch behavior has changed between versions:
    • Airflow 2.x: Continue using Airflow Variables to disable the plugin (set the Airflow Variable datahub_airflow_plugin_disable_listener to true)
    • Airflow 3.x: Use environment variable AIRFLOW_VAR_DATAHUB_AIRFLOW_PLUGIN_DISABLE_LISTENER=true to disable the plugin. This change is required to comply with Airflow 3.x's strict database access restrictions during listener initialization.
  • Configuration Changes: Airflow 3.x moved some configuration keys (e.g., WEBSERVER__BASE_URL → API__BASE_URL). The plugin automatically detects and uses the correct configuration for each version.
  • SubDAG Removal: SubDAG support has been removed in Airflow 3.x. Users should migrate to TaskGroups for visual grouping of tasks.
  • New SQLParser Integration: Airflow 3.x uses a unified SQLParser patch architecture for lineage extraction, replacing operator-specific extractors. This provides better consistency and column-level lineage across all SQL operators.
• #15118: (Ingestion) The Oracle source now includes stored procedures, functions, packages, and materialized views with automatic lineage generation. Use procedure_pattern to filter procedures if needed. See the Oracle source documentation for permissions and configuration details.
• #14717: The Tableau ingestion source now enables extract_lineage_from_unsupported_custom_sql_queries by default. This improves the quality of lineage extracted by using DataHub's SQL parser in cases where the Tableau Catalog API fails to return lineage for Custom SQL queries.
• #14824: DataHub now supports CDC (Change Data Capture) mode for generating MetadataChangeLogs with guaranteed ordering based on database transaction commits. CDC mode is optional and disabled by default. When enabled via CDC_MCL_PROCESSING_ENABLED=true, MCLs are generated from Debezium-captured database changes rather than directly from GMS. This provides stronger ordering guarantees and decoupled processing. Requires MySQL 5.7+ or PostgreSQL 10+ with replication enabled. See CDC Configuration Guide for setup instructions.
• Added multi-client search engine shim for Elasticsearch and OpenSearch support. This enables DataHub to work with ES 7.17 (with API compatibility mode for ES 8.x servers), ES 8.x, and OpenSearch 2.x through a unified interface. The shim includes auto-detection of search engine types and backward compatibility with existing RestHighLevelClient usage. See elasticsearch-search-client-shim.md for configuration details.
• #14953: Unity Catalog ingestion source now supports extracting usage query history from the system.query.history table for improved performance with large query volumes. The new usage_data_source configuration (default: AUTO) automatically uses system tables when warehouse_id is configured, otherwise falls back to the REST API. This change is not breaking as the AUTO mode gracefully handles configurations without warehouse_id by using the existing REST API approach. Users can explicitly force system tables mode by setting usage_data_source: SYSTEM_TABLES (requires SELECT permission on system.query.history) or continue using the REST API with usage_data_source: API.

1.2.0

Breaking Changes

• All DataHub Python packages now require Python 3.9+. This affects the following packages:
  • acryl-datahub (DataHub CLI and SDK)
  • acryl-datahub-actions
  • acryl-datahub-airflow-plugin
  • acryl-datahub-prefect-plugin
  • acryl-datahub-gx-plugin
  • acryl-datahub-dagster-plugin (already required Python 3.9+)
• #13619: The acryl-datahub-airflow-plugin has dropped support for Airflow versions less than 2.7.
• #14054: The v1 plugin in acryl-datahub-airflow-plugin has been removed. The v2 plugin has been the default for a while already, so this should not impact most users. Users who were explicitly setting DATAHUB_AIRFLOW_PLUGIN_USE_V1_PLUGIN=true will need to either upgrade or pin to an older version to continue using the v1 plugin.
• #14015: In the sql-queries source, the default_dialect configuration parameter has been renamed to override_dialect. This also affects the Python SDK methods:
  • DataHubGraph.parse_sql_lineage(default_dialect=...) → DataHubGraph.parse_sql_lineage(override_dialect=...)
  • LineageClient.add_lineage_via_sql(default_dialect=...) → LineageClient.add_lineage_via_sql(override_dialect=...)
• #14059: The acryl-datahub-gx-plugin now requires pydantic v2, which means the effective minimum supported version of GX is 0.17.15 (from Sept 2023).
• #13601: The use_queries_v2 flag is now enabled by default for Snowflake and BigQuery ingestion. This improves the quality of lineage and quantity of queries extracted.

Known Issues

• Internal Schema Registry - The internal schema registry does not supply a compatible schema for older MCP messages. The short term recommendation is to process all MCPs before upgrading to this release.

Potential Downtime

Deprecations

• #13858: For folks using bigquery and redshift connectors please update schema_pattern to match against fully qualified schema name <database_name>.<schema_name> and set config match_fully_qualified_names : True. Current default match_fully_qualified_names: False is only to maintain backward compatibility. The config option match_fully_qualified_names will be removed in future and the default behavior will be like match_fully_qualified_names: True.

Other Notable Changes

• The acryl-datahub-actions package now requires Pydantic V2, while it previously was compatible with both Pydantic V1 and V2.
• #14123: Adds a new environment variable DATAHUB_REST_EMITTER_BATCH_MAX_PAYLOAD_BYTES to control batch size limits when using the RestEmitter in ingestions. Default is 15MB but configurable.

1.1.0

Breaking Changes

• #12795: slack source v2 - now ingests all user and channels. Make sure the ingestion source and GMS are on the same version of DataHub.
• #13004: The acryl-datahub-airflow-plugin dropped support for Airflow 2.3 and 2.4.
• #13186: NoCode Migration Removed - This code hasn't been required in many years. If needed, a user should upgrade to DataHub 1.0.x prior to upgrading to a later version.
• #13397: async_flag removed from rest emitter, replaced with emit mode ASYNC
• #13120: DataHub Actions Integration: Moved datahub-actions into OSS DataHub. Users building their own DataHub Actions docker images can now build from the OSS project.

Known Issues

• #13397: Ingestion Rest Emitter
  • ASYNC_WAIT/ASYNC - Async modes are impacted by kafka lag.
  • SYNC_WAIT - Only available with OpenAPI ingestion
• OpenAPI Reports OpenAPI Spec 3.1.0 when it only supports 3.0.1

Potential Downtime

Deprecations

Other Notable Changes

• SDK Improvements: Added URN to URL helpers, lineage client, ML model support, and search enhancements.
• Connector Improvements: Significant enhancements to PowerBI, Superset, MLflow, and Redshift connectors.
• New Connectors: Added connectors for Hex notebooks and VertexAI.
• Iceberg Connector Refactoring: The Iceberg connector has been completely refactored to use MCPWs instead of MCEs.
• Entity Subtypes Model Change: The addition of subtypes to most entities.
• #13151 #13255: UI Search Bar Redesign: search bar with new autocomplete functionality has been added.
• #12976: Edit Lineage Feature: Added ability to edit lineage directly in the UI.
• #12983 #13107: Manage Tags Interface: Added a dedicated "Manage Tags" navigation page.
• #13361: Theme Support: Added foundation for basic theme support with primary color configuration.
• #13165 - OpenAPI v3 Patching Improvements
• #13397 - Ingestion Rest Emitter - Added EmitMode parameter for write guarantees.
  • SYNC_WAIT: Synchronously updates the primary storage (SQL) but asynchronously updates search storage (Elasticsearch). Provides a balance between consistency and performance. Suitable for updates that need to be immediately reflected in direct entity retrievals but where search index consistency can be slightly delayed.
  • SYNC_PRIMARY: Synchronously updates the primary storage (SQL) but asynchronously updates search storage (Elasticsearch). Provides a balance between consistency and performance. Suitable for updates that need to be immediately reflected in direct entity retrievals but where search index consistency can be slightly delayed.
  • ASYNC: Queues the metadata change for asynchronous processing and returns immediately. The client continues execution without waiting for the change to be fully processed. Best for high-throughput scenarios where eventual consistency is acceptable.
  • ASYNC_WAIT: Queues the metadata change asynchronously but blocks until confirmation that the write has been fully persisted. More efficient than fully synchronous operations due to backend parallelization and batching while still providing strong consistency guarantees. Useful when you need confirmation of successful persistence without sacrificing performance.
• #13426 - Added support for extracting column transformation logic when using use_queries_v2 with warehouse ingestion.
• #13499 - Added ELASTICSEARCH_MIN_SEARCH_FILTER_LENGTH configuration for ElasticSearch index config. If modified from the default, this configuration can have significant impact on search performance if changed and will trigger reindexing causing large delays in updates. Most users will not want to modify this.

1.0.0

Breaking Changes

• #12673: Business Glossary ID generation has been modified to handle special characters and URL cleaning. When enable_auto_id is false (default), IDs are now generated by cleaning the name (converting spaces to hyphens, removing special characters except periods which are used as path separators) while preserving case. This may result in different IDs being generated for terms with special characters.

• #12580: The OpenAPI source handled nesting incorrectly. 12580 fixes it to create proper nested field paths, however, this will re-write the incorrect schemas of existing OpenAPI runs.

• #12408: The platform field in the DataPlatformInstance GraphQL type is removed. Clients need to retrieve the platform via the optional dataPlatformInstance field.

• #12671: The priority field of the Incident entity is changed from an integer to an enum. This field was previously completely unused in UI and API, so this change should not affect existing deployments.

• #12716: Fix the platform_instance being added twice to the URN. If you want to have the previous behavior back, you need to add your platform_instance twice (i.e. plat.plat).

• #12797: Previously endpoints when used in ASYNC mode would not validate URNs, entity & aspect names immediately. Starting with this release, even in ASYNC mode, these requests will be returned with http code 4xx. This includes URN, Entity Type names, Entity & Aspect names.

Known Issues

• #12601: Jetty 12 introduces a stricter handling of url encoding. We are currently applying a workaround to prevent a regression, while technically breaking the official specifications.
• #12714: API Tracing requires at least one mutation of the aspect being updated using this version of DataHub.
• #12797: See Breaking Change above. Entity Type names are case sensitive, this will result in 4xx exceptions when this rule is violated.
• Python SDK v1.0.0.3 - direct accesses to the server_config property on DataHubRestEmitter can throw an unknown attribute error if test_connection is not called prior to directly accessing it as the default empty map initialization was removed. This is resolved in v1.1.0.

Potential Downtime

Deprecations

Other Notable Changes

• #12641: Adds a new MCP validator that prevents deletes of any CorpUser entity that has a newly introduced CorpUserInfo#system flag set to true.
• #12433: Fixes the searchable annotations in the model supporting Dashboard to Dashboard lineage within the DashboardInfo aspect. Mainly, users of Sigma and PowerBI Apps ingestion may be affected by this adjustment. Consequently, a reindex will be automatically triggered during the system upgrade.

0.15.0

• OpenAPI Update: PIT Keep Alive parameter added to scroll endpoints. NOTE: This parameter requires the pointInTimeCreationEnabled feature flag to be enabled and the elasticSearch.implementation configuration to be elasticsearch. This feature is not supported for OpenSearch at this time and the parameter will not be respected without both of these set.
• OpenAPI Update 2: Previously there was an incorrectly marked parameter named sort on the generic list entities endpoint for v3. This parameter is deprecated and only supports a single string value while the documentation indicates it supports a list of strings. This documentation error has been fixed and the correct field, sortCriteria, is now documented which supports a list of strings.

Known Issues

• Persistence Exception: No Rows Updated may occur if a transaction does not change any aspect's data.

Breaking Changes

• #12223: For dbt Cloud ingestion, the "View in dbt" link will point at the "Explore" page in the dbt Cloud UI. You can revert to the old behavior of linking to the dbt Cloud IDE by setting `external_url_mode: ide".
• #12191 - Configs include_view_lineage and include_view_column_lineage are removed from snowflake ingestion source. View and External Table DDL lineage will always be ingested when definitions are available.
• #12181 - Configs include_view_lineage, include_view_column_lineage and lineage_parse_view_ddl are removed from bigquery ingestion source. View and Snapshot lineage will always be ingested when definitions are available.
• #12077: Kafka source no longer ingests schemas from schema registry as separate entities by default, set ingest_schemas_as_entities to true to ingest them
• #11486 - Criterion's value parameter has been previously deprecated. Use of value instead of values is no longer supported and will be completely removed on the next major version.
• #11484 - Metadata service authentication enabled by default
• #11484 - Rest API authorization enabled by default
• #10472 - SANDBOX added as a FabricType. No rollbacks allowed once metadata with this fabric type is added without manual cleanups in databases.
• #11619 - schema field/column paths can no longer be empty strings
• #11619 - schema field/column paths can no longer be duplicated within the schema
• #11570 - The DatahubClientConfig's server field no longer defaults to http://localhost:8080. Be sure to explicitly set this.
• #11570 - If a datahub_api is explicitly passed to a stateful ingestion config provider, it will be used. We previously ignored it if the pipeline context also had a graph object.
• #11518 - DataHub Garbage Collection: Various entities that are soft-deleted (after 10d) or are timeseries entities (dataprocess, execution requests) will be removed automatically using logic in the datahub-gc ingestion source.
• #12020 - Removed sql_parser configuration from the Redash source, as Redash now exclusively uses the sqlglot-based parser for lineage extraction.
• #12020 - Removed datahub.utilities.sql_parser, datahub.utilities.sql_parser_base and datahub.utilities.sql_lineage_parser_impl module along with SqlLineageSQLParser and DefaultSQLParser. Use create_lineage_sql_parsed_result from datahub.sql_parsing.sqlglot_lineage module instead.
• #11518 - DataHub Garbage Collection: Various entities that are soft-deleted (after 10d) or are timeseries entities (dataprocess, execution requests) will be removed automatically using logic in the datahub-gc ingestion source.
• #12067 - Default behavior of DataJobPatchBuilder in Python sdk has been changed to NOT fill out created and lastModified auditstamps by default for input and output dataset edges. This should not have any user-observable impact (time-based lineage viz will still continue working based on observed time), but could break assumptions previously being made by clients.
• #12158 - Users provisioned with user.props will need to be enabled before login in order to be granted access to DataHub.
• #13273 - When using Opensearch, we will use 'zstd-no-dict' codec instead of 'default'. Elasticsearch still uses 'default'

Potential Downtime

Deprecations

• #12056: The DataHub Airflow plugin no longer supports Airflow 2.1 and Airflow 2.2.

• #11701: The Fivetran sources_to_database field is deprecated in favor of setting directly within sources_to_platform_instance.<key>.database.

• #11560 - The PowerBI ingestion source configuration option include_workspace_name_in_dataset_urn determines whether the workspace name is included in the PowerBI dataset's URN.
  PowerBI allows to have identical name of semantic model and their tables across the workspace, It will overwrite the semantic model in-case of multi-workspace ingestion.
  Entity urn with include_workspace_name_in_dataset_urn: false

   urn:li:dataset:(urn:li:dataPlatform:powerbi,[<PlatformInstance>.]<SemanticModelName>.<TableName>,<ENV>)

  Entity urn with include_workspace_name_in_dataset_urn: true

   urn:li:dataset:(urn:li:dataPlatform:powerbi,[<PlatformInstance>.].<WorkspaceName>.<SemanticModelName>.<TableName>,<ENV>)

  The config include_workspace_name_in_dataset_urn is default to false for backward compatibility, However, we recommend enabling this flag after performing the necessary cleanup. If stateful ingestion is enabled, running ingestion with the latest CLI version will handle the cleanup automatically. Otherwise, we recommend soft deleting all powerbi data via the DataHub CLI: datahub delete --platform powerbi --soft and then re-ingest with the latest CLI version, ensuring the include_workspace_name_in_dataset_urn configuration is set to true.

Other Notable Changes

• #12236: Data flow and data job entities may additionally produce container aspect that will require a corresponding upgrade of server. Otherwise server can reject the aspect.
• #12056: The DataHub Airflow plugin now defaults to the v2 plugin implementation.
• #11742: For PowerBi ingestion, use_powerbi_email is now enabled by default when extracting ownership information.
• #11549 - Manage Operations Privilege is extended from throttle control to all system management and operations APIs.

0.14.1

Breaking Changes

• #9857 (#10773) lower method was removed from get_db_name of SQLAlchemySource class. This change will affect the urns of all related to SQLAlchemySource entities.

  Old urn, where data_base_name is Some_Database:

  - urn:li:dataJob:(urn:li:dataFlow:(mssql,demodata.Foo.stored_procedures,PROD),Proc.With.SpecialChar)

  New urn, where data_base_name is Some_Database:

  - urn:li:dataJob:(urn:li:dataFlow:(mssql,DemoData.Foo.stored_procedures,PROD),Proc.With.SpecialChar)

  Re-running with stateful ingestion should automatically clear up the entities with old URNS and add entities with new URNs, therefore not duplicating the containers or jobs.

• #11313 - datahub get will no longer return a key aspect for entities that don't exist.

• #11369 - The default datahub-rest sink mode has been changed to ASYNC_BATCH. This requires a server with version 0.14.0+.

• #11214 Container properties aspect will produce an additional field that will require a corresponding upgrade of server. Otherwise server can reject the aspects.

• #10190 - extractor_config.set_system_metadata of datahub source has been moved to be a top level config in the recipe under flags.set_system_metadata

Potential Downtime

Deprecations

Other Notable Changes

• Downgrade to previous version is not automatically supported.
• Data Product Properties Unset side effect introduced
  • Previously, Data Products could be set as linked to multiple Datasets if modified directly via the REST API rather than linked through the UI or GraphQL. This side effect aligns the REST API behavior with the GraphQL behavior by introducing a side effect that enforces the 1-to-1 constraint between Data Products and Datasets
  • NOTE: There is a pathological pattern of writes for Data Products that can introduce issues with write processing that can occur with this side effect. If you are constantly changing all of the Datasets associated with a Data Product back and forth between multiple Data Products it will result in a high volume of writes due to the need to unset previous associations.

0.14.0.2

Breaking Changes

• Protobuf CLI will no longer create binary encoded protoc custom properties. Flag added -protocProp in case this behavior is required.
• #10814 Data flow info and data job info aspect will produce an additional field that will require a corresponding upgrade of server. Otherwise server can reject the aspects.
• #10868 - OpenAPI V3 - Creation of aspects will need to be wrapped within a value key and the API is now symmetric with respect to input and outputs.

Example Global Tags Aspect:

Previous:

{
  "tags": [
    {
      "tag": "string",
      "context": "string"
    }
  ]
}

New (optional fields systemMetadata and headers):

{
  "value": {
    "tags": [
      {
        "tag": "string",
        "context": "string"
      }
    ]
  },
  "systemMetadata": {},
  "headers": {}
}

• #10858 Profiling configuration for Glue source has been updated.

  Previously, the configuration was:

  profiling: {}

  Now, it needs to be:

  profiling:
    enabled: true

Potential Downtime

Deprecations

• OpenAPI v1: OpenAPI v1 is collectively defined as all endpoints which are not prefixed with /v2 or /v3. The v1 endpoints will be deprecated in no less than 6 months. Endpoints will be replaced with equivalents in the /v2 or /v3 APIs. No loss of functionality expected unless explicitly mentioned in Breaking Changes.

Other Notable Changes

• #10498 - Tableau ingestion can now be configured to ingest multiple sites at once and add the sites as containers. The feature is currently only available for Tableau Server.
• #10466 - Extends configuration in ~/.datahubenv to match DatahubClientConfig object definition. See full configuration in https://docs.datahub.com/docs/python-sdk/clients/. The CLI should now respect the updated configurations specified in ~/.datahubenv across its functions and utilities. This means that for systems where ssl certification is disabled, setting disable_ssl_verification: true in ~./datahubenv will apply to all CLI calls.
• #11002 - We will not auto-generate a ~/.datahubenv file. You must either run datahub init to create that file, or set environment variables so that the config is loaded.
• #11023 - Added a new parameter to datahub's put cli command: --run-id. This parameter is useful to associate a given write to an ingestion process. A use-case can be mimic transformers when a transformer for aspect being written does not exist.
• #11051 - Ingestion reports will now trim the summary text to a maximum of 800k characters to avoid generating dataHubExecutionRequestResult that are too large for GMS to handle.

0.13.3

Breaking Changes

• #10419 - aws_region is now a required configuration in the DynamoDB connector. The connector will no longer loop through all AWS regions; instead, it will only use the region passed into the recipe configuration.
• #10389 - Custom validators, mutators, side-effects dropped a previously required constructor
• #10472 - RVW added as a FabricType. No rollbacks allowed once metadata with this fabric type is added without manual cleanups in databases.

Potential Downtime

Deprecations

Other Notable Change

0.13.1

Breaking Changes

• #9934 and #10075 - Stateful ingestion is now enabled by default if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified. It will still be disabled by default when any other sink type is used or if there is no pipeline name set.
• #10002 - The DataHubGraph client no longer makes a request to the backend during initialization. If you want to preserve the old behavior, call graph.test_connection() after constructing the client.
• #10026 - The dbt use_compiled_code option has been removed, because we now support capturing both source and compiled dbt SQL. This can be configured using include_compiled_code, which will be default enabled in 0.13.1.
• #10055 - Assertion entities generated by dbt are now associated with the dbt dataset entity, and not the entity in the data warehouse.
• #10090 - For Redshift ingestion, use_lineage_v2 is now enabled by default.
• #10147 - For looker ingestion, the browse paths for looker Dashboard, Chart, View, Explore have been updated to align with Looker UI. This does not affect URNs or lineage but primarily affects (improves) browsing experience.
• #10164 - For dbt ingestion, entities_enabled.model_performance and include_compiled_code are now both enabled by default. Upgrading dbt ingestion will also require upgrading the backend to 0.13.1.
• #10066 - For view access controls, SEARCH_AUTHORIZATION_ENABLED replaced by VIEW_AUTHORIZATION_ENABLED to more accurately represent the feature.
• #8231 - Google Analytics 3 has been fully sunsetted by Google as of July 2023, so we now support GA4 thanks to this PR and no longer support GA3 (which would have been broken since last year anyways).
• #10278 - Renaming Presto-On-Hive Source to Hive Metastore source to reflect better its purpose

Potential Downtime

Deprecations

Other Notable Changes

0.13.0

Breaking Changes

• Updating MySQL version for quickstarts to 8.2, may cause quickstart issues for existing instances.

• Neo4j 5.x, may require migration from 4.x

• Build requires JDK17 (Runtime Java 11)

• Build requires Docker Compose > 2.20

• #9731 - The acryl-datahub CLI now requires Python 3.8+

• #9601 - The Unity Catalog(UC) ingestion source config include_metastore is now disabled by default. This change will affect the urns of all entities in the workspace.
  Entity Hierarchy with include_metastore: true (Old)

  - UC Metastore
    - Catalog
      - Schema
        - Table

  Entity Hierarchy with include_metastore: false (New)

  - Catalog
    - Schema
      - Table

  We recommend using platform_instance for differentiating across metastores.

  If stateful ingestion is enabled, running ingestion with latest cli version will perform all required cleanup. Otherwise, we recommend soft deleting all databricks data via the DataHub CLI: datahub delete --platform databricks --soft and then reingesting with latest cli version.

• #9601 - The Unity Catalog(UC) ingestion source config include_hive_metastore is now enabled by default. This requires config warehouse_id to be set. You can disable include_hive_metastore by setting it to False to avoid ingesting legacy hive metastore catalog in Databricks.

• #9904 - The default Redshift table_lineage_mode is now MIXED, instead of STL_SCAN_BASED. Improved lineage generation is also available by enabling use_lineaege_v2. This v2 implementation will become the default in a future release.

Potential Downtime

Deprecations

• Spark 2.x (including previous JDK8 build requirements)

Other Notable Changes

0.12.1

Breaking Changes

• #9244: The redshift-legacy and redshift-legacy-usage sources, which have been deprecated for >6 months, have been removed. The new redshift source is a superset of the functionality provided by those legacy sources.
• database_alias config is no longer supported in SQL sources namely - Redshift, MySQL, Oracle, Postgres, Trino, Presto-on-hive. The config will automatically be ignored if it's present in your recipe. It has been deprecated since v0.9.6.
• #9257: The Python SDK urn types are now autogenerated. The new classes are largely backwards compatible with the previous, manually written classes, but many older methods are now deprecated in favor of a more uniform interface. The only breaking change is that the signature for the director constructor e.g. TagUrn("tag", ["tag_name"]) is no longer supported, and the simpler TagUrn("tag_name") should be used instead. The canonical place to import the urn classes from is datahub.metadata.urns.*. Other import paths, like datahub.utilities.urns.corpuser_urn.CorpuserUrn are retained for backwards compatibility, but are considered deprecated.
• #9286: The DataHubRestEmitter.emit method no longer returns anything. It previously returned a tuple of timestamps.
• #8951: A great expectations based profiler has been added for the Unity Catalog source. To use the old profiler, set method: analyze under the profiling section in your recipe. To use the new profiler, set method: ge. Profiling is disabled by default, so to enable it, one of these methods must be specified.

Potential Downtime

Deprecations

Other Notable Changes

0.12.0

Breaking Changes

• #8687 (datahub-helm #365 #353) - If Helm is used for installation and Neo4j is enabled, update the prerequisites Helm chart to version >=0.1.2 and adjust your value overrides in the neo4j: section according to the new structure.
• #9044 - GraphQL APIs for adding ownership now expect either an ownershipTypeUrn referencing a customer ownership type or a (deprecated) type. Where before adding an ownership without a concrete type was allowed, this is no longer the case. For simplicity you can use the type parameter which will get translated to a custom ownership type internally if one exists for the type being added.
• #9010 - In Redshift source's config incremental_lineage is set default to off.
• #8810 - Removed support for SQLAlchemy 1.3.x. Only SQLAlchemy 1.4.x is supported now.
• #8942 - Removed urn:li:corpuser:datahub owner for the Measure, Dimension and Temporal tags emitted by Looker and LookML source connectors.
• #8853 - The Airflow plugin no longer supports Airflow 2.0.x or Python 3.7. See the docs for more details.
• #8853 - Introduced the Airflow plugin v2. If you're using Airflow 2.3+, the v2 plugin will be enabled by default, and so you'll need to switch your requirements to include pip install 'acryl-datahub-airflow-plugin[plugin-v2]'. To continue using the v1 plugin, set the DATAHUB_AIRFLOW_PLUGIN_USE_V1_PLUGIN environment variable to true.
• #8943 - The Unity Catalog ingestion source has a new option include_metastore, which will cause all urns to be changed when disabled. This is currently enabled by default to preserve compatibility, but will be disabled by default and then removed in the future. If stateful ingestion is enabled, simply setting include_metastore: false will perform all required cleanup. Otherwise, we recommend soft deleting all databricks data via the DataHub CLI: datahub delete --platform databricks --soft and then reingesting with include_metastore: false.
• #8846 - Changed enum values in resource filters used by policies. RESOURCE_TYPE became TYPE and RESOURCE_URN became URN. Any existing policies using these filters (i.e. defined for particular urns or types such as dataset) need to be upgraded manually, for example by retrieving their respective dataHubPolicyInfo aspect and changing part using filter i.e.

   "resources": {
     "filter": {
       "criteria": [
         {
           "field": "RESOURCE_TYPE",
           "condition": "EQUALS",
           "values": [
             "dataset"
           ]
         }
       ]
     }

into

   "resources": {
     "filter": {
       "criteria": [
         {
           "field": "TYPE",
           "condition": "EQUALS",
           "values": [
             "dataset"
           ]
         }
       ]
     }

for example, using datahub put command. Policies can be also removed and re-created via UI.

• #9077 - The BigQuery ingestion source by default sets match_fully_qualified_names: true. This means that any dataset_pattern or schema_pattern specified will be matched on the fully qualified dataset name, i.e. <project_name>.<dataset_name>. We attempt to support the old pattern format by prepending .*\\. to dataset patterns lacking a period, so in most cases this should not cause any issues. However, if you have a complex dataset pattern, we recommend you manually convert it to the fully qualified format to avoid any potential issues.
• #9110 - The Unity Catalog source will now generate urns based on env properly. If you have been setting env in your recipe to something besides PROD, we will now generate urns with that new env variable, invalidating your existing urns.

Potential Downtime

Deprecations

Other Notable Changes

• Session token configuration has changed, all previously created session tokens will be invalid and users will be prompted to log in. Expiration time has also been shortened which may result in more login prompts with the default settings. There should be no other interruption due to this change.

0.11.0

Breaking Changes

Potential Downtime

• #8611 Search improvements requires reindexing indices. A system-update job will run which will set indices to read-only and create a backup/clone of each index. During the reindexing new components will be prevented from start-up until the reindex completes. The logs of this job will indicate a % complete per index. Depending on index sizes and infrastructure this process can take 5 minutes to hours however as a rough estimate 1 hour for every 2.3 million entities.

Deprecations

• #8525: In LDAP ingestor, the manager_pagination_enabled changed to general pagination_enabled
• MAE Events are no longer produced. MAE events have been deprecated for over a year.

Other Notable Changes

• In this release we now enable you to create and delete pinned announcements on your DataHub homepage! If you have the “Manage Home Page Posts” platform privilege you’ll see a new section in settings called “Home Page Posts” where you can create and delete text posts and link posts that your users see on the home page.
• The new search and browse experience, which was first made available in the previous release behind a feature flag, is now on by default. Check out our release notes for v0.10.5 to get more information and documentation on this new Browse experience.
• In addition to the ranking changes mentioned above, this release includes changes to the highlighting of search entities to understand why they match your query. You can also sort your results alphabetically or by last updated times, in addition to relevance. In this release, we suggest a correction if your query has a typo in it.
• #8300: Clickhouse source now inherited from TwoTierSQLAlchemy. In old way we have platform_instance -> container -> co container db (None) -> container schema and now we have platform_instance -> container database.
• #8300: Added uri_opts argument; now we can add any options for clickhouse client.
• #8659: BigQuery ingestion no longer creates DataPlatformInstance aspects by default. This will only affect users that were depending on this aspect for custom functionality, and can be enabled via the include_data_platform_instance config option.
• OpenAPI entity and aspect endpoints expanded to improve developer experience when using this API with additional aspects to be added in the near future.
• The CLI now supports recursive deletes.
• Batching of default aspects on initial ingestion (SQL)
• Improvements to multi-threading. Ingestion recipes, if previously reduced to 1 thread, can be restored to the 15 thread default.
• Gradle 7 upgrade moderately improves build speed
• DataHub Ingestion slim images reduced in size by 2GB+
• Glue Schema Registry fixed

0.10.5

Breaking Changes

• #8201: Python SDK: In the DataFlow class, the cluster argument is deprecated in favor of env.
• #8263: Okta source config option okta_profile_to_username_attr default changed from login to email. This determines which Okta profile attribute is used for the corresponding DataHub user and thus may change what DataHub users are generated by the Okta source. And in a follow up okta_profile_to_username_regex has been set to .* which taken together with previous change brings the defaults in line with OIDC.
• #8331: For all sql-based sources that support profiling, you can no longer specify profile_table_level_only together with include_field_xyz config options to ingest certain column-level metrics. Instead, set profile_table_level_only to false and individually enable / disable desired field metrics.
• #8451: The bigquery-beta and snowflake-beta source aliases have been dropped. Use bigquery and snowflake as the source type instead.
• #8472: Ingestion runs created with Pipeline.create will show up in the DataHub ingestion tab as CLI-based runs. To revert to the previous behavior of not showing these runs in DataHub, pass no_default_report=True.
• #8513: snowflake connector will use user's email attribute as is in urn. To revert to previous behavior disable email_as_user_identifier in recipe.

Potential Downtime

• BrowsePathsV2 upgrade will now be handled by the system-update job in non-blocking mode. This process generates data needed for the new search and browse feature. This process must complete before enabling the new search and browse UI and while upgrading entities will be missing from the UI. If not using the new search and browse UI, there will be no impact and the update will complete in the background.

Deprecations

• #8198: In the Python SDK, the PlatformKey class has been renamed to ContainerKey.

Other Notable Changes

0.10.5 introduces the new Unified Search & Browse experience and is disabled by default. You can control whether or not you want to see just the new search filtering experience, the new search and browse experience together, or keep the existing search and browse experiences by toggling the two environment variable feature flags SHOW_SEARCH_FILTERS_V2 and SHOW_BROWSE_V2 in your GMS container.

Upgrade Considerations:

• With the release of Browse V2, we have created a job to run in GMS that will backfill your existing data with new browsePathsV2 aspects. This job loops over entity types that need a browsePathsV2 aspect (Dataset, Dashboard, Chart, DataJob, DataFlow, MLModel, MLModelGroup, MLFeatureTable, and MLFeature) and generates one for them. For entities that may have Container parents (Datasets and Dashboards) we will try to fetch their parent containers in order to generate this new aspect. For those deployments with large amounts of data, consider whether running this upgrade job makes sense as it may be a heavy operation and take some time to complete. If you wish to skip this job, simply set the BACKFILL_BROWSE_PATHS_V2 environment variable flag to false in your GMS container. Without this backfill job, though, you will need to rely on the newest CLI of ingestion to create these browsePathsV2 aspects when running ingestion otherwise your browse sidebar will be out-of-sync.
• Since the new browse experience replaces the old, consider whether having the SHOW_BROWSE_V2 environment variable feature flag on is the right decision for your organization. If you’re creating custom browse paths with the browsePaths aspect, you can continue to do the same with the new experience, however you will have to generate browsePathsV2 aspects instead which are documented here.

0.10.4

Breaking Changes

Potential Downtime

Deprecations

• #8045: With the introduction of custom ownership types, the Owner aspect has been updated where the type field is deprecated in favor of a new field typeUrn. This latter field is an urn reference to the new OwnershipType entity. GraphQL endpoints have been updated to use the new field. For pre-existing ownership aspect records, DataHub now has logic to map the old field to the new field.

Other notable Changes

• #8191: Updates GMS's health check endpoint to account for its dependency on external components. Notably, at this time, elasticsearch. This means that DataHub operators can now use GMS health status more reliably.

0.10.3

Breaking Changes

• #7900: The catalog_pattern and schema_pattern options of the Unity Catalog source now match against the fully qualified name of the catalog/schema instead of just the name. Unless you're using regex ^ in your patterns, this should not affect you.
• #7942: Renaming the containerPath aspect to browsePathsV2. This means any data with the aspect name containerPath will be invalid. We had not exposed this in the UI or used it anywhere, but it was a model we recently merged to open up other work. This should not affect many people if anyone at all unless you were manually creating containerPath data through ingestion on your instance.
• #8068: In the datahub delete CLI, if an --entity-type filter is not specified, we automatically delete across all entity types. The previous behavior was to use a default entity type of dataset.
• #8068: In the datahub delete CLI, the --start-time and --end-time parameters are not required for timeseries aspect hard deletes. To recover the previous behavior of deleting all data, use --start-time min --end-time max.

Potential Downtime

Deprecations

• The signature of Source.get_workunits() is changed from Iterable[WorkUnit] to the more restrictive Iterable[MetadataWorkUnit].
• Legacy usage creation via the UsageAggregation aspect, /usageStats?action=batchIngest GMS endpoint, and UsageStatsWorkUnit metadata-ingestion class are all deprecated.

Other notable Changes

0.10.2

Breaking Changes

• #7016 Add add_database_name_to_urn flag to Oracle source which ensure that Dataset urns have the DB name as a prefix to prevent collision (.e.g. {database}.{schema}.{table}). ONLY breaking if you set this flag to true, otherwise behavior remains the same.
• The Airflow plugin no longer includes the DataHub Kafka emitter by default. Use pip install acryl-datahub-airflow-plugin[datahub-kafka] for Kafka support.
• The Airflow lineage backend no longer includes the DataHub Kafka emitter by default. Use pip install acryl-datahub[airflow,datahub-kafka] for Kafka support.
• Java SDK PatchBuilders have been modified in a backwards incompatible way to align more with the Python SDK and support more use cases. Any application utilizing the Java SDK for patch building may be affected on upgrading this dependency.

Deprecations

• The docker image and script for updating from Elasticsearch 6 to 7 is no longer being maintained and will be removed from the /contrib section of the repository. Please refer to older releases if needed.

0.10.0

Breaking Changes

• #7103 This should only impact users who have configured explicit non-default names for DataHub's Kafka topics. The environment variables used to configure Kafka topics for DataHub used in the kafka-setup docker image have been updated to be in-line with other DataHub components, for more info see our docs on Configuring Kafka in DataHub . They have been suffixed with _TOPIC where as now the correct suffix is _TOPIC_NAME. This change should not affect any user who is using default Kafka names.
• #6906 The Redshift source has been reworked and now also includes usage capabilities. The old Redshift source was renamed to redshift-legacy. The redshift-usage source has also been renamed to redshift-usage-legacy will be removed in the future.

Potential Downtime

• #6894 Search improvements requires reindexing indices. A system-update job will run which will set indices to read-only and create a backup/clone of each index. During the reindexing new components will be prevented from start-up until the reindex completes. The logs of this job will indicate a % complete per index. Depending on index sizes and infrastructure this process can take 5 minutes to hours however as a rough estimate 1 hour for every 2.3 million entities.

Helm Notes

Helm without --atomic: The default timeout for an upgrade command is 5 minutes. If the reindex takes longer (depending on data size) it will continue to run in the background even though helm will report a failure. Allow this job to finish and then re-run the helm upgrade command.

Helm with --atomic: In general, it is recommended to not use the --atomic setting for this particular upgrade since the system update job will be terminated before completion. If --atomic is preferred, then increase the timeout using the --timeout flag to account for the reindexing time (see note above for estimating this value).

Deprecations

0.9.6

Breaking Changes

• #6742 The metadata file sink's output format no longer contains nested JSON strings for MCP aspects, but instead unpacks the stringified JSON into a real JSON object. The previous sink behavior can be recovered using the legacy_nested_json_string option. The file source is backwards compatible and supports both formats.
• #6901 The env and database_alias fields have been marked deprecated across all sources. We recommend using platform_instance where possible instead.

Potential Downtime

Deprecations

• #6851 - Sources bigquery-legacy and bigquery-usage-legacy have been removed

Other notable Changes

• If anyone faces issues with login please clear your cookies. Some security updates are part of this release. That may cause login issues until cookies are cleared.

0.9.4 / 0.9.5

Breaking Changes

• #6243 apache-ranger authorizer is no longer the core part of DataHub GMS, and it is shifted as plugin. Please refer updated documentation Configuring Authorization with Apache Ranger for configuring apache-ranger-plugin in DataHub GMS.
• #6243 apache-ranger authorizer as plugin is not supported in DataHub Kubernetes deployment.
• #6243 Authentication and Authorization plugins configuration are removed from application.yaml. Refer documentation Migration Of Plugins From application.yaml for migrating any existing custom plugins.
• datahub check graph-consistency command has been removed. It was a beta API that we had considered but decided there are better solutions for this. So removing this.
• graphql_url option of powerbi-report-server source deprecated as the options is not used.
• #6789 BigQuery ingestion: If enable_legacy_sharded_table_support is set to False, sharded table names will be suffixed with _yyyymmdd to make sure they don't clash with non-sharded tables. This means if stateful ingestion is enabled then old sharded tables will be recreated with a new id and attached tags/glossary terms/etc will need to be added again. This behavior is not enabled by default yet, but will be enabled by default in a future release.

Potential Downtime

Deprecations

Other notable Changes

• #6611 - Snowflake schema_pattern now accepts pattern for fully qualified schema name in format <catalog_name>.<schema_name> by setting config match_fully_qualified_names : True. Current default match_fully_qualified_names: False is only to maintain backward compatibility. The config option match_fully_qualified_names will be deprecated in future and the default behavior will assume match_fully_qualified_names: True."
• #6636 - Sources snowflake-legacy and snowflake-usage-legacy have been removed.

0.9.3

Breaking Changes

• The beta datahub check graph-consistency command has been removed.

Potential Downtime

Deprecations

• PowerBI source: workspace_id_pattern is introduced in place of workspace_id. workspace_id is now deprecated and set for removal in a future version.

Other notable Changes

0.9.2

• LookML source will only emit views that are reachable from explores while scanning your git repo. Previous behavior can be achieved by setting emit_reachable_views_only to False.
• LookML source will always lowercase urns for lineage edges from views to upstream tables. There is no fallback provided to previous behavior because it was inconsistent in application of lower-casing earlier.
• dbt config node_type_pattern which was previously deprecated has been removed. Use entities_enabled instead to control whether to emit metadata for sources, models, seeds, tests, etc.
• The dbt source will always lowercase urns for lineage edges to the underlying data platform.
• The DataHub Airflow lineage backend and plugin no longer support Airflow 1.x. You can still run DataHub ingestion in Airflow 1.x using the PythonVirtualenvOperator.

Breaking Changes

• #6570 snowflake connector now populates created and last modified timestamps for snowflake datasets and containers. This version of snowflake connector will not work with datahub-gms version older than v0.9.3

Potential Downtime

Deprecations

Other notable Changes

0.9.1

Breaking Changes

• We have promoted bigquery-beta to bigquery. If you are using bigquery-beta then change your recipes to use the type bigquery.

Potential Downtime

Deprecations

Other notable Changes

0.9.0

Breaking Changes

• Java version 11 or greater is required.
• For any of the GraphQL search queries, the input no longer supports value but instead now accepts a list of values. These values represent an OR relationship where the field value must match any of the values.

Potential Downtime

Deprecations

Other notable Changes

v0.8.45

Breaking Changes

• The getNativeUserInviteToken and createNativeUserInviteToken GraphQL endpoints have been renamed to getInviteToken and createInviteToken respectively. Additionally, both now accept an optional roleUrn parameter. Both endpoints also now require the MANAGE_POLICIES privilege to execute, rather than MANAGE_USER_CREDENTIALS privilege.
• One of the default policies shipped with DataHub (urn:li:dataHubPolicy:7, or All Users - All Platform Privileges) has been edited to no longer include MANAGE_POLICIES. Its name has consequently been changed to All Users - All Platform Privileges (EXCEPT MANAGE POLICIES). This change was made to prevent all users from effectively acting as superusers by default.

Potential Downtime

Deprecations

Other notable Changes

v0.8.44

Breaking Changes

• Browse Paths have been upgraded to a new format to align more closely with the intention of the feature. Learn more about the changes, including steps on upgrading, here: https://docs.datahub.com/docs/advanced/browse-paths-upgrade
• The dbt ingestion source's disable_dbt_node_creation and load_schema options have been removed. They were no longer necessary due to the recently added sibling entities functionality.
• The snowflake source now uses newer faster implementation (earlier snowflake-beta). Config properties provision_role and check_role_grants are not supported. Older snowflake and snowflake-usage are available as snowflake-legacy and snowflake-usage-legacy sources respectively.

Potential Downtime

• [Helm] If you're using Helm, please ensure that your version of the datahub-actions container is bumped to v0.0.7 or head. This version contains changes to support running ingestion in debug mode. Previous versions are not compatible with this release. Upgrading to helm chart version 0.2.103 will ensure that you have the compatible versions by default.

Deprecations

Other notable Changes

v0.8.42

Breaking Changes

• Python 3.6 is no longer supported for metadata ingestion
• #5451 GMS_HOST and GMS_PORT environment variables deprecated in v0.8.39 have been removed. Use DATAHUB_GMS_HOST and DATAHUB_GMS_PORT instead.
• #5478 DataHub CLI delete command when used with --hard option will delete soft-deleted entities which match the other filters given.
• #5471 Looker now populates userEmail in dashboard user usage stats. This version of looker connector will not work with older version of datahub-gms if you have extract_usage_history looker config enabled.
• #5529 - ANALYTICS_ENABLED environment variable in datahub-gms is now deprecated. Use DATAHUB_ANALYTICS_ENABLED instead.
• #5485 --include-removed option was removed from delete CLI

Potential Downtime

Deprecations

Other notable Changes

v0.8.41

Breaking Changes

• The should_overwrite flag in csv-enricher has been replaced with write_semantics to match the format used for other sources. See the documentation for more details

• Closing an authorization hole in creating tags adding a Platform Privilege called Create Tags for creating tags. This is assigned to datahub root user, along with default All Users policy. Notice: You may need to add this privilege (or Manage Tags) to existing users that need the ability to create tags on the platform.

• #5329 Below profiling config parameters are now supported in BigQuery:

  • profiling.profile_if_updated_since_days (default=1)
  • profiling.profile_table_size_limit (default=1GB)
  • profiling.profile_table_row_limit (default=50000)

  Set above parameters to null if you want older behaviour.

Potential Downtime

Deprecations

Other notable Changes

v0.8.40

Breaking Changes

• #5240 lineage_client_project_id in bigquery source is removed. Use storage_project_id instead.

Potential Downtime

Deprecations

Other notable Changes

v0.8.39

Breaking Changes

• Refactored the health field of the Dataset GraphQL Type to be of type list of HealthStatus (was type HealthStatus). See this PR for more details.

Potential Downtime

Deprecations

• #4875 Lookml view file contents will no longer be populated in custom_properties, instead view definitions will be always available in the View Definitions tab.
• #5208 GMS_HOST and GMS_PORT environment variables being set in various containers are deprecated in favour of DATAHUB_GMS_HOST and DATAHUB_GMS_PORT.
• KAFKA_TOPIC_NAME environment variable in datahub-mae-consumer and datahub-gms is now deprecated. Use METADATA_AUDIT_EVENT_NAME instead.
• KAFKA_MCE_TOPIC_NAME environment variable in datahub-mce-consumer and datahub-gms is now deprecated. Use METADATA_CHANGE_EVENT_NAME instead.
• KAFKA_FMCE_TOPIC_NAME environment variable in datahub-mce-consumer and datahub-gms is now deprecated. Use FAILED_METADATA_CHANGE_EVENT_NAME instead.

Other notable Changes

• #5132 Profile tables in snowflake source only if they have been updated since configured (default: 1) number of day(s). Update the config profiling.profile_if_updated_since_days as per your profiling schedule or set it to None if you want older behaviour.

v0.8.38

Breaking Changes

Potential Downtime

Deprecations

Other notable Changes

• Create & Revoke Access Tokens via the UI
• Create and Manage new users via the UI
• Improvements to Business Glossary UI
• FIX - Do not require reindexing to migrate to using the UI business glossary

v0.8.36

Breaking Changes

• In this release we introduce a brand new Business Glossary experience. With this new experience comes some new ways of indexing data in order to make viewing and traversing the different levels of your Glossary possible. Therefore, you will have to restore your indices in order for the new Glossary experience to work for users that already have existing Glossaries. If this is your first time using DataHub Glossaries, you're all set!

Potential Downtime

Deprecations

Other notable Changes

• #4961 Dropped profiling is not reported by default as that caused a lot of spurious logging in some cases. Set profiling.report_dropped_profiles to True if you want older behaviour.

v0.8.35

Breaking Changes

Potential Downtime

Deprecations

• #4875 Lookml view file contents will no longer be populated in custom_properties, instead view definitions will be always available in the View Definitions tab.

Other notable Changes

v0.8.34

Breaking Changes

• #4644 Remove database option from snowflake source which was deprecated since v0.8.5
• #4595 Rename confusing config report_upstream_lineage to upstream_lineage_in_report in snowflake connector which was added in 0.8.32

Potential Downtime

Deprecations

• #4644 host_port option of snowflake and snowflake-usage sources deprecated as the name was confusing. Use account_id option instead.

Other notable Changes

• #4760 check_role_grants option was added in snowflake to disable checking roles in snowflake as some people were reporting long run times when checking roles.