PowerBI
Overview
Microsoft Power BI is a business intelligence and analytics platform. Learn more in the official Microsoft Power BI documentation.
The DataHub integration for Microsoft Power BI covers BI entities such as dashboards, charts, datasets, and related ownership context. It also captures table- and column-level lineage, data profiling, ownership, tags, and stateful deletion detection.
Concept Mapping
| PowerBI | Datahub |
|---|---|
Dashboard | Dashboard |
Dataset's Table | Dataset |
Tile | Chart |
Report.webUrl | Chart.externalUrl |
Workspace | Container |
Report | Dashboard |
PaginatedReport | Dashboard |
Page | Chart |
App | Dashboard |
- If
Tileis created from report thenChart.externalUrlis set to Report.webUrl. - The
Pageis unavailable for PowerBI PaginatedReport.
Module powerbi
Important Capabilities
| Capability | Status | Notes |
|---|---|---|
| Asset Containers | ✅ | Enabled by default. Supported for types - Workspace, Semantic Model. |
| Column-level Lineage | ✅ | Enabled by default, configured using extract_column_level_lineage. |
| Data Profiling | ✅ | Optionally enabled via configuration profiling.enabled. |
| Descriptions | ✅ | Enabled by default. |
| Detect Deleted Entities | ✅ | Enabled by default via stateful ingestion. |
| Extract Ownership | ✅ | Disabled by default, configured using extract_ownership. In metadata_api_mode: workspace_apis owners come from workspace roles. |
| Extract Tags | ✅ | Enabled by default. |
| Operation Capture | ✅ | Optionally enabled via extract_refresh_history: the latest dataset refresh as an Operation aspect. |
| Platform Instance | ✅ | Enabled by default. |
| Probe | ✅ | Check what a recipe would ingest before running it, with datahub recipe probe. |
| Schema Metadata | ✅ | Enabled by default. |
| Table-Level Lineage | ✅ | Enabled by default, configured using extract_lineage. In metadata_api_mode: workspace_apis lineage comes from the semantic-model definitions (TMDL partitions) without admin API access. |
| Test Connection | ✅ | Enabled by default. |
Overview
The powerbi module ingests metadata from Powerbi into DataHub. It is intended for production ingestion workflows and module-specific capabilities are documented below.
This plugin extracts the following:
- Power BI dashboards, tiles and datasets
- Names, descriptions and URLs of dashboard and tile
- Owners of dashboards
Prerequisites
In order to execute this source, you will need to have a Microsoft Entra Application service principal and grant permissions to it inside Power BI.
Power BI's APIs can be categorized into two sets of API methods, with different permission structures:
- Public APIs are designed for developers to interact with specific resources within a tenant, and require the Entra application to be explicitly granted access to individual Workspaces.
- The Admin APIs are designed for administrators to interact with the entire Power BI tenant at a high level, and return metadata on all Power BI resources.
The recommended way to execute Power BI ingestion is to do both: add your Entra application to the workspaces you want to ingest, andgrant it access to the public and Admin APIs. That way ingestion can extract the most metadata.
Public APIs ingestion
To grant public API access to your Entra application:
Grant permissions to access Fabric public APIs: Add your Entra Application's parent Entra Group under your Power BI/Fabric tenant settings in order to grant API access.
a. In Power BI or Fabric, go to
Settings->Admin portalb. In the
Admin portal, navigate toTenant settingsd. Under
Developer Settings, enable the optionService principals can call Fabric Public APIs(orAllow service principals to use Power BI APIsin older versions of Power BI), and add your application's Entra group underSpecific security groups.Add your Entra application as a member of your Power BI workspaces: For workspaces which you want to ingest into DataHub, add the Entra application as a member. For most cases
Viewerrole is enough, but for profiling theContributorrole is required.
If you have granted your Entra application permissions to the public APIs and added it as a member in a workspace, then the Power BI Source will be able to ingest the below metadata of that particular workspace:
- Dashboards
- Dashboard Tiles
- Reports
- Report Pages
If you don't want to add an Entra application as a member in your workspace, then you can enable admin_apis_only: true in your recipe to use the Power BI Admin API only. Caveats of setting admin_apis_only to true:
- Report Pages will not get ingested as the page API is not available in the Power BI Admin API
- Power BI Parameters will not get resolved to actual values while processing M-Query for table lineage
- Dataset profiling is unavailable, as it requires access to the non-admin workspace API
Admin APIs ingestion
To grant admin API access to the Entra application:
Grant permissions to access Admin APIs: Add your Entra Application's parent Entra Group under your Power BI/Fabric tenant settings in order to grant API access.
a. In Power BI or Fabric, go to
Settings->Admin portalb. In the
Admin portal, navigate toTenant settingsd. For each of the following options, enable the option and add your Entra application's Group under
Specific security groups:Service principals can access read-only admin APIsEnhance admin APIs responses with detailed metadataEnhance admin APIs responses with DAX and mashup expressions
If you have granted your Entra application permissions to the Admin APIs, then the Power BI Source will be able to ingest the below listed metadata of that particular workspace:
- Lineage
- Datasets
- Endorsement as tag
- Dashboards
- Dashboard Tiles
- Reports
- Report Pages
- App
Workspace APIs ingestion (no admin API access)
If granting tenant-level admin API access is not an option, set metadata_api_mode: workspace_apis. In this mode the connector never calls the Admin Scanner API; semantic-model schemas, M expressions and lineage come from the Fabric item-definition APIs (TMDL for semantic models, PBIR for reports) and ownership comes from workspace role assignments.
Requirements, all scoped to the workspaces you ingest:
| Requirement | Why |
|---|---|
Tenant setting Allow service principals to use Power BI APIs (Developer settings) | Every Power BI REST call (workspaces, datasets, reports, dashboards, refresh history, workspace users). |
Tenant setting Service principals can use Fabric APIs (Developer settings) | Fabric item listing and getDefinition for semantic models, reports and paginated reports. |
| Contributor role on each workspace (a security group containing the application is fine) | getDefinition and the full dataset listing need read and write item permission; Viewer only returns dataset ids and names. |
| The same role on workspaces that host shared semantic models used by your reports | Report-to-dataset lineage is resolved across workspaces only for datasets the principal can read. |
| No encrypted (protected) sensitivity labels on the semantic models and reports, or usage rights to remove them | Fabric refuses getDefinition for items with an encrypted label. |
environment: COMMERCIAL | The connector only knows the commercial Fabric API endpoint. |
If the principal only has the Viewer role on a workspace, the dataset listing still returns each dataset's id and name, so the workspace, its reports and dashboards are ingested and linked to their datasets, but getDefinition is refused: those datasets have no tables, columns, measures or upstream lineage, and the run reports a failure for each refused semantic-model definition (a refused report definition only loses visual field usage and is a warning). Grant Contributor on every workspace you expect lineage from.
No Microsoft Entra application permissions need to change: both API families are authorized through the tenant settings and workspace roles above. The application requests a second token audience (https://api.fabric.microsoft.com/.default) in addition to the Power BI one.
What workspace_apis mode ingests: workspaces, dashboards, tiles, reports, report pages, datasets with tables, columns, measures and their descriptions, M expressions, table and column lineage (including DirectLake lineage to Fabric Lakehouses and Warehouses), paginated-report dataset bindings, and owners derived from workspace roles.
What stays admin-only (the connector rejects these options together with workspace_apis): extract_endorsements_to_tags, filter_dataset_endorsements, extract_app, modified_since, admin_apis_only, and per-report or per-dashboard owners (owners are the workspace's members instead, filtered by ownership.owner_criteria against Admin, Member, Contributor and Viewer).
Install the Plugin
pip install 'acryl-datahub[powerbi]'
Starter Recipe
Check out the following recipe to get started with ingestion! See below for full configuration options.
For general pointers on writing and running a recipe, see our main recipe guide.
source:
type: "powerbi"
config:
# Your Power BI tenant identifier
tenant_id: a949d688-67c0-4bf1-a344-e939411c6c0a
# Microsoft Entra Application identifier
client_id: 12345678-abcd-abcd-abcd-123456789012
# Microsoft Entra Application client secret value
client_secret: Abc12d~efg3hijkl_45Abcdefg67aBcdef89
# Ingest elements of below PowerBi Workspace into Datahub
workspace_name_pattern:
allow:
- MyWorkspace
deny:
- PrivateWorkspace
# Optional -- ingest without tenant admin API access. Needs the Contributor role on
# each workspace and the tenant setting "Service principals can use Fabric APIs".
# Endorsements, apps and modified_since are not available in this mode: when you
# enable it, also remove modified_since and filter_dataset_endorsements below
# (and extract_app / extract_endorsements_to_tags if set), or the recipe is rejected.
# metadata_api_mode: workspace_apis
# Enrichment read from the semantic-model and report definitions (workspace_apis only)
# extract_model_relationships: true
# extract_report_field_usage: true
# Latest dataset refresh as an Operation aspect (works in both modes)
# extract_refresh_history: true
# Enable / Disable ingestion of ownership information for dashboards
extract_ownership: true
# Enable/Disable extracting workspace information to DataHub containers
extract_workspaces_to_containers: true
# Enable / Disable ingestion of endorsements.
# Please notice that this may overwrite any existing tags defined to ingested entities!
extract_endorsements_to_tags: false
# Optional -- This mapping is optional and only required to configure platform-instance for upstream tables
# A mapping of PowerBI datasource's server i.e host[:port] to data platform instance.
# :port is optional and only needed if your datasource server is running on non-standard port.
# For Google BigQuery the datasource's server is google bigquery project name.
# For Fabric OneLake (DirectLake lineage), use PowerBI workspace ID as the key.
# The platform_instance typically represents the Fabric tenant identifier, matching how
# the OneLake connector uses platform_instance to group workspaces by tenant.
server_to_platform_instance:
ap-south-1.snowflakecomputing.com:
platform_instance: operational_instance
env: DEV
oracle-server:1920:
platform_instance: high_performance_production_unit
env: PROD
big-query-sales-project:
platform_instance: sn-2
env: QA
# Fabric OneLake: workspace ID -> (platform_instance, env)
# platform_instance is typically the Fabric tenant identifier
ff23fbe3-7418-42f8-a675-9f10eb2b78cb: # PowerBI workspace ID
platform_instance: contoso-tenant # Fabric tenant/platform instance
env: PROD
# Need admin_api (not available with metadata_api_mode: workspace_apis), only ingest workspace that are modified since...
modified_since: "2023-02-10T00:00:00.0000000Z"
ownership:
# create powerbi user as datahub corpuser, false will still extract ownership of workspace/ dashboards
create_corp_user: false
# use email to build user urn instead of powerbi user identifier
use_powerbi_email: true
# remove email suffix like @acryl.io
remove_email_suffix: true
# only ingest user with certain authority
owner_criteria: ["ReadWriteReshareExplore","Owner","Admin"]
# wrap powerbi tables (datahub dataset) under 1 powerbi dataset (datahub container)
extract_datasets_to_containers: true
# only ingest dataset that are endorsed, like "Certified" (not available with metadata_api_mode: workspace_apis)
filter_dataset_endorsements:
allow:
- Certified
# extract powerbi dashboards and tiles
extract_dashboards: false
# extract powerbi dataset table schema
extract_dataset_schema: true
# Enable PowerBI dataset profiling
profiling:
enabled: false
# Pattern to limit which resources to profile
# Matched resource format is following:
# workspace_name.dataset_name.table_name
profile_pattern:
deny:
- .*
sink:
# sink configs
Config Details
- Options
- Schema
Note that a . is used to denote nested fields in the YAML recipe.
| Field | Description |
|---|---|
client_id ✅ string | Azure app client identifier |
client_secret ✅ string(password) | Azure app client secret |
tenant_id ✅ string | PowerBI tenant identifier |
admin_apis_only boolean | Retrieve metadata using PowerBI Admin API only. If this is enabled, then Report Pages will not be extracted. Admin API access is required if this setting is enabled Default: False |
app_url_pattern Enum | One of: "WORKSPACE_BASED", "REDIRECT_BASED" |
convert_lineage_urns_to_lowercase boolean | Whether to convert the urns of ingested lineage dataset to lowercase Default: True |
convert_urns_to_lowercase boolean | Whether to convert the PowerBI assets urns to lowercase Default: False |
dsn_to_database_schema map(str,string) | |
dsn_to_platform_name map(str,string) | |
enable_advance_lineage_sql_construct boolean | Whether to enable advance native sql construct for parsing like join, sub-queries. along this flag , the native_query_parsing should be enabled. By default convert_lineage_urns_to_lowercase is enabled, in-case if you have disabled it in previous ingestion execution then it may break lineageas this option generates the upstream datasets URN in lowercase. Default: True |
environment Enum | One of: "COMMERCIAL", "GOVERNMENT" |
extract_app boolean | Whether to ingest workspace app. Requires DataHub server 0.14.2+. Default: False |
extract_column_level_lineage boolean | Whether to extract column level lineage. Works only if configs native_query_parsing, enable_advance_lineage_sql_construct & extract_lineage are enabled. Works for M-Query where native SQL is used for transformation. Default: True |
extract_dashboards boolean | Whether to ingest PBI Dashboard and Tiles as Datahub Dashboard and Chart Default: True |
extract_dataset_schema boolean | Whether to ingest PBI Dataset Table columns and measures. Note: this setting must be true for schema extraction and column lineage to be enabled. Default: True |
extract_datasets_to_containers boolean | PBI tables will be grouped under a Datahub Container, the container reflect a PBI Dataset Default: False |
extract_endorsements_to_tags boolean | Whether to extract endorsements to tags, note that this may overwrite existing tags. Admin API access is required if this setting is enabled. Default: False |
extract_independent_datasets boolean | Whether to extract datasets not used in any PowerBI visualization Default: False |
extract_lineage boolean | Whether lineage should be ingested between X and Y. Admin API access is required if this setting is enabled, except with metadata_api_mode: workspace_apis, where lineage is read from the semantic-model definitions (TMDL partitions). Default: True |
extract_model_relationships boolean | Emit semantic-model relationships as foreign-key constraints on the from table's schema. Requires metadata_api_mode: workspace_apis and extract_dataset_schema. Default: False |
extract_ownership boolean | Whether ownership should be ingested. Admin API access is required if this setting is enabled, except with metadata_api_mode: workspace_apis, where owners are derived from workspace role assignments. Note that enabling this may overwrite owners that you've added inside DataHub's web application. Default: False |
extract_refresh_history boolean | Fetch each dataset's most recent refresh and emit it as an Operation aspect on the dataset's tables. Adds one API call per dataset; works in both metadata API modes and needs only the Viewer role, but uses the workspace API, so it cannot be combined with admin_apis_only. Default: False |
extract_report_field_usage boolean | Read each report's definition to record which model columns and measures its visuals use, emitted as inputFields on the page charts together with visual counts and types. Adds one Fabric getDefinition call per report. Requires metadata_api_mode: workspace_apis, extract_reports and extract_dataset_schema. Default: False |
extract_reports boolean | Whether reports should be ingested Default: True |
extract_workspaces_to_containers boolean | Extract workspaces to DataHub containers Default: True |
include_workspace_name_in_dataset_urn boolean | It is recommended to set this to true, as it helps prevent the overwriting of datasets.Read section #11560 at https://docs.datahub.com/docs/how/breaking-changes/ before enabling this option.To maintain backward compatibility, this is set to False. Default: False |
incremental_lineage boolean | When enabled, emits lineage as incremental to existing lineage already in DataHub. When disabled, re-states lineage on each run. Default: False |
m_query_parse_timeout integer | Timeout for PowerBI M-query parsing in seconds. Table-level lineage is determined by analyzing the M-query expression. Increase this value if you encounter the 'M-Query Parsing Timeout' message in the connector report. Default: 70 |
metadata_api_mode Enum | One of: "ADMIN_SCAN", "WORKSPACE_APIS" |
modified_since One of string, null | Get only recently modified workspaces based on modified_since datetime '2023-02-10T00:00:00.0000000Z', excludeInActiveWorkspaces limit to last 30 days Default: None |
native_query_parsing boolean | Whether PowerBI native query should be parsed to extract lineage Default: True |
patch_metadata boolean | Patch dashboard metadata Default: True |
platform_instance One of string, null | The instance of the platform that all assets produced by this recipe belong to Default: None |
scan_batch_size integer | batch size for sending workspace_ids to PBI, 100 is the limit Default: 1 |
scan_timeout integer | Seconds to wait for a PowerBI admin scan to complete, and, with metadata_api_mode: workspace_apis, for each Fabric getDefinition operation. Default: 60 |
workspace_id_as_urn_part boolean | It is recommended to set this to True only if you have legacy workspaces based on Office 365 groups, as those workspaces can have identical names. To maintain backward compatibility, this is set to False which uses workspace name Default: False |
env string | The environment that all assets produced by this connector belong to Default: PROD |
athena_table_platform_override array | List of platform overrides for Athena federated queries. Use this to override the platform when Athena queries data from federated sources (e.g., MySQL, PostgreSQL) via ODBC. The lineage will point to the actual source platform instead of Athena. This override is applied AFTER catalog stripping, so use 2-part names (database.table), not 3-part names (catalog.database.table). Overrides with a DSN specified take precedence over those without. Default: [] |
athena_table_platform_override.AthenaPlatformOverride AthenaPlatformOverride | Configuration for overriding the platform of Athena federated tables. Use this when Athena queries data from federated sources (e.g., MySQL, PostgreSQL) and you want the lineage to point to the actual source platform instead of Athena. |
athena_table_platform_override.AthenaPlatformOverride.database ❓ string | The database name in the Athena query (after catalog stripping). |
athena_table_platform_override.AthenaPlatformOverride.platform ❓ string | The target DataHub platform name (e.g., 'mysql', 'postgres'). |
athena_table_platform_override.AthenaPlatformOverride.table ❓ string | The table name in the Athena query. |
athena_table_platform_override.AthenaPlatformOverride.dsn One of string, null | Optional DSN to scope this override to a specific data source. If specified, this override only applies when the query comes from this DSN. Default: None |
bigquery_external_query_connection_to_platform map(str,BigQueryExternalQueryPlatformDetail) | Maps a BigQuery EXTERNAL_QUERY connection to the external source it federates to. BigQuery federation ( EXTERNAL_QUERY(connection, sql)) runs the inner SQL on an external engine such as Cloud SQL or AlloyDB (which expose MySQL/PostgreSQL). The connection id ( project.region.connection) does not reveal the external platform, so this mapping supplies it, letting PowerBI lineage point at the real upstream table on that platform instead of failing to resolve a URN. platform must be a recognized DataHub platform (see SupportedDataPlatform). |
bigquery_external_query_connection_to_platform. key.platform_instanceOne of string, null | DataHub platform instance name. To generate correct urn for upstream dataset, this should match with platform instance name used in ingestion recipe of other datahub sources. Default: None |
bigquery_external_query_connection_to_platform. key.platform ❓string | Target DataHub platform of the external source the EXTERNAL_QUERY federates to (e.g. 'postgres', 'mysql', 'snowflake'). |
bigquery_external_query_connection_to_platform. key.default_databaseOne of string, null | Database prepended to unqualified or 2-part table names in the federated (inner) SQL. The EXTERNAL_QUERY connection id does not carry the external database name, so set this when your external source's ingestion emits 3-part database.schema.table URNs so the lineage URNs match. Default: None |
bigquery_external_query_connection_to_platform. key.default_schemaOne of string, null | Schema applied to unqualified table references in the federated (inner) SQL. Default: None |
bigquery_external_query_connection_to_platform. key.envstring | The environment that all assets produced by DataHub platform ingestion source belong to Default: PROD |
filter_dataset_endorsements AllowDenyPattern | A class to store allow deny regexes. Patterns are matched against the start of the string only, not the entire string - a pattern does not need to match to the end to be considered a match. For example, the pattern "prod" matches "prod", "prod_east", and "production". To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
filter_dataset_endorsements.ignoreCase One of boolean, null | Whether to ignore case sensitivity during pattern matching. Default: True |
filter_dataset_endorsements.allow array | List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'. Default: ['.*'] |
filter_dataset_endorsements.allow.string string | |
filter_dataset_endorsements.deny array | List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'. Default: [] |
filter_dataset_endorsements.deny.string string | |
ownership OwnershipMapping | |
ownership.create_corp_user boolean | Whether to create user entities from PowerBI data. When False (RECOMMENDED): PowerBI emits ownership URNs only (soft references). User profiles must come from LDAP/SCIM/Okta. When True (OPT-IN): PowerBI creates users with displayName and email from PowerBI. WARNING: May overwrite existing user profiles from other sources. Use only if PowerBI is your authoritative user source. Default: True |
ownership.dataset_configured_by_as_owner boolean | Take PBI dataset configuredBy as dataset owner if exist Default: False |
ownership.remove_email_suffix boolean | Remove PowerBI User email suffix for example, @acryl.io Default: False |
ownership.use_powerbi_email boolean | Use PowerBI User email to ingest as corpuser, default is powerbi user identifier Default: True |
ownership.owner_criteria One of array, null | Need to have certain authority to qualify as owner for example ['ReadWriteReshareExplore','Owner','Admin'] Default: None |
ownership.owner_criteria.string string | |
profile_pattern AllowDenyPattern | A class to store allow deny regexes. Patterns are matched against the start of the string only, not the entire string - a pattern does not need to match to the end to be considered a match. For example, the pattern "prod" matches "prod", "prod_east", and "production". To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
profile_pattern.ignoreCase One of boolean, null | Whether to ignore case sensitivity during pattern matching. Default: True |
server_to_platform_instance One of map(str,union), map(str,union), map(str,union) | |
server_to_platform_instance. key.platform_instanceOne of string, null, union(anyOf), string, null, union(anyOf), string, null | DataHub platform instance name. To generate correct urn for upstream dataset, this should match with platform instance name used in ingestion recipe of other datahub sources. Default: None |
server_to_platform_instance. key.metastore ❓string | Databricks Unity Catalog metastore name. |
server_to_platform_instance. key.default_databaseOne of string, null | Database segment prepended to the table name when the Oracle.Database connection is a bare TNS alias or descriptor (which carries no database). Set this to match the database segment your Oracle ingestion uses, only when that ingestion emits 3-part database.schema.table URNs (add_database_name_to_urn: true); leave unset for the default 2-part URNs and for EZ-Connect host:port/service connections. Default: None |
server_to_platform_instance. key.default_schemaOne of string, null | Owner/schema applied to unqualified table references inside Oracle.Database(…, Query="…") inline native SQL, so they resolve to your ingested Oracle datasets. Not used by hierarchical navigation. Default: None |
server_to_platform_instance. key.envOne of string, string | The environment that all assets produced by DataHub platform ingestion source belong to Default: PROD |
workspace_id_pattern AllowDenyPattern | A class to store allow deny regexes. Patterns are matched against the start of the string only, not the entire string - a pattern does not need to match to the end to be considered a match. For example, the pattern "prod" matches "prod", "prod_east", and "production". To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
workspace_id_pattern.ignoreCase One of boolean, null | Whether to ignore case sensitivity during pattern matching. Default: True |
workspace_name_pattern AllowDenyPattern | A class to store allow deny regexes. Patterns are matched against the start of the string only, not the entire string - a pattern does not need to match to the end to be considered a match. For example, the pattern "prod" matches "prod", "prod_east", and "production". To require an exact match, anchor your pattern explicitly, e.g. "^prod$". |
workspace_name_pattern.ignoreCase One of boolean, null | Whether to ignore case sensitivity during pattern matching. Default: True |
workspace_type_filter array | Ingest the metadata of the workspace where the workspace type corresponds to the specified workspace_type_filter. Note: This field works in conjunction with 'workspace_id_pattern'. Both must be matched for a workspace to be processed. Default: ['Workspace'] |
workspace_type_filter.enum Enum | One of: "Workspace", "PersonalGroup", "Personal", "AdminWorkspace", "AdminInsights" |
profiling PowerBiProfilingConfig | |
profiling.enabled boolean | Whether profiling of PowerBI datasets should be done Default: False |
stateful_ingestion One of StatefulStaleMetadataRemovalConfig, null | PowerBI Stateful Ingestion Config. Default: None |
stateful_ingestion.enabled boolean | Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False Default: False |
stateful_ingestion.fail_safe_threshold number | Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'. Default: 75.0 |
stateful_ingestion.remove_stale_metadata boolean | Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled. Default: True |
The JSONSchema for this configuration is inlined below.
{
"$defs": {
"AllowDenyPattern": {
"additionalProperties": false,
"description": "A class to store allow deny regexes.\n\nPatterns are matched against the start of the string only, not the entire\nstring - a pattern does not need to match to the end to be considered a match.\nFor example, the pattern \"prod\" matches \"prod\", \"prod_east\", and \"production\".\nTo require an exact match, anchor your pattern explicitly, e.g. \"^prod$\".",
"properties": {
"allow": {
"default": [
".*"
],
"description": "List of regex patterns to include in ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.",
"items": {
"type": "string"
},
"title": "Allow",
"type": "array"
},
"deny": {
"default": [],
"description": "List of regex patterns to exclude from ingestion. Patterns match from the start of the string only, not the entire string - anchor with '^...$' for an exact match, e.g. '^prod$'.",
"items": {
"type": "string"
},
"title": "Deny",
"type": "array"
},
"ignoreCase": {
"anyOf": [
{
"type": "boolean"
},
{
"type": "null"
}
],
"default": true,
"description": "Whether to ignore case sensitivity during pattern matching.",
"title": "Ignorecase"
}
},
"title": "AllowDenyPattern",
"type": "object"
},
"AthenaPlatformOverride": {
"additionalProperties": false,
"description": "Configuration for overriding the platform of Athena federated tables.\n\nUse this when Athena queries data from federated sources (e.g., MySQL, PostgreSQL)\nand you want the lineage to point to the actual source platform instead of Athena.",
"properties": {
"database": {
"description": "The database name in the Athena query (after catalog stripping).",
"minLength": 1,
"title": "Database",
"type": "string"
},
"table": {
"description": "The table name in the Athena query.",
"minLength": 1,
"title": "Table",
"type": "string"
},
"platform": {
"description": "The target DataHub platform name (e.g., 'mysql', 'postgres').",
"minLength": 1,
"title": "Platform",
"type": "string"
},
"dsn": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Optional DSN to scope this override to a specific data source. If specified, this override only applies when the query comes from this DSN.",
"title": "Dsn"
}
},
"required": [
"database",
"table",
"platform"
],
"title": "AthenaPlatformOverride",
"type": "object"
},
"BigQueryExternalQueryPlatformDetail": {
"additionalProperties": false,
"description": "Maps a BigQuery ``EXTERNAL_QUERY`` connection to the external source it federates to.\n\nBigQuery federation (``EXTERNAL_QUERY(connection, sql)``) runs the inner SQL on an\nexternal engine such as Cloud SQL or AlloyDB (which expose MySQL/PostgreSQL). The\nconnection id (``project.region.connection``) does not reveal the external platform,\nso this mapping supplies it, letting PowerBI lineage point at the real upstream table\non that platform instead of failing to resolve a URN. ``platform`` must be a\nrecognized DataHub platform (see ``SupportedDataPlatform``).",
"properties": {
"platform_instance": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "DataHub platform instance name. To generate correct urn for upstream dataset, this should match with platform instance name used in ingestion recipe of other datahub sources.",
"title": "Platform Instance"
},
"env": {
"default": "PROD",
"description": "The environment that all assets produced by DataHub platform ingestion source belong to",
"title": "Env",
"type": "string"
},
"platform": {
"description": "Target DataHub platform of the external source the EXTERNAL_QUERY federates to (e.g. 'postgres', 'mysql', 'snowflake').",
"minLength": 1,
"title": "Platform",
"type": "string"
},
"default_database": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Database prepended to unqualified or 2-part table names in the federated (inner) SQL. The EXTERNAL_QUERY connection id does not carry the external database name, so set this when your external source's ingestion emits 3-part database.schema.table URNs so the lineage URNs match.",
"title": "Default Database"
},
"default_schema": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Schema applied to unqualified table references in the federated (inner) SQL.",
"title": "Default Schema"
}
},
"required": [
"platform"
],
"title": "BigQueryExternalQueryPlatformDetail",
"type": "object"
},
"DataBricksPlatformDetail": {
"additionalProperties": false,
"description": "metastore is an additional field used in Databricks connector to generate the dataset urn",
"properties": {
"platform_instance": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "DataHub platform instance name. To generate correct urn for upstream dataset, this should match with platform instance name used in ingestion recipe of other datahub sources.",
"title": "Platform Instance"
},
"env": {
"default": "PROD",
"description": "The environment that all assets produced by DataHub platform ingestion source belong to",
"title": "Env",
"type": "string"
},
"metastore": {
"description": "Databricks Unity Catalog metastore name.",
"title": "Metastore",
"type": "string"
}
},
"required": [
"metastore"
],
"title": "DataBricksPlatformDetail",
"type": "object"
},
"OraclePlatformDetail": {
"additionalProperties": false,
"properties": {
"platform_instance": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "DataHub platform instance name. To generate correct urn for upstream dataset, this should match with platform instance name used in ingestion recipe of other datahub sources.",
"title": "Platform Instance"
},
"env": {
"default": "PROD",
"description": "The environment that all assets produced by DataHub platform ingestion source belong to",
"title": "Env",
"type": "string"
},
"default_schema": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Owner/schema applied to unqualified table references inside ``Oracle.Database(\u2026, Query=\"\u2026\")`` inline native SQL, so they resolve to your ingested Oracle datasets. Not used by hierarchical navigation.",
"title": "Default Schema"
},
"default_database": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Database segment prepended to the table name when the ``Oracle.Database`` connection is a bare TNS alias or descriptor (which carries no database). Set this to match the database segment your Oracle ingestion uses, only when that ingestion emits 3-part ``database.schema.table`` URNs (``add_database_name_to_urn: true``); leave unset for the default 2-part URNs and for EZ-Connect ``host:port/service`` connections.",
"title": "Default Database"
}
},
"title": "OraclePlatformDetail",
"type": "object"
},
"OwnershipMapping": {
"additionalProperties": false,
"properties": {
"create_corp_user": {
"default": true,
"description": "Whether to create user entities from PowerBI data. When False (RECOMMENDED): PowerBI emits ownership URNs only (soft references). User profiles must come from LDAP/SCIM/Okta. When True (OPT-IN): PowerBI creates users with displayName and email from PowerBI. WARNING: May overwrite existing user profiles from other sources. Use only if PowerBI is your authoritative user source.",
"title": "Create Corp User",
"type": "boolean"
},
"use_powerbi_email": {
"default": true,
"description": "Use PowerBI User email to ingest as corpuser, default is powerbi user identifier",
"title": "Use Powerbi Email",
"type": "boolean"
},
"remove_email_suffix": {
"default": false,
"description": "Remove PowerBI User email suffix for example, @acryl.io",
"title": "Remove Email Suffix",
"type": "boolean"
},
"dataset_configured_by_as_owner": {
"default": false,
"description": "Take PBI dataset configuredBy as dataset owner if exist",
"title": "Dataset Configured By As Owner",
"type": "boolean"
},
"owner_criteria": {
"anyOf": [
{
"items": {
"type": "string"
},
"type": "array"
},
{
"type": "null"
}
],
"default": null,
"description": "Need to have certain authority to qualify as owner for example ['ReadWriteReshareExplore','Owner','Admin']",
"title": "Owner Criteria"
}
},
"title": "OwnershipMapping",
"type": "object"
},
"PlatformDetail": {
"additionalProperties": false,
"properties": {
"platform_instance": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "DataHub platform instance name. To generate correct urn for upstream dataset, this should match with platform instance name used in ingestion recipe of other datahub sources.",
"title": "Platform Instance"
},
"env": {
"default": "PROD",
"description": "The environment that all assets produced by DataHub platform ingestion source belong to",
"title": "Env",
"type": "string"
}
},
"title": "PlatformDetail",
"type": "object"
},
"PowerBiAppUrlPattern": {
"enum": [
"WORKSPACE_BASED",
"REDIRECT_BASED"
],
"title": "PowerBiAppUrlPattern",
"type": "string"
},
"PowerBiEnvironment": {
"enum": [
"COMMERCIAL",
"GOVERNMENT"
],
"title": "PowerBiEnvironment",
"type": "string"
},
"PowerBiMetadataApiMode": {
"enum": [
"ADMIN_SCAN",
"WORKSPACE_APIS"
],
"title": "PowerBiMetadataApiMode",
"type": "string"
},
"PowerBiProfilingConfig": {
"additionalProperties": false,
"properties": {
"enabled": {
"default": false,
"description": "Whether profiling of PowerBI datasets should be done",
"title": "Enabled",
"type": "boolean"
}
},
"title": "PowerBiProfilingConfig",
"type": "object"
},
"StatefulStaleMetadataRemovalConfig": {
"additionalProperties": false,
"description": "Base specialized config for Stateful Ingestion with stale metadata removal capability.",
"properties": {
"enabled": {
"default": false,
"description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False",
"title": "Enabled",
"type": "boolean"
},
"remove_stale_metadata": {
"default": true,
"description": "Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.",
"title": "Remove Stale Metadata",
"type": "boolean"
},
"fail_safe_threshold": {
"default": 75.0,
"description": "Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.",
"maximum": 100.0,
"minimum": 0.0,
"title": "Fail Safe Threshold",
"type": "number"
}
},
"title": "StatefulStaleMetadataRemovalConfig",
"type": "object"
}
},
"additionalProperties": false,
"properties": {
"incremental_lineage": {
"default": false,
"description": "When enabled, emits lineage as incremental to existing lineage already in DataHub. When disabled, re-states lineage on each run.",
"title": "Incremental Lineage",
"type": "boolean"
},
"env": {
"default": "PROD",
"description": "The environment that all assets produced by this connector belong to",
"title": "Env",
"type": "string"
},
"platform_instance": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "The instance of the platform that all assets produced by this recipe belong to",
"title": "Platform Instance"
},
"stateful_ingestion": {
"anyOf": [
{
"$ref": "#/$defs/StatefulStaleMetadataRemovalConfig"
},
{
"type": "null"
}
],
"default": null,
"description": "PowerBI Stateful Ingestion Config."
},
"environment": {
"$ref": "#/$defs/PowerBiEnvironment",
"default": "COMMERCIAL",
"description": "PowerBI environment to connect to. Options: 'commercial' (default) for commercial PowerBI, 'government' for PowerBI Government Community Cloud (GCC)"
},
"tenant_id": {
"description": "PowerBI tenant identifier",
"title": "Tenant Id",
"type": "string"
},
"workspace_id_pattern": {
"$ref": "#/$defs/AllowDenyPattern",
"default": {
"allow": [
".*"
],
"deny": [],
"ignoreCase": true
},
"description": "Regex patterns to filter PowerBI workspaces in ingestion by ID. By default all IDs are allowed unless they are filtered by name using 'workspace_name_pattern'. Note: This field works in conjunction with 'workspace_type_filter' and both must be considered when filtering workspaces."
},
"workspace_name_pattern": {
"$ref": "#/$defs/AllowDenyPattern",
"default": {
"allow": [
".*"
],
"deny": [],
"ignoreCase": true
},
"description": "Regex patterns to filter PowerBI workspaces in ingestion by name. By default all names are allowed unless they are filtered by ID using 'workspace_id_pattern'. Note: This field works in conjunction with 'workspace_type_filter' and both must be considered when filtering workspaces."
},
"server_to_platform_instance": {
"additionalProperties": {
"anyOf": [
{
"$ref": "#/$defs/OraclePlatformDetail"
},
{
"$ref": "#/$defs/DataBricksPlatformDetail"
},
{
"$ref": "#/$defs/PlatformDetail"
}
]
},
"default": {},
"description": "Mapping from a PowerBI datasource server to the DataHub platform instance (and env) of its upstream tables, so lineage URNs match your other DataHub sources. The key is the server as it appears in the M-query, i.e. `host[:port]` (`:port` only for non-standard ports); for Google BigQuery it is the project name, for Databricks Unity Catalog the workspace FQDN, and for Oracle the EZ-Connect host, bare TNS alias, or descriptor SERVICE_NAME (case-insensitive). The value is a platform-detail object; Oracle servers may add `default_schema`/`default_database` and Databricks servers `metastore`.",
"title": "Server To Platform Instance",
"type": "object"
},
"dsn_to_platform_name": {
"additionalProperties": {
"type": "string"
},
"default": {},
"description": "A mapping of ODBC DSN to DataHub data platform name. For example with an ODBC connection string 'DSN=database' where the database type is 'PostgreSQL' you would configure the mapping as 'database: postgres'.",
"title": "Dsn To Platform Name",
"type": "object"
},
"dsn_to_database_schema": {
"additionalProperties": {
"type": "string"
},
"default": {},
"description": "A mapping of ODBC DSN to database names with optional schema names (some database platforms such a MySQL use the table name pattern 'database.table', while others use the pattern 'database.schema.table'). This mapping is used in conjunction with ODBC SQL query parsing. If SQL queries used with ODBC do not reference fully qualified tables names, then you should configure mappings for your DSNs. For example with an ODBC connection string 'DSN=database' where the database is 'prod' you would configure the mapping as 'database: prod'. If the database is 'prod' and the schema is 'data' then mapping would be 'database: prod.data'.",
"title": "Dsn To Database Schema",
"type": "object"
},
"athena_table_platform_override": {
"default": [],
"description": "List of platform overrides for Athena federated queries. Use this to override the platform when Athena queries data from federated sources (e.g., MySQL, PostgreSQL) via ODBC. The lineage will point to the actual source platform instead of Athena. This override is applied AFTER catalog stripping, so use 2-part names (database.table), not 3-part names (catalog.database.table). Overrides with a DSN specified take precedence over those without.",
"items": {
"$ref": "#/$defs/AthenaPlatformOverride"
},
"title": "Athena Table Platform Override",
"type": "array"
},
"bigquery_external_query_connection_to_platform": {
"additionalProperties": {
"$ref": "#/$defs/BigQueryExternalQueryPlatformDetail"
},
"default": {},
"description": "Mapping from a BigQuery ``EXTERNAL_QUERY`` connection id (``project.region.connection``, the first argument of EXTERNAL_QUERY) to the external source it federates to. BigQuery federation runs the inner SQL on an external engine such as Cloud SQL or AlloyDB (which expose MySQL/PostgreSQL); configure this so PowerBI lineage resolves to the real upstream table on that platform instead of failing. The value sets the target `platform` (required, must be one of `athena`, `bigquery`, `databricks`, `fabric-onelake`, `hive`, `mssql`, `mysql`, `odbc`, `oracle`, `postgres`, `redshift` or `snowflake`, and its PowerBI name must remain in `dataset_type_mapping` if you narrow that mapping) plus optional `platform_instance`, `env`, `default_database`, and `default_schema`. Requires `extract_lineage`, `native_query_parsing`, and `enable_advance_lineage_sql_construct`. If the outer native query cannot be parsed, lineage is skipped for the whole table (native upstreams included), not just the federated part.",
"title": "Bigquery External Query Connection To Platform",
"type": "object"
},
"client_id": {
"description": "Azure app client identifier",
"title": "Client Id",
"type": "string"
},
"client_secret": {
"description": "Azure app client secret",
"format": "password",
"title": "Client Secret",
"type": "string",
"writeOnly": true
},
"scan_timeout": {
"default": 60,
"description": "Seconds to wait for a PowerBI admin scan to complete, and, with `metadata_api_mode: workspace_apis`, for each Fabric getDefinition operation.",
"title": "Scan Timeout",
"type": "integer"
},
"scan_batch_size": {
"default": 1,
"description": "batch size for sending workspace_ids to PBI, 100 is the limit",
"exclusiveMinimum": 0,
"maximum": 100,
"title": "Scan Batch Size",
"type": "integer"
},
"workspace_id_as_urn_part": {
"default": false,
"description": "It is recommended to set this to True only if you have legacy workspaces based on Office 365 groups, as those workspaces can have identical names. To maintain backward compatibility, this is set to False which uses workspace name",
"title": "Workspace Id As Urn Part",
"type": "boolean"
},
"extract_ownership": {
"default": false,
"description": "Whether ownership should be ingested. Admin API access is required if this setting is enabled, except with `metadata_api_mode: workspace_apis`, where owners are derived from workspace role assignments. Note that enabling this may overwrite owners that you've added inside DataHub's web application.",
"title": "Extract Ownership",
"type": "boolean"
},
"extract_reports": {
"default": true,
"description": "Whether reports should be ingested",
"title": "Extract Reports",
"type": "boolean"
},
"ownership": {
"$ref": "#/$defs/OwnershipMapping",
"default": {
"create_corp_user": true,
"use_powerbi_email": true,
"remove_email_suffix": false,
"dataset_configured_by_as_owner": false,
"owner_criteria": null
},
"description": "Configure how is ownership ingested"
},
"modified_since": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "Get only recently modified workspaces based on modified_since datetime '2023-02-10T00:00:00.0000000Z', excludeInActiveWorkspaces limit to last 30 days",
"title": "Modified Since"
},
"extract_dashboards": {
"default": true,
"description": "Whether to ingest PBI Dashboard and Tiles as Datahub Dashboard and Chart",
"title": "Extract Dashboards",
"type": "boolean"
},
"extract_dataset_schema": {
"default": true,
"description": "Whether to ingest PBI Dataset Table columns and measures. Note: this setting must be `true` for schema extraction and column lineage to be enabled.",
"title": "Extract Dataset Schema",
"type": "boolean"
},
"extract_lineage": {
"default": true,
"description": "Whether lineage should be ingested between X and Y. Admin API access is required if this setting is enabled, except with `metadata_api_mode: workspace_apis`, where lineage is read from the semantic-model definitions (TMDL partitions).",
"title": "Extract Lineage",
"type": "boolean"
},
"extract_endorsements_to_tags": {
"default": false,
"description": "Whether to extract endorsements to tags, note that this may overwrite existing tags. Admin API access is required if this setting is enabled.",
"title": "Extract Endorsements To Tags",
"type": "boolean"
},
"filter_dataset_endorsements": {
"$ref": "#/$defs/AllowDenyPattern",
"default": {
"allow": [
".*"
],
"deny": [],
"ignoreCase": true
},
"description": "Filter and ingest datasets which are 'Certified' or 'Promoted' endorsement. If both are added, dataset which are 'Certified' or 'Promoted' will be ingested . Default setting allows all dataset to be ingested"
},
"extract_workspaces_to_containers": {
"default": true,
"description": "Extract workspaces to DataHub containers",
"title": "Extract Workspaces To Containers",
"type": "boolean"
},
"extract_datasets_to_containers": {
"default": false,
"description": "PBI tables will be grouped under a Datahub Container, the container reflect a PBI Dataset",
"title": "Extract Datasets To Containers",
"type": "boolean"
},
"native_query_parsing": {
"default": true,
"description": "Whether PowerBI native query should be parsed to extract lineage",
"title": "Native Query Parsing",
"type": "boolean"
},
"convert_urns_to_lowercase": {
"default": false,
"description": "Whether to convert the PowerBI assets urns to lowercase",
"title": "Convert Urns To Lowercase",
"type": "boolean"
},
"convert_lineage_urns_to_lowercase": {
"default": true,
"description": "Whether to convert the urns of ingested lineage dataset to lowercase",
"title": "Convert Lineage Urns To Lowercase",
"type": "boolean"
},
"admin_apis_only": {
"default": false,
"description": "Retrieve metadata using PowerBI Admin API only. If this is enabled, then Report Pages will not be extracted. Admin API access is required if this setting is enabled",
"title": "Admin Apis Only",
"type": "boolean"
},
"extract_independent_datasets": {
"default": false,
"description": "Whether to extract datasets not used in any PowerBI visualization",
"title": "Extract Independent Datasets",
"type": "boolean"
},
"enable_advance_lineage_sql_construct": {
"default": true,
"description": "Whether to enable advance native sql construct for parsing like join, sub-queries. along this flag , the native_query_parsing should be enabled. By default convert_lineage_urns_to_lowercase is enabled, in-case if you have disabled it in previous ingestion execution then it may break lineageas this option generates the upstream datasets URN in lowercase.",
"title": "Enable Advance Lineage Sql Construct",
"type": "boolean"
},
"extract_column_level_lineage": {
"default": true,
"description": "Whether to extract column level lineage. Works only if configs `native_query_parsing`, `enable_advance_lineage_sql_construct` & `extract_lineage` are enabled. Works for M-Query where native SQL is used for transformation.",
"title": "Extract Column Level Lineage",
"type": "boolean"
},
"profile_pattern": {
"$ref": "#/$defs/AllowDenyPattern",
"default": {
"allow": [
".*"
],
"deny": [],
"ignoreCase": true
},
"description": "Regex patterns to filter tables for profiling during ingestion. Note that only tables allowed by the `table_pattern` will be considered. Matched format is 'workspacename.datasetname.tablename'"
},
"profiling": {
"$ref": "#/$defs/PowerBiProfilingConfig",
"default": {
"enabled": false
}
},
"patch_metadata": {
"default": true,
"description": "Patch dashboard metadata",
"title": "Patch Metadata",
"type": "boolean"
},
"workspace_type_filter": {
"default": [
"Workspace"
],
"description": "Ingest the metadata of the workspace where the workspace type corresponds to the specified workspace_type_filter. Note: This field works in conjunction with 'workspace_id_pattern'. Both must be matched for a workspace to be processed.",
"items": {
"enum": [
"Workspace",
"PersonalGroup",
"Personal",
"AdminWorkspace",
"AdminInsights"
],
"type": "string"
},
"title": "Workspace Type Filter",
"type": "array"
},
"include_workspace_name_in_dataset_urn": {
"default": false,
"description": "It is recommended to set this to true, as it helps prevent the overwriting of datasets.Read section #11560 at https://docs.datahub.com/docs/how/breaking-changes/ before enabling this option.To maintain backward compatibility, this is set to False.",
"title": "Include Workspace Name In Dataset Urn",
"type": "boolean"
},
"extract_app": {
"default": false,
"description": "Whether to ingest workspace app. Requires DataHub server 0.14.2+.",
"title": "Extract App",
"type": "boolean"
},
"app_url_pattern": {
"$ref": "#/$defs/PowerBiAppUrlPattern",
"default": "WORKSPACE_BASED",
"description": "URL pattern for Power BI App external links. 'workspace_based' uses /groups/{workspace-id}/apps/{app-id} (default). 'redirect_based' uses /Redirect?action=OpenApp&appId={app-id}."
},
"m_query_parse_timeout": {
"default": 70,
"description": "Timeout for PowerBI M-query parsing in seconds. Table-level lineage is determined by analyzing the M-query expression. Increase this value if you encounter the 'M-Query Parsing Timeout' message in the connector report.",
"title": "M Query Parse Timeout",
"type": "integer"
},
"metadata_api_mode": {
"$ref": "#/$defs/PowerBiMetadataApiMode",
"default": "ADMIN_SCAN",
"description": "Which API family supplies semantic-model metadata (tables, columns, measures, M expressions, lineage). `admin_scan` (default) uses the Admin Scanner API and requires tenant-level admin API access. `workspace_apis` instead reads semantic-model (TMDL) and report definitions through the Fabric item APIs and needs only the Contributor role on each ingested workspace plus the tenant setting 'Service principals can use Fabric APIs'. In `workspace_apis` mode endorsements, apps, `modified_since` and per-report owners are unavailable (ownership comes from workspace roles), and the `environment` must be COMMERCIAL."
},
"extract_model_relationships": {
"default": false,
"description": "Emit semantic-model relationships as foreign-key constraints on the `from` table's schema. Requires `metadata_api_mode: workspace_apis` and `extract_dataset_schema`.",
"title": "Extract Model Relationships",
"type": "boolean"
},
"extract_report_field_usage": {
"default": false,
"description": "Read each report's definition to record which model columns and measures its visuals use, emitted as `inputFields` on the page charts together with visual counts and types. Adds one Fabric getDefinition call per report. Requires `metadata_api_mode: workspace_apis`, `extract_reports` and `extract_dataset_schema`.",
"title": "Extract Report Field Usage",
"type": "boolean"
},
"extract_refresh_history": {
"default": false,
"description": "Fetch each dataset's most recent refresh and emit it as an Operation aspect on the dataset's tables. Adds one API call per dataset; works in both metadata API modes and needs only the Viewer role, but uses the workspace API, so it cannot be combined with `admin_apis_only`.",
"title": "Extract Refresh History",
"type": "boolean"
}
},
"required": [
"tenant_id",
"client_id",
"client_secret"
],
"title": "PowerBiDashboardSourceConfig",
"type": "object"
}
Capabilities
Use the Important Capabilities table above as the source of truth for supported features and whether additional configuration is required.
User and Ownership Handling
PowerBI Source supports two modes for handling user ownership:
Soft References Mode (Recommended - Default)
When ownership.create_corp_user: false (default), PowerBI will:
- Extract ownership information as URN references only
- NOT create user entities in DataHub
- User profiles must come from your identity provider (LDAP/SCIM/Okta)
This is the recommended approach as it prevents PowerBI from overwriting user profiles from your identity provider.
ownership:
create_corp_user: false # Default - soft references only
Full User Creation Mode (Opt-in)
When ownership.create_corp_user: true, PowerBI will:
- Create user entities with
displayNameandemailfrom PowerBI - This may overwrite existing user profiles from LDAP/Okta/SCIM
Warning: Only use this if PowerBI is your authoritative source for user information.
ownership:
create_corp_user: true # Opt-in - creates user entities
Filtering Owners by Access Rights
You can limit which users become owners using owner_criteria. Only users with at least one of the specified access rights will be assigned as owners:
ownership:
owner_criteria:
- ReadWriteReshareExplore
- Owner
- Admin
Valid values depend on the PowerBI access right types for your resources (e.g., dataset, report, dashboard). If owner_criteria is not set or is an empty list, all users with principalType: User qualify as owners.
With metadata_api_mode: workspace_apis, dashboard and report owners come from workspace roles, so a non-empty owner_criteria is matched against the role names Admin, Member, Contributor and Viewer and must include at least one of them; item-level rights such as ReadWriteReshareExplore and Owner never match in this mode. In the example above only Admin applies, so it keeps workspace Admins. An unset or empty owner_criteria still makes every user principal an owner.
Admin scan vs. workspace listing
PowerBI ingestion uses two metadata paths per workspace:
| Path | When | What it fetches | API volume |
|---|---|---|---|
| Admin scan | The configured principal has admin-API access and the workspace scan returns data. | Reports, users, datasets, lineage, endorsements — all from a single workspace scan response. | 1 scan request per workspace batch |
| Workspace listing fallback | The admin scan returns no data for a workspace (permissions, throttling, etc.). | Reports via the per-workspace /reports endpoint; users via /admin/reports/{id}/users per report (only when extract_ownership: true). | O(workspaces + reports) requests |
When the fallback engages, the ingestion report includes a Report Scan Fallback Active entry per workspace so you can correlate that workspace's slower ingestion with the missing scan output.
Workspace APIs mode
metadata_api_mode: workspace_apis replaces the admin scan with workspace-scoped calls (see the prerequisites above for the permissions it needs):
| Metadata | admin_scan (default) | workspace_apis |
|---|---|---|
| Datasets, tables, columns, measures | Scanner API (datasetSchema, datasetExpressions) | GET /groups/{id}/datasets plus the semantic model's TMDL definition (getDefinition) |
| Table and column lineage | M expressions from the scan | M expressions from TMDL partitions, through the same parser, so upstream URNs are identical |
| DirectLake lineage | Scan relations and Fabric artifact lists | Fabric item listing; the lakehouse SQL endpoint referenced by the model is resolved to its lakehouse |
| Table, column and measure descriptions | Dataset description only | TMDL /// doc comments on tables, columns and measures |
| Ownership | Per-item users from the scan | Workspace role assignments (GET /groups/{id}/users) |
| Paginated report to dataset binding | /reports/{id}/datasources | The report's .rdl definition, falling back to /reports/{id}/datasources |
Endorsements, apps, modified_since | Supported | Not available; the options are rejected |
getDefinition is a long-running operation: the connector polls it and gives up after scan_timeout seconds per item. A semantic model whose definition is not fetched is reported as a failure under Semantic Model Definition Not Fetched (or Fabric API Permission Denied when it was refused); a report as a warning under Report Definition Not Fetched (or Report Definition Permission Denied). Push and streaming semantic models have no definition to fetch: they are ingested without tables, schema or lineage, reported as a warning under Semantic Model Definition Not Supported, and counted in semantic_model_definitions_unsupported. The report also carries fabric_definitions_fetched, fabric_definitions_failed, semantic_model_definitions_fetched and semantic_model_definitions_failed (the run fails as No Item Definitions Fetched when every semantic model definition failed, that is when the second is non-zero and the first is zero), fabric_lro_timeouts, tmdl_tables_parsed and related counters.
Call volume replaces one scan per scan_batch_size workspaces with per-item requests:
- per workspace: the Fabric item listing and the Power BI dataset listing, the report and dashboard listings when
extract_reportsandextract_dashboardsare enabled, and the workspace users whenextract_ownershipis enabled; - per semantic model: one
getDefinition(a long-running operation, so also its status polls and result fetch) and its parameters, plus its refresh history whenextract_refresh_historyis enabled; - per report: its pages, plus one
getDefinitionwhenextract_report_field_usageis enabled; a paginated report instead fetches its.rdldefinition, and its data sources when that definition yields no dataset binding; - per dashboard: its tiles.
Fabric throttles at 500 calls per minute per principal; throttled calls are retried.
Enrichment from item definitions
These options are off by default and, except for extract_refresh_history, require workspace_apis mode:
extract_model_relationships: semantic-model relationships become foreign-key constraints on thefromtable's schema, pointing at thetotable and column. Inactive relationships are included and marked(inactive)in the constraint name.extract_report_field_usage: each report's definition is read to find the columns and measures its visuals use. Every report page chart gets aninputFieldsaspect pointing at those schema fields, and thevisualCountandvisualTypescustom properties. Fields that reference tables outside the report's dataset are counted underreport_field_references_unresolved.extract_refresh_history: the dataset's most recent refresh (GET /groups/{id}/datasets/{id}/refreshes) is emitted as anOperationaspect (customOperationType: REFRESH) on each of its tables when the refresh completed. Works in both modes and needs only theViewerrole.
In workspace_apis mode the connector also emits lineage inside a semantic model: DAX calculated tables depend on the model tables they read, and, when extract_column_level_lineage is enabled, each measure gets fine-grained lineage from the columns of other model tables its DAX references. References to columns of the measure's own table and to other measures ([Total Premium]) are not emitted as edges; the measure's DAX is kept in its schema-field description instead.
Permission problems are reported as failures rather than warnings, because the missing entities would otherwise be soft-deleted by stale-entity removal on a green run: a 401/403 from a semantic model's getDefinition removes that model's tables and lineage from the run, and a 401/403 from the dataset listing (GET /groups/{id}/datasets) removes the tables and lineage of every semantic model in that workspace.
Lineage
This source extracts table lineage for tables present in PowerBI Datasets. Lets consider a PowerBI Dataset SALES_REPORT and a PostgreSQL database is configured as data-source in SALES_REPORT dataset.
Consider SALES_REPORT PowerBI Dataset has a table SALES_ANALYSIS which is backed by SALES_ANALYSIS_VIEW of PostgreSQL Database then in this case SALES_ANALYSIS_VIEW will appear as upstream dataset for SALES_ANALYSIS table.
You can control table lineage ingestion using extract_lineage configuration parameter, by default it is set to true.
PowerBI Source extracts the lineage information by parsing PowerBI M-Query expressions and from dataset data returned by the PowerBI API.
The source will attempt to extract information from ODBC connection strings in M-Query expressions to determine the database type. If the database type matches a supported platform and the source is able to extract enough information to construct a valid Dataset URN, it will extract lineage for that data source.
PowerBI Source will extract lineage for the below listed PowerBI Data Sources:
- Snowflake
- Oracle
- PostgreSQL
- Microsoft SQL Server
- Google BigQuery
- Databricks
- MySQL
- Amazon Redshift
- Amazon Athena
- Starburst / Trino
Native SQL query parsing is supported for Snowflake, Amazon Redshift, Oracle, and ODBC data sources.
Oracle TNS Aliases and Inline Native Queries
PowerBI users can connect to Oracle via a TNS alias (any tnsnames.ora-based deployment) and embed SQL inline using a Query= argument:
let
Source = Oracle.Database(
"EDWPSFN",
[HierarchicalNavigation = true,
Query = "SELECT … FROM PS_VENDOR, PS_COR_CNTRCT_PROJ …"])
in
Source
Three Oracle.Database first-argument forms are recognized:
- EZ-Connect:
host:port/service[.domain] - Bare TNS alias: e.g.
EDWPSFN,MYDB.WORLD - Full TNS descriptor:
(DESCRIPTION=(ADDRESS=…)(CONNECT_DATA=(SERVICE_NAME=foo)))
For the bare-alias and descriptor forms, the alias / SERVICE_NAME becomes the server key for server_to_platform_instance lookup. The lookup is case-insensitive, so EDWPSFN in the M-Query matches an EDWPSFN (or edwpsfn) entry in the recipe.
Matching the Oracle URN shape (database segment)
The generated upstream URN must match the URN that your Oracle ingestion produces for the same table, otherwise the lineage edge points at a dataset that does not exist. Oracle ingestion emits 2-part schema.table URNs by default, and 3-part database.schema.table URNs only when it runs with add_database_name_to_urn: true.
PowerBI derives the database segment from the connection form:
| Oracle.Database form | Database segment |
|---|---|
EZ-Connect host:port/service | the service (always 3-part) |
| Bare TNS alias / descriptor | none — 2-part by default |
So a bare TNS alias produces a 2-part URN, which is correct for a default Oracle ingestion. If your Oracle ingestion uses add_database_name_to_urn: true, set default_database on the alias entry to supply the missing segment (use the same value as the Oracle ingestion's database / urn_db_name):
source:
type: powerbi
config:
server_to_platform_instance:
EDWPSFN:
default_database: edwprd # only if Oracle ingestion uses 3-part URNs
default_schema: sysadm # owner schema for unqualified inline-SQL tables
# platform_instance: EDWPSFN # only set this if your Oracle ingestion uses one
Qualifying unqualified inline SQL (default_schema)
Because inline Query= SQL often references unqualified tables, declare a default_schema on the alias entry so those references resolve to the ingested Oracle datasets. default_schema only applies to inline native SQL — hierarchical navigation takes the schema from the M-Query itself.
If default_schema is missing and the inline SQL references unqualified tables, lineage will still be drawn for any qualified tables in the SQL, and a structured warning will appear in the ingestion report telling you exactly which alias needs the knob set.
At least one of default_schema / default_database must be set for an Oracle entry; a mapping that needs neither should be a plain platform_instance / env entry.
BigQuery EXTERNAL_QUERY Federation
If you query BigQuery from PowerBI, you may use BigQuery federation
(EXTERNAL_QUERY("project.region.connection", "<sql>")) to run SQL on an external
engine such as Cloud SQL or AlloyDB. The generic SQL parser cannot resolve those
calls — the arguments are string literals, not table identifiers — so the federated
upstream is skipped (and reported in the ingestion report) unless you map each
connection to its external platform.
Requirements: native_query_parsing: true,
enable_advance_lineage_sql_construct: true, and extract_lineage: true (the
default). Federation is resolved while extracting lineage, so configuring the
mapping with any of these disabled fails config validation rather than silently
doing nothing. The target platform must be one of the platforms PowerBI
lineage supports: athena, bigquery, databricks, fabric-onelake, hive,
mssql, mysql, odbc, oracle, postgres, redshift, snowflake, or
trino.
Other DataHub platforms such as cloudsql, alloydb, spanner, or mariadb
are rejected at config validation even though they exist in DataHub, so point
the mapping at the engine's wire-compatible platform instead (Cloud SQL and
AlloyDB expose postgres or mysql). If you narrow dataset_type_mapping,
keep the target platform's PowerBI name in that mapping so the resolved upstream
is not filtered out.
Configuration:
source:
type: powerbi
config:
native_query_parsing: true
enable_advance_lineage_sql_construct: true
# ... other config ...
bigquery_external_query_connection_to_platform:
"my-gcp-project.us-east1.my_cloudsql_connection":
platform: postgres # required; e.g. postgres, mysql
default_database: ext_db # optional; match your external source's URN shape
default_schema: public # optional
# platform_instance: pg-prod # optional
# env: PROD # optional
The map key is the connection id exactly as it appears as the first
EXTERNAL_QUERY argument (project.region.connection). The mapping is a no-op for
queries that do not contain EXTERNAL_QUERY calls. Unmapped connections are
reported as info (BigQuery EXTERNAL_QUERY connection not mapped); parse failures
and empty resolutions are reported as warnings under SQL Parsing Failure.
Athena Federated Query Platform Override
When using Amazon Athena via ODBC that queries federated data sources (e.g., Athena querying MySQL or PostgreSQL via federated connectors), the lineage URNs will default to the Athena platform. Use athena_table_platform_override to point lineage to the actual source platform instead of Athena.
Configuration:
source:
type: powerbi
config:
# ... other config ...
dsn_to_platform_name:
MyAthenaDSN: athena
athena_table_platform_override:
# DSN-scoped key (takes precedence)
"MyAthenaDSN:analytics.users": mysql
# Global key (fallback for any DSN)
"reporting.orders": postgres
Key format:
- DSN-scoped:
"DSN_NAME:database.table"- applies only to specific DSN - Global:
"database.table"- applies to all DSNs
DSN-scoped keys take precedence over global keys, allowing different overrides for the same table name across different Athena data sources.
Note: This override only applies to Athena ODBC connections. For other ODBC platforms, lineage will use the platform determined from the DSN configuration.
For example, consider the SQL query shown below. The table OPERATIONS_ANALYTICS.TRANSFORMED_PROD.V_UNIT_TARGET will be ingested as an upstream table.
let
Source = Value.NativeQuery(
Snowflake.Databases(
"sdfsd788.ws-east-2.fakecomputing.com",
"operations_analytics_prod",
[Role = "OPERATIONS_ANALYTICS_MEMBER"]
){[Name = "OPERATIONS_ANALYTICS"]}[Data],
"select #(lf)UPPER(REPLACE(AGENT_NAME,\'-\',\'\')) AS Agent,#(lf)TIER,#(lf)UPPER(MANAGER),#(lf)TEAM_TYPE,#(lf)DATE_TARGET,#(lf)MONTHID,#(lf)TARGET_TEAM,#(lf)SELLER_EMAIL,#(lf)concat((UPPER(REPLACE(AGENT_NAME,\'-\',\'\'))), MONTHID) as AGENT_KEY,#(lf)UNIT_TARGET AS SME_Quota,#(lf)AMV_TARGET AS Revenue_Quota,#(lf)SERVICE_QUOTA,#(lf)BL_TARGET,#(lf)SOFTWARE_QUOTA as Software_Quota#(lf)#(lf)from OPERATIONS_ANALYTICS.TRANSFORMED_PROD.V_UNIT_TARGETS#(lf)#(lf)where YEAR_TARGET >= 2020#(lf)and TEAM_TYPE = \'foo\'#(lf)and TARGET_TEAM = \'bar\'",
null,
[EnableFolding = true]
),
#"Added Conditional Column" = Table.AddColumn(
Source,
"Has PS Software Quota?",
each
if [TIER] = "Expansion (Medium)" then
"Yes"
else if [TIER] = "Acquisition" then
"Yes"
else
"No"
)
in
#"Added Conditional Column"
Use full-table-name in from clause. For example dev.public.category
M-Query Pattern Supported For Lineage Extraction
Lets consider a M-Query which combine two PostgreSQL tables. Such M-Query can be written as per below patterns.
Pattern-1
let
Source = PostgreSQL.Database("localhost", "book_store"),
book_date = Source{[Schema="public",Item="book"]}[Data],
issue_history = Source{[Schema="public",Item="issue_history"]}[Data],
combine_result = Table.Combine({book_date, issue_history})
in
combine_result
Pattern-2
let
Source = PostgreSQL.Database("localhost", "book_store"),
combine_result = Table.Combine({Source{[Schema="public",Item="book"]}[Data], Source{[Schema="public",Item="issue_history"]}[Data]})
in
combine_result
Pattern-2 is not supported for upstream table lineage extraction as it uses nested item-selector i.e. {Source{[Schema="public",Item="book"]}[Data], Source{[Schema="public",Item="issue_history"]}[Data]} as argument to M-QUery table function i.e. Table.Combine
Pattern-1 is supported as it first assigns the table from schema to variable and then variable is used in M-Query Table function i.e. Table.Combine
Extract endorsements to tags
By default, extracting endorsement information to tags is disabled. The feature may be useful if organization uses endorsements to identify content quality.
Please note that the default implementation overwrites tags for the ingested entities, if you need to preserve existing tags, consider using a transformer with semantics: PATCH tags instead of OVERWRITE.
Profiling
The profiling implementation is done through querying DAX query endpoint. Therefore, the principal needs to have permission to query the datasets to be profiled. Usually this means that the service principal should have Contributor role for the workspace to be ingested. Profiling is done with column-based queries to be able to handle wide datasets without timeouts.
Take into account that the profiling implementation executes a fairly big number of DAX queries, and for big datasets this is a significant load to the PowerBI system.
The profiling_pattern setting may be used to limit profiling actions to only a certain set of resources in PowerBI. Both allowed and deny rules are matched against the following pattern for every table in a PowerBI Dataset: workspace_name.dataset_name.table_name. Users may limit profiling with these settings at table level, dataset level or workspace level.
Probing a PowerBI recipe
datahub recipe probe checks a PowerBI recipe against the live tenant before a run. It authenticates with the recipe's own service principal and environment, uses the same REST endpoints, paging and retries as ingestion, and returns metadata only. It never starts the admin workspace scan. Run it with the read-only service principal that ingestion uses.
| Command | Parameters | Returns |
|---|---|---|
workspaces | limit | Each listed workspace's name, id, type, type_allowed (the workspace_type_filter verdict) and, with the admin API, state |
reports | workspace | The reports in one workspace, by workspace name, with type Report or PaginatedReport, and the workspace's id, type and state |
dashboards | workspace | The dashboards in one workspace, by workspace name, with the workspace's id, type and state |
admin_api_access | none | Whether the credential can call the read-only admin APIs |
api | path | One raw response from /groups, /groups/{id}/reports or /groups/{id}/dashboards |
The workspace listing follows the recipe: the regular API's member workspaces by default, or the whole tenant with admin_apis_only. It is narrowed by modified_since the same way ingestion narrows it. If no workspace was modified since that date, ingestion applies no filter and lists every workspace, and the probe does the same and says so. If PowerBI rejects the date, for example because it is more than 30 days ago, the probe fails with an error that names modified_since. If the modified-workspaces list cannot be read at all, for example without admin API access, ingestion lists every workspace, and the probe does the same with a warning.
Ingestion keeps a workspace only when it passes workspace_name_pattern, workspace_id_pattern and workspace_type_filter together, and its admin scan also skips any workspace whose state is not Active. A name alone answers only the first of these, so save the listing and judge them all from it. The state is judged only when the listing has one, which means with admin_apis_only:
datahub recipe probe run workspaces --recipe recipe.yml --report-to workspaces.json
datahub recipe probe filter --recipe recipe.yml --kind Workspace --from-run workspaces.json
datahub recipe probe run reports --recipe recipe.yml --workspace "Sales"
The id and type rules are judged only from a record that carries the workspace's id and type. With --name instead of --from-run, there is none, so the result judges the name alone and warns that the id and type rules need such a record. Reports and dashboards have no filters of their own: their workspace's verdict decides, and extract_reports and extract_dashboards switch them off. Each report and dashboard record carries its workspace's workspace_id, workspace_type and workspace_state, so a saved reports or dashboards run judged with --from-run applies all three workspace rules and the state rule. The same warning still appears there, because the --parent workspace itself is judged by name, but the verdicts use the facts on each row. A saved record with an id but no type is a workspace whose type is null, and it is excluded by workspace_type_filter, as ingestion excludes it. reports and dashboards also warn when ingestion would read nothing in the workspace, and name the rule that drops it. With --parent and --name alone, only the workspace's name is judged.
admin_api_access: denied means ingestion still runs but gets none of the workspace scan's metadata: no scan-derived lineage, endorsements or apps, because the scan always uses the admin APIs, even with admin_apis_only: false. Paginated-report datasource lineage still comes through the regular API.
Personal workspaces are named after their owner. Unless workspace_type_filter includes their type, ingestion skips them, and the probe counts them in a warning without listing them, in workspaces and in the api command's /groups response alike. In the /groups response, a record with no type is withheld too, since it may be a personal workspace. The api command refuses a path that would be sent differently from how it is written, such as /./groups or /group%73: write the path as it is sent, with spaces and non-ASCII characters percent-encoded. The api command does not allow the admin endpoints, $expand, datasets, datasources, parameters or users, so it cannot return these names, email addresses or connection details.
Limitations
- Some metadata and lineage fields are only available through admin APIs or specific tenant settings. In
workspace_apismode endorsements, apps,modified_sinceand per-item owners are not available, and the service principal needs theContributorrole rather than read-only access. - Lineage quality depends on available model metadata and supported query/source patterns.
- Large tenants with many workspaces can require longer extraction windows.
Troubleshooting
- Authentication failures: verify
tenant_id,client_id, andclient_secret, and confirm the app has the required Power BI API permissions. - Missing workspaces/assets: check service principal access to target workspaces or enable the required admin API mode/settings.
- Lineage gaps: confirm lineage-related config is enabled and that semantic models expose supported upstream source details.
Code Coordinates
- Class Name:
datahub.ingestion.source.powerbi.powerbi.PowerBiDashboardSource - Browse on GitHub
If you've got any questions on configuring ingestion for PowerBI, feel free to ping us on our Slack.
This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.
Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.