Skip to main content

Matillion

Overview​

Matillion Data Productivity Cloud (DPC) is a cloud-native data integration platform for building, orchestrating, and monitoring data pipelines. Learn more in the official Matillion documentation.

The DataHub integration for Matillion DPC ingests pipelines, streaming pipelines, projects, and environments as DataHub entities. It captures table- and column-level lineage via the Matillion OpenLineage API, pipeline execution history as operational metadata, and child pipeline dependency relationships for end-to-end orchestration visibility.

Concept Mapping​

Source ConceptDataHub ConceptNotes
ProjectContainerTop-level grouping of pipelines within a Matillion account.
EnvironmentContainerDeployment environment within a project (e.g. Production, Staging).
PipelineDataFlowAn orchestration pipeline that transforms or moves data.
Pipeline Component / StepDataJobAn individual step within a pipeline.
Streaming PipelineDataFlowA CDC or streaming pipeline, emitted with pipeline_type=streaming.
Pipeline ExecutionDataProcessInstanceA single run of a pipeline, including status and timing.
OpenLineage table referenceDatasetUpstream or downstream dataset referenced via OpenLineage events.
Table/column lineage edgeLineage edgeExtracted from OpenLineage events; column-level via SQL parsing.

Module matillion-dpc​

Incubating

Important Capabilities​

CapabilityStatusNotes
Column-level Lineage✅Enabled by default, can be disabled via configuration parse_sql_for_lineage.
Detect Deleted Entities✅Enabled via stateful ingestion.
Platform Instance✅Enabled by default.
Table-Level Lineage✅Enabled by default via OpenLineage data from pipeline executions.

Overview​

The matillion-dpc module ingests metadata from Matillion Data Productivity Cloud (DPC) into DataHub. It extracts pipelines, streaming pipelines, projects, environments, execution history, and table and column-level lineage via the Matillion OpenLineage API.

Prerequisites​

Obtain API Credentials​

The connector uses OAuth2 client credentials and automatically handles token generation and refresh.

  1. Log into Matillion Data Productivity Cloud as a Super Admin
  2. Navigate to Profile & Account → API credentials
  3. Click Set an API Credential
  4. Provide a descriptive name (e.g., "DataHub Integration")
  5. Assign an Account Role with read permissions to required APIs
  6. Click Save and immediately copy the Client Secret (not shown again)
  7. Note the Client ID (remains visible)

For detailed instructions, see Matillion API Authentication.

Required Permissions​

The API credentials must have an Account Role with Read permissions to:

  • Projects (/v1/projects)
  • Environments (/v1/environments)
  • Pipelines (/v1/pipelines)
  • Schedules (/v1/schedules)
  • Lineage Events (/v1/lineage/events)
  • Pipeline Executions (/v1/pipeline-executions) - optional
  • Streaming Pipelines (/v1/streaming-pipelines) - optional

If using an account role other than Super Admin, grant project and environment-level roles as needed.

See Matillion RBAC documentation for details.

Lineage and Dependencies​

The connector automatically extracts:

  1. Table and Column-Level Lineage - From OpenLineage Events API (/v1/lineage/events) (docs)
  2. Operational Metadata - Pipeline execution history from Pipeline Executions API (/v1/pipeline-executions) emitted as DataProcessInstance entities (docs)
  3. Child Pipeline Dependencies - Automatically tracks when pipelines call other pipelines, creating step-to-step dependency relationships for comprehensive pipeline orchestration visibility

OpenLineage Namespace Mapping (Optional)​

Optional: Map OpenLineage namespace URIs to DataHub platform instances for lineage connections. If not configured, the connector extracts platform type from URIs (e.g., postgresql://... → postgres) with default environment (PROD).

When to use: Configure this when you need lineage to connect to existing datasets with platform instances.

Example namespaces: postgresql://host:5432, snowflake://account.snowflakecomputing.com, bigquery://project

namespace_to_platform_instance:
  "postgresql://prod-db.us-east-1.rds.amazonaws.com:5432":
    platform_instance: postgres_prod
    env: PROD
    database: analytics
    schema: public

  "snowflake://prod-account.snowflakecomputing.com":
    platform_instance: snowflake_prod
    env: PROD
    convert_urns_to_lowercase: true

Platform instances must match those used when ingesting the source data platforms.

Install the Plugin​

pip install 'acryl-datahub[matillion-dpc]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: matillion-dpc
  config:
    api_config:
      client_id: "${MATILLION_CLIENT_ID}"
      client_secret: "${MATILLION_CLIENT_SECRET}"
      region: "EU1" # EU1, US1, or AU1

    env: "PROD"

    # Optional: Map OpenLineage namespaces to DataHub platform instances
    # Required if existing datasets use platform instances
    namespace_to_platform_instance:
      "postgresql://prod-db.us-east-1.rds.amazonaws.com:5432":
        platform_instance: postgres_prod
        env: PROD
        database: analytics
        schema: public

      "snowflake://prod-account.snowflakecomputing.com":
        platform_instance: snowflake_prod
        env: PROD
        convert_urns_to_lowercase: true

      "bigquery://my-gcp-project":
        platform_instance: bigquery_prod
        env: PROD

    include_streaming_pipelines: true
    include_unpublished_pipelines: true
    max_executions_per_pipeline: 10
    extract_projects_to_containers: true

    # Optional: Filter projects, environments, pipelines using regex patterns
    # project_patterns:
    #   allow: ["^prod-.*", "^staging-.*"]
    #   deny: [".*-deprecated$", ".*-archived$"]

    # environment_patterns:
    #   allow: ["^production$", "^staging$"]

    # pipeline_patterns:
    #   deny: ["^test_.*", ".*_backup$"]

    # streaming_pipeline_patterns:
    #   allow: ["^cdc_.*"]

    stateful_ingestion:
      enabled: true

sink:
  type: datahub-rest
  config:
    server: "http://localhost:8080"

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
api_config ✅
MatillionAPIConfig
api_config.client_id ❓
string(password)
Matillion API Client ID for OAuth2 authentication.
api_config.client_secret ❓
string(password)
Matillion API Client Secret for OAuth2 authentication.
api_config.custom_base_url
One of string, null
Custom API base URL for VPC endpoints or on-premise installations.
Default: None
api_config.custom_oauth_token_url
One of string, null
Custom OAuth2 token endpoint URL for VPC endpoints or on-premise installations.
Default: None
api_config.region
Enum
One of: "EU1", "US1", "AU1"
api_config.request_timeout_sec
integer
Per-request timeout in seconds. On timeouts, prefer narrowing start_time/end_time or enabling stateful ingestion over a large timeout.
Default: 30
bucket_duration
Enum
One of: "DAY", "HOUR"
end_time
string(date-time)
Latest date of lineage/usage to consider. Default: Current time in UTC
extract_projects_to_containers
boolean
Whether to add the top-level Matillion project as a DataHub container. Environment and folder containers are always created; this only controls whether they are nested under a project container or hang at the root.
Default: True
extract_run_history
boolean
Emit run history (DataProcessInstances) for published pipelines by fetching their executions in the time window. Implied when include_unpublished_pipelines is enabled.
Default: False
include_dependent_pipelines
boolean
Whether to ingest dependent (child) pipelines that are only referenced via lineage events and were not discovered as project pipelines (published or via execution history). When disabled, lineage is still emitted for discovered pipelines, but these lineage-only dependencies are not created as their own DataFlows/DataJobs.
Default: True
include_external_urls
boolean
Whether to emit links back to the Matillion console (project, run, and pipeline). Off by default because the pipeline link searches the observability dashboard by file name and only resolves for pipelines that ran recently and whose editor name matches their file name. Enable to add all Matillion external links.
Default: False
include_streaming_pipelines
boolean
Whether to ingest Matillion streaming pipelines (CDC pipelines). Streaming pipelines are emitted as separate DataFlows with pipeline_type='streaming'.
Default: True
include_unpublished_pipelines
boolean
Whether to discover and ingest unpublished pipelines from recent execution history. When enabled, the connector will discover pipelines that have been executed but not yet published. Disable this to only ingest published pipelines from the published-pipelines API.
Default: True
lineage_platform_mapping
One of string, null
Override platform name mappings from OpenLineage namespaces to DataHub platforms. Only needed for non-standard platforms. See documentation for list of pre-mapped platforms. Example: {"customdb": "postgres", "mywarehouse": "snowflake"}
Default: None
max_executions_per_pipeline
integer
Maximum number of recent pipeline executions to ingest per pipeline. Set to 0 to disable execution ingestion.
Default: 10
parse_sql_for_lineage
boolean
Whether to parse SQL from OpenLineage events to extract additional column-level lineage. Requires DataHub graph access. When enabled, SQL queries are parsed to infer lineage beyond what's explicitly provided in OpenLineage column mappings.
Default: True
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to
Default: None
start_time
string(date-time)
Earliest date of lineage/usage to consider. Default: Last full day in UTC (or hour, depending on bucket_duration). You can also specify relative time with respect to end_time such as '-7 days' Or '-7d'.
Default: None
env
string
The environment that all assets produced by DataHub platform ingestion source belong to
Default: PROD
environment_patterns
AllowDenyPattern
A class to store allow deny regexes
environment_patterns.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
environment_patterns.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
environment_patterns.allow.string
string
environment_patterns.deny
array
List of regex patterns to exclude from ingestion.
Default: []
environment_patterns.deny.string
string
namespace_to_platform_instance
One of NamespacePlatformMapping, null
Maps OpenLineage namespace prefixes to platform instance/environment using longest prefix matching. Unmapped namespaces extract platform from URI with defaults (env=PROD). Example: {"snowflake://prod-account": {"platform_instance": "snowflake_prod", "env": "PROD"}}
Default: None
namespace_to_platform_instance.key.platform_instance
One of string, null
DataHub platform instance to use for datasets from this namespace
Default: None
namespace_to_platform_instance.key.convert_urns_to_lowercase
boolean
Whether to convert dataset URNs to lowercase for this namespace.
Default: False
namespace_to_platform_instance.key.database
One of string, null
Default database name to prepend if dataset name doesn't include database context
Default: None
namespace_to_platform_instance.key.schema
One of string, null
Default schema name to prepend if dataset name doesn't include schema context
Default: None
namespace_to_platform_instance.key.env
string
Environment (PROD, DEV, etc.) to use for datasets from this namespace
Default: PROD
pipeline_patterns
AllowDenyPattern
A class to store allow deny regexes
pipeline_patterns.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
pipeline_patterns.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
pipeline_patterns.allow.string
string
pipeline_patterns.deny
array
List of regex patterns to exclude from ingestion.
Default: []
pipeline_patterns.deny.string
string
project_patterns
AllowDenyPattern
A class to store allow deny regexes
project_patterns.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
project_patterns.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
project_patterns.allow.string
string
project_patterns.deny
array
List of regex patterns to exclude from ingestion.
Default: []
project_patterns.deny.string
string
streaming_pipeline_patterns
AllowDenyPattern
A class to store allow deny regexes
streaming_pipeline_patterns.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
streaming_pipeline_patterns.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
streaming_pipeline_patterns.allow.string
string
streaming_pipeline_patterns.deny
array
List of regex patterns to exclude from ingestion.
Default: []
streaming_pipeline_patterns.deny.string
string
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Stateful ingestion configuration.
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Capabilities​

OpenLineage Namespace Mapping​

Optional configuration to map OpenLineage namespace URIs to DataHub platform information. Without this, the connector extracts platform type from URIs with default environment.

Fields:

  • platform_instance: Platform instance identifier (must match source ingestion)
  • database / schema: Defaults for incomplete dataset names from OpenLineage
    • 3-tier platforms (Snowflake, Postgres, Redshift): database.schema.table
    • 2-tier platforms (MySQL, Hive): schema.table
  • convert_urns_to_lowercase: Normalize URNs to lowercase (use true for Snowflake)
  • env: Environment tag (PROD, DEV, etc.)

Fallback behavior: Unmapped namespaces extract platform type from the URI (e.g., postgresql://... → postgres) without platform instance assignment.

SQL Parsing for Column-Level Lineage​

Enable parse_sql_for_lineage: true to parse SQL queries from OpenLineage events for additional column-level lineage.

Requirements:

  • DataHub graph connection configured
  • Schema information in OpenLineage events

Platform-Specific Handling​

Snowflake: Use convert_urns_to_lowercase: true in namespace mapping

BigQuery: 3-tier naming (project.dataset.table). Set database: project-id, schema: dataset-name

MySQL / 2-tier: 2-tier naming (schema.table). Set schema only

Postgres / Redshift: 3-tier naming (database.schema.table). Set both database and schema

Container Hierarchy​

Pipelines and their components are organized into a browsable container hierarchy that mirrors their path in Matillion:

Project › Environment › <folder> › … › Pipeline › Component

The folder levels come from the pipeline's path (e.g. ingest/staging/orders/load.orch.yaml yields ingest › staging › orders folders), so the browse tree lines up with the paths you match on in pipeline_patterns. Components (DataJobs) live in their pipeline's folder and browse directly under the pipeline.

The environment and folder levels are always built. extract_projects_to_containers (default true) only controls whether the top-level Project container is added to the hierarchy; set it to false to hang environments (and their folders) directly at the root instead of under a project.

Filtering Options​

The connector supports flexible regex-based filtering to control what metadata is ingested.

Project Filtering​
project_patterns:
  allow: ["^prod-.*", "^staging-.*"]
  deny: [".*-deprecated$"]
Environment Filtering​
environment_patterns:
  allow: ["^production$", "^staging$"]
  deny: ["^sandbox.*"]
Pipeline Filtering​
pipeline_patterns:
  allow: [".*"]
  deny: ["^test_.*", ".*_backup$"]
Streaming Pipeline Filtering​
streaming_pipeline_patterns:
  allow: ["^cdc_.*"]
  deny: [".*_test$"]

All patterns are case-insensitive by default and support full regex syntax. Deny patterns take precedence over allow patterns.

Child Pipeline Dependencies​

The connector automatically detects and tracks when pipelines call other pipelines (via "Run Pipeline" components). This creates step-level dependency relationships in DataHub, showing:

  • Which pipeline steps trigger child pipelines
  • Complete execution lineage across pipeline orchestrations
  • Cross-pipeline data flow for comprehensive impact analysis

No configuration needed — this feature is automatic when execution history is ingested.

Child pipelines that only appear in lineage events (and were not discovered as project pipelines themselves) are, by default, created as their own DataFlows/DataJobs so the full dependency graph is captured. To suppress these lineage-only dependencies and keep ingestion scoped to discovered pipelines, disable:

include_dependent_pipelines: false

Lineage is still emitted for discovered pipelines when this is disabled — only the lineage-only dependent pipelines are skipped.

Published vs Unpublished Pipelines​

The connector can discover pipelines from two sources:

  1. Published Pipelines — Pipelines explicitly published in Matillion DPC (fetched from /published-pipelines API)
  2. Unpublished Pipelines — Pipelines discovered from recent execution history (fetched from /pipeline-executions API)

By default, both types are ingested. To only ingest published pipelines:

include_unpublished_pipelines: false

This is useful when:

  • You want to control what appears in DataHub via Matillion's publish workflow
  • You have many development/test pipelines that run but shouldn't be documented
  • You want to reduce ingestion time and API calls

Run History (DataProcessInstances)​

Run history — per-execution DataProcessInstance entities with status and timing — is produced from the pipeline-executions API. Each execution surfaces a run on the pipeline (DataFlow) itself as well as on each of its components (DataJobs), so the "Runs" tab is populated at both levels. When include_unpublished_pipelines: true, this happens automatically as part of discovery.

When include_unpublished_pipelines: false, discovery only lists published pipelines and does not fetch executions, so no runs are emitted by default. To get run history for your published pipelines without also ingesting unpublished ones, enable:

include_unpublished_pipelines: false
extract_run_history: true

This fetches executions within the configured time window and attaches runs to the matching published pipelines. It is off by default because it calls the pipeline-executions and per-execution steps APIs, which are slower and degrade over wider time windows — pair it with a narrow start_time / end_time and stateful ingestion.

Enabling it has a second benefit: lineage often references unpublished child orchestrations (e.g. SRC_*_ORCH pipelines invoked by a published schedule). The OpenLineage namespace only carries an opaque environment UUID, so when such a pipeline is neither published nor seen in executions, its environment cannot be resolved and the connector skips it rather than placing it in a pipeline with no environment. Because executions report the environment name, enabling extract_run_history (or include_unpublished_pipelines) lets these child orchestrations resolve their environment and nest correctly under it.

Time Window and Incremental Ingestion​

Pipeline-execution discovery and lineage are bounded by start_time / end_time. If you do not set them, end_time defaults to now and start_time defaults to the start (00:00 UTC) of the previous day — i.e. at least the last 24 hours of jobs. Set start_time to backfill more history on the first run, especially if your pipelines do not run daily:

start_time: "2024-01-01T00:00:00Z" # absolute
# start_time: "-30d"                # or relative to end_time

Enable stateful ingestion so subsequent runs only fetch new lineage instead of re-reading the whole window:

stateful_ingestion:
  enabled: true

Lineage endpoint performance: the Matillion lineage events API paginates by offset and gets progressively slower the further back it reads. Wide time windows therefore both increase total runtime and make individual requests more likely to time out. Lineage requests are automatically split into sub-windows of at most 31 days (the API's hard limit per request), but each sub-window still pages through its full result set. Prefer a narrower window plus stateful ingestion over a single very large backfill, and only raise api_config.request_timeout_sec when a genuinely large window is unavoidable (a high timeout multiplies the worst-case wait, since failed requests are retried).

Limitations​

  • SQL parsing for column-level lineage requires a DataHub graph connection and schema information in OpenLineage events. Unsupported SQL dialects or complex queries are skipped with a warning.
  • Column-level lineage is only available when Matillion pipelines emit SQL via OpenLineage; transformations without SQL output will have coarse-grained lineage only.
  • Matillion console links (project, run, and pipeline) are off by default; set include_external_urls: true to emit them. Pipeline names and the pipeline link use the pipeline file name — the only name the API exposes; a different display name set inside the Maia editor is not retrievable. The pipeline link opens the observability dashboard pre-filtered by that file name (there is no per-pipeline deep-link), so the pipeline must have run within the dashboard's time window to appear.

Troubleshooting​

Lineage Not Showing Up​

  1. Verify namespace mapping matches source ingestion platform instances
  2. Check logs for Processing OpenLineage event messages
  3. Confirm dataset names in OpenLineage match actual tables

Column-Level Lineage Missing​

Enable parse_sql_for_lineage: true (requires DataHub graph connection).

Execution History Not Appearing​

  1. Adjust start_time to query further back in time if needed
  2. Verify API permissions for Pipeline Executions API

Performance Issues or Request Timeouts​

The lineage events endpoint paginates by offset and slows down the further back in time it reads, so wide windows are the most common cause of slow runs and request_timeout_sec timeouts. In order of preference:

  1. Narrow the time window by adjusting start_time (e.g., last 7 days instead of 30) and enable stateful_ingestion so later runs stay incremental.
  2. Use filtering patterns to reduce scope:
    • project_patterns to filter projects
    • environment_patterns to filter environments
    • pipeline_patterns to filter pipelines
    • streaming_pipeline_patterns to filter streaming pipelines
  3. Disable include_streaming_pipelines if not needed.
  4. As a last resort, raise api_config.request_timeout_sec. Keep it as low as practical — failed requests are retried, so a very high timeout multiplies the worst-case wait on a slow endpoint.

Code Coordinates​

  • Class Name: datahub.ingestion.source.matillion_dpc.matillion.MatillionSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for Matillion, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.