Skip to main content

Recipes

A recipe is a YAML file that configures one metadata ingestion job. It names a source (the system metadata is read from) and a sink (where that metadata is written, usually DataHub).

# recipe.dhub.yaml — the smallest valid recipe
source:
type: mysql
config:
host_port: localhost:3306
username: root
password: ${MYSQL_PASSWORD} # read from the environment, never hard-coded

Run it with datahub ingest -c recipe.dhub.yaml. New to ingestion? Start at Minimal recipe. Looking up a field? Go to Recipe structure. Hitting an error? Go to Troubleshooting.

Minimal recipe​

Only source is required. Everything else has a default.

# recipe.dhub.yaml
source:
type: mysql
config:
host_port: localhost:3306
username: root
password: ${MYSQL_PASSWORD}

With no sink, metadata is by default written to datahub-rest at the DataHub instance configured by datahub init. To be explicit, name the sink yourself:

# recipe.dhub.yaml
source:
type: mysql
config:
host_port: localhost:3306
username: root
password: ${MYSQL_PASSWORD}

sink:
type: datahub-rest
config:
server: http://localhost:8080

Install the connector plugin for your source before running — connectors ship separately from the CLI:

pip install --upgrade 'acryl-datahub[mysql]'
datahub ingest -c recipe.dhub.yaml

A recipe contains exactly one source and one sink. To ingest from several systems, write several recipes, and give each one its own pipeline_name.

More examples live in examples/recipes. Every connector's page also carries a complete starter recipe — see the list of integrations.

Recipe structure​

A recipe has the following top-level fields. Everything else is nested underneath them. In particular, a connector's own options always go under source.config, never at the top level.

FieldTypeRequiredDefaultDescriptionExample
sourceobjectYes—The system metadata is read from, and its connector options. See Source.type: mysql
sinkobjectNodatahub-restWhere metadata is written. See Sink.type: datahub-rest
transformerslist of objectsNo[]Transformations applied in order before the sink. See Transformers.- type: simple_add_dataset_tags
pipeline_namestringNononeStable identifier for this recurring job. Must be unique per source configuration. See Pipeline name.mysql-prod-orders
datahub_apiobjectNotaken from a datahub-rest sinkConnection used to read from DataHub. See Connecting to DataHub.server: http://localhost:8080
run_idstringNoauto-generated per runIdentifies one execution. See Run identification.leave unset
reportinglist of objectsNo[]Where to send run telemetry. See Reporting.- type: datahub
failure_logobjectNodisabledWrites rejected records to a local file. See Failure logging.enabled: true
recordingobjectNodisabledRecords a run for later replay. Beta. See Recording runs.enabled: true

Unknown top-level fields are rejected rather than ignored, so a typo fails the run instead of silently doing nothing. See Common mistakes.

Source​

The source is the system metadata is read from. It is the only required field. type selects the connector; config holds that connector's own options.

FieldTypeRequiredDefaultDescriptionExample
typestringYes—Connector to run. Valid values are the connector names listed on the integrations page.snowflake
configobjectNo{}Connector-specific options. Each connector's page documents its own fields.host_port: localhost:3306
extractorstringNogenericAdvanced. How source records become work units. Change only if a connector's docs say to.generic
extractor_configobjectNo{}Advanced. Options passed to the extractor named above.{}
source:
type: mysql
config:
host_port: localhost:3306 # replace with your MySQL host and port
username: root
password: ${MYSQL_PASSWORD}
database: orders # optional: limit ingestion to one database

Install the plugin for your connector before running, or the run fails with is disabled due to a missing dependency:

pip install --upgrade 'acryl-datahub[mysql]'

Connector options are documented per connector, not here: see Sources and the list of integrations.

Sink​

The sink is where extracted metadata is written. If you omit sink, DataHub uses datahub-rest with the connection from datahub init.

typeWhat it doesWhen to use it
datahub-restWrites to DataHub over HTTP. The default.Almost always, including DataHub Cloud.
datahub-kafkaPublishes metadata events to Kafka.High-volume ingestion that must not block GMS.
fileWrites the metadata events to a local JSON file.Inspecting output without writing to DataHub.
consolePrints the metadata events to stdout.Quick debugging.

Fields for datahub-rest:

FieldTypeRequiredDefaultDescriptionExample
serverstringYes—GMS endpoint of your DataHub instance.http://localhost:8080
tokenstringNononePersonal access token. Required when authentication is enabled.${DATAHUB_TOKEN}
timeout_secfloatNononePer-request timeout in seconds.30
retry_max_timesintegerNononeHow many times to retry a failed request.3
max_threadsintegerNo15Concurrent requests, in async modes.15
ca_certificate_pathstringNononeCustom CA bundle for TLS verification./etc/ssl/ca.pem
disable_ssl_verificationbooleanNofalseSkip TLS verification. Not recommended outside local testing.false
sink:
type: datahub-rest
config:
server: http://localhost:8080

Full details for each sink are in Sinks.

Transformers​

Transformers modify metadata between the source and the sink — for example attaching default ownership or tags to everything one recipe produces. They run in the order listed.

FieldTypeRequiredDefaultDescriptionExample
transformers[].typestringYes—Name of the transformer to run.simple_add_dataset_tags
transformers[].configobjectNo{}Options for that transformer.tag_urns: [...]
source:
type: mysql
config:
host_port: localhost:3306
username: root
password: ${MYSQL_PASSWORD}

transformers:
- type: simple_add_dataset_tags
config:
tag_urns:
- "urn:li:tag:Important"

See the transformers guide for the full list and for writing your own.

Pipeline name​

pipeline_name is a stable, unique name for a recurring ingestion job. It identifies the job across runs; run_id identifies a single execution of it.

pipeline_name: mysql-prod-orders

source:
type: mysql
config:
host_port: localhost:3306
username: root
password: ${MYSQL_PASSWORD}
database: orders
FieldTypeRequiredDefaultDescriptionExample
pipeline_namestringYes, for stateful ingestionnoneUnique, stable name for this recurring job. Unique per source configuration.mysql-prod-orders

It is optional, but four features depend on it:

  • Stateful ingestion. Checkpoints — the record of what each run saw — are stored against this name. Without it, stateful ingestion cannot be enabled, so stale entity removal and redundant run elimination are unavailable.
  • Run telemetry. The reporting framework groups a job's run history under this name.
  • Provenance. Every aspect the run emits is stamped with systemMetadata.pipelineName, so you can tell which recipe wrote a given piece of metadata.
  • datahub ingest deploy. When --urn is not given, the command falls back to pipeline_name to decide whether to update an existing ingestion source or create a new one.

Stateful ingestion turns on automatically when you omit the stateful_ingestion block entirely, pipeline_name is set, and the pipeline can reach DataHub (a datahub-rest sink, or an explicit datahub_api). If you write a stateful_ingestion block, it is disabled unless you set enabled: true in it.

Use a unique pipeline name for every source configuration​

One pipeline_name per source configuration — never shared

Two recipes must never use the same pipeline_name. Ingestion state is keyed on the pipeline name, so sharing one makes separate recipes read and overwrite each other's checkpoints.

Checkpoints are stored per pipeline name and per job, where the job id is derived from the connector — for most connectors <platform>_stale_entity_removal, though several keep legacy names (see StaleEntityRemovalHandler.compute_job_id):

urn:li:dataJob:(urn:li:dataFlow:(datahub,<pipeline_name>,PROD),<job_id>)

Two recipes running the same connector under one name share that state, so each run reads what the other wrote:

FeatureWhat goes wrong when the name is shared
Stale entity removalEach run compares what it just saw against the other recipe's entity list and soft-deletes the difference.
Redundant run eliminationA run skips a time window the other recipe already covered, leaving a permanent gap in usage or query data.
Incremental / resumable sourcesA run resumes from a checkpoint written against different data and silently skips records.

Whether those deletions happen quietly depends on how much the two scopes overlap. stateful_ingestion.fail_safe_threshold (default 75.0) measures the change against the previous state as (1 - overlap / previous entity count) × 100 and blocks deletion above that:

  • Nested scopes — a broad recipe plus a narrower one inside it — keep enough overlap to stay under the threshold when the narrow scope is at least a quarter of the broad one. Everything outside the narrow scope is then soft-deleted silently. This is the dangerous case. A much smaller narrow scope trips the guard instead.
  • Disjoint scopes — two separate schemas, say — share no entities, so every run scores 100% and trips the guard: nothing is deleted, but the run reports a failure each time.

Two recipes running different connectors under one name do not collide in state, because the job id differs. They do still interleave run telemetry, leave systemMetadata.pipelineName ambiguous about which recipe wrote an entity, and point datahub ingest deploy at the wrong ingestion source.

Give each of these its own pipeline_name:

  • different accounts, hosts, projects, or platform_instance values;
  • different env values (PROD and DEV);
  • different filter patterns, when you deliberately split one system across several recipes (for example one recipe per database);
  • different connector types (snowflake and bigquery) — for telemetry, provenance and deploy, as above.

Keep the pipeline name stable across runs​

The name is the key to the job's history. Renaming it orphans everything stored under the old name: the next run starts from an empty checkpoint, re-scans from scratch, and never cleans up entities the old name was responsible for. Telemetry history is detached too.

Do not put anything that varies per run — timestamps, hostnames, random suffixes — in the name. Use run_id for that.

To force a full re-ingest, set stateful_ingestion.ignore_old_state: true for that run instead of changing the name:

pipeline_name: mysql-prod-orders

source:
type: mysql
config:
host_port: localhost:3306
username: root
password: ${MYSQL_PASSWORD}
stateful_ingestion:
enabled: true
ignore_old_state: true # this run only — remove afterwards

Choosing a pipeline name​

Use a descriptive, stable name that captures what makes the source configuration unique, such as <platform>-<instance>-<scope>:

pipeline_name: snowflake-prod-finance

Pipeline names in UI ingestion​

Ingestion sources created in the DataHub UI get a pipeline_name injected automatically: the URN of the ingestion source, for example urn:li:dataHubIngestionSource:f38bd060-4ea8-459c-8f24-a773286a2927. That guarantees uniqueness, so you do not need to set one yourself. An explicit pipeline_name in the recipe is kept and not overwritten.

Connecting to DataHub​

datahub_api is the connection the pipeline uses to read from DataHub — checkpoints, reporting, and any connector feature that resolves entities already in the catalog.

You usually do not need it. When the sink is datahub-rest, that connection is reused. Set datahub_api explicitly when the sink is something else (datahub-kafka, file) but you still want stateful ingestion or reporting.

FieldTypeRequiredDefaultDescriptionExample
serverstringYes—GMS endpoint.http://localhost:8080
tokenstringNononePersonal access token, when authentication is enabled.${DATAHUB_TOKEN}
timeout_secfloatNononePer-request timeout in seconds.30
retry_status_codeslist of integersNononeHTTP status codes to retry.[429, 502]
retry_max_timesintegerNononeMaximum retry attempts.3
extra_headersobjectNononeExtra HTTP headers sent with each request.X-Env: prod
ca_certificate_pathstringNononeCustom CA bundle for TLS verification./etc/ssl/ca.pem
client_certificate_pathstringNononeClient certificate for mutual TLS./etc/ssl/client.pem
disable_ssl_verificationbooleanNofalseSkip TLS verification. Not recommended outside local testing.false
pipeline_name: mysql-prod-orders

source:
type: mysql
config:
host_port: localhost:3306
username: root
password: ${MYSQL_PASSWORD}

# Sink is Kafka, so reads need their own connection.
sink:
type: datahub-kafka
config:
connection:
bootstrap: localhost:9092
schema_registry_url: http://localhost:8081

datahub_api:
server: http://localhost:8080
token: ${DATAHUB_TOKEN}

Run identification​

run_id identifies a single execution, where pipeline_name identifies the recurring job. It is stamped on every emitted aspect as systemMetadata.runId, and it is what rollback operates on.

FieldTypeRequiredDefaultDescriptionExample
run_idstringNo<source type>-<timestamp>-<random suffix>Identifier for this one execution.leave unset

Leave it unset in almost all cases. The generated default keeps every run distinguishable. Setting a fixed run_id in a recipe makes all runs share one identifier, which defeats per-run rollback.

Reporting​

reporting lists destinations for run telemetry: whether a run succeeded, how long it took, what it produced. Telemetry is sent to every entry in the list.

FieldTypeRequiredDefaultDescriptionExample
reporting[].typestringYes—Valid values: datahub (report to DataHub), file (write the summary to disk).datahub
reporting[].configobjectNothe pipeline's datahub_apiOptions for that reporter.{}
reporting[].requiredbooleanNofalseWhen false, reporting errors are warnings. true fails the run if telemetry can't be sent.false
pipeline_name: mysql-prod-orders

source:
type: mysql
config:
host_port: localhost:3306
username: root
password: ${MYSQL_PASSWORD}

reporting:
- type: datahub
config:
report_recipe: false # omit the recipe itself from the report

The datahub reporter requires pipeline_name to be set. See the reporting framework guide.

Failure logging​

failure_log writes records the sink rejected to a local file, so you can inspect exactly what failed instead of digging through logs. Disabled by default.

FieldTypeRequiredDefaultDescriptionExample
enabledbooleanNofalseWhether rejected records are written to disk.true
log_config.filenamestringYes, when enabled—File the rejected records are written to../failures.json
source:
type: mysql
config:
host_port: localhost:3306
username: root
password: ${MYSQL_PASSWORD}

failure_log:
enabled: true
log_config:
filename: ./failures.json

Recording runs​

recording captures a full run — the recipe and the metadata it produced — into an encrypted archive that can be replayed later for debugging. Equivalent to the datahub ingest --record flag.

Beta

Recording and replay is in beta. The archive format may change between releases.

Recording needs an optional extra; without it the run fails when recording starts:

pip install --upgrade 'acryl-datahub[debug-recording]'
FieldTypeRequiredDefaultDescriptionExample
enabledbooleanNofalseWhether to record this run.true
passwordstringYes, when enabled—Encrypts the archive, which contains metadata from your systems.${RECORDING_PASSWORD}
output_pathstringYes, when s3_uploadINGESTION_ARTIFACT_DIR env var, else a temp dirLocal path, or an s3:// URL when s3_upload is true../recordings
s3_uploadbooleanNofalseUpload the archive straight to S3. Requires an s3:// output_path.false
source:
type: mysql
config:
host_port: localhost:3306
username: root
password: ${MYSQL_PASSWORD}

recording:
enabled: true
password: ${RECORDING_PASSWORD}
output_path: ./recordings

Secrets and environment variables​

Never hard-code credentials in a recipe. DataHub expands environment variables in the recipe before parsing it, using ${VAR} syntax, similar to bash or docker-compose:

source:
type: mysql
config:
host_port: localhost:3306
username: root
password: ${MYSQL_PASSWORD} # resolved from the environment at run time
export MYSQL_PASSWORD='...'
datahub ingest -c recipe.dhub.yaml

Constraints:

  • A variable that is referenced but not set fails the run with unbound variable. It does not resolve to an empty string.
  • Variable names may contain letters, digits and underscores only — no whitespace or other special characters.

For UI ingestion, store the value as a DataHub secret (Ingestion → Secrets) and reference it with the same ${VAR} syntax. See the Secret Resolution guide.

Loading values as files​

Some connectors (kafka, bigquery, mysql) need a path to a file rather than a value. That does not work for UI ingestion, where the recipe has to be self-contained. The __DATAHUB_TO_FILE_ directive writes a value to a temporary file and substitutes the path.

__DATAHUB_TO_FILE_<property>: <value> becomes <property>: <path to a file containing value>. The value can be inline or come from an environment variable or secret.

source:
type: mysql
config:
host_port: localhost:3306
username: root
password: ${MYSQL_PASSWORD}
options:
connect_args:
__DATAHUB_TO_FILE_ssl_key: "${MYSQL_SSL_KEY}"
# becomes:
# ssl_key: /tmp/path/to/file # file contains the value of ${MYSQL_SSL_KEY}

Running a recipe​

Install the CLI and the plugin for your connector, then run the recipe:

python3 -m pip install --upgrade acryl-datahub
pip install --upgrade 'acryl-datahub[mysql]'
datahub ingest -c recipe.dhub.yaml

Useful flags when testing a new recipe:

FlagWhat it does
--dry-run / -nExtracts from the source as normal but does not write to the sink. Disables stateful ingestion.
--previewStops after 10 work units, for a fast sanity check. Disables stateful ingestion.
--preview-workunits NChanges how many work units --preview produces.
--test-source-connectionOnly tests that the source credentials and connectivity work. Extracts nothing.
--strict-warningsExit non-zero if the run produced warnings. Useful in CI.

There is no offline "validate only" flag: --dry-run still connects to the source. To check a recipe without writing to DataHub, use --dry-run, or point the recipe at a file sink and inspect the output.

See the CLI Ingestion Guide and the datahub ingest reference.

Scheduling a recipe​

A recipe describes one job; it does not schedule itself. Run it on a schedule with:

Each source system needs its own recipe file and its own pipeline_name, so schedules can be managed independently.

Common mistakes​

MistakeWhat happensFix
Reusing one pipeline_name across recipesThe recipes overwrite each other's ingestion state and soft-delete each other's entities.Give every source configuration its own name. See Pipeline name.
Changing pipeline_name between runsState is orphaned: a full re-scan, and entities from the old name are never cleaned up.Keep the name stable; use stateful_ingestion.ignore_old_state: true for a one-off full re-ingest.
Putting connector options at the top levelExtra inputs are not permitted — the run fails immediately.Nest connector options under source.config.
Two sources in one recipeOnly one source key is allowed; YAML keeps the last one silently.Write one recipe per source.
Hard-coding passwordsCredentials end up in version control.Use ${VAR} and set the environment variable, or a DataHub secret for UI ingestion.
Forgetting the connector pluginis disabled due to a missing dependency.pip install 'acryl-datahub[<connector>]'.
Enabling stateful ingestion without pipeline_nameThe run fails at startup.Set a unique pipeline_name.
Setting a fixed run_idEvery run shares one id, so per-run rollback no longer works.Leave run_id unset.

Troubleshooting​

Find the error text from your run below.

Extra inputs are not permitted [type=extra_forbidden]​

A field name is not recognised at the level you put it. The error names the offending field and the model it was checked against — PipelineConfig means a top-level field, a connector config class such as MySQLConfig means a field under source.config.

Common causes: a typo (pipelineName instead of pipeline_name), a connector option placed at the top level instead of under source.config, or an option that belongs to a different connector.

Did not find a registered class for <type>​

The value of source.type (or sink.type) is not a known connector. Check the spelling against the list of integrations.

<connector> is disabled due to a missing dependency​

The connector's plugin is not installed. The message includes the command to run:

pip install --upgrade 'acryl-datahub[<connector>]'

'<VAR>: unbound variable'​

The recipe references ${<VAR>} but that environment variable is not set. Export it before running, or define it as a DataHub secret for UI ingestion. See Secrets and environment variables.

Field required [type=missing]​

A required connector option is absent. The error names the field and the connector config class. Check that connector's page for which options are required.

pipeline_name must be set when stateful ingestion is enabled​

Stateful ingestion needs a name to store state against. Add a pipeline_name that is unique to this source configuration and stable across runs — see Pipeline name.

Entities are being soft-deleted unexpectedly​

Usually two recipes sharing one pipeline_name, so each run treats the other's entities as stale. Give each source configuration its own name. See Use a unique pipeline name for every source configuration.

Skipping stateful ingestion / stale entity removal​

The entity count changed by more than stateful_ingestion.fail_safe_threshold (default 75%) since the last run, so deletions were blocked as a safety measure. This is reported as a failure, so the run shows as failed even though ingestion itself completed.

Check whether the source configuration changed. If it did not, check whether another recipe writes to the same pipeline_name over a different scope — two recipes sharing a name over disjoint scopes score 100% change and trip this on every run. If the change is intentional, raise the threshold for that run.

FAQ​

How do I write my first recipe? Name a source and run it. See Minimal recipe.

Can one recipe ingest from two systems? No. A recipe has exactly one source and one sink. Write one recipe per system, each with its own pipeline_name.

Do I have to set pipeline_name? Not for a one-off run. You do need it for stateful ingestion, run telemetry, and datahub ingest deploy. When you set one, it must be unique to that source configuration — see Pipeline name.

Can two recipes share a pipeline_name if they ingest different systems? No. Different connectors do not overwrite each other's ingestion state, but they still interleave run telemetry, make it ambiguous which recipe wrote an entity, and send datahub ingest deploy to the wrong ingestion source. Two recipes on the same connector additionally corrupt each other's state.

What is the difference between pipeline_name and run_id? pipeline_name names the recurring job and stays the same across runs. run_id identifies one execution and is generated fresh each time.

How do I keep secrets out of the recipe? Reference environment variables with ${VAR}, or DataHub secrets for UI ingestion. See Secrets and environment variables.

How do I test a recipe without writing to DataHub? Run it with --dry-run, or point it at a file sink and inspect the output. See Running a recipe.

How do I run ingestion on a schedule? Use UI ingestion's cron schedule, Airflow, cron, or Kubernetes. See Scheduling a recipe.

Where do I find the options for my connector? On that connector's page — see the list of integrations. They all go under source.config.

Why did my run fail validation? Find the error text in Troubleshooting.

Editor autocomplete​

Name your recipe with the .dhub.yaml extension — for example myrecipe.dhub.yaml — to get autocomplete and schema validation in VS Code or IntelliJ. Install the YAML plugin for your editor: