Remote Executor best practices

Feature Availability

Self-Hosted DataHub

DataHub Cloud

Operational guidance for running Remote Executors reliably — how to size CPU, memory, and storage for your ingestion workload, and how the executor uses ephemeral disk so you can keep it from filling up. This is a living document; sections are added as recurring operational questions come up.

Resource allocation

An executor runs multiple ingestion and observe tasks in parallel. CPU, memory, and disk each scale differently with that concurrency, so size them independently rather than picking a single "instance size".

CPU and parallelism

Core count should track how many ingestions you want to run at once. Each concurrent task is largely single-threaded from the executor's point of view, so a reasonable starting point is roughly one core per parallel ingestion, plus one for the executor's own coordination.

The number of parallel tasks is capped by environment variables:

• DATAHUB_EXECUTOR_INGESTION_MAX_WORKERS (default 4) — maximum concurrent Ingestion tasks.
• DATAHUB_EXECUTOR_MONITORS_MAX_WORKERS (default 10) — maximum concurrent Observe monitor tasks.

In the DataHub-provided Helm chart these map to datahub.executor_ingestions_workers / datahub.executor_monitors_workers. If you raise the worker count, raise cores to match — otherwise tasks contend for CPU and every run gets slower.

The general recommendation would be to assign cores to the below formula:

1 + n + k

Where:

• n - number of expected concurrent ingestion runs handled by the executor
• k - 1 if Observe functionalities are enabled for resources ingested by the executor, else 0

Memory

Memory is the hardest resource to size, because per-ingestion usage varies enormously — anywhere from ~100 MB for a small source to 4–6 GB (occasionally more, in rare cases) for a large one. It scales with both the type of source and the amount of metadata pulled in a single run.

As a rule of thumb, data-warehouse ingestions — Snowflake, BigQuery, Redshift — are the heavy hitters, routinely peaking around 4–6 GB each when profiling or extracting large schemas and lineage.

Because tasks run concurrently, size memory for the expected peak of all runs happening at the same time, not the average of a single run: n warehouse ingestions scheduled together can momentarily need roughly n × 6 GB.

Because of this, Stagger large ingestions. Schedule big warehouse runs so they don't overlap. This is the simplest and most effective lever — two 6 GB runs an hour apart need 6 GB of headroom; the same two at once need 12 GB.

Note that the executor itself consumes some memory, as does each running Observe task.

Storage

Two things consume the executor's ephemeral disk (under /tmp/datahub/) during a run, and they are reclaimed differently:

• Dynamic virtual environments under /tmp/datahub/ingest/<execution-id>/, built per run for non-bundled CLI versions and connectors. Each is deleted when its run ends, so venv disk is released continuously as runs finish. The shared uv cache that keeps this bounded is covered in the uv cache below.
• Per-execution logs under /tmp/datahub/logs/. Unlike venvs, logs are kept after the run so you can inspect them directly (logs visible via UI are stored in DataHub and are truncated)

By default nothing prunes the logs. Enable the in-process log garbage collector with DATAHUB_EXECUTOR_LOG_GC_ENABLED=true: it scans the log directory on a timer and deletes per-execution directories older than the retention window, with a size cap as a safety net. Tunables:

Variable	Default	Purpose
DATAHUB_EXECUTOR_LOG_GC_ENABLED	false	Master switch for the log GC.
DATAHUB_EXECUTOR_LOG_DIR	/tmp/datahub/logs	Directory scanned.
DATAHUB_EXECUTOR_LOG_GC_RETENTION_DAYS	14	Age threshold for deletion.
DATAHUB_EXECUTOR_LOG_GC_MAX_DIR_SIZE_MB	10000	Size-cap safety net; oldest logs are removed first when exceeded. 0 disables the cap.
DATAHUB_EXECUTOR_LOG_GC_INTERVAL_SECONDS	3600	How often the GC runs.
DATAHUB_EXECUTOR_LOG_GC_IN_FLIGHT_GRACE_SECONDS	3600	Protects logs of recently or still-running executions from deletion.

Size ephemeral storage — Fargate ephemeral_storage, or the node / emptyDir backing /tmp on Kubernetes — for the largest single venv, plus the uv cache, plus your retained-log budget.

Ingestion virtual environments and the uv cache

Runs that target a non-bundled CLI version or connector build a dynamic virtual environment per execution under /tmp/datahub/ingest/<execution-id>/, removed when the run ends. To stop repeated installs from filling ephemeral storage, package installs go through the uv cache: each package is unpacked once into UV_CACHE_DIR (by default: $HOME/.cache/uv) and shared across venvs, so many runs that share a dependency pay for its bytes roughly once rather than once per run.

The DataHub-provided Helm chart, Terraform module, and CloudFormation template set UV_LINK_MODE=hardlink by default, which links files from the cache instead of copying them — the most space-efficient way to share it. Other modes (copy, clone, symlink) trade space for portability. See Dynamic venvs and the uv cache for the full mechanism and the UV_LINK_MODE options.

For the cache to actually save space, it has to sit on the same filesystem as the venvs it feeds:

Keep the uv cache and /tmp on the same filesystem

Hardlinks (and clone reflinks) only work within one filesystem. If you mount a separate volume or an emptyDir at /tmp — for a bigger ingestion disk, or to satisfy a read-only root — but leave the cache on the container root, uv cannot link and falls back to copying, so every venv consumes the full size of its dependencies. Put the cache on the same volume as /tmp/datahub/ingest with UV_CACHE_DIR, or keep both on the container root.

Read-only root filesystem

The default cache path ($HOME/.cache/uv, i.e. /home/datahub/.cache/uv) lives on the container root. If you set readOnlyRootFilesystem: true (Kubernetes) or readonly_root_filesystem = true (ECS), uv cannot write its cache there and dynamic venv builds fail outright — non-bundled runs will not execute at all. This is not about losing the storage savings; the runs stop working.

Point UV_CACHE_DIR at a writable volume. Mounting a single emptyDir at /tmp and placing the cache under it satisfies both the writability and the same-filesystem requirement at once:

# Kubernetes (datahub-executor-worker values.yaml)
extraVolumes:
  - name: ingest-tmp
    emptyDir: {}
extraVolumeMounts:
  - name: ingest-tmp
    mountPath: /tmp
extraEnvs:
  - name: UV_CACHE_DIR
    value: /tmp/uv-cache # same emptyDir as /tmp/datahub/ingest → links work

Deployments that pin every source to the bundled CLI version don't build dynamic venvs and are unaffected.