Skip to main content

Google Cloud Storage

Overview​

Google Cloud Storage is a storage and lakehouse platform. Learn more in the official Google Cloud Storage documentation.

The DataHub integration for Google Cloud Storage covers file/lakehouse metadata entities such as datasets, paths, and containers. It also captures stateful deletion detection.

Concept Mapping​

Source ConceptDataHub ConceptNotes
"Google Cloud Storage"Data Platform
GCS object / Folder containing GCS objectsDataset
GCS bucketContainerSubtype GCS bucket
GCS folderContainerSubtype Folder

Module gcs​

Incubating

Important Capabilities​

CapabilityStatusNotes
Asset Containers✅Enabled by default. Supported for types - GCS bucket, Folder.
Data Profiling✅Optionally enabled via configuration.
Detect Deleted Entities✅Enabled by default via stateful ingestion.
Schema Metadata✅Enabled by default.

Overview​

The gcs module ingests metadata from Gcs into DataHub. It is intended for production ingestion workflows and module-specific capabilities are documented below.

This connector ingests Google Cloud Storage datasets into DataHub. It allows mapping an individual file or a folder of files to a dataset in DataHub. To specify the group of files that form a dataset, use path_specs configuration in ingestion recipe. This source leverages Interoperability of GCS with S3 and uses DataHub S3 Data Lake integration source under the hood. Refer section Path Specs from S3 connector for more details.

Prerequisites​

Before running ingestion, ensure network connectivity to the source, valid authentication credentials, and read permissions for metadata APIs required by this module.

The GCS source supports three authentication methods:

  1. HMAC Keys (hmac, default): Long-lived key-based authentication using Google Cloud Storage HMAC keys. Suitable for simple setups and service accounts outside GCP.
  2. GKE Workload Identity (workload_identity): Keyless authentication using Application Default Credentials. The recommended option when DataHub runs inside GKE with Workload Identity enabled — no credential config required.
  3. Workload Identity Federation (workload_identity_federation): Keyless, token-based authentication for workloads running outside GCP (AWS, Azure, on-premises). Exchanges an external identity token for short-lived GCP credentials via GCP's STS endpoint.

HMAC Authentication​

  1. Create a service account with "Storage Object Viewer" role — Create a service account.
  2. Ensure you meet the requirements to generate an HMAC key.
  3. Create an HMAC key for the service account — Create HMAC keys.

GKE Workload Identity​

This is the simplest option for DataHub running inside GKE. No credential files or secrets are needed — the pod's Kubernetes Service Account is used automatically.

  1. Enable Workload Identity on your GKE cluster — Enable Workload Identity.

  2. Create a Google Service Account (GSA) with "Storage Object Viewer" role.

  3. Bind your Kubernetes Service Account (KSA) to the GSA:

    gcloud iam service-accounts add-iam-policy-binding GSA_EMAIL \
      --role roles/iam.workloadIdentityUser \
      --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"
    kubectl annotate serviceaccount KSA_NAME \
      --namespace NAMESPACE \
      iam.gke.io/gcp-service-account=GSA_EMAIL

  4. Set auth_type: workload_identity in your recipe. No credential or WIF config fields are needed.

Workload Identity Federation​

Use this when DataHub runs outside GCP and you want keyless authentication without distributing service account key files.

  1. Set up a Workload Identity Pool and Provider in Google Cloud — Workload Identity Federation.
  2. Grant the external identity permission to impersonate a GCP service account with "Storage Object Viewer" role.
  3. Download or generate the WIF credential configuration JSON from the Google Cloud Console.
  4. Supply the configuration to the connector via one of three options: a file path (gcp_wif_configuration), inline dict (gcp_wif_configuration_json), or JSON string (gcp_wif_configuration_json_string). The JSON string option is useful for injecting from a secrets manager.

Install the Plugin​

pip install 'acryl-datahub[gcs]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

# HMAC Authentication (default)
source:
  type: gcs
  config:
    path_specs:
      - include: gs://gcs-ingestion-bucket/parquet_example/{table}/year={partition[0]}/*.parquet
    credential:
      hmac_access_id: <hmac access id>
      hmac_access_secret: <hmac access secret>

---

# GKE Workload Identity (Application Default Credentials)
# Recommended for DataHub running inside GKE with Workload Identity enabled.
# No credential config needed — credentials are sourced automatically from the pod's service account.
source:
  type: gcs
  config:
    auth_type: workload_identity
    path_specs:
      - include: gs://gcs-ingestion-bucket/parquet_example/{table}/year={partition[0]}/*.parquet

---

# Workload Identity Federation with configuration file
source:
  type: gcs
  config:
    auth_type: workload_identity_federation
    gcp_wif_configuration: "/path/to/gcp_wif_configuration.json"
    path_specs:
      - include: gs://gcs-ingestion-bucket/parquet_example/{table}/year={partition[0]}/*.parquet

---

# Workload Identity Federation with inline configuration (dict)
source:
  type: gcs
  config:
    auth_type: workload_identity_federation
    gcp_wif_configuration_json:
      type: external_account
      audience: "//iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID"
      subject_token_type: "urn:ietf:params:oauth:token-type:jwt"
      token_url: "https://sts.googleapis.com/v1/token"
      credential_source:
        file: "/var/run/secrets/tokens/gcp-ksa/token"
      service_account_impersonation_url: "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/SERVICE_ACCOUNT_EMAIL:generateAccessToken"
    path_specs:
      - include: gs://gcs-ingestion-bucket/parquet_example/{table}/year={partition[0]}/*.parquet

---

# Workload Identity Federation with JSON string (copy-paste from file)
source:
  type: gcs
  config:
    auth_type: workload_identity_federation
    gcp_wif_configuration_json_string: |
      {
        "type": "external_account",
        "audience": "//iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID",
        "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
        "token_url": "https://sts.googleapis.com/v1/token",
        "credential_source": {
          "file": "/var/run/secrets/tokens/gcp-ksa/token"
        },
        "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/SERVICE_ACCOUNT_EMAIL:generateAccessToken"
      }
    path_specs:
      - include: gs://gcs-ingestion-bucket/parquet_example/{table}/year={partition[0]}/*.parquet

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
path_specs ✅
array
List of PathSpec. See below the details about PathSpec
path_specs.PathSpec
PathSpec
path_specs.PathSpec.include ❓
string
Path to table. Name variable {table} is used to mark the folder with dataset. In absence of {table}, file level dataset will be created. Check below examples for more details.
path_specs.PathSpec.allow_double_stars
boolean
Allow double stars in the include path. This can affect performance significantly if enabled
Default: False
path_specs.PathSpec.autodetect_partitions
boolean
Autodetect partition(s) from the path. If set to true, it will autodetect partition key/value if the folder format is {partition_key}={partition_value} for example year=2024
Default: True
path_specs.PathSpec.default_extension
One of string, null
For files without extension it will assume the specified file type. If it is not set the files without extensions will be skipped.
Default: None
path_specs.PathSpec.emit_folders_only
boolean
If set, this path_spec emits each matching storage folder as a DataHub Container and creates no dataset entities. Folder depth is set by the number of wildcard levels in include (e.g. .../*/ one level deep, .../*/*/ two levels). exclude and include_hidden_folders still apply to the folder walk; file/dataset fields (file_types, default_extension, table_name, tables_filter_pattern) are rejected since no datasets are produced. include must end at a folder and must not contain {table} or **.
Default: False
path_specs.PathSpec.enable_compression
boolean
Enable or disable processing compressed files. Currently .gz and .bz files are supported.
Default: True
path_specs.PathSpec.include_hidden_folders
boolean
Include hidden folders in the traversal (folders starting with . or _
Default: False
path_specs.PathSpec.sample_files
boolean
Not listing all the files but only taking a handful amount of sample file to infer the schema. File count and file size calculation will be disabled. This can affect performance significantly if enabled
Default: True
path_specs.PathSpec.table_name
One of string, null
Display name of the dataset.Combination of named variables from include path and strings
Default: None
path_specs.PathSpec.traversal_method
Enum
One of: "ALL", "MIN_MAX", "MAX"
path_specs.PathSpec.exclude
One of array, null
list of paths in glob pattern which will be excluded while scanning for the datasets
Default: []
path_specs.PathSpec.exclude.string
string
path_specs.PathSpec.file_types
array
Files with extenstions specified here (subset of default value) only will be scanned to create dataset. Other files will be omitted.
Default: ['csv', 'tsv', 'json', 'parquet', 'avro']
path_specs.PathSpec.file_types.string
string
path_specs.PathSpec.tables_filter_pattern
AllowDenyPattern
A class to store allow deny regexes
path_specs.PathSpec.tables_filter_pattern.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
auth_type
Enum
One of: "hmac", "workload_identity_federation", "workload_identity"
convert_urns_to_lowercase
boolean
Whether to convert dataset urns to lowercase. This value is part of each dataset's URN identity, so it must stay fixed for the life of a deployment. Changing it after data has been ingested re-keys every dataset (e.g. MyDb.MyTable becomes mydb.mytable); with stateful ingestion enabled the old-cased URNs are then soft-deleted as stale while the new-cased ones are created, producing duplicate or orphaned entities. Pick one value before the first run and leave it unchanged.
Default: False
gcp_wif_configuration
One of string, null
Path to the GCP Workload Identity Federation configuration JSON file. Mutually exclusive with gcp_wif_configuration_json and gcp_wif_configuration_json_string.
Default: None
gcp_wif_configuration_json
One of object, null
GCP Workload Identity Federation configuration as a dict or a JSON string. Mutually exclusive with gcp_wif_configuration and gcp_wif_configuration_json_string.
Default: None
gcp_wif_configuration_json_string
One of string, null
GCP Workload Identity Federation configuration as a JSON string (contents of the configuration file). Useful for injecting configuration from secrets managers. Mutually exclusive with gcp_wif_configuration and gcp_wif_configuration_json. Note: WIF configuration typically contains public endpoint URLs rather than private keys, so SecretStr masking is not applied. If your WIF config contains sensitive material, ensure it is not logged at DEBUG level.
Default: None
max_rows
integer
Maximum number of rows to use when inferring schemas for TSV and CSV files.
Default: 100
number_of_files_to_sample
integer
Number of files to list to sample for schema inference. This will be ignored if sample_files is set to False in the pathspec.
Default: 100
platform_instance
One of string, null
The instance of the platform that all assets produced by this recipe belong to. This should be unique within the platform. See https://docs.datahub.com/docs/platform-instances/ for more details.
Default: None
env
string
The environment that all assets produced by this connector belong to
Default: PROD
credential
One of HMACKey, null
Google cloud storage HMAC keys. Required when auth_type is 'hmac'.
Default: None
credential.hmac_access_id ❓
string
Access ID
credential.hmac_access_secret ❓
string(password)
Secret
profile_patterns
AllowDenyPattern
A class to store allow deny regexes
profile_patterns.ignoreCase
One of boolean, null
Whether to ignore case sensitivity during pattern matching.
Default: True
profile_patterns.allow
array
List of regex patterns to include in ingestion
Default: ['.*']
profile_patterns.allow.string
string
profile_patterns.deny
array
List of regex patterns to exclude from ingestion.
Default: []
profile_patterns.deny.string
string
profiling
DataLakeProfilerConfig
profiling.enabled
boolean
Whether profiling should be done.
Default: False
profiling.include_field_distinct_value_frequencies
boolean
Whether to profile for distinct value frequencies.
Default: True
profiling.include_field_histogram
boolean
Whether to profile for the histogram for numeric fields.
Default: True
profiling.include_field_max_value
boolean
Whether to profile for the max value of numeric columns.
Default: True
profiling.include_field_mean_value
boolean
Whether to profile for the mean value of numeric columns.
Default: True
profiling.include_field_median_value
boolean
Whether to profile for the median value of numeric columns.
Default: True
profiling.include_field_min_value
boolean
Whether to profile for the min value of numeric columns.
Default: True
profiling.include_field_null_count
boolean
Whether to profile for the number of nulls for each column.
Default: True
profiling.include_field_quantiles
boolean
Whether to profile for the quantiles of numeric columns.
Default: True
profiling.include_field_sample_values
boolean
Whether to profile for the sample values for all columns.
Default: True
profiling.include_field_stddev_value
boolean
Whether to profile for the standard deviation of numeric columns.
Default: True
profiling.max_number_of_fields_to_profile
One of integer, null
A positive integer that specifies the maximum number of columns to profile for any table. None implies all columns. The cost of profiling goes up significantly as the number of columns to profile goes up.
Default: None
profiling.profile_table_level_only
boolean
Whether to perform profiling at table-level only or include column-level profiling as well.
Default: False
profiling.operation_config
OperationConfig
profiling.operation_config.lower_freq_profile_enabled
boolean
Whether to do profiling at lower freq or not. This does not do any scheduling just adds additional checks to when not to run profiling.
Default: False
profiling.operation_config.profile_date_of_month
One of integer, null
Number between 1 to 31 for date of month (both inclusive). If not specified, defaults to Nothing and this field does not take affect.
Default: None
profiling.operation_config.profile_day_of_week
One of integer, null
Number between 0 to 6 for day of week (both inclusive). 0 is Monday and 6 is Sunday. If not specified, defaults to Nothing and this field does not take affect.
Default: None
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Capabilities​

Use the Important Capabilities table above as the source of truth for supported features and whether additional configuration is required.

Path Specs​

Example - Dataset per file

Bucket structure:

test-gs-bucket
├── employees.csv
└── food_items.csv

Path specs config

path_specs:
    - include: gs://test-gs-bucket/*.csv

Example - Datasets with partitions

Bucket structure:

test-gs-bucket
├── orders
│   └── year=2022
│       └── month=2
│           ├── 1.parquet
│           └── 2.parquet
└── returns
    └── year=2021
        └── month=2
            └── 1.parquet

Path specs config:

path_specs:
    - include: gs://test-gs-bucket/{table}/{partition_key[0]}={partition[0]}/{partition_key[1]}={partition[1]}/*.parquet

Example - Datasets with partition and exclude

Bucket structure:

test-gs-bucket
├── orders
│   └── year=2022
│       └── month=2
│           ├── 1.parquet
│           └── 2.parquet
└── tmp_orders
    └── year=2021
        └── month=2
            └── 1.parquet


Path specs config:

path_specs:
    - include: gs://test-gs-bucket/{table}/{partition_key[0]}={partition[0]}/{partition_key[1]}={partition[1]}/*.parquet
      exclude:
        - **/tmp_orders/**

Example - Datasets of mixed nature

Bucket structure:

test-gs-bucket
├── customers
│   ├── part1.json
│   ├── part2.json
│   ├── part3.json
│   └── part4.json
├── employees.csv
├── food_items.csv
├── tmp_10101000.csv
└──  orders
    └── year=2022
        └── month=2
            ├── 1.parquet
            ├── 2.parquet
            └── 3.parquet

Path specs config:

path_specs:
    - include: gs://test-gs-bucket/*.csv
      exclude:
        - **/tmp_10101000.csv
    - include: gs://test-gs-bucket/{table}/*.json
    - include: gs://test-gs-bucket/{table}/{partition_key[0]}={partition[0]}/{partition_key[1]}={partition[1]}/*.parquet

Valid path_specs.include

gs://my-bucket/foo/tests/bar.avro # single file table
gs://my-bucket/foo/tests/*.* # mulitple file level tables
gs://my-bucket/foo/tests/{table}/*.avro #table without partition
gs://my-bucket/foo/tests/{table}/*/*.avro #table where partitions are not specified
gs://my-bucket/foo/tests/{table}/*.* # table where no partitions as well as data type specified
gs://my-bucket/{dept}/tests/{table}/*.avro # specifying keywords to be used in display name
gs://my-bucket/{dept}/tests/{table}/{partition_key[0]}={partition[0]}/{partition_key[1]}={partition[1]}/*.avro # specify partition key and value format
gs://my-bucket/{dept}/tests/{table}/{partition[0]}/{partition[1]}/{partition[2]}/*.avro # specify partition value only format
gs://my-bucket/{dept}/tests/{table}/{partition[0]}/{partition[1]}/{partition[2]}/*.* # for all extensions
gs://my-bucket/*/{table}/{partition[0]}/{partition[1]}/{partition[2]}/*.* # table is present at 2 levels down in bucket
gs://my-bucket/*/*/{table}/{partition[0]}/{partition[1]}/{partition[2]}/*.* # table is present at 3 levels down in bucket

Valid path_specs.exclude

  • **/tests/**
  • gs://my-bucket/hr/**
  • */tests/.csv
  • gs://my-bucket/foo/*/my_table/**

Notes

  • {table} represents folder for which dataset will be created.
  • include path must end with (*.* or *.[ext]) to represent leaf level.
  • if *.[ext] is provided then only files with specified type will be scanned.
  • /*/ represents single folder.
  • {partition[i]} represents value of partition.
  • {partition_key[i]} represents name of the partition.
  • While extracting, "i" will be used to match partition_key to partition.
  • all folder levels need to be specified in include. Only exclude path can have ** like matching.
  • exclude path cannot have named variables ( {} ).
  • Folder names should not contain {, }, *, / in their names.
  • {folder} is reserved for internal working. please do not use in named variables.

If you would like to write a more complicated function for resolving file names, then a {transformer} would be a good fit.

caution

Specify as long fixed prefix ( with out /*/ ) as possible in path_specs.include. This will reduce the scanning time and cost, specifically on Google Cloud Storage.

caution

If you are ingesting datasets from Google Cloud Storage, we recommend running the ingestion on a server in the same region to avoid high egress costs.

Limitations​

Module behavior is constrained by source APIs, permissions, and metadata exposed by the platform. Refer to capability notes for unsupported or conditional features.

Supported file types​

Supported file types are as follows:

  • CSV
  • TSV
  • JSONL
  • JSON
  • Parquet
  • Apache Avro

Schemas for Parquet and Avro files are extracted as provided.

Schemas for schemaless formats (CSV, TSV, JSONL, JSON) are inferred. For CSV, TSV and JSONL files, we consider the first 100 rows by default, which can be controlled via the max_rows recipe parameter (see below) JSON file schemas are inferred on the basis of the entire file (given the difficulty in extracting only the first few objects of the file), which may impact performance. We are working on using iterator-based JSON parsers to avoid reading in the entire JSON object.

Profiling​

Profiling is supported for GCS and, when enabled, extracts:

  • Row and column counts for each dataset
  • For each column, if profiling is enabled:
    • null counts and proportions
    • distinct counts and proportions
    • minimum, maximum, mean, median, standard deviation, some quantile values
    • histograms or frequencies of unique values

Profiling is a pure-Python implementation (built on pyarrow and Apache DataSketches) and requires no Spark, Hadoop, or JVM. Distinct counts and quantiles/histograms are approximate (DataSketches). GCS files are read over the S3-interoperability endpoint using the same credentials configured for ingestion, so no extra setup is needed beyond enabling profiling.

Enabling profiling will slow down ingestion runs.

Troubleshooting​

If ingestion fails, validate credentials, permissions, connectivity, and scope filters first. Then review ingestion logs for source-specific errors and adjust configuration accordingly.

Code Coordinates​

  • Class Name: datahub.ingestion.source.gcs.gcs_source.GCSSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for Google Cloud Storage, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.