Skip to main content

GitHub

Overview​

GitHub hosts source files, documentation, and knowledge bases in git repositories. Learn more in the official GitHub documentation.

The DataHub github-documents integration ingests markdown and text files from a GitHub repository as Document entities. Folder structure in the repository is preserved as parent-child document relationships in DataHub.

Concept Mapping​

Source ConceptDataHub ConceptNotes
Repository folderDocumentOptional folder documents for navigation.
Markdown / text fileDocumentNative (editable) or external (read-only) depending on config.
Repository + branchCustom metadataStored on documents for traceability.

Module github-documents​

Alpha

Important Capabilities​

CapabilityStatusNotes
Detect Deleted Entities✅Enabled by default via stateful ingestion.
Test Connection✅Enabled by default.

Overview​

The github-documents source ingests files from a GitHub repository branch using the GitHub REST API. It is designed for scheduled, repeatable imports of documentation into DataHub—especially context documents nested under a parent folder.

Prerequisites​

GitHub access token​

Create a GitHub Personal Access Token (classic or fine-grained) with read access to the target repository:

  • repo (private repositories) or contents: read (fine-grained)

Store the token in a DataHub secret and reference it from your ingestion recipe.

Repository access​

Ensure the token can read the repository, branch, and paths you configure.

Supported imports​

  • Import .md, .txt, and other configured extensions
  • Preserve folder hierarchy as parent-child documents
  • Optionally attach imported trees under a parent document URN
  • Import as native (editable) or external (read-only) documents
  • Schedule recurring syncs via ingestion sources

Install the Plugin​

pip install 'acryl-datahub[github-documents]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
type: github-documents
config:
github_token: "${GITHUB_TOKEN}"
repository: acme/handbook
branch: main
path_prefix: docs
file_extensions:
- .md
- .txt
parent_document_urn: null
create_repo_root_document: true
max_files: 500
document_import_mode: NATIVE
show_in_global_context: true
stateful_ingestion:
enabled: true

sink:
type: datahub-rest
config:
server: "http://localhost:8080"

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
github_token ✅
string(password)
GitHub access token (PAT or GitHub App installation token).
repository ✅
string
Repository to ingest, as 'owner/repo' or 'https://github.com/owner/repo'.
branch
string
Branch to read files from.
Default: main
create_repo_root_document
boolean
When True, create a folder document named after the repository and nest imported files beneath it.
Default: True
document_import_mode
Enum
One of: "NATIVE", "EXTERNAL"
enable_okf
boolean
Opt in to importing OKF concepts inside discovered or configured bundles. Requires a server that supports the okfProperties aspect; enabling this on an unsupported server fails the run. Disabled by default to import every file verbatim without OKF metadata.
Default: False
include_organization_in_browse_path
boolean
Include the GitHub organization/owner name as the top-most entry in the browse path, above the repository name.
Default: False
include_repository_in_browse_path
boolean
Include the repository name as a top-level entry in the browse path. When a repository root document is created, that document already provides the repository entry; this adds it for configurations that skip the root document or nest under a configured parent document.
Default: True
max_files
integer
Maximum number of matching files to import per run. Additional matches are skipped with a report warning.
Default: 500
parent_document_urn
One of string, null
Optional parent document URN. Top-level imported items are nested beneath this document while preserving the GitHub folder hierarchy.
Default: None
path_prefix
string
Only ingest files under this path (e.g. 'docs').
Default:
show_in_global_context
boolean
Whether imported documents appear in global search and navigation.
Default: True
document_mapping
DocumentMappingConfig
Document entity mapping configuration.
document_mapping.id_pattern
string
Pattern for generating document IDs
Default: {source_type}-{directory}-{basename}
document_mapping.status
Enum
One of: "PUBLISHED", "UNPUBLISHED"
Default: PUBLISHED
document_mapping.id_normalization
IdNormalizationConfig
Document ID normalization rules.
document_mapping.id_normalization.lowercase
boolean
Convert to lowercase
Default: True
document_mapping.id_normalization.max_length
integer
Maximum ID length
Default: 200
document_mapping.id_normalization.remove_special_chars
boolean
Remove special characters except _ and -
Default: True
document_mapping.id_normalization.replace_spaces_with
string
Replace spaces with this character
Default: -
document_mapping.source
SourceConfig
Document source configuration.
document_mapping.source.include_external_id
boolean
Include external ID in DocumentSource
Default: True
document_mapping.source.include_external_url
boolean
Include external URL in DocumentSource
Default: True
document_mapping.source.type
Enum
One of: "NATIVE", "EXTERNAL"
Default: EXTERNAL
document_mapping.title
TitleExtractionConfig
Title extraction configuration.
document_mapping.title.extract_from_content
boolean
Try to extract title from document content
Default: True
document_mapping.title.fallback_to_filename
boolean
Use filename as title if not found in content
Default: True
document_mapping.title.max_length
integer
Maximum title length
Default: 500
file_extensions
array
File extensions to include (include the leading dot).
file_extensions.string
string
hierarchy
HierarchyConfig
Hierarchy configuration.
hierarchy.enabled
boolean
Enable parent-child relationships
Default: True
hierarchy.parent_strategy
Enum
One of: "folder", "none", "custom", "notion", "confluence"
Default: folder
hierarchy.custom_mapping
One of CustomMappingConfig, null
Custom mapping configuration
Default: None
hierarchy.custom_mapping.rules
array
Custom parent mapping rules
hierarchy.custom_mapping.rules.CustomParentRule
CustomParentRule
Custom parent mapping rule.
hierarchy.custom_mapping.rules.CustomParentRule.parent_id ❓
string
Parent document ID for matching files
hierarchy.custom_mapping.rules.CustomParentRule.pattern ❓
string
Glob pattern to match file paths
hierarchy.folder_mapping
FolderMappingConfig
Folder hierarchy mapping configuration.
hierarchy.folder_mapping.create_parent_docs
boolean
Create Document entities for folders
Default: True
hierarchy.folder_mapping.max_depth
integer
Maximum hierarchy depth
Default: 10
hierarchy.folder_mapping.parent_id_pattern
string
Pattern for parent document IDs
Default: {source_type}-{directory}
hierarchy.folder_mapping.root_parent
One of string, null
Optional root document URN
Default: None
okf_bundle_roots
array
Additional bundle roots relative to the repository, for bundles without a versioned index.md. Use an empty string for the repository root. Directories with an index.md declaring okf_version are also discovered.
okf_bundle_roots.string
string
stateful_ingestion
One of StatefulStaleMetadataRemovalConfig, null
Stateful Ingestion Config
Default: None
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False
Default: False
stateful_ingestion.fail_safe_threshold
number
Prevents large amount of soft deletes & the state from committing from accidental changes to the source configuration if the relative change percent in entities compared to the previous state is above the 'fail_safe_threshold'.
Default: 75.0
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Capabilities​

Context documents under a parent folder​

source:
type: github-documents
config:
github_token: "${GITHUB_TOKEN}"
repository: example-org/analytics
branch: main
path_prefix: docs
parent_document_urn: "urn:li:document:context-handbook"
document_import_mode: NATIVE
show_in_global_context: true
stateful_ingestion:
enabled: true

sink:
type: datahub-rest
config:
server: "http://localhost:8080"

Read-only external references​

source:
type: github-documents
config:
github_token: "${GITHUB_TOKEN}"
repository: https://github.com/example-org/analytics
document_import_mode: EXTERNAL
stateful_ingestion:
enabled: true

sink:
type: datahub-rest
config:
server: "http://localhost:8080"

Plain-body serialization​

Files are imported as plain markdown or text by default. Set enable_okf: true to import OKF concepts (below). Other document metadata lives in DataHub aspects and customProperties.

Open Knowledge Format bundles​

Opt in with enable_okf: true to import Open Knowledge Format (OKF) 0.2 concepts alongside ordinary documentation. A concept must be a Markdown file inside an OKF bundle and have a type key in its YAML frontmatter. A type key alone does not enable OKF handling for a Hugo page or another document outside a bundle.

With OKF enabled, the source discovers bundle roots from index.md files whose frontmatter contains okf_version:

---
okf_version: "0.2"
---

# Analytics knowledge bundle

- [Monthly active users](guides/mau.md)

An index may define a bundle at the repository root or in a subdirectory, such as context/index.md. If bundles nest, the deepest enclosing root determines each concept's identity and version. The source checks ancestor indexes from the existing GitHub tree, even when they sit above path_prefix. It fetches each needed index once per run.

For bundles without a versioned index, declare roots in your ingestion recipe:

source:
type: github-documents
config:
github_token: "${GITHUB_TOKEN}"
repository: example-org/analytics
enable_okf: true
okf_bundle_roots: [context]

Use okf_bundle_roots: [""] for a repository-root bundle. Roots must be repository-relative POSIX paths. Explicit roots supplement discovered roots. A bundle without a declared version has no typed okfVersion. Unknown version strings are preserved and consumed on a best-effort basis.

A concept at context/guides/mau.md can contain:

---
id: guide:mau
type: Guide
title: Monthly active users
owners: [alice@example.com]
tags: [metrics]
glossary_terms: [Active User]
status: stable
stale_after: 2026-09-23T00:00:00Z
generated: { by: reference_agent/model, at: 2026-06-20T22:53:05Z }
verified: { by: human:alice, at: 2026-06-25T09:00:00Z }
audience: analysts
---

# Monthly active users

With context/ as its bundle root, this file has conceptId: guides/mau. The identity is scoped to bundle.location and bundle.rootPath. For GitHub, bundle.location is the canonical repository URI, such as https://github.com/example-org/analytics, without a branch or trailing slash. rootPath is relative to that location, without leading or trailing slashes, and is . for a repository-root bundle (an empty value could not be used as a search filter). The optional producer extension id: guide:mau becomes declaredId. Resolve a declared ID within its bundle, rather than treating it as a global identifier. Only the bundle-root index's okf_version supplies bundle.okfVersion. OKF permits unversioned bundles; declare these in okf_bundle_roots. Index files with frontmatter but no okf_version increment okf_unversioned_index_files and produce an info entry suggesting this setting. Concept-level version keys remain extensions in the raw frontmatter.

For each concept the source:

  • imports the text after the frontmatter block as the document body;
  • stores the original YAML block, including delimiters, in okfProperties.raw as the only stored copy of the frontmatter. Typed fields are read from it, and write-back edits it in place;
  • records bundle identity, path-based conceptId, optional declaredId, type, and description. The optional revision holds the branch/ref, commit, and blob SHA read at ingestion, providing the merge base for write-back;
  • maps title to the document title, type to its subtype, and owners, tags, and glossary_terms to native associations;
  • stores status, staleAfter, generated, and verified as OKF metadata only. These fields do not change the document's native status, owners, or permissions.

The file location is the bundle location followed by its root path and conceptId with .md appended, joining path components with slashes. The GitHub document source and existing github_* custom properties still retain the import location and revision details. mappingVersion is recorded in value attribution and source hashes, rather than in okfProperties. Bundle location and root path support exact search through okfBundleLocation and okfBundleRootPath. okfConceptId and okfDeclaredId also support case-insensitive and text search on Search V3; okfType and okfStatus provide type and lifecycle filters.

GraphQL exposes a read-only, nullable okfProperties.frontmatter field containing the keys as a JSON object parsed from raw when read. It returns null if parsing fails. Dates remain strings. Ingestion validates JSON-compatible YAML values, rejecting duplicate keys, non-string keys, anchors, and aliases, but stores no JSON copy. Frontmatter blocks are limited to 64 KiB, including delimiters. Unsafe or oversized frontmatter is imported as plain text with a parse warning. Unknown keys remain in raw.

Status strings are kept as written, including unknown values. An omitted status remains unset in the aspect. OKF consumers can apply the spec's implicit stable default. Timestamps require ISO 8601 datetimes with an explicit offset and are stored as epoch milliseconds. An invalid timestamp produces a warning and leaves the typed timestamp null. Its original value stays in raw YAML. generated.by and verified[].by retain the producer's actor strings. A single verified object is normalized to a one-element list.

index.md and log.md are reserved at every level inside a bundle. They remain imported documents and receive no concept aspect or native OKF mapping, even if they contain type. All index and log files retain their full text, including frontmatter. Files outside bundles, non-Markdown files, and files without type retain their original text. Malformed concept frontmatter is also imported verbatim with a warning.

Mapped owners, tags, and terms carry attribution with origin: okf, bundle (the canonical location URI), rootPath, conceptId, key, and mappingVersion. The commit is included only when known. On each document, re-import adds listed values and removes previously imported values identified by origin: okf; the bundle and revision fields describe provenance and do not limit reconciliation. Changes to the bundle root or ingestion source do not leave old copies behind. DataHub-added associations without OKF attribution are preserved. A null or empty association key means no values. Unresolved values prevent missing-value removals for that key, while resolved values can still move from an old source key to the current source key. Without a graph connection, imports emit additions only and cannot remove existing values.

An owner maps when it is an email address, a corpuser or corpGroup URN, or a user that exists in DataHub. Tag names map directly to tag URNs without an existence check. A glossary term maps when it is a URN or the name of exactly one catalog term. Values that cannot be mapped remain in raw frontmatter and increment okf_unresolved_values. Invalid typed OKF values increment okf_invalid_values and produce warnings. Catalog changes alone do not retry unresolved values on unchanged files; edit the file to trigger a retry. Owner-existence and term-name lookups are cached for each run.

If a file stops being a concept, a graph connection lets the source UPSERT an empty okfProperties record, remove its OKF-attributed associations, and UPSERT an empty subtype list only when the stored subtype still equals the previous OKF type. The source never emits aspect DELETEs. An aspect without raw is neutral and is treated like an absent aspect; GraphQL returns null for it. Subsequent runs leave an already neutral aspect alone. All top-level fields are optional to permit this neutral value; when raw is present, bundle, conceptId, and type are set. A subtype edited in DataHub is preserved. Without a graph, previous OKF metadata cannot be cleared. Setting enable_okf: false disables discovery and mapping and also clears previous OKF state when a graph is available.

The source resolves the configured branch head once and lists and reads files at that commit SHA. It stops if the head cannot be resolved, so the content, blob SHA, and revision commit refer to one snapshot. GitHub links and branch custom properties retain the configured branch.

content_hash is the sync-back contract: it hashes exactly documentInfo.contents.text plus the extraction version, including only the body for a concept. source_content_hash hashes the complete source file and, for enabled OKF files inside a bundle, the mapping version and bundle metadata. Frontmatter edits and bundle changes trigger re-ingestion; mapping changes trigger re-ingestion only inside bundles.

Each imported file document stores these customProperties keys:

KeyPurpose
import_source_idStable external key for upserts
content_hashHash of stored document text (sync-back contract)
source_content_hashHash of raw source and applicable OKF interpretation (change detection)
extraction_algo_versionBump when hash algorithm changes
github_blob_shaGit blob SHA at last import
github_commit_shaPinned commit SHA read during this import

Cloud sync-back may additionally write last_exported_content_hash to prevent re-import loops after export. The source skips a file when its new source hash matches stored source_content_hash, or its whole-file text hash matches last_exported_content_hash. Documents without source_content_hash are re-imported once unless the sync-back loop check matches.

Server compatibility​

OKF import is disabled by default. Set enable_okf: true only when your DataHub server supports the okfProperties aspect. The config flag alone controls OKF import; the source does not check the server registry or silently disable OKF. Enabling OKF on a server without the aspect fails the ingestion run when the server rejects the writes.

Stateful ingestion​

Enable stateful_ingestion.enabled: true (recommended for scheduled syncs) to soft-delete documents that disappear from the GitHub tree between runs. Unchanged files are skipped when a graph connection is available and the source-hash or sync-back loop check matches.

Limitations​

  • Maximum file size is 1 MB per file (larger files are skipped).
  • Requires network access from the ingestion executor to api.github.com.
  • Binary formats are not parsed; use text/markdown files.
  • GitHub may truncate recursive tree listings for very large repositories; narrow path_prefix or split across sources.
  • Imports are capped by max_files (default 500) per run.

Troubleshooting​

  • 401 / 403 from GitHub: Verify the token, repository name, and permissions.
  • Branch not found: Confirm the branch exists and is spelled correctly.
  • No files imported: Check path_prefix and file_extensions filters.
  • Documents not removed after GitHub delete: Ensure stateful_ingestion.enabled: true and the pipeline has a graph connection for checkpoint storage.
  • Partial import on large repos: Check ingestion report warnings for github-tree-truncated or github-files-truncated.

Code Coordinates​

  • Class Name: datahub.ingestion.source.github_documents.github_documents_source.GitHubDocumentsSource
  • Browse on GitHub
Questions?

If you've got any questions on configuring ingestion for GitHub, feel free to ping us on our Slack.

💡 Contributing to this documentation

This page is auto-generated from the underlying source code. To make changes, please edit the relevant source files in the metadata-ingestion directory.

Tip: For quick typo fixes or documentation updates, you can click the ✏️ Edit icon directly in the GitHub UI to open a Pull Request. For larger changes and PR naming conventions, please refer to our Contributing Guide.