Software Engineer Digital Brain CI/CD Asset Deployment Automation

Passport File Ingestion and Validation

Ingest multiple passport files received for asset deployment or update.

Validate each passport file to ensure all required deployment data is present.

Scan the passport file format and register valid passport files for processing.

Generate validation reports in JSON format.

Dependency DAG and Execution Order

Analyze dependencies between assets across multiple passport files.

Build an asset dependency DAG based on upstream and downstream relationships.

Implement topological sorting to determine the safe asset deployment execution order.

Ensure downstream assets wait until all required upstream dependencies are successfully completed.

Handle success dependency, control flow, data/artifact flow, and loop execution per passport file.

Asset Deployment Loop

Build the full deployment loop for each passport file in DAG execution order.

Extract changes from the passport file by comparing with previously deployed assets.

Support change detection for new, changed, and existing assets.

Generate required DDL scripts and deployment logs.

Deploy or update DDL changes into Databricks and Unity Catalog.

Ensure each passport file completes the full deployment loop before dependent assets are triggered.

Downstream System Integration

Develop integrations with the following enterprise systems:

Databricks Unity Catalog Alation SharePoint Knowledge Graph platforms Vector databases Ontology or semantic model platforms REST APIs and metadata services

Build payload generation logic for each downstream system, including:

Alation metadata payloads SharePoint document payloads Knowledge graph payloads Vector index payloads Ontology or semantic model payloads

Implement parallel downstream deployment where applicable, while ensuring dependency and success/failure rules are respected.

Post-DDL Deployment Processing

Extract changes to push to Alation only after successful DDL deployment.

Extract changes to push to SharePoint only after successful DDL deployment.

Extract changes to push to downstream systems only after successful DDL deployment.

Support downstream updates for:

Alation metadata update or upsert SharePoint document and metadata update Knowledge Graph ingest or graph update Vector Database vector upsert and indexing Optional Ontological Model update and relationship management

Failure Handling and Auditability

Stop the asset deployment loop if any required extraction or deployment step fails.

Prevent downstream deployment when upstream deployment is unsuccessful.

Generate deployment logs, payload files, validation evidence, and audit artifacts.

Maintain traceability of generated artifacts for every passport file and deployment step.

Strong software engineering experience using Python or a similar backend programming language.

Experience building configuration-driven automation using JSON and YAML.

Strong understanding of DAGs, dependency resolution, graph processing, and topological sorting.

Experience developing CI/CD or deployment automation workflows.

Hands-on experience with Databricks and Unity Catalog deployment automation.

Experience integrating with enterprise platforms using REST APIs and metadata services.

Ability to build payload generation logic for multiple downstream systems.

Strong knowledge of validation, logging, error handling, audit tracking, and deployment status management.

Experience working with Git-based source control and environment-based deployments.