Databricks Engineer with Test Data Management

Role details

Contract type

Contract

Employment type

Full-time (> 32 hours)

Experience level

Experienced

Experience required

3 years minimum

Working hours

Regular working hours

Job source

www.dice.com

Tech stack

Artificial Intelligence Amazon Web Services Amazon S3 Microsoft Azure Continuous Integration Data Architecture Data Discovery Information Engineering Data Governance Data Infrastructure Data Masking DevOps

+14 more

Github Python (Programming Language) Meta-Data Management Role-Based Access Control DataOps Test Data Data Classification Generative AI Data Lakes Pyspark Data Management Terraform Databricks Data Generation

Job description

Roll out TDM framework across projects on Databricks
Establish governance controls for sensitive data in lower environments
Build scalable pipelines for test data generation, masking, and refresh
Implement metadata management and lineage capabilities
Design synthetic data strategies for non-production environments
Enable self-service access to compliant test datasets
Develop reusable, automated data provisioning and lifecycle frameworks
Partner with QA, application teams, data governance, and security

Requirements

8+ years in Data Engineering, Data Architecture, or Data Platform roles
3+ years hands-on Databricks (Lakehouse, Delta Lake, Unity Catalog, PySpark)
Medallion Architecture implementation (Bronze/Silver/Gold)
Unity Catalog / data governance rollout experience
Enterprise data governance - Role based access control, metadata management, data classification, lineage, access control
Scalable ingestion, transformation, and provisioning pipelines
Azure (ADF, DevOps, Synapse) and/or AWS (S3, Lambda, Glue, Redshift, MWAA)
Python and PySpark development

Highly Desired

Test Data Management (data masking, de-identification, synthetic data generation)
AI-powered data discovery and classification
Generative AI / Databricks Mosaic AI
Data observability and monitoring frameworks
Cross-team framework rollout and enablement

Nice to Have

CI/CD and IaC (Terraform, Azure DevOps, GitHub Actions)
Regulatory compliance (PII, GDPR, CCPA, HIPAA)

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com

Prepare application