DataOps Engineer (Local to Charlotte, NC)

The DataOps Engineer will support the design, implementation, automation, and operational management of enterprise data platforms leveraging Databricks and AWS cloud services. The role will focus on building scalable and reliable data pipelines, supporting Databricks platform operations, and implementing DataOps and Infrastructure as Code (IaC) best practices to enable secure and efficient data processing across the enterprise. The engineer will work closely with data engineering, analytics, AI/ML, and platform teams to support data integration, operational monitoring, governance, and deployment automation initiatives.

Core Responsibilities Databricks Platform & DataOps

• Develop, maintain, and optimize ETL/ELT pipelines within Databricks using PySpark, Spark SQL, and Databricks Workflows.
• Support batch and streaming data processing workloads within Databricks environments.
• Configure and manage Databricks clusters to support scalability, reliability, and cost optimization.
• Implement Delta Lake best practices including partitioning, schema evolution, optimization, and performance tuning.
• Support Unity Catalog administration including access controls, governance policies, lineage, and auditing.
• Contribute to medallion/lakehouse architecture implementations across bronze, silver, and gold data layers.
• Monitor and troubleshoot Databricks jobs, workflows, pipelines, and cluster operations using platform monitoring and observability tools.
• Support enterprise analytics, reporting, and AI/ML workloads running on Databricks.

Data Engineering & Integration

• Develop and maintain scalable data ingestion and transformation pipelines using Python, PySpark, SQL, AWS Glue, and related AWS services.
• Integrate structured, semi-structured, unstructured, and streaming data from multiple enterprise and cloud data sources.
• Support real-time and event-driven integrations using AWS Kinesis, Firehose, and related streaming technologies.
• Collaborate with cross-functional teams to deliver scalable and reliable enterprise data solutions.

Infrastructure Automation & CI/CD

• Support Infrastructure as Code (IaC) initiatives using Terraform for provisioning and managing Databricks and cloud infrastructure components.
• Assist with automating deployment processes, configuration management, and operational workflows.
• Support CI/CD pipelines for Databricks code deployments and infrastructure automation.
• Maintain version-controlled repositories and deployment automation processes following DataOps best practices.

Governance, Security & Operations

• Support implementation of data governance, privacy, security, and compliance controls across the platform.
• Implement and maintain data quality checks, lineage tracking, and operational monitoring processes.
• Contribute to operational documentation, runbooks, and support procedures.
• Participate in troubleshooting, root cause analysis, and continuous platform improvement initiatives.

Deliverables

• Production-ready Databricks ETL/ELT pipelines and workflows.
• Scalable batch and streaming data integration solutions.
• Terraform scripts and Infrastructure as Code templates for platform provisioning.
• Monitoring dashboards and operational alerts for Databricks workloads and pipelines.
• Data lineage, metadata, and operational documentation.
• CI/CD deployment automation and operational support documentation.
• Weekly status reports and participation in Agile sprint ceremonies.

Do you have experience in Version control?, 5+ years of experience in Data Engineering, DataOps, or Platform Engineering.

• 3+ years of hands-on experience with Databricks in enterprise environments.
• Strong proficiency in Python, PySpark, Spark SQL, and SQL.
• Hands-on experience with Databricks Workflows, Delta Lake, and Unity Catalog.
• Experience building and supporting scalable ETL/ELT pipelines and distributed data processing solutions.
• Working knowledge of Terraform and Infrastructure as Code (IaC) practices.
• Experience with AWS cloud services including AWS Glue, Kinesis, Firehose, S3, and IAM.
• Understanding of data governance, security, monitoring, and operational best practices.

Pulled from the full job description

• 401(k)
• Health insurance
• Disability insurance
• Paid holidays, General Benefits:
• Complete and comprehensive benefits package including Med/Dent/Vision
• Employer paid STD/LTD/Life
• 401k Retirement program
• Generous paid vacation/sick/holidays
• Creativity encouraged in a fun, friendly work environment

With the strength of a global network of over 14,500 colleagues in 50+ locations, Bertrandt US combines deep expertise in Electronics, Product Engineering, Physical, and Production & After Sales. Join us in engineering tomorrow's mobility today.