Databricks Engineer

The Senior Databricks Engineer will design, develop, and optimize enterprise-scale data solutions on the Databricks platform. This role is responsible for building high-performance ETL/ELT pipelines, implementing modern Lakehouse architectures, developing scalable data models, and delivering analytics solutions using Databricks SQL, Databricks Apps, Delta Lake, and Lakeflow Declarative Pipelines. The engineer will ensure data quality, governance, security, scalability, and operational efficiency while supporting enterprise reporting and analytics initiatives., * Design, develop, and optimize scalable data solutions using Databricks.

• Build and maintain ETL/ELT data pipelines using Apache Spark and PySpark or Scala.
• Develop and manage Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT).
• Design and implement Delta Lake and Medallion Architecture using:
• Bronze Layer
• Silver Layer
• Gold Layer
• Develop enterprise data models using dimensional modeling techniques.
• Design Star Schema and Snowflake Schema data warehouses.
• Create and maintain Databricks SQL Dashboards and Databricks Apps.
• Develop analytical solutions that provide actionable business insights.
• Implement enterprise data governance, security, and data quality frameworks.
• Optimize Spark workloads for performance, scalability, and cost efficiency.
• Schedule and orchestrate data pipelines using:
• Lakeflow Jobs (formerly Databricks Workflows)
• Airflow or similar orchestration tools
• Monitor production workloads and resolve performance issues.
• Collaborate with architects, analysts, developers, and business stakeholders.

Additional Responsibilities:

• Support enterprise Lakehouse architecture initiatives.
• Develop reusable data engineering frameworks and best practices.
• Implement CI/CD pipelines for data engineering projects.
• Maintain technical documentation and operational procedures.
• Participate in architecture reviews and technical design sessions.
• Support production deployments and ongoing platform optimization.

• Minimum 8 years of IT experience designing, developing, or delivering technology solutions.
• Minimum 8 years of experience with Databricks.
• Strong experience building ETL/ELT pipelines using Apache Spark.
• Proficiency with:
• PySpark
• Python
• SQL
• Scala
• Experience designing enterprise data warehouses.
• Strong knowledge of:
• Star Schema
• Snowflake Schema
• Dimensional Data Modeling
• Experience implementing:
• Delta Lake
• Medallion Architecture
• Bronze, Silver, Gold Layers
• Experience with Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT).
• Experience with Lakeflow Jobs (formerly Databricks Workflows) or Airflow.
• Experience creating Databricks SQL Dashboards and Databricks Apps.
• Experience implementing:
• Data Governance
• Data Quality
• Data Validation
• Data Security
• Excellent verbal and written communication skills.

Preferred Skills:

• Experience working in public-sector or state government environments.
• Databricks Certified Data Engineer Associate or Professional Certification.
• Experience implementing CI/CD pipelines.
• Experience with Git-based DevOps workflows.
• Knowledge of cloud-native data engineering best practices.
• Experience optimizing enterprise-scale Spark workloads.

Qualifications:

• Proven experience delivering enterprise data engineering solutions.
• Strong understanding of modern data lakehouse architecture.
• Excellent analytical, troubleshooting, and problem-solving skills.
• Ability to communicate effectively with technical and business stakeholders.
• Ability to manage multiple priorities in a fast-paced environment.

Education:

• Bachelor's degree in Computer Science, Information Systems, Data Engineering, or a related field.
• Master's degree preferred.

Certifications:

• Databricks Certified Data Engineer Associate Preferred
• Databricks Certified Data Engineer Professional Preferred
• Microsoft Azure Data Engineer Associate Preferred
• AWS Certified Data Analytics Specialty Preferred
• Google Professional Data Engineer Preferred

Skills: Amazon Web Services (AWS), Analysis Skills, Apache Spark, Architectural Analysis, Best Practices, Cloud Computing, Communication Skills, Computer Science, Continuous Deployment/Delivery, Continuous Integration, Data Analysis, Data Management, Data Modeling, Data Quality, Data Warehousing, Database Design, Database Extract Transform and Load (ETL), Dimensional Modeling, Engineering, Enterprise Architecture, Identify Issues, Information Technology & Information Systems, Information/Data Security (InfoSec), Microsoft Windows Azure, Multitasking, Operational Support, Operations Processes, Performance Tuning/Optimization, Presentation/Verbal Skills, Problem Solving Skills, Production Control, Production Support, Reporting Dashboards, SQL (Structured Query Language), Sales Pipeline, Scala Programming Language, Scalable System Development, Snowflake Schema, Software Engineering, Star Schema, Technical Delivery, Technical Writing, Technical/Engineering Design, Writing Skills