Databricks Engineer - Systems Analyst 3

US CONSULTANTS & IT SERVICES LTD. CO
Cedar Park, United States of America
2 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior
Compensation
$ 200K

Job location

Cedar Park, United States of America

Tech stack

Airflow
Big Data
Continuous Integration
Information Engineering
Data Governance
ETL
Data Security
Data Systems
Data Warehousing
DevOps
Python
Scala
SQL Databases
Spark
Data Lake
PySpark
Git Flow
Star Schema
Data Lakehouse
Data Pipelines
Databricks

Job description

We are seeking an experienced Databricks Engineer to design, develop, optimize, and maintain scalable data solutions using the Databricks platform. The ideal candidate will have extensive experience with Databricks, Apache Spark, PySpark or Scala, SQL, Python, Delta Lake, Lakeflow Declarative Pipelines, Medallion Architecture, and data lakehouse solutions. The candidate will work on complex data engineering initiatives and collaborate with technical and business stakeholders to develop reliable data solutions supporting enterprise analytics, reporting, and decision-making., * Design, develop, and optimize scalable data solutions using Databricks.

  • Build and maintain ETL/ELT pipelines using Apache Spark, PySpark, Python, or Scala.
  • Develop and manage Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT).
  • Implement Medallion Architecture using Bronze, Silver, and Gold layers.
  • Design and develop enterprise data models for analytics and reporting.
  • Implement dimensional data modeling, including Star and Snowflake schemas.
  • Work extensively with Delta Lake and modern data lakehouse architectures.
  • Build, schedule, monitor, and optimize jobs using Lakeflow Jobs (formerly Databricks Workflows) or tools such as Airflow.
  • Implement data quality, validation, governance, and security frameworks.
  • Develop Databricks SQL dashboards, Databricks Apps, and analytical solutions.
  • Optimize workloads for scalability, performance, reliability, and cost efficiency.
  • Analyze business and technical requirements and translate them into effective data solutions.
  • Prepare technical documentation covering data requirements, processing logic, and implementation steps.
  • Evaluate system capabilities and limitations to determine appropriate technical solutions.
  • Present technical findings and analytical insights to technical and business stakeholders.

Requirements

  • 8+ years of experience in IT supporting the design, development, deployment, or delivery of technology solutions.
  • 8+ years of experience with Databricks, including building and optimizing ETL/ELT pipelines using Apache Spark.
  • 8+ years of experience with data warehousing and dimensional data modeling.
  • 8+ years of experience with SQL and Python or Scala for large-scale data processing.
  • 8+ years of experience developing Databricks-native dashboards and applications, including Databricks SQL dashboards and Databricks Apps.
  • 8+ years of experience implementing data governance, data quality, and data security practices.
  • 8+ years of experience with Lakeflow Declarative Pipelines / DLT.
  • 8+ years of experience with Delta Lake, Medallion Architecture, data lakehouse design, and Lakeflow Jobs / Databricks Workflows or similar orchestration tools.
  • Excellent verbal and written communication skills.
  • Strong analytical, troubleshooting, and problem-solving abilities.
  • Ability to independently handle complex technical assignments., * Experience working in public sector or state government environments.
  • Databricks certification, such as Databricks Certified Data Engineer Associate or Professional.
  • Experience with CI/CD practices for data pipelines.
  • Experience with DevOps and Git-based workflows.

Apply for this position