Senior Data Engineer

We are seeking a highly capable Senior Data Engineer to design, build, and optimize modern data pipelines and analytics platforms. This role is focused on leveraging Databricks as a development environment for large-scale data engineering, transformation, and analytics workloads-not platform administration.

In support of a Department of Energy client, you will work closely with business stakeholders, analytics teams, software engineers, and AI/ML practitioners to build reliable, scalable, and high-performance data solutions that support reporting, operational intelligence, and advanced analytics initiatives., * Design, develop, and maintain scalable data pipelines using Azure Databricks and Apache Spark.

• Build and optimize batch and streaming data processing solutions.
• Develop robust ETL/ELT frameworks for structured and semi-structured data from SQL, APIs, SaaS apps, and Data Live Lake Storage.
• Implement transformation logic using PySpark, SQL, and Delta Lake within a Lakehouse / Medallion architecture.
• Integrate and model data from SQL, SaaS systems, and cloud storage into unified, analytics-ready and AI-ready structures.
• Partner with analytics, BI, and AI/ML teams to deliver trusted datasets, semantic layers, and AI-ready data (features, vector-ready patterns, embeddings).
• Improve data quality, observability, lineage, governance, and security in alignment with Azure and enterprise standards.
• Optimize data performance, storage costs, and processing efficiency in cloud environments.
• Participate in architecture discussions and contribute to the evolution of the organization’s data platform and engineering best practices.

Do you have experience in Technical architecture?, Do you have a Bachelor’s degree?, * U.S. Citizenship is required and non-negotiable

• 2 days a week in the Rockville, MD office are required and non-negotiable.
• Ability to pass multiple background and employment checks - federal and local.
• Bachelor’s degree Technology, Business, or Related Field, * 5+ years of hands-on data engineering experience (or equivalent depth), including production-grade pipelines.
• Strong experience building solutions using Databricks, Spark/PySpark, SQL, and Delta Lake.
• Solid understanding of Lakehouse and Medallion architectures and when to apply them.
• Advanced SQL skills and experience supporting analytics, BI, or operational reporting workloads.
• Experience developing and operating ETL/ELT pipelines on Azure or another major cloud platform.
• Familiarity with Git-based development and CI/CD practices for data pipelines (Azure DevOps or GitHub preferred).
• Strong understanding of data modeling (relational + dimensional) for both operational and analytical workloads.
• Working knowledge of data quality, lineage, governance, and security concepts in cloud environments.
• Note: This role focuses on building data products on Databricks, not platform administration., * Experience with Databricks features such as Unity Catalog, Delta Live Tables, Workflows, Lakeflow, SQL Warehouse, or Mosaic AI.
• Experience using tools that schedule and run data pipelines, such as Data Factory, Synapse pipelines, Airflow, Dagster.
• Experience using modern SQL-based transformation tools such as dbt, Databricks SQL (SQL Warehouse), or Delta Live Tables
• Exposure to AI/ML workflows: Azure OpenAI, embeddings, vector stores, feature engineering, or model training pipelines.
• Knowledge of streaming technologies such as Kafka, Event Hubs, or Kinesis.
• Familiarity with modern data warehouses (Snowflake, Synapse, Redshift) and how they integrate with a Lakehouse.

Technical Environment

Candidates should have experience with several of the following:

• Databricks
• Apache Spark
• PySpark
• SQL
• Delta Lake
• Python
• dbt
• Airflow
• Azure Data Factory
• Kafka
• Snowflake
• Git
• CI/CD Pipelines
• Cloud Platforms (Azure, AWS, or GCP)

Pulled from the full job description

• 401(k)
• Health insurance
• Paid time off
• Vision insurance, Within the first 6-12 months, you will:
• Deliver reliable, production-ready data pipelines supporting critical business use cases.
• Improve data processing performance and scalability.
• Establish reusable engineering patterns and best practices.
• Enable analytics and AI teams with high-quality, trusted datasets.
• Contribute to the modernization and growth of the organization’s data platform capabilities.

Education & Certifications

• B.S. in Business/Technology-related field

Legal

• IvoryCloud is an Equal Opportunity Employer (EEO)

Pay: $155,000.00 - $200,000.00 per year

Benefits:

• 401(k)
• Health insurance
• Paid time off
• Vision insurance

Application Question(s):

• This role requires U.S. Citizenship. Do you meet this requirement?
• Can you meet the hybrid requirement of 2 days a week in office in Rockville, Maryland?
• Are you able to come into the office for in-person interviews?