Sr Data Engineer - Databricks

ADASTRA DESIGNS, INCORPORATED
Irvine, CA, United States
5 days ago

Role details

Contract type
Contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
5 years minimum
Working hours
Regular working hours

Tech stack

Application Programming Interfaces (APIs) Amazon Web Services Microsoft Azure Cloud Storage Code Review Databases Continuous Integration Customer Data Management Information Engineering Data Governance Data Integration Extract Transform Load (ETL)
+14 more
DevOps Apache Hive Python (Programming Language) Operational Databases Performance Tuning Power BI Data Streaming Apache Spark Git Data Lakes Pyspark Information Technology Apache Kafka Databricks

Job description

  • Design, build, and maintain scalable batch and streaming pipelines on Databricks using PySpark and Spark SQL within a medallion (Bronze/Silver/Gold) architecture
  • Ingest and unify data from operational databases, cloud storage, APIs, and marketing platforms such as Braze
  • Model and optimize Delta Lake tables for performance, reliability, and cost efficiency using techniques such as partitioning, liquid clustering, OPTIMIZE, and VACUUM
  • Deliver analytics-ready data for reporting on sales, shipments, orders, and other key business metrics, supporting Power BI dashboards and downstream analytics
  • Orchestrate jobs using Databricks Workflows, managing dependencies, retries, alerting, and SLAs
  • Apply Unity Catalog for data governance, lineage, and access control
  • Work to a high engineering standard using Git, CI/CD, testing, code reviews, and clear documentation
  • Monitor pipeline health, troubleshoot issues, and tune Spark jobs and cluster configurations for speed and cost optimization
  • Partner with business and technical stakeholders to deliver reliable, high-quality data products

Requirements

  • Bachelor’s degree in Computer Science, Information Technology, Engineering, Mathematics, or a related field
  • 5+ years of data engineering experience building and operating production data pipelines
  • Hands-on expertise with the Databricks Lakehouse Platform, including Delta Lake, Databricks Workflows, and Unity Catalog
  • Advanced experience with PySpark and Spark SQL
  • Strong SQL development and performance tuning skills
  • Solid Python development skills and experience with modern engineering practices
  • Experience using Git, CI/CD pipelines, testing frameworks, and code review processes
  • Experience working with AWS, Azure, or GCP and their core compute and storage services
  • Strong understanding of data modeling concepts, including dimensional and medallion architectures
  • Experience designing ELT/ETL solutions and integrating data from databases, APIs, files, and event streams
  • Strong communication skills and the ability to work independently within a client-facing environment

NICE TO HAVE

  • Databricks Certified Data Engineer Associate or Professional certification
  • Experience with Structured Streaming, Auto Loader, Kafka, or Kinesis
  • Experience working with marketing or customer data platforms such as Braze
  • Retail, grocery, or consumer packaged goods (CPG) industry experience
  • Experience supporting Power BI reporting and self-service analytics initiatives

Benefits & conditions

  • Opportunity for advancement and career progression
  • Competitive compensation package
  • Comprehensive benefits plan
  • Successful referral program
  • The opportunity to work with one of Canada’s 50 Best Managed Companies
  • Satisfaction of working for a reputable company
  • A flexible, dynamic, and diverse workplace

EQUAL OPPORTUNITY EMPLOYER

In our commitment to promote fair and equitable treatment of all employees and applicants, Adastra Corporation provides equal employment opportunities for all individuals regardless of age, sex, disability, race, ethnic origin, citizenship, creed, sexual orientation, marital status, or any other ground as described in the Ontario Human Rights Code.

In addition, accommodation will be provided during the hiring process. Adastra Corporation’s implementation and support of employment initiatives, encourage diversified labour force participation and equal access to opportunities based on merit and performance.

AI Usage - Our hiring process includes the use of AI-enabled tools to screen applications (e.g., keyword matching, qualification ranking). A human recruiter reviews all AI-generated shortlists to make informed hiring decisions.

About the company

Adastra is looking for a hands-on Data Engineer to join our team supporting a modern enterprise data platform. You’ll design and run Databricks pipelines that turn raw operational, marketing, and retail data into trusted, analytics-ready data products the business relies on every day.

Working as an embedded member of the data team, you’ll own pipelines end to end on the Databricks Lakehouse Platform. If you love clean data, well-tuned Spark jobs, and shipping solutions that people actually use, this role is for you., Adastra is a global leader in AI and data-driven transformation, helping organizations lead with artificial intelligence-responsibly, strategically, and at scale. With over 25 years of experience, Adastra empowers enterprise clients to unlock business value through data innovation, operational excellence, and smart customer engagement.

Trusted by some of the world’s most prominent brands, Adastra delivers end-to-end solutions grounded in thoughtful strategy, robust governance, and deep technical expertise. From defining vision to ensuring execution, Adastra guides organizations through every stage of their AI, data and cloud journey-building future-ready capabilities and delivering measurable, lasting impact.

Adastra serves clients across key industries including financial services, automotive, manufacturing, technology, media and telecom (TMT), healthcare, retail, and professional services. The company employs more than 2,000 professionals across North America, Europe, and Asia.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.techcareers.com

Good distractions

Talks and stories from around this role — technically off-topic, practically not.

6:21 min

Investigating push inefficiencies with upstream Git experts

Jonathan Creamer · Coffee With Developers

2:17 min

Mapping the maturity roadmap for scaled devops adoption

Dominik Krichbaum Dominik Krichbaum · World Congress 2026 Europe

4:32 min

Harnessing Spark with Python using PySpark and Py4J

Ayon Roy · LIVE

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

56 sec

Favorite git commands and the importance of patch commits

Eileen Uchitelle Eileen Uchitelle +1 · Coffee With Developers

3:09 min

Balancing data science skillings alongside systems engineering rigor

Nico Schmidt · LIVE

Videos

See all

Related articles

See all