Data Engineering Technical Lead

job summary: We are seeking a hands-on technical leader to architect, modernize, and scale our enterprise data platform. This pivotal role drives the transformation of legacy data systems into a modern, cloud-native architecture that powers advanced analytics, machine learning, and business intelligence across the organization. As a technical lead, you will design scalable pipelines, establish engineering standards, and utilize AI-assisted development tools to increase velocity and minimize technical debt. You will also spearhead proofs of concept to evaluate emerging data frameworks, ensuring our platform remains highly efficient and future-ready.

location: San Diego, California job type: Permanent salary: $130,000 - 150,000 per year work hours: 8am to 5pm education: Bachelors

responsibilities: Incorporate AI-driven tools such as GitHub Copilot, Cursor AI, or similar platforms for automated testing, rapid code generation, and pipeline optimization.

Guide data engineers by providing technical mentorship, fostering continuous skill development, and establishing modern data engineering best practices.

Establish and enforce platform-wide engineering excellence, including robust CI/CD practices, version control, peer reviews, and observability.

Design and implement standardized, reusable architectural frameworks for data ingestion, curation, and transformation across the cloud ecosystem.

Collaborate with data architects, analysts, and business product owners to translate complex commercial needs into scalable technical solutions.

Lead the migration and modernization of legacy SQL-based server integration workloads into modular, high-performance cloud pipelines.

Conduct proof-of-concept evaluations on emerging technologies like streaming ingestion and AI-driven observability to guide the enterprise technology roadmap.

Apply predictive modeling and machine learning techniques to monitor compute scheduling, identify pipeline anomalies, and track data drift.

Optimize system performance and cost-efficiency across multi-cloud environments by fine-tuning storage tiers and orchestration logic.

qualifications: Bachelor's degree in a relevant technical field.

At least 8 years of experience in data engineering or related technical domains, with a minimum of 3 years serving in a senior or leadership capacity.

Deep expertise with modern data platforms, data transformation tools, and orchestration engines (such as Databricks, Spark, dbt, Airflow, Fivetran, or Kafka).

Proven track record designing data patterns, including CDC, SCD, and multi-layered cloud data storage architectures (e.g., Medallion).

Solid experience developing and deploying cloud-native pipelines across major cloud provider networks (such as Azure or GCP).

Demonstrated experience converting legacy relational database integration tasks into modern cloud architectures.

Strong proficiency in SQL along with at least one major programming language such as Python, Scala, or Java.

Practical experience executing technical proofs of concept to validate new infrastructure frameworks and tools.

Familiarity with AI-assisted software development and optimization utilities.

Excellent communication skills with the ability to articulate highly technical data architecture concepts to non-technical stakeholders.

Equal Opportunity Employer: Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.

At Randstad Digital, we welcome people of all abilities and want to ensure that our hiring and interview process meets the needs of all applicants. If you require a reasonable accommodation to make your application or interview experience a great one, please contact HRsupport@randstadusa.com.

Pay offered to a successful candidate will be based on several factors including the candidate's education, work experience, work location, specific job duties, certifications, etc. In addition, Randstad Digital offers a comprehensive benefits package, including: medical, prescription, dental, vision, AD&D, and life insurance offerings, short-term disability, and a 401K plan (all benefits are based on eligibility).

This posting is open for thirty (30) days.

Qualified applicants in San Francisco with criminal histories will be considered for employment in accordance with the San Francisco Fair Chance Ordinance.

Qualified applicants with arrest or conviction records will be considered for employment in accordance with the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act.

We will consider for employment all qualified Applicants, including those with criminal histories, in a manner consistent with the requirements of applicable state and local laws, including the City of Los Angeles' Fair Chance Initiative for Hiring Ordinance.

,

Incorporate AI-driven tools such as GitHub Copilot, Cursor AI, or similar platforms for automated testing, rapid code generation, and pipeline optimization.

Guide data engineers by providing technical mentorship, fostering continuous skill development, and establishing modern data engineering best practices.

Establish and enforce platform-wide engineering excellence, including robust CI/CD practices, version control, peer reviews, and observability.

Design and implement standardized, reusable architectural frameworks for data ingestion, curation, and transformation across the cloud ecosystem.

Collaborate with data architects, analysts, and business product owners to translate complex commercial needs into scalable technical solutions.

Lead the migration and modernization of legacy SQL-based server integration workloads into modular, high-performance cloud pipelines.

Conduct proof-of-concept evaluations on emerging technologies like streaming ingestion and AI-driven observability to guide the enterprise technology roadmap.

Apply predictive modeling and machine learning techniques to monitor compute scheduling, identify pipeline anomalies, and track data drift.

Optimize system performance and cost-efficiency across multi-cloud environments by fine-tuning storage tiers and orchestration logic.

Bachelor's degree in a relevant technical field. At least 8 years of experience in data engineering or related technical domains, with a minimum of 3 years serving in a senior or leadership capacity. Deep expertise with modern data platforms, data transformation tools, and orchestration engines (such as Databricks, Spark, dbt, Airflow, Fivetran, or Kafka). Proven track record designing data patterns, including CDC, SCD, and multi-layered cloud data storage architectures (e.g., Medallion). Solid experience developing and deploying cloud-native pipelines across major cloud provider networks (such as Azure or GCP). Demonstrated experience converting legacy relational database integration tasks into modern cloud architectures. Strong proficiency in SQL along with at least one major programming language such as Python, Scala, or Java. Practical experience executing technical proofs of concept to validate new infrastructure frameworks and tools. Familiarity with AI-assisted software development and optimization utilities. Excellent communication skills with the ability to articulate highly technical data architecture concepts to non-technical stakeholders.