Lead Data Engineer

The Doyle Group
Shelton, CT, United States
8 days ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
6 years minimum
Working hours
Regular working hours
Job source

Tech stack

Query Performance Artificial Intelligence Amazon Web Services Amazon S3 Architectural Patterns Code Review Data Architecture Information Engineering Data Integration Extract Transform Load (ETL) Relational Databases Dimensional Modeling
+33 more
Django Web Framework Identity and Access Management Python (Programming Language) PostgreSQL OAuth Operational Databases Query Optimization Raw Data Role-Based Access Control Power BI Software Requirements Analysis SQL Databases Data Logging Google Data Studio Flask (Web Framework) Large Language Models Snowflake Virtual Environment Backend Cloudformation Fastapi Build Management Amazon Relational Database Service AWS Glue AWS Fargate Functional Programming Cloudwatch Restful APIs Amazon Simple Queue Service (SQS) Terraform Pagination Data Pipelines Call Tracing

Job description

  • Own the full ETL/ELT lifecycle across dozens of media, CRM, and vendor data sources: extraction, transformation, loading into Snowflake, orchestration, scheduling, retries, alerting, and backfills, and define the patterns the rest of the team builds to.
  • Design and build scalable, reliable, idempotent data pipelines that move raw data through to modeled, analysis-ready tables in a raw-to-curated (medallion-style) data architecture.
  • Set the architectural direction for how new pipelines, connectors, and integrations get designed, reviewed, and shipped across the team.
  • Integrate external systems via REST APIs, including ad platforms, CRMs, call tracking, and vendor feeds, handling OAuth 2.0 and token refresh, pagination, rate limits, exponential backoff, schema drift, and partial failures without losing or duplicating data.
  • Configure and manage ELT connectors such as Airbyte, Funnel.io, and AWS Glue, including sync scheduling, schema-change handling, monitoring, and row-volume cost management.
  • Manage AWS infrastructure as code using Terraform or CloudFormation so environments stay versioned, peer-reviewed, and reproducible, and serve as the final technical call on infrastructure decisions.
  • Own core Snowflake functionality: schema design, dimensional and analytics modeling, partitioning and clustering, warehouse sizing, role-based access control, and query performance tuning.
  • Build data quality and observability into every pipeline: freshness and volume checks, row-level validation, reconciliation against source-of-truth platforms, lineage, and alerting that surfaces problems before stakeholders do.
  • Treat platform cost, including Snowflake credit consumption and AWS spend, as a design constraint rather than an afterthought, and set the cost guardrails other engineers design against.
  • Design LLM-readable, MCP-compatible interfaces with clear action-based endpoints and strongly typed schemas so AI agents and automation can reliably query and act on the data platform, and define how those agents access and maintain context.
  • Serve as the primary technical point of contact for analytics, media buying, account, and finance stakeholders, translating their needs into system requirements and resolving data and application issues.
  • Lead code review, pair with other engineers on hard problems, unblock the team, and help set the shared technical standards the rest of the group codes to.
  • Mentor engineers, help define coding standards and architectural patterns, and weigh in on technical hiring decisions, without owning formal people-management or performance responsibilities.
  • Help keep the team building reusable, leverageable systems rather than one-off, ad hoc fixes that don’‘t compound over time.

Requirements

This is a full-time, direct-hire position. Candidates must be authorized to work in the United States without current or future visa sponsorship., * 6+ years of professional experience in data engineering, analytics engineering, or a backend data-focused role, including time spent setting technical direction or acting as a de facto technical lead.

  • Expert-level Python for production data work, including:
  • Clean, modular, testable code; virtual environments and dependency management; error handling and logging
  • Building and consuming REST APIs (FastAPI, Flask, or similar)
  • Extending existing frameworks (e.g., Django) or writing custom connectors rather than building everything from scratch
  • Expert-level SQL, including CTEs, window functions, and query optimization.
  • Deep, hands-on Snowflake expertise - a core requirement, not a nice-to-have:
  • Schema design, dimensional modeling, partitioning and clustering
  • Warehouse sizing and credit/cost management
  • Role-based access control and query performance tuning
  • Comfort working in a raw-to-curated (medallion-style) data architecture
  • Comprehensive hands-on AWS experience - a requirement, not a preference:
  • Core comfort with S3, Lambda, and IAM
  • Working experience with several of: Glue, Step Functions, EventBridge, ECS/Fargate, Secrets Manager, CloudWatch, RDS, and SQS/SNS
  • Infrastructure as code via Terraform or CloudFormation, with a clear point of view on structuring state, modules, and environments
  • Experience with a managed ELT/ETL / data integration platform such as Airbyte or Funnel.io.
  • Pipeline orchestration experience with at least one of Temporal, Prefect, or AWS Step Functions.
  • Meaningful big-data / cloud-warehouse experience - a background limited to a single relational database (e.g., Postgres) is not sufficient depth for this role.
  • Hands-on experience building with LLMs and AI agents in production - not just using AI coding assistants day to day:
  • MCP-compatible interfaces, structured tool schemas, or agent orchestration workflows
  • Demonstrated technical leadership:
  • Setting architectural direction, running design reviews, and mentoring engineers through code review and pairing (this is a lead role measured by technical influence, not headcount managed)
  • Clear communication with both technical and non-technical stakeholders, with strong attention to detail, project management, and organizational skills.
  • Bachelor’’s degree or equivalent professional experience.
  • Must be legally authorized to work in the United States without sponsorship now or in the future.

ADDITIONAL PLUS

  • Background in media, advertising, or direct-response marketing (channels, sources, conversions, performance signals), or an adjacent domain such as CRM, sales enablement, or a DSP/media platform
  • Experience managing a high-volume connector environment (30-50+ integrations) with a monitoring and cost-management discipline
  • Prior experience mentoring or growing a data engineering team toward a formal management track
  • Experience with Power BI, Looker Studio, or similar BI tools consuming the pipelines you build
  • Comfort operating in a fast-paced, deliverable-driven, daily-standup team culture
  • Product-building experience - having shipped something from scratch, not just extended existing systems

About the company

ABOUT THE DOYLE GROUP The Doyle Group is a proven partner for Placement and Consulting services, headquartered in Denver, CO. Our core mission is to forge genuine partnerships with our clients who seek strategic talent solutions and to assist highly skilled candidates looking for their next career opportunity. With over 30 years of industry experience, our consultative approach allows us to provide a higher level of guidance and insight, empowering our clients to secure top IT talent that fits seamlessly into their team and culture. We look forward to collaborating to help you achieve your career goals., Our client is a high-growth, private equity-backed marketing and advertising technology agency that delivers direct-response campaigns across TV, audio, digital, and direct mail, with a rapidly expanding data and analytics function supporting both its legacy media business and a growing slate of digital campaigns. This is a hands-on technical lead role on a data engineering team of roughly six engineers, created to backfill a departing team member and give the group a technical anchor: someone who leads through code, architecture, and example rather than headcount. You’‘ll set technical direction for the data platform, own key architectural decisions, and raise the bar for the engineers around you through code review, pairing, and shared standards, while still spending the large majority of your week hands-on in the pipelines, integrations, and Snowflake environment the business runs on. The ideal candidate wants to own systems end to end, is comfortable being roughly 60-80% hands-on while also being the person the team turns to for architecture calls, and is genuinely energized by the idea of building the interfaces that let the business’s growing use of AI agents reliably read from and act on the data platform. This is a technical-influence role, not a people-management role, though there is room to grow toward formal management over time for the right person.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Talks and stories from around this role — technically off-topic, practically not.

2:49 min

Adopting OAuth best practices and removing outdated grants

Alexander Schwartz Alexander Schwartz · World Congress 2026 Europe

3:33 min

Connecting frontends via a FastAPI proxy backend layer

Saoussen Chaabnia Saoussen Chaabnia · Europe 2026 Virtual

1:52 min

Structuring and scaling the backend engineering team

Stefan Lingler Stefan Lingler +1 · Coffee With Developers

3:03 min

Career evolution in data engineering and AI platforms

Maria Apazoglou · Coffee With Developers

1:34 min

Analyzing vulnerabilities in standard OAuth 2.0 authorization flows

Alexander Schwartz Alexander Schwartz · World Congress 2026 Europe

3:09 min

Balancing data science skillings alongside systems engineering rigor

Nico Schmidt · LIVE

Videos

See all

Related articles

See all