> Markdown version of [/jobs/ext/3451292-data-engineer-lead](https://www.wearedevelopers.com/jobs/ext/3451292-data-engineer-lead). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Data Engineer Lead - **Company:** AIG View all jobs - **Location:** Parsippany-Troy Hills, NJ, United States - **Experience:** Expert - **Salary:** $125,000.0 - $135,000.0 - **Contract:** Permanent contract - **Skills:** Query Performance, Amazon Web Services, Amazon S3, ARM Architecture, Code Review, Information Systems, Continuous Integration, Data as a Services, Data Dictionary, Information Engineering, Data Governance, Software Design Patterns, Fault Tolerance, Identity and Access Management, Python (Programming Language), Operational Databases, Scrum Methodology, Release Management, SQL Databases, Snowflake, Apache Spark, Pyspark, Information Technology, Functional Programming, Cloudwatch, Data Pipelines, Amazon Elastic Mapreduce (EMR) - **Published:** September 1, 2026 - **Apply:** https://www.careerjet.com/job/us0b8963a42aba72dd32cb26eaa716c6b5/eaa ## About the Role * Bachelor's degree in Computer Science, Information Systems, Engineering, or a related technical field. * 7+ years of hands-on data engineering experience with a consistent track record delivering production data pipelines end-to-end. * Strong command of PySpark, Python, and SQL - including Spark optimization, partition management, and query performance tuning in Snowflake. * Hands-on experience with AWS data services: S3, Glue, EMR, Aurora, Lambda, IAM, and CloudWatch. * Working knowledge of architecture and design principles - Lakehouse patterns, data modeling, fault-tolerant ingestion - with demonstrated ability to apply them in practice. * Strong analytical and problem-solving skills: able to decompose complex requirements, reason through design trade-offs, and arrive at practical, well-justified solutions. * Clear communicator - able to explain technical decisions and design rationale in plain language to both engineering peers and business stakeholders. * Self-motivated learner with a genuine interest in staying current with evolving tools, frameworks, and engineering best practices. Preferred/Nice To Have * AWS certification - Solutions Architect Associate or equivalent. * Domain familiarity with insurance or financial services data - claims, policy, risk, or actuarial structures. * Experience writing ADRs, design notes, or technical documentation in a collaborative engineering team. * Demonstrated use of GenAI tooling (e.g. Claude Code CLI or Snowflake Cortex) to improve productivity or solution quality. ## Description The Data Engineer Lead is a hands-on engineering role within AIG's Data Engineering organization. You will design, build, and operate production-grade data pipelines and platforms - and bring sound engineering judgment to every solution you deliver. We actively leverage GenAI tooling - including Claude Code and Snowflake Cortex - as a genuine productivity multiplier across the development lifecycle. We want someone who is motivated to learn, keeps up with a fast-moving space, and continuously finds new ways to put these tools to work. You will collaborate closely with the product owner, business stakeholders, and fellow engineers. Strong communication and analytical skills matter here as much as technical depth - the ability to ask the right questions, think through trade-offs clearly, and explain your reasoning is part of the job. Responsibilities * Design, build, and operate production-grade PySpark and Python pipelines on AWS EMR, Glue, and S3 with integrated data quality checks and observability. * Own end-to-end delivery of data products from requirements through production, working within an Agile/Scrum framework - including sprint planning, CI/CD, release management, and production readiness * Translate business and product requirements into concrete technical designs, defining pipeline structure, data models, and SLAs. * Demonstrate a sound understanding of architecture and design patterns, evaluate trade-offs of design decisions, and articulate clear rationale to peers and stakeholders. * Set and enforce engineering standards - coding conventions, data quality frameworks, reusable pipeline patterns, and observability hooks - across the team. * Conduct thorough code reviews and pair with engineers on hard problems, raising the technical quality of the whole team. * Leverage GenAI tooling - Claude Code CLI, Snowflake Cortex - throughout the development lifecycle to write better code faster and improve solution quality. * Collaborate with product owners to refine requirements, surface trade-offs, and push back constructively when scope is technically unworkable. * Communicate design decisions and delivery updates clearly to both engineering peers and non-technical stakeholders. * Mentor junior engineers through review, pairing, and knowledge sharing, building team capability alongside product delivery. * Produce clear technical documentation - Architecture Decision Record (ADR), runbooks, data dictionaries, and pipeline lineage notes - as a standard part of delivery, not an afterthought. * Evaluate and drive adoption of productivity tooling and best practices, setting team norms for safe and effective use., Well established real estate company seeking and an experienced Leasing Manager to lead our growing talented team! This Jobot Job is hosted by: Josh Rabinowitz Are you a fit? Ea… + 7 days ago ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [How Cisco embraced a DevOps culture within its network engineering team](https://www.wearedevelopers.com/videos/99-how-cisco-embraced-a-devops-culture-within-its-network-engineering-team) - [WeAreDevelopers LIVE - CSS is DOOMed](https://www.wearedevelopers.com/videos/1838-wearedevelopers-live-css-is-doomed) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production)