Lead Data Engineer

Polaris I/O is hiring a Lead Data Engineer to provide technical leadership and delivery support for our growing Data Engineering team.

Reporting directly to the Director of Data Engineering, this role will help transform business and technical priorities into well-defined, actionable engineering work. The Lead Data Engineer will partner with the Director to identify upcoming work, perform technical analysis and decomposition, create and refine JIRA tickets, identify dependencies and risks, and help maintain a healthy backlog of development-ready work for the team.

This is a technical leadership position without formal direct reports. The Lead Data Engineer will mentor engineers, review technical designs and code, establish development standards, support hiring, and provide input during performance reviews. Prioritization and formal people-management responsibilities will remain with the Director of Data Engineering.

Approximately 70% of this role will focus on technical leadership, planning, architecture, mentorship, and delivery support, with approximately 30% focused on hands-on development. The allocation may vary based on team and project needs.

Our stack includes:

• Python microservices using FastAPI
• Async Python
• AWS SQS and EventBridge
• Kubernetes and Amazon EKS
• Aurora PostgreSQL
• SQLAlchemy, Alembic, and Pydantic
• Configuration-driven workflows using YAML
• Event-driven data pipelines

The ideal candidate combines deep hands-on engineering experience with the ability to clarify complex technical work, guide other engineers, and improve the consistency and predictability of team delivery.

What You’ll Do

Technical Planning and Delivery

· Partner with the Director of Data Engineering to identify, analyze, and prepare upcoming engineering work.

· Translate technical initiatives and business requirements into clearly defined JIRA epics, stories, tasks, and technical acceptance criteria.

· Break large or ambiguous initiatives into appropriately scoped units of work for Data Engineers and Senior Data Engineers.

· Identify technical dependencies, risks, assumptions, sequencing requirements, and potential delivery constraints.

· Help maintain a healthy backlog of technically refined and development-ready work.

· Facilitate or support backlog refinement, technical planning, estimation, and sprint-planning activities.

· Assist engineers when requirements, acceptance criteria, or implementation approaches require clarification.

· Track technical progress and proactively surface delivery risks or blockers to the Director. Support prioritization decisions by providing technical context, effort assessments, and recommendations. Final prioritization will remain with the Director of Data Engineering.

Technical Leadership and Architecture

· Provide system-level and component-level architectural guidance for event-driven data platforms and Python microservices.

· Review proposed technical designs for scalability, reliability, maintainability, security, performance, and operational supportability.

· Establish and promote consistent engineering patterns, coding standards, development practices, and shared components.

· Guide the design of asynchronous workflows, service boundaries, message contracts, database models, and error-handling strategies.

· Evaluate technical tradeoffs and communicate recommendations clearly to engineers and engineering leadership.

· Identify opportunities to reduce technical debt, simplify systems, improve developer productivity, and increase platform reliability.

· Develop a strong understanding of dependencies and integrations across the systems supported by the Data Engineering team.

· Help the team balance near-term delivery needs with long-term architectural health.

· Resolve technical disagreements by facilitating fact-based discussion and documenting decisions.

Team Leadership and Mentorship

· Mentor Data Engineers and Senior Data Engineers through technical guidance, pairing, design discussions, and constructive feedback.

· Perform meaningful code reviews focused on correctness, maintainability, performance, security, and adherence to team standards.

· Help engineers develop stronger system-design, debugging, database, cloud, and software-development skills.

· Encourage ownership, documentation, knowledge sharing, and effective engineering practices across the team.

· Provide the Director of Data Engineering with relevant technical observations and feedback for use during performance reviews.

· Participate in interviewing and evaluating Data Engineer candidates as the team grows.

· Help onboard new engineers and improve team documentation, development workflows, and technical training materials.

· Serve as a trusted escalation point for complex technical problems without becoming a bottleneck for team delivery.

Hands-On Development

· Contribute to production development when team priorities and leadership responsibilities permit.

· Build and maintain Python microservices using FastAPI and async Python.

· Design and maintain Aurora PostgreSQL data models using SQLAlchemy and Alembic.

· Develop event-driven data pipelines using AWS services such as SQS, EventBridge, and Lambda.

· Deploy and support containerized workloads running in Kubernetes and Amazon EKS.

· Implement data-processing capabilities including:

o Normalization

o Deduplication

o Data hygiene

o Enrichment

o Record matching

· Tune PostgreSQL performance through query optimization, indexing, data-model improvements, and partitioning where appropriate.

· Improve observability through structured logging, monitoring, alerting, metrics, and actionable operational diagnostics.

· Debug complex issues across distributed cloud systems.

· Develop reference implementations, shared libraries, automation, and technical prototypes that other engineers can build upon.

· Bachelor’s degree in Computer Science, Engineering, or related technical field

· 2+ years of professional software engineering or data engineering experience.

· 7+ years of hands-on Python development experience.

· 3+ years of experience serving in a technical leadership capacity, such as Lead Engineer,.

· Technical Lead, Staff Engineer, Principal Engineer, or an equivalent role.

· Demonstrate experience leading technical work without relying on formal people-management authority.

· Experience translating large or ambiguous technical initiatives into actionable engineering plans and development tasks.

· Experience creating and refining JIRA epics, stories, tasks, technical requirements, and acceptance criteria.

· Strong experience with system design and architecture for event-driven or distributed systems.

· Production experience with:

o Python and FastAPI

o Async Python

o PostgreSQL performance tuning

o AWS event-driven services such as SQS and EventBridge

o Kubernetes and Amazon EKS

o SQLAlchemy, Alembic, and Pydantic

· Strong understanding of REST APIs, microservices, asynchronous processing, message-driven workflows, and service-to-service communication.

· Experience designing reliable database schemas, migrations, indexes, constraints, and data-access patterns.

· Strong debugging and incident-analysis skills in distributed cloud environments.

· Experience establishing coding standards, design patterns, code-review practices, and engineering quality expectations.

· Experience mentoring experienced engineers and providing constructive technical feedback.

· Ability to identify technical risks, dependencies, delivery constraints, and architectural tradeoffs.

· Ability to communicate technical decisions clearly in writing and in team discussions.

· Strong organizational, analytical, prioritization, and problem-solving skills.

· Strong written and verbal communication skills in English. This role includes significant documentation and collaboration with U.S.-based team members.

Preferred Qualifications

· Experience leading technical planning, backlog refinement, estimation, or sprint-planning activities.

· Experience supporting the hiring and technical evaluation of software or data engineers.

· Experience providing technical feedback that contributes to employee-development or performance-review discussions.

· Experience creating shared frameworks, libraries, templates, or automation used by multiple engineers.

· Experience improving CI/CD pipelines, automated testing, deployment processes, and developer workflows.

· Experience designing observability strategies using structured logs, metrics, dashboards, monitoring, and alerting.

· Experience evaluating cloud infrastructure performance and cost.

· Experience with configuration-driven workflows using YAML.

· Experience supporting production systems that process high-volume, sequential, or asynchronous data.

Work Authorization Requirement

Candidates must be legally authorized to work in the United States on a full-time, ongoing basis without the need for current or future employer sponsorship (for example, we are not able to sponsor employment visas now or in the future).

Work Expectations

Work Hours:This is a full-time position that requires employees to be online and actively engaged during standard business hours, and available for occasional after-hours support.

Conflict of Interest:You are expected to devote your full professional time and attention to Polaris I/O and not take on other employment or contract work that interferes with your responsibilities, competes with the company, or creates a potential conflict of interest.

Remote Work:This position is fully remote, and all work must be performed within the United States. Candidates must be legally authorized to work in the United States. Regular working hours will align with the time zone., * serving in a technical leader capacity: 3 years (Preferred)

$150,000 - $172,000 a year Health insurance, Paid time off, Vision insurance, Dental insurance Full-time Remote, Polaris I/O offers a competitive salary, a comprehensive benefit package including medical, dental, and vision coverage, a generous PTO plan, and opportunities for professional growth and development.

Polaris I/O is an Equal Opportunity Employer. We consider all qualified applicants without regard to any characteristic protected by applicable federal, state, or local law.

Pay: $150,000.00 - $172,000.00 per year

Benefits:

• Dental insurance
• Health insurance
• Paid time off
• Vision insurance

Polaris I/O provides the world’s only integrated go-to-customer platform that combines transformation services, industry insights, and powerful technology to enable B2B companies to protect, retain, and grow large accounts. Our platforms take an outside-in approach to improving commercial health, utilizing executive buyer insights to inform the orchestration of commercial activities that improve relationships, maximize growth, and scale beyond our engagement.