Senior Data Engineer
Role details
Job location
Tech stack
Job description
As a Senior Data Engineer, you will work to define the data ontology for all of Yahoo Mail, establish standard methodologies for data operations and lifecycle management, design and build analytics tooling and frameworks, and influence event instrumentation. Additionally, this role is highly multi-functional, requiring close collaboration with Data Science and Machine Learning teams to understand customer requirements and analytics applications, as well as with other Mail engineering teams to develop integrated solutions.
As part of the Mail Analytics Infrastructure & Data Engineering team, you will be working on large-scale batch pipelines, data serving, data lakehouse, and analytics systems, enabling mission critical decision making, downstream, AI-powered capabilities, and more.
If you thrive on building data infrastructure and platforms that power modern data- and AI-driven businesses at scale, we'd love to hear from you!
Your Day
- Partner with Data Science, Product, and Engineering to collect requirements to define the data ontology for Mail Data & Analytics
- Lead and mentor junior Data Engineers to support Yahoo Mail's ever-evolving data needs
- Design, build, and maintain efficient and reliable batch data pipelines to populate core data sets
- Develop scalable frameworks and tooling to automate analytics workflows and streamline users interactions with data products
- Establish and promote standard methodologies for data operations and lifecycle management
- Develop new or improve and maintain existing large-scale data infrastructures and systems for data processing or serving, optimizing complex code through advanced algorithmic concepts and in-depth understanding of underlying data system stacks
- Create and contribute to frameworks that improve the efficacy of the management and deployment of data platforms and systems, while working with data infrastructure to triage and resolve issues
- Prototype new metrics or data systems
- Define and manage Service Level Agreements for all data sets in allocated areas of ownership
- Develop complex queries, very large volume data pipelines, and analytics applications to solve analytics and data engineering problems
- Collaborate with engineers, data scientists, and product managers to understand business problems, technical requirements to deliver data solutions
- Engineering consulting on large and complex data lakehouse data
Requirements
- BS in Computer Science/Engineering, relevant technical field, or equivalent practical experience, with specialization in Data Engineering
- 6+ years of experience building scalable ETL pipelines on industry standard ETL orchestration tools (Airflow, Composer, Oozie) with deep expertise in SQL, PySpark, or scala.
- Built, scaled, and maintained Multi-Terabyte data sets and having an expansive toolbox for debugging and unblocking large scale analytics challenges (skew mitigation, sampling strategies, accumulation patterns, data sketches, etc.)
- Experience with at least one major cloud's suite of offerings (AWS, GCP, Azure).
- Developed or enhanced ETL orchestrations tools or framework
- Worked within standard GitOps workflow (branch and merge, PRs, CI / CD systems)
- Experience working with GDPR
- Highly self-motivated with a strong sense of ownership
- Detail-oriented with a commitment to quality and accuracy
- Collaborative team player who contributes positively to group success
- Strong written and verbal communication skills
- Able to prioritize effectively, manage multiple tasks, and set clear expectations
Preferred
- 3+ years experience in Google Cloud Platform technologies (BiqQuery, Dataproc, Dataflow, Composer, Looker)
The material job duties and responsibilities of this role include those listed above as well as adhering to Yahoo policies ; exercising sound judgment ; working effectively, safely and inclusively with others ; exhibiting trustworthiness and meeting expectations ; and safeguarding business operations and brand integrity.
Benefits & conditions
The compensation for this position ranges from $128,250.00 - $266,875.00/yr and will vary depending on factors such as your location, skills and experience.The compensation package may also include incentive compensation opportunities in the form of discretionary annual bonus or commissions. Our comprehensive benefits include healthcare, a great 401k, backup childcare, education stipends and much (much) more., $90,000.00 - $180,000.00 per year Engineering 2 minutes ago (USA) Staff, Cyber Intelligence Engineer Walmart Herndon, Virginia $132,000.00 - $264,000.00 per year Engineering 4 minutes ago (USA) Manager II, Process Engineer - Supply Chain Walmart Olney, Illinois $104,000.00 - $156,000.00 per year
