Data Pipeline Engineer - School of Computer Science - MLD

We are seeking a Data Pipeline Engineer to join the team! As a Data Pipeline Engineer, your role is vital in ensuring the integrity and reliability of our data pipelines. This position is responsible for monitoring, troubleshooting, and conducting root cause analysis of data quality issues within our pipelines, but as a part time team member, you will consult and assist rather than lead in these areas. Your contributions are crucial to maintaining the high standard of our epidemiological tracking and forecasting tools. This role will report directly to the Delphi Engineering Manager.

Core Responsibilities

• Monitor and maintain the health and efficiency of data pipelines.
• Troubleshoot and perform root cause analysis for data discrepancies and pipeline issues.
• Communicate with data providers to understand data discrepancies and manage changes in data delivery.
• Implement fixes and enhancements to improve data quality and pipeline performance.
• Collaborate with data scientists and analysts to understand data needs and implement effective data solutions.
• Develop strategies for data validation and quality assurance.
• Optimize data flow and collection to improve system efficiency.
• Document and manage data pipeline architectures, including maintenance and update protocols.
• Use tools such as SQL, version control and CI/CD, containerization, task schedulers, python frameworks, and cloud services for data pipeline management.
• Ensure compliance with data governance and security standards.

Adaptability, excellence, and passion are vital qualities within Carnegie Mellon University. We are in search of a team member who can effectively interact with a varied population of internal and external partners at a high level of integrity. We are looking for someone who shares our values and who will support the mission of the university through their work.

• Bachelor's Degree required.
• Minimum one year of research computing experience required.
• Basic Linux use and administration: system layout, file permissions, shell, utilities (syslog, cron), diagnostic tools (ps, htop, grep, lsof)
• Experience in Apache Airflow, preferably version 3.0
• Basic database use, especially in Postgres
• Rough script programming (Python, bash)
• Team software development (git/GitHub, Jira, code reviews, agile methodologies)
• Data analysis: diagnosing and fixing runtime errors and logic bugs; performing basic growth projections to predict future problems; communicating results
• Required technologies: Python, MySQL/Postgres, Linux, git & GitHub, Apache Airflow
• A combination of education and proven experience from which comparable knowledge is demonstrated may be considered.

Preferred Technologies and Languages:

• Linux, Ubuntu, Bash, Make
• Apache Airflow
• Python, pandas, Flask, PyPI publishing
• SQL, Postgres
• git, GitHub, GitHub Actions, GitHub Issues
• Docker, Docker Compose
• Elastic, Kibana, FileBeat
• G Suite (Calendar, Mail, Docs, Sheets, Slides, Forms, AppsScript, Groups)
• Jira Software

Requirements :

• Successful completion of a pre-employment background check

Carnegie Mellon University is a private, global research university that stands among the world's most renowned education institutions. With ground-breaking brain science, path-breaking performances, creative start-ups, big data, big ambitions, hands-on learning, and a whole lot of robots, CMU doesn't imagine the future, we invent it. If you're passionate about joining a community that challenges the curious to deliver work that matters, your journey starts here! The Machine Learning Department (MLD) at Carnegie Mellon University is a leading hub for research and education in artificial intelligence and machine learning. It focuses on developing innovative algorithms and models to address complex problems in diverse fields such as robotics, healthcare, and finance. The department offers a range of undergraduate and graduate programs, fostering a collaborative environment that bridges theoretical research and practical applications. Faculty and students frequently collaborate with industry and other academic disciplines to push the boundaries of what is possible with machine learning., Benefits eligible (https://www.cmu.edu/hr/benefits/eligibility/index.html) employees enjoy a wide array of benefits including comprehensive medical, prescription, dental, and vision insurance (https://www.cmu.edu/hr/benefits/health-welfare/index.html) as well as a generous retirement savings program (https://www.cmu.edu/hr/benefits/retirement-savings/index.html) with employer contributions. Unlock your potential with tuition benefits (https://www.cmu.edu/hr/benefits/tuition/index.html) , take well-deserved breaks with ample paid time off (https://www.cmu.edu/hr/benefits/time-away/pto.html) and observed holidays (https://www.cmu.edu/hr/benefits/time-away/holidays.html) , and rest easy with life and accidental death and disability insurance. Additional perks include a free Pittsburgh Regional Transit bus pass, access to our Family Concierge Team (https://www.cmu.edu/hr/work-life/support/family-child-care-resources/index.html) to help navigate childcare needs, fitness center access (https://athletics.cmu.edu/recreation/facilities) , and much more! For a comprehensive overview of the benefits available, explore our Benefits page (https://www.cmu.edu/hr/benefits/index.html) . At Carnegie Mellon, we value the whole package when extending offers of employment. Beyond credentials, we evaluate the role and responsibilities, your valuable work experience, and the knowledge gained through education and training. We appreciate your unique skills and the perspective you bring. Your journey with us is about more than just a job; it's about finding the perfect fit for your professional growth and personal aspirations., Interested in a career with Carnegie Mellon University but not finding anything that currently aligns with your interests, background, or experience? Learn how to sign up for Job Alerts (https://www.cmu.edu/jobs/external-applicants.html#job-alerts) through your candidate profile. If your heart is in your work, come work with us. Carnegie Mellon University isn't just one of the world's most renowned educational institutions - it's also a hotspot for some of the most talented doers, dreamers, and difference-makers on the planet. When you join our staff, you'll become an important part of our mission to create a healthier, safer, and more just life for all. No matter what your role or location, you'll connect and collaborate with dedicated, passionate colleagues - and you'll have the satisfaction of delivering work that truly matters. We cultivate a vibrant, welcoming environment where everyone is valued and encouraged to contribute and achieve. In addition to competitive benefits and a robust support network, you'll have access to many tools and resources to sharpen your abilities and professional skills, as well as opportunities to engage and share perspectives with a dynamic and inspiring community of uniquely talented staff, faculty, students, and alumni.