Data Engineer - AI Infrastructure

This Senior Data Engineer - AI Infrastrucute role focuses on designing and implementing data pipelines, ensuring correctness, and building scalable data models. You will work closely with data scientists and platform engineers to ensure that data is accurate, reliable, and usable for downstream decision-making.

We are looking for engineers who care deeply about data correctness, understand how systems behave at scale, and can translate complex data into well-structured, reliable datasets.

Responsibilities

• Design and implement large-scale data pipelines using PySpark and distributed processing frameworks

• Build and maintain data models that accurately represent underlying system behavior and business logic

• Ensure high standards of data correctness, completeness, and consistency across datasets

• Develop validation, monitoring, and alerting mechanisms to detect data quality issues

• Partner with data scientists to support experimentation and analytics use cases

• Collaborate with platform engineers to ensure efficient data ingestion, processing, and storage

• Optimize pipelines for performance, scalability, and cost efficiency

• Define and enforce best practices for schema design, data transformations, and pipeline reliability

Master's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 3+ years experience in business analytics, data science, software development, data modeling, or data engineering OR Bachelor's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 4+ years experience in business analytics, data science, software development, data modeling, or data engineering OR equivalent experience.

Other Requirements:

Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter.

Preferred Qualifications:

• Experience with Azure technologies such as:

• ADLS Gen2 (Blob Storage)

• Synapse Spark

• Azure Data Explorer (ADX)

• Experience working with structured and semi-structured data (e.g., JSON logs)

• Familiarity with experimentation and analytics workflows

• Experience with orchestration tools (e.g., Airflow)

• Exposure to privacy, compliance, and secure data handling practices

• 5+ years of experience in data engineering or software engineering with a strong focus on data systems

• Strong experience with PySpark or similar distributed data processing frameworks

• Experience building and operating large-scale data pipelines

• Strong understanding of data modeling and schema design

• Experience ensuring data quality and correctness in production systems

• Proficiency in Python

• Experience working with cloud-based data platforms (Azure, AWS, or Google Cloud Platform)

• Ability to reason about data at scale, including performance and failure modes

#aiinfra

Data Engineering IC4 - The typical base pay range for this role across the U.S. is USD $119,800 - $234,700 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $158,400 - $258,000 per year.

Microsoft is a global technology company headquartered in Redmond, Washington. Our mission is to empower every person and every organization on the planet to achieve more. We develop, license, and support a wide range of software products, services, and devices that help individuals and businesses realize their full potential.

Our flagship products include the Microsoft 365 productivity cloud, Windows operating system, Azure cloud platform, and Dynamics 365 business applications. We are also a leader in areas such as artificial intelligence, cybersecurity, developer tools, and gaming through Xbox and Game Pass.

With operations in more than 190 countries and over 220,000 employees worldwide, Microsoft is committed to responsible innovation, inclusive economic growth, and sustainability. We work closely with governments, industries, and communities to ensure that technology serves the public good and helps address some of the world’s most pressing challenges.

As we celebrate our 50th anniversary in 2025, we continue to look forward—investing in AI, cloud, and quantum computing to shape the future of work, education, and society at large scale.