Azure Databricks Data Engineer

• Design, develop, and maintain ETL/data pipelines using Azure Data Factory and Databricks.
• Develop scalable data-processing solutions using PySpark and Python.
• Perform data validation, data-quality checks, reconciliation, and troubleshooting.
• Transform and prepare operational data for analytics and reporting.
• Identify and resolve pipeline failures, performance issues, and data inconsistencies.
• Work with business analysts, data engineers, and other stakeholders to understand data requirements.
• Participate in Agile ceremonies, including sprint planning, stand-ups, reviews, and retrospectives.
• Document pipeline designs, data mappings, validation rules, and technical processes.
• Support Airline Crew Operations data and analytics initiatives.
• Follow cloud engineering, security, testing, and deployment best practices.

Pay Range: $60 - $65

The specific compensation for this position will be determined by several factors, including the scope, complexity, and location of the role, as well as the cost of labor in the market; the skills, education, training, credentials, and experience of the candidate; and other conditions of employment. Our full-time consultants have access to benefits, including medical, dental, vision, and 401K contributions, as well as PTO, sick leave, and other benefits mandated by applicable state or localities where you reside or work.

We are seeking a Senior Azure Databricks Data Engineer to support Airline Crew Operations Data Analytics projects. The selected candidate will design, develop, test, and maintain cloud-based ETL pipelines using Azure Data Factory, Databricks, PySpark, and Python.The ideal candidate is a self-driven problem solver who can work independently, handle complex data requirements, and deliver reliable data solutions in an Agile environment. Required:

• 7+ years of overall experience in data engineering, software development, or a related field.
• 5+ years of experience working in an Agile development environment.
• 3-5 years of hands-on experience with Databricks, PySpark, and Python.
• At least 3 years of experience with Microsoft Azure and Azure Data Factory.
• 3-5 years of experience with data validation, data-quality testing, and reconciliation.
• Strong experience designing and supporting cloud-based ETL pipelines.
• Ability to analyze data issues and independently develop practical solutions.

Preferred:

• Experience optimizing Databricks workloads and large-scale data pipelines.
• Familiarity with Azure-based data architecture and deployment processes.