Azure Data Engineer

• Design, develop, and maintain complex and scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks.
• Build and optimize large-scale data processing solutions using PySpark and Databricks.
• Develop robust ETL/ELT solutions in Python for data extraction, transformation, and loading.
• Collaborate with cross-functional teams to understand data requirements and deliver efficient data engineering solutions.
• Implement and maintain CI/CD pipelines using Azure DevOps for automated testing, integration, and deployment.
• Develop, test, and maintain reliable data solutions aligned with technical and business requirements.
• Monitor, troubleshoot, and optimize data pipelines to ensure performance, scalability, and reliability.
• Work with large datasets and use SQL for data querying and data management.
• Ensure data solutions follow applicable security, compliance, and engineering best practices.
• Stay updated with the latest trends, technologies, and best practices in Azure cloud data engineering., * Design and develop scalable data pipelines using Azure Data Factory and Azure Databricks.
• Develop efficient PySpark applications for processing and transforming large volumes of data.
• Write clean, maintainable, and reusable Python-based data engineering solutions.
• Build and manage automated CI/CD workflows in Azure DevOps.
• Perform data extraction, transformation, loading, validation, and integration across different data sources.
• Optimize data pipelines and Databricks workloads for performance and scalability.
• Monitor pipelines, identify failures or bottlenecks, and implement appropriate solutions.
• Troubleshoot data processing and pipeline-related issues effectively.
• Write SQL queries for data analysis, transformation, and validation.
• Work with stakeholders to understand requirements and translate them into technical data solutions.
• Follow appropriate data security, governance, compliance, and development standards.
• Document technical solutions, data processes, pipelines, and deployment procedures.
• Adapt to new Azure services, cloud data engineering technologies, and industry best practices.

What We Bring to the Table:

• An opportunity to work on enterprise-scale Azure cloud data engineering projects.
• Exposure to Azure Data Factory, Azure Databricks, PySpark, Python, Azure DevOps, and SQL.
• Opportunities to design and optimize large-scale data pipelines and processing frameworks.
• A collaborative environment involving cross-functional technical and business teams.
• Opportunities to contribute to cloud data modernization, automation, and CI/CD initiatives.

• 8-10 years of overall professional experience in data engineering, cloud data engineering, or related technology roles.
• Strong hands-on experience with Azure Data Factory for data orchestration, movement, and transformation.
• Strong experience with Azure Databricks for developing and managing scalable data processing solutions.
• Strong programming skills in Python.
• Hands-on experience with PySpark for large-scale data processing.
• Experience implementing CI/CD pipelines using Azure DevOps.
• Good knowledge of SQL and experience working with large datasets.
• Experience developing and maintaining ETL/ELT data pipelines.
• Understanding of data security, compliance, and cloud data engineering best practices.
• Strong problem-solving and analytical skills.
• Experience working independently as well as collaboratively within cross-functional teams.