Senior Software Engineer- Data

The Data Engineer will be responsible for building robust data pipelines, transforming and processing large datasets, implementing data integration solutions, and working closely with data architects, analysts, application teams, and business stakeholders to deliver reliable and high-quality data solutions., * Design, develop, and maintain scalable data pipelines and ETL/ELT processes using Azure Data Factory and Databricks.

• Develop data processing and transformation solutions using Python and PySpark .
• Build and optimize data pipelines for batch and incremental data processing.
• Develop data ingestion frameworks to integrate data from databases, APIs, files, cloud storage, and other data sources.
• Develop and manage Azure Data Factory pipelines , datasets, linked services, triggers, parameters, and integration runtimes.
• Develop and optimize Databricks notebooks, jobs, workflows, and Spark-based data processing applications .
• Implement data transformations, cleansing, validation, aggregation, and enrichment processes.
• Work with Azure Data Lake Storage (ADLS) and other Azure data services.
• Design and implement scalable data lake and data warehouse solutions.
• Optimize PySpark jobs and Databricks workloads for performance and cost efficiency.
• Implement incremental data loading and change data capture (CDC) patterns where required.
• Monitor data pipelines and troubleshoot failures, performance issues, and data quality problems.
• Implement data quality checks and validation frameworks to ensure data accuracy and completeness.
• Work closely with data architects and technical leads to implement enterprise data architecture standards.
• Collaborate with business analysts and stakeholders to understand data requirements and translate them into technical solutions.
• Participate in code reviews and follow established coding, testing, documentation, and deployment standards.
• Implement CI/CD processes for data engineering solutions using Azure DevOps or similar tools.
• Maintain technical documentation for data pipelines, data flows, transformations, and integration processes.
• Support production deployments and resolve data-related incidents and issues.
• Mentor junior data engineers and contribute to improving data engineering best practices.

• designing, developing, and maintaining scalable data solutions.

• strong hands-on experience with Microsoft Azure, Azure Data Factory (ADF),

• Python/PySpark, Azure Databricks .

• Azure Data Lake Storage, * 6+ years of professional experience in Data Engineering, ETL development, or related roles.
• Strong hands-on experience with Microsoft Azure data services.
• Strong experience with Azure Data Factory (ADF) .
• Strong programming experience in Python .
• Hands-on experience with PySpark and Apache Spark .
• Strong hands-on experience with Databricks .
• Experience working with Azure Data Lake Storage (ADLS) .
• Strong SQL skills and experience working with relational databases.
• Experience designing and developing complex ETL/ELT pipelines.
• Experience working with large-volume and complex datasets.
• Strong understanding of data warehousing and data lake concepts.
• Experience with batch and incremental data processing.
• Experience with Git and source control.
• Experience with CI/CD and deployment processes, preferably using Azure DevOps .
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration skills.