Senior Data Engineer

We are seeking a highly skilled Senior Data Engineer with 10+ years of experience in designing, developing, and maintaining scalable, high-performance data engineering solutions. The ideal candidate will have strong expertise in data architecture, ETL/ELT development, cloud data platforms, data modeling, and big data technologies. This role requires hands-on technical leadership, excellent problem-solving skills, and the ability to collaborate with architects, data scientists, business stakeholders, and cross-functional engineering teams to deliver reliable, secure, and cost-effective data solutions., * Design, develop, and maintain scalable, reliable, and high-performance data pipelines for batch and real-time data processing.

• Architect and implement end-to-end ETL/ELT workflows using modern data engineering tools and frameworks.
• Build and optimize data solutions on cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform.
• Develop complex SQL queries, stored procedures, data transformations, and data models to support analytics and reporting requirements.
• Work with cloud data warehouses and lakehouse platforms such as Snowflake, Azure Synapse Analytics, Databricks, and Microsoft Fabric.
• Implement data ingestion, transformation, integration, and orchestration processes using tools such as Azure Data Factory, Apache Airflow, or equivalent technologies.
• Develop data processing and automation solutions using Python, PySpark, SQL, and other relevant programming languages.
• Implement incremental data loading, Change Data Capture (CDC), data replication, and real-time streaming solutions.
• Design and maintain data lakes, data warehouses, data marts, and enterprise data platforms.
• Establish data quality checks, validation frameworks, data lineage, metadata management, and monitoring mechanisms.
• Optimize data pipelines and queries for performance, scalability, reliability, and cost efficiency.
• Implement data security, access controls, encryption, governance, and compliance standards.
• Collaborate with data architects, business analysts, data scientists, BI developers, and application teams to translate business requirements into technical solutions.
• Implement CI/CD pipelines, version control, automated testing, and Infrastructure as Code practices for data engineering workflows.
• Troubleshoot complex production issues, identify root causes, and implement permanent solutions.
• Mentor junior and mid-level engineers, conduct code reviews, and establish engineering best practices.
• Lead technical discussions, contribute to architectural decisions, and drive continuous improvement across data engineering initiatives.

• Advanced SQL, including complex joins, CTEs, window functions, query optimization, and stored procedures.
• Python for data processing, automation, API integration, and ETL/ELT development.
• PySpark and Apache Spark for large-scale distributed data processing.

1. Cloud Platforms * Strong hands-on experience with at least one major cloud platform: Microsoft Azure, AWS, or Google Cloud Platform. * Experience with cloud storage, data integration services, compute resources, identity management, and monitoring. * Knowledge of cloud security, scalability, availability, and cost optimization.

2. Data Engineering and Integration * Strong experience building enterprise-grade ETL/ELT pipelines. * Expertise in batch processing, incremental loads, CDC, and streaming data integration. * Experience with Azure Data Factory, Apache Airflow, AWS Glue, or similar orchestration tools. * Familiarity with REST APIs, file-based ingestion, database integration, and event-driven architectures.

3. Data Warehousing and Lakehouse * Experience with Snowflake, Azure Synapse Analytics, Databricks, Microsoft Fabric, Amazon Redshift, or equivalent platforms. * Strong understanding of dimensional modeling, star and snowflake schemas, data marts, and data warehouse architecture. * Experience with Delta Lake, data lakes, lakehouse architecture, partitioning, and schema evolution.

4. Big Data and Streaming Technologies * Experience with Apache Spark and PySpark. * Knowledge of Apache Kafka, Azure Event Hubs, AWS Kinesis, or similar streaming technologies. * Understanding of distributed computing, parallel processing, and large-scale data transformation.

5. DevOps and DataOps * Experience with Git, GitHub, GitLab, Azure DevOps, or equivalent version control and CI/CD tools. * Knowledge of automated testing, deployment pipelines, workflow monitoring, and environment management. * Familiarity with Docker, Kubernetes, Terraform, or other infrastructure automation tools is an advantage., * 10+ years of overall experience in data engineering, data integration, data warehousing, or related technical roles. * Strong hands-on experience designing and implementing complex, enterprise-scale data pipelines. * Proven experience working with large datasets and optimizing data processing workloads. * Experience delivering cloud-based data engineering solutions in production environments. * Strong understanding of data architecture, data modeling, data governance, and data quality principles. * Experience handling production support, performance tuning, and troubleshooting complex data issues. * Demonstrated ability to lead technical initiatives and collaborate with cross-functional teams. * Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent practical experience., * Experience with Snowflake features such as Streams, Tasks, Dynamic Tables, and performance tuning. * Experience with Microsoft Fabric, Azure Databricks, or Azure Synapse Analytics. * Knowledge of metadata-driven pipelines, data lineage, and automated data quality frameworks. * Experience implementing data mesh, domain-oriented data platforms, or modern lakehouse architectures. * Familiarity with data catalog and governance tools such as Microsoft Purview or equivalent platforms. * Experience with machine learning data pipelines and feature engineering workflows. * Relevant cloud certifications in Azure, AWS, Google Cloud Platform, or Databricks are a plus.

Depends on Experience