Data Engineer

We are seeking an experienced Data Engineer with strong hands-on expertise in AWS, Databricks, PySpark, and Python. The ideal candidate will have experience designing and developing scalable data pipelines and data processing solutions using AWS cloud services and Databricks., * Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and AWS.

• Develop robust ETL/ELT pipelines for ingesting and transforming large volumes of data.
• Build and optimize PySpark/Spark SQL jobs within Databricks.
• Work with AWS S3 for data storage and data lake solutions.
• Develop data processing workflows using Databricks Workflows and related AWS services.
• Work with Delta Lake for reliable data storage, incremental processing, and data transformation.
• Develop reusable Python frameworks and utilities for data engineering processes.
• Perform data validation, quality checks, error handling, and reconciliation.
• Troubleshoot production pipeline issues and perform Root Cause Analysis (RCA).
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Integrate data from databases, APIs, files, and other enterprise data sources.
• Implement CI/CD and source-control practices for data engineering applications.
• Collaborate with Data Architects, Data Scientists, Analysts, and business stakeholders.
• Participate in design, development, testing, deployment, and production support.

• 10+ years of overall Data Engineering experience preferred.
• Strong hands-on experience with AWS.
• Strong experience with Databricks.
• Strong hands-on experience with PySpark / Apache Spark.
• Strong Python programming experience.
• Strong SQL skills.
• Experience developing enterprise-scale ETL/ELT pipelines.
• Experience with AWS S3 and AWS data services.
• Experience with Delta Lake.
• Experience working with large-scale datasets.
• Strong understanding of data lake/lakehouse architecture.
• Experience with Spark performance tuning and optimization.
• Experience with Git and CI/CD.

AWS Skills

Candidates should have hands-on experience with AWS services such as:

• Amazon S3
• AWS Glue
• AWS Lambda
• Amazon Redshift
• Amazon EMR
• AWS CloudWatch
• AWS IAM

Preferred Skills

• Databricks certification
• Experience with Unity Catalog
• Experience with Airflow
• Experience with Kafka
• Experience with AWS Glue/Airflow orchestration
• Experience with Terraform
• Experience with data governance and data quality frameworks