Senior Data Engineer - Job Description

We are seeking a highly experienced Senior Data Engineer with 12+ years of experience in designing, developing, and maintaining scalable data platforms and data pipelines. The ideal candidate will have strong hands-on expertise in Python, SQL, Spark, Databricks, cloud technologies, ETL/ELT, data warehousing, and data integration., * Design, develop, and maintain scalable ETL/ELT data pipelines for enterprise applications.

• Develop complex data transformation and processing workflows using Python, SQL, and Apache Spark.
• Build and optimize data pipelines using Databricks and PySpark.
• Design and implement data solutions on AWS, Azure, or Google Cloud Platform cloud platforms.
• Develop data ingestion processes from databases, APIs, files, and other structured/unstructured sources.
• Implement data quality, validation, reconciliation, and monitoring processes.
• Design and optimize data warehouses, data lakes, and lakehouse architectures.
• Work with relational and NoSQL databases to support data engineering requirements.
• Perform performance tuning and optimization of SQL queries, Spark jobs, and data pipelines.
• Implement orchestration using Apache Airflow, Azure Data Factory, AWS Glue, or similar tools.
• Develop reusable frameworks and components for data ingestion and transformation.
• Integrate data engineering solutions with CI/CD pipelines and DevOps processes.
• Implement data security, access controls, encryption, and governance best practices.
• Troubleshoot production data pipelines and resolve data processing issues.
• Collaborate with data architects, analysts, data scientists, application developers, and business stakeholders.
• Participate in Agile/Scrum ceremonies and contribute to technical design and architecture discussions.
• Mentor junior and mid-level data engineers and provide technical guidance.

• 12+ years of experience in Data Engineering, ETL, or related data technologies.
• Strong hands-on experience with Python and SQL.
• Extensive experience with Apache Spark / PySpark.
• Strong experience with Databricks and Delta Lake.
• Experience building enterprise-scale ETL/ELT pipelines.
• Strong knowledge of data warehousing concepts, dimensional modeling, and data lake architectures.
• Experience with one or more cloud platforms: AWS, Azure, or Google Cloud Platform.
• Experience with databases such as SQL Server, Oracle, PostgreSQL, MySQL, Snowflake, or similar.
• Experience with workflow orchestration tools such as Airflow, Azure Data Factory, AWS Glue, or similar.
• Strong understanding of batch and near-real-time data processing.
• Experience with Git, CI/CD, Jenkins, Azure DevOps, or GitHub Actions.
• Strong understanding of data quality, data validation, and performance optimization.
• Excellent analytical, troubleshooting, and communication skills.

Preferred Skills

• Experience with Azure Data Lake, AWS S3, ADLS, or Google Cloud Storage.
• Experience with Snowflake or other modern cloud data warehouses.
• Knowledge of Kafka, Event Hubs, or other streaming technologies.
• Experience with Terraform or Infrastructure as Code.
• Knowledge of Docker and Kubernetes.
• Experience implementing data governance, metadata management, and lineage.
• Experience working with large-scale distributed data environments.
• Knowledge of Medallion Architecture (Bronze, Silver, Gold).
• Experience with real-time/streaming data pipelines.
• Experience in designing highly available and fault-tolerant data platforms.