Senior Data Engineer

We are seeking a highly skilled and experienced Senior Data Engineer with 10+ years of expertise in designing, developing, and optimizing enterprise-scale data platforms across Azure, AWS, and GCP cloud ecosystems. The ideal candidate should possess strong hands-on experience with modern data engineering frameworks, cloud-native services, big data technologies, real-time data processing, data warehousing, and AI-ready analytics platforms.

The candidate will play a key role in building scalable, secure, and high-performance data pipelines, enabling advanced analytics, reporting, machine learning, and enterprise data modernization initiatives.

Key Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines across multi-cloud platforms.
• Develop enterprise-grade data solutions using Azure, AWS, and GCP services.
• Build and optimize modern Lakehouse architectures using Databricks, Microsoft Fabric, Synapse, Snowflake, and BigQuery.
• Implement batch and real-time data ingestion pipelines using Kafka, Spark Streaming, Event Hub, Kinesis, and Pub/Sub.
• Create reusable and automated workflows using orchestration tools such as Airflow, ADF, Glue Workflows, and Fabric Pipelines.
• Optimize cloud data warehouse performance, partitioning strategies, indexing, and cost management.
• Work closely with Data Scientists, BI teams, Architects, and Business stakeholders.
• Implement data governance, cataloging, lineage, security, and compliance standards.
• Build CI/CD-enabled deployment frameworks for data engineering solutions.
• Monitor and troubleshoot large-scale distributed data processing systems.
• Support AI/ML data preparation pipelines and GenAI-enabled analytics initiatives.
• Lead architecture discussions and mentor junior engineers., * Microsoft Azure
• Amazon Web Services (AWS)
• Google Cloud Platform (GCP)

Azure Stack

• Microsoft Fabric
• Azure Synapse Analytics
• Azure Data Factory (ADF)
• Azure Databricks
• Azure Data Lake Storage (ADLS)
• Azure Event Hub
• Azure Functions
• Power BI
• Purview

AWS Stack

• AWS Glue
• Redshift
• EMR
• Athena
• Lambda
• Kinesis
• S3
• Lake Formation
• Step Functions

GCP Stack

• BigQuery
• Dataflow
• Dataproc
• Pub/Sub
• Composer
• Cloud Storage
• Vertex AI
• Looker

Big Data & Modern Data Tools

• Apache Spark
• PySpark
• Scala
• Hadoop Ecosystem
• Kafka
• Delta Lake
• Iceberg
• Hudi
• dbt
• Snowflake
• Trino
• Presto

Programming & Scripting

• Python
• SQL
• Scala
• Shell Scripting

Orchestration & DevOps

• Apache Airflow
• Terraform
• Docker
• Kubernetes
• Jenkins
• GitHub Actions
• Azure DevOps
• CI/CD Pipelines

Data Modeling & Warehousing

• Dimensional Modeling
• Data Vault
• Star/Snowflake Schema
• Lakehouse Architecture
• Medallion Architecture

Monitoring & Governance

• Microsoft Purview
• Collibra
• Apache Atlas
• Data Quality Frameworks
• Observability Tools

Do you have experience in Terraform?, Do you have a Bachelor’s degree?, * Bachelor’s or Master’s degree in Computer Science, Information Systems, or related field.

• Cloud certifications in Azure, AWS, or GCP are highly preferred.
• Experience working in Banking, Healthcare, Retail, Telecom, or Insurance domains.
• Strong understanding of AI/ML-ready data platforms and GenAI integrations.
• Experience with real-time analytics and streaming architectures.
• Exposure to modern semantic models and Fabric OneLake concepts.

Nice to Have

• Experience with AI-powered analytics platforms
• Knowledge of Microsoft Copilot / Fabric AI capabilities
• Exposure to MLOps platforms
• Experience with OpenAI integrations and vector databases
• Hands-on with Neo4j, MongoDB, Cassandra, or other NoSQL databases
• Experience implementing data mesh architecture

Key Traits

• Strong problem-solving and analytical skills
• Excellent communication and stakeholder management
• Ability to work in fast-paced Agile environments
• Leadership and mentoring capabilities
• Strong understanding of enterprise-scale distributed systems

$40 - $50 an hour - Contract