Senior Data Engineer

SUMMARY OF THE JOBWe are seeking a seasoned Senior Data Engineer to design, build, and optimize our next-generation data platform. You will be responsible for architecting scalable data pipelines, managing large-scale distributed systems, and ensuring our data infrastructure in AWS and Databricks is robust and efficient. The ideal candidate is a Spark expert with a deep understanding of the AWS ecosystem and a passion for automation.MAIN ACTIVITIES / RESPONSIBILITIESPipeline Architecture:Design and implement complex batch and streaming ETL/ELT pipelines usingPython, SQL, and Sparkto process massive datasets.Cloud Infrastructure:LeverageAWS Data Analyticsservices to build scalable, secure, and cost-effective data solutions.Orchestration & DevOps:Manage and automate data workflows usingAirflow, while utilizingDocker and ECSfor containerized application deployment.System Optimization:Monitor and tune the performance of distributed systems (Spark Cluster) to ensure high availability and low latency.Infrastructure as Code:UtilizeAWS CloudFormationorTerraformto manage data infrastructure, ensuring repeatable and version-controlled environments.Cost Optimization:Monitor and optimize AWS spend by selecting appropriate instance types (Spot vs. On-Demand) and refining data storage strategies.Security & Compliance:Implement IAM roles, bucket policies, and encryption (KMS) to ensure data is secure at rest and in transit.Collaboration:Work within anAgileframework to deliver iterative value, collaborating closely with Data Scientists and Stakeholders to translate business needs into technical reality.JOB DIMENSIONSList of direct reports:Up to 2 Direct Reports, and around 15 externalsKey interfaces, stakeholders and relationships:Internal:GDS: product manager, application manager, data & analytics & AI teamCountry business stakeholdersExternal : 3rd party vendorsPROFILE REQUIREDExperience:Minimum4+ yearsof hands-on experience in activeBig Dataenvironments and2+ yearsspecializing inData Analytics within AWS.Compute & Processing:Amazon EMR:Architecting and managing Spark clusters for large-scale distributed processing.AWS Glue:Developing serverless ETL jobs, managing the Data Catalog, and implementing Glue Crawlers.Storage & Warehousing:Amazon S3:Implementing "Data Lake" best practices, including partitioning, compression (Parquet/Avro), and lifecycle policies.Amazon Redshift:Designing star/snowflake schemas and optimizing query performance for high-volume data warehousing.Amazon Athena:Performing ad-hoc SQL analysis directly on S3 data.Experience with open table formats (iceberg/delta)Orchestration & Integration:Amazon MWAA (Managed Workflows for Apache Airflow):Deploying and scaling Airflow environments.AWS Lambda:Building event-driven data triggers and micro-services.Streaming (Advantage):Amazon KinesisorMSK (Managed Streaming for Kafka)for real-time data ingestion.Core Engineering:Expert-level proficiency inSpark,Python, andSQL.Infrastructure & Tooling:Proven experience withAirflowfor orchestration andDocker/ECSfor containerization.Good knowledge inDatabricksand data mesh architectures. Good understanding in how to implement and maintainLakehouse data models(bronze / silver / gold layers) usingDelta Lakefor reliability, ACID transactions, time travel and schema evolution.Solid software engineering practices:Git, CI/CD for data pipelines, automated testing,code qualityanddocumentation.Communication:Excellent written and oral English communication skills, with the ability to explain complex technical concepts to non-technical audiences.Degree in Computer Science, Engineering, Mathematics or related field, or equivalent practical experience.PREFERRED “PLUS” QUALIFICATIONReal-time Processing:Experience with streaming and distributed messaging applications likeFlinkandKafka.Core Tech:Javaprogramming.Industrialise ML use casesData Visualization:Experience withQlikVieworQlikSenseto support BI initiatives.Agile:Experience working in a fast-paced Scrum or Kanban environment.Certifications:AWS Certified Data Engineer - Associate/Professional or AWS Certified Solutions Architect, Databricks Data engineer (Associated/Professional) certificationDevOps:Experience withOpenshift, Github Actionsor Jenkins for CI/CD of data workflows.#J-*****-Ljbffr

Excellent written and oral English communication skills, with the ability to explain complex technical concepts to non-technical audiences. Degree in Computer Science, Engineering, Mathematics or related field, or equivalent practical experience. PREFERRED “PLUS” QUALIFICATION Real-time Processing: Experience with streaming and distributed messaging applications like Flink and Kafka . Core Tech: Java programming. Industrialise ML use cases Data Visualization: Experience with QlikView or QlikSense to support BI initiatives. Agile: Experience working in a fast-paced Scrum or Kanban environment. Certifications: AWS Certified Data Engineer - Associate/Professional or AWS Certified Solutions Architect, Databricks Data engineer (Associated/Professional) certification DevOps: Experience with Openshift, Github Actions or Jenkins for CI/CD of data workflows. #J-*****-Ljbffr