Lead Cloud Data Engineer

• Design and develop scalable ETL pipelines using AWS Glue and PySpark.
• Integrate data from various sources into Amazon Redshift, ensuring data quality and consistency.
• Optimize the performance of data processing jobs and Redshift queries.
• Deploy infrastructure as code (IaC) with Terraform
• Monitor and troubleshoot data pipelines to ensure reliability and efficiency.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements.
• Ensure compliance with security policies and regulations.
• Document data workflows and best practices.
• Mentor and guide junior engineers on the team.
• Continuously explore new AWS services and technologies to improve existing solutions.

• Expert level experience with AWS Cloud Services is preferred, knowledge of other Cloud Providers is a plus
• Experience and understanding of AWS Cloud Services components like Redshift, S3, Kafka, Glue, AirFlow, SageMaker, EMR, Athena, Lake Formation, DynamoDB, Aurora, VPC, IAM and many other Cloud Service modules.
• Working experience in Python is preferred, Java and other languages are a plus
• Nice to have Terraform or other Infrastructure as Code experience
• Knowledge and experience with Analytics, Data Warehouse and Big Data concepts across both batch and streaming patterns
• Solid understanding of networking, infrastructure, cost management, SRE concepts
• Creating patterns for adoption from scratch and being able to document and articulate coding and architecture standards for others to follow.
• Performance tuning and optimization of code and infrastructure
• Creating a culture of continuous improvement, performs code reviews and makes recommendations for improvement or streamlining patterns
• Relevant experience in various database platforms, ETL solutions/products, Infrastructure, ETL architecture and On-Prem or Cloud Services.
• Financial Services Industry knowledge is a plus.

Ability to:

• Identify and understand issues, problems and opportunities for automation.
• Good communication and organization skills that clearly convey information and ideas through a variety of media to individuals or groups in a manner that engages the audience and helps them understand and retain the message.
• Use effective approaches for choosing a course of action or developing appropriate solutions; recommend or take action that is consistent with available facts, constraints and probable consequences.
• Demonstrate a satisfactory level of technical and professional skill or knowledge in position-related areas; remains current with developments and trends in areas of expertise.
• Develop and use collaborative relationships to facilitate the accomplishment of work goals.
• Make internal and external clients and their needs a primary focus of actions; develop and sustain productive client relationships.

Certifications

Preferred , not required

• AWS Certified Solutions Architect
• AWS Certified Big Data
• AWS Certified Data Analytics
• AWS Certified Developer
• AWS Certified Machine Learning