Lead Data Engineer
Role details
Job location
Tech stack
Job description
This position requires a Lead Software Developer who will drive and contribute to the design and implementation of the Bronze and Silver layers of Swift's Data Platform using the Medallion Architecture. The ideal candidate should have combination of technical leadership qualities and hands-on experience with Apache Spark, Java, Kafka an established experience of data engineering principles and ETL concepts, and an interest or background in the payment's domain. The role involves collaborating closely with technical and business teams to deliver a reliable, enriched and integrated common set of tables to enable the creation of business-ready datasets in the Gold layer.
- Act as technical lead in the team for the design, development, testing, and post-deployment support of the Bronze & Silver Layer components in line with the agreed functional, non-functional and security requirements
- Drive the implementation and adherence of best practices by the team for quality, performance, and scalability to ensure accuracy, consistency, and reliability of the Bronze & Silver layer datasets.
- Implement or lead the implementation of complex ETL work with data analysts to deliver quality-checked, enriched and integrated datasets, and implement transformations and business rules per requirements.
- Define established data governance, security, and coding standards to ensure compliance and maintainability.
- Contribute to documentation and continuous improvement efforts, including process refinement and knowledge sharing.
- Effectively communicate development progress to the team leadership and raise the alarm immediately when deviating from the plan
- Provide coaching, technical assistance and support to more junior team members and internal customers.
- Run or assist in transversal initiatives within the Tribe or beyond - act as a trusted and recognized point of contact for the team and give presentations to higher management within or beyond the Tribe.
- Partner with the Team Lead to guide the team, proactively identifying dependencies, risks, and priorities to ensure smooth delivery.
Requirements
- Minimum of 10+ years of relevant software development experience in an international setting.
- Proven expertise in data engineering and software development practices., * Hands-on experience in programming languages, particularly Java, and their design patterns.
- Hands-on experience with data management tools such as: Spark, Kafka, Airflow.
- Hands-on experience with large-scale object storage services (Amazon S3 or equivalent is a plus).
- Deep understanding of Data Engineering, data hub, data lake, and data warehousing concepts.
- Hands-on experience with continuous integration and delivery build environments, including GIT, Jenkins, Maven, and coverage and static/dynamic code analysis tools.
- Fluent with Agile tools like JIRA, Confluence and Bitbucket.
Good to Have:
- Understanding of containerization technologies such as OpenShift and Kubernetes.
- Knowledge of Red Hat Linux.
- Experience with Elasticsearch/Kibana., * Adaptability and responsiveness to change.
- Accountability and leadership for delivering results.
- Strong written and oral communication skills.
- Time management, planning, and organizing abilities.
- Ability to thrive in both multi-cultural team and individual environments.
- Quality-oriented approach.
- Engaged and team-driver attitude.
- Open communication style.
- Ready to perform on-call support.
- Fluent in English.