Data Engineer

Source Group
2 days ago

Role details

Contract type
Temporary contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Senior

Job location

Tech stack

API
Agile Methodologies
Airflow
Apache HTTP Server
Business Logic
Big Data
C Sharp (Programming Language)
Continuous Delivery
Data Architecture
Data Governance
Data Infrastructure
ETL
Data Warehousing
Database Queries
Hadoop
Hive
Python
PostgreSQL
Microsoft SQL Server
NoSQL
Open Source Technology
DataOps
Cloudera
Simple Object Access Protocol (SOAP)
SQL Databases
Data Streaming
Workflow Management Systems
Snowflake
Spark
Data Lake
Cassandra
Kafka
Apache Nifi
Spark Streaming
Azure
Data Pipelines
Databricks
Microservices

Job description

  • Design, develop, and maintain end-to-end data pipelines (ETL/ELT) using Spark, SQL, and related technologies.
  • Implement business logic, data governance, security, and retention requirements within pipelines.
  • Translate business and technical requirements into scalable data solutions and detailed designs.
  • Collaborate with business stakeholders, product owners, and technical teams to address data needs and issues.
  • Contribute to data architecture, ensuring scalable and high-performance system design.
  • Support continuous delivery through effective governance, transparency, and communication.
  • Stay up to date with emerging data technologies and support proof-of-concept initiatives.

Requirements

  • 5+ years of experience working with modern data technologies.
  • Strong experience building data ingestion pipelines for data warehouse, data lake, or lakehouse architectures.
  • Hands-on expertise with big data and open-source tools (eg, Spark, Hadoop, Hive, Kafka, dbt, ELK), ideally in a Cloudera environment.
  • Proficiency in Python and strong SQL skills.
  • Solid understanding of data modelling, scalable system design, and data engineering best practices.
  • Experience in financial services (eg, trade life cycle, risk management).
  • Familiarity with lakehouse technologies such as Apache Iceberg.
  • Experience with relational and NoSQL databases (eg, Postgres, SQL Server, Cassandra).
  • Exposure to workflow orchestration (Airflow, NiFi), streaming (Kafka, Spark Streaming), and CI/CD pipelines.
  • Knowledge of Agile methodologies and testing practices.
  • Experience with cloud platforms (Databricks, Snowflake, Azure Synapse) - nice-to-have
  • Understanding of microservices and APIs (REST/SOAP) - nice-to-have
  • Exposure to .NET/C# - nice-to-have
  • You bring strong communication skills

About the company

Source Group International Ltd is acting as an Employment Business in relation to this vacancy

Apply for this position