Data Pipeline Developer

Apex Group
2 days ago

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English

Job location

Tech stack

Java
API
Agile Methodologies
Amazon Web Services (AWS)
Azure
Cloud Computing
Cloud Database
Computer Programming
Data Cleansing
Data Integration
ETL
Data Warehousing
Software Design Patterns
Graph Database
Python
Enterprise Messaging Systems
Microsoft SQL Server
Neo4j
NoSQL
RabbitMQ
Power BI
Azure
Scala
SQL Databases
SQL Server Reporting Services
SQL Server Integration Services
SQL Server Analysis Services
Apex Code
Azure
Delivery Pipeline
Snowflake
Multi-Cloud
AWS Lambda
Data Lake
Amazon Web Services (AWS)
Kafka
Cosmos DB
Amazon Web Services (AWS)
Azure
Data Pipelines
Api Management
Serverless Computing
Redshift

Job description

As a Senior Data Engineer, you will be a driving force towards data engineering excellence. Working with other data engineers, analysts, and the architecture function, you'll be involved in building a modern data platform using a number of cutting-edge technologies in a multi-cloud environment. You'll have the opportunity to expand your knowledge and skills across multiple areas, including data pipelines, data integrations, data preparation, data models, and analytical and reporting marts.

As the business grows, we want our staff to grow with us, so there will be plenty of opportunities to learn and upskill. While much of the work follows business requirements and design concepts, you will also play a key role in the continuous improvement of design patterns and automation processes., * Help break down large problems into smaller iterative steps.

  • Contribute to defining the prioritisation of your squad's backlog.
  • Build the modern data platform (data pipelines, data integrations, data preparation, data models, analytical and reporting marts) based on business requirements using agreed design patterns.
  • Determine the most appropriate tools, methods, and design patterns to meet business requirements.
  • Proactively identify and suggest improvements.
  • Prepare data for exposure through APIs.
  • Document, support, manage, and maintain the modern data platform built within your squad.
  • Provide guidance and training to downstream data consumers on best practices.
  • Support and build new data APIs.
  • Promote the effective use and value of data across the organisation.
  • Comply with all company policies.
  • Perform any other duties assigned by management.

Requirements

  • Experience with Data Warehousing and Data Modelling.
  • Experience with Data Lakes (AWS Lake Formation, Azure Data Lake).
  • Experience with Cloud Data Warehouses (AWS Redshift, Azure Synapse, Snowflake).
  • Experience with ETL/ELT/Pipeline tools (AWS Glue, Azure Data Factory, Fivetran, Stitch).
  • Knowledge of messaging systems (AWS SNS & SQS, Azure ASQ, Kafka, RabbitMQ).
  • Programming skills in SQL, Python, Scala, or Java.
  • Experience with cloud workflow services (AWS Step Functions, Azure Logic Apps, Camunda).
  • Experience with event and schedule management (AWS Lambda, Azure Functions).
  • Knowledge of the Microsoft BI Stack (SQL Server, SSIS, SSAS, SSRS).
  • Experience with reporting and visualisation tools such as Power BI, QuickSight, or Mode.
  • Knowledge of NoSQL and Graph Databases (AWS Neptune, Azure Cosmos DB, Neo4j) - desirable.
  • API Management experience - desirable.
  • Excellent communication and interpersonal skills.
  • Strong critical thinking, analytical, and problem-solving skills.
  • Ability to prioritise and manage workload independently.
  • Ability to work both independently and collaboratively within a team.
  • Working knowledge of Agile Software Development Lifecycles.

Apply for this position