Senior Data Engineer

Response Informatics

Salford, UK

18 days ago

Apply on www.collegerecruiter.com

Prepare application

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Working hours

Regular working hours

Job source

www.collegerecruiter.com

Tech stack

Microsoft Azure Big Data Computer Programming Continuous Integration Information Engineering Data Systems Distributed Computing Environment Job Scheduling Python (Programming Language) Metadata Query Optimization Data Processing

+7 more

Azure Data Factory Apache Spark Git Pandas Deployment Automation Data Pipelines Databricks

Job description

Design, develop, and maintain metadata-driven data pipelines using Azure Data Factory and Databricks.
Build and implement end-to-end metadata frameworks, ensuring scalability and reusability.
Optimize data workflows leveraging SparkSQL and Pandas for large-scale data processing.
Collaborate with cross-functional teams to integrate data solutions into enterprise architecture.
Implement CI/CD pipelines for automated deployment and testing of data solutions.
Ensure data quality, governance, and compliance with organizational standards.
Provide technical leadership and take complete ownership of assigned projects.

Requirements

Azure Data Factory (ADF): Expertise in building and orchestrating data pipelines.
Databricks: Hands-on experience with notebooks, clusters, and job scheduling.
Pandas: Advanced data manipulation and transformation skills.
SparkSQL: Strong knowledge of distributed data processing and query optimization.
CI/CD: Experience with Azure DevOps, Git, or similar for automated deployments.
Metadata-driven architecture: Proven experience designing and implementing metadata frameworks.
Programming: Proficiency in Python and/or Scala for data engineering tasks.