Data Engineer

Square One Resources Limited

Sheffield, UK

15 days ago

Apply on www.careerboard.com

Prepare application

Role details

Contract type

Temporary contract

Employment type

Full-time (> 32 hours)

Experience level

Expert

Compensation

£78,000.0 - £110,500.0

Working hours

Regular working hours

Job source

www.careerboard.com

Tech stack

Data Analysis Apache HTTP Server Computing Platforms Microsoft Azure Bash Shell Cloud Computing Cloud Database Databases Continuous Integration Information Engineering Data Integration Extract Transform Load (ETL)

+33 more

Data Warehousing Elasticsearch Github Identity and Access Management Python (Programming Language) Linux System Administration Log Analysis Nginx Windows PowerShell Ansible Prometheus Kusto Query Language Standard Sql Azure Data Lake Data Streaming Scripting Cloud Platform System Azure Data Factory Delivery Pipeline Grafana Azure Powershell Kubernetes Deployment Automation Real Time Data Apache Kafka Cosmos DB Spark Streaming Terraform Stream Analytics Software Version Control Data Pipelines Serverless Computing Databricks

Job description

Design, build and maintain large-scale data pipelines and ETL workflows across enterprise datasets
Ingest, provision, and manage raw, enriched, and curated data assets for analytics platforms
Improve reliability, scalability and Real Time data ingestion capabilities
Support and enhance cloud-based data ingestion infrastructure and platform services
Acquire, transport, clean, transform and integrate data from internal and external sources
Perform exploratory data analysis and onboard new data sources
Develop automated data engineering workflows and optimise data processing pipelines
Manage data modelling, data warehousing, data integration and data cataloguing activities
Support Real Time analytics and streaming data platforms
Contribute to cloud infrastructure engineering, platform operations and production support
Implement cloud cost optimisation and platform efficiency improvements
Collaborate with engineering, analytics, infrastructure and cybersecurity teams
Support DevOps automation, monitoring and operational excellence initiatives
Maintain documentation, governance standards and operational procedures

Requirements

Strong experience with Azure and Azure DevOps
Strong Python and SQL skills, including KQL
Experience with Bash, PowerShell and Azure CLI
Strong Linux administration and Scripting
Experience with GitHub, source control and CI/CD pipelines
Understanding of networking, operating systems, storage and infrastructure

Data Engineering

Experience designing and maintaining large-scale data pipelines and ETL workflows
Strong experience in data acquisition, transport, cleaning and integration
Experience with data modelling, data warehousing and data cataloguing
Experience with pipeline automation, productionisation and optimisation
Experience supporting Real Time analytics and streaming data platforms

Azure Data Platform

Azure Data Factory
Azure Databricks and Unity Catalog
Azure Data Lake and Synapse Analytics
Azure Functions and Azure Logic Apps
Azure Monitor and Azure Log Analytics
Power BI

Cloud & Infrastructure

Strong knowledge of Azure IAM, networking, compute, storage, databases, and containers
Experience with Kubernetes and cloud-native platforms
Experience with Infrastructure as Code, including Terraform, ARM, Chef or Ansible
Experience with cloud cost optimisation
Understanding of SRE principles

Streaming & Platform Technologies

Experience with Azure Event Hubs, Kafka and Spark Streaming
Knowledge of Nginx, Apache, Cosmos DB, Prometheus, Grafana and Elasticsearch

About the company

Square One is acting as both an employment agency and an employment business, and is an equal opportunities recruitment business. Square One embraces diversity and will treat everyone equally. Please see our website for our full diversity statement.