Data Engineer

bigspark

Glasgow, UK

10 days ago

Role details

Contract type

Permanent contract

Employment type

Full-time (> 32 hours)

Experience level

Experienced

Experience required

3 years minimum

Compensation

£65,280.0

Working hours

Regular working hours

Job source

www.adzuna.co.uk

Tech stack

Clean Code Principles Java (Programming Language) Artificial Intelligence Airflow Amazon Web Services Amazon S3 Apache HTTP Server CA Workload Automation Ae Big Data BigQuery Cloud Computing Computer Programming

+36 more

Databases Continuous Integration Data Vault Modeling DevOps Amazon DynamoDB Github Identity and Access Management Python (Programming Language) PostgreSQL Linux System Administration MongoDB MySQL NoSQL OpenShift Simple Data Format SQL Databases Data Streaming Virtualization Technology Parquet Snowflake Apache Spark Gitlab Git Data Lakes Kubernetes Apache Flink Cassandra Avro Apache Kafka Data Management Data Lakehouse Terraform Azure Synapse Analytics Docker Jenkins Databricks

Job description

Were looking for a Data Engineer to developer enterprise-scale data platforms and pipelines that power analytics, AI, and business decision-making. You’ll work in a hybrid capacity which may require up to 2 days per week on a client premises.

Requirements

3+ years commercial data engineering experience
Strong programming skills in Python, Scala, or Java, with clean coding and testing practices.
Big Data & Analytics Platforms: Hands-on experience with Apache Spark (core, SQL, streaming), Databricks, Snowflake, Flink, Beam.
Data Lakehouse & Storage Formats: Expert knowledge of Delta Lake, Apache Iceberg, Hudi, and file formats like Parquet, ORC, Avro.
Streaming & Messaging: Experience with Kafka (including Schema Registry & Kafka Streams), Pulsar, AWS Kinesis, or Azure Event Hubs.
Data Modelling & Virtualisation: Knowledge of dimensional, Data Vault, and semantic modelling; tools like Denodo or Starburst/Trino.
Cloud Platforms: Strong AWS experience (Glue, EMR, Athena, S3, Lambda, Step Functions), plus awareness of Azure Synapse, GCP BigQuery.
Databases: Proficient with SQL and NoSQL stores (PostgreSQL, MySQL, DynamoDB, MongoDB, Cassandra).
Orchestration & Workflow: Experience with Autosys/CA7/Control-M, Airflow, Dagster, Prefect, or managed equivalents.
Observability & Lineage: Familiarity with OpenLineage, Marquez, Great Expectations, Monte Carlo, or Soda for data quality.
DevOps & CI/CD: Proficient in Git (GitHub/GitLab), Jenkins, Terraform, Docker, Kubernetes (EKS/AKS/GKE, OpenShift).
Security & Governance: Experience with encryption, tokenisation (e.g., Protegrity), IAM policies, and GDPR compliance.
Linux administration skills and strong infrastructure-as-code experience.