> Markdown version of [/jobs/ext/2729892-core-data-engineer](https://www.wearedevelopers.com/jobs/ext/2729892-core-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Core Data Engineer - **Company:** Optima Partners - **Location:** Edinburgh, UK - **Experience:** Experienced - **Salary:** £51,382.0 - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Agile Methodology, Airflow, Amazon Web Services, Microsoft Azure, Big Data, BigQuery, Cloud Computing, Computer Programming, Continuous Integration, Data Architecture, Information Engineering, Data Integration, Extract Transform Load (ETL), Data Systems, Data Warehousing, Relational Databases, Apache Hadoop, Python (Programming Language), PostgreSQL, Microsoft SQL Server, Performance Tuning, Scrum Methodology, Scala (Programming Language), Talend, Google Cloud, Data Storage Technologies, Cloud Platform System, Snowflake, Apache Spark, Git, Data Lakes, Core Data, Apache Nifi, Data Management, Software Version Control, Data Pipelines, Docker, Amazon Redshift, Databricks - **Published:** September 5, 2026 - **Apply:** https://www.adzuna.co.uk/jobs/details/5869807199 ## About the Role * Programming: Proficiency in Python, Java, Scala, or similar languages. * Big Data Technologies: Hands-on experience with big data tools e.g. (Databricks, Apache Spark, Hadoop). * Cloud Platforms: Familiarity with AWS, Azure, GCP, or other cloud ecosystems for data engineering tasks. * Expertise in relational databases (e.g. postgres, sql server) * Data Integration Tools: Knowledge of platforms like Airflow, Apache NiFi, or Talend. * Data Storage and Modelling: Experience with data warehousing tools (e.g. Snowflake, Redshift, BigQuery) and schema design. * Version Control and CI/CD: Familiarity with Git, Docker, and CI/CD pipelines for deployment. * 2+ years of experience in data engineering or a similar role. * Proven track record of delivering data solutions in a consulting or client-facing environment is a plus. * Experience with Agile or Scrum methodologies is beneficial. * * Strong problem-solving and analytical thinking abilities. * Excellent communication skills to explain complex technical concepts to non-technical stakeholders. * A team-oriented mindset with a focus on collaboration and adaptability ## Description As a Core Data Engineer, you will play a pivotal role in designing, building, and optimising data pipelines, architectures, and workflows for our diverse client base. You'll work within dynamic, multi-disciplinary project teams to implement scalable data solutions while adhering to best practices and emerging trends in data engineering., * Data Pipeline Development: Design and implement robust, scalable, and efficient data pipelines to collect, transform, and integrate data from various sources. * Data Architecture: Develop and optimise data architectures, including data warehouses, data lakes, and other storage solutions. * ETL/ELT Processes: Create and maintain reliable ETL/ELT workflows that ensure data quality and accessibility. * Collaboration: Work closely with data scientists, analysts, and client stakeholders to understand requirements and deliver impactful solutions. * Performance Optimisation: Monitor and enhance the performance of data systems to ensure minimal downtime and fast query responses. * Automation and Tools: Identify opportunities for automation and recommend tools to improve data engineering workflows. * Documentation: Maintain detailed technical documentation for all solutions and processes. ## Related Videos - [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Docker build without Docker](https://www.wearedevelopers.com/videos/100114-docker-build-without-docker) - [Git for Code Reviews](https://www.wearedevelopers.com/videos/429-git-for-code-reviews) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)