Data Engineer

I am currently recruiting for a Data Engineer to join a forward-thinking organisation based in Peterborough, where you will play a pivotal role in the ongoing transformation of their data infrastructure. Currently transitioning to Databricks, you will ensure seamless migration and optimisation of data pipelines.

Data Engineer - What will you be doing?

• Collaborating with team members to migrate existing data pipelines from legacy systems to the new Databricks platform.
• Supporting the ongoing build-out of the Databricks environment by developing robust, scalable data solutions that meet current and future business needs.
• Maintaining dual operations across both legacy and modern stacks, ensuring reliable data flow and system integrity during the transition period.
• Contributing to the strategic roadmap for data engineering by providing insights, feedback, and technical recommendations that align with organisational goals.
• Troubleshooting issues related to data integration, pipeline performance, and platform dependencies.
• Working closely with other engineers and business users to understand requirements, translate them into technical specifications, and deliver effective solutions.

• Experience working with Databricks or a similar cloud-based data platform is highly desirable for this role as it involves significant migration projects.
• Solid understanding of building, maintaining, and optimising ETL/ELT pipelines using modern tools and technologies is essential for success.
• Familiarity with managing dual environments involving both legacy systems (such as on-premise databases) and contemporary cloud solutions is beneficial.
• Strong problem-solving skills combined with an analytical mindset enable you to address technical challenges efficiently while considering broader business impacts.
• Demonstrated ability to design lakehouse architectures.
• Experience implementing CI/CD pipeline management for reliable solution delivery.