Data Engineer

Whitehall Resources are currently looking for a Data Engineer for an initial 6 month contract. ** INSIDE IR35 ** *This role will require on site work in Cambridge 2 days per week. Job Spec: Our client are seeking a motivated and detail-oriented Data Engineer with a passion for designing and delivering high-quality data solutions in Databricks. You will be responsible for building and optimising data pipelines that form the foundation of our reporting and analytics ecosystem. Using your technical expertise, you will design and maintain efficient ETL processes to integrate data from systems such as ServiceNow, JIRA, and Dynatrace, ensuring accuracy, performance, and scalability. Main Responsibilities: Design, build, and maintain scalable ETL pipelines in Databricks to integrate data from multiple business systems such as ServiceNow, JIRA, and ADO. Optimise data workflows for performance, scalability, and reliability. Implement data validation and quality checks to ensure trustworthy reporting in downstream tools such as Power BI. Create data modelling and schema design at backend for analytics and reporting. Collaborate with the Staff Data Engineer and Visualisation Developer to align technical delivery with reporting needs. Manage code in Git and support CI/CD processes for Databricks deployments. Contribute to data lineage documentation, standards, and governance best practices.

Proven experience (minimum 4 years) in building and maintaining ETL pipelines using Databricks. Strong knowledge of SQL and Python for data transformation and automation. Proven experience in data modelling, schema design, and data quality validation. Solid understanding of data performance tuning and pipeline optimisation in Databricks. Experience working with Git-based version control and collaborative development workflows. Strong analytical and problem-solving skills, with an eye for efficiency and accuracy. Excellent communication and collaboration skills, comfortable working with both technical and non-technical partners.

Desirable Skills: Familiarity with Agile delivery methods and iterative development practices Knowledge of data governance and data lineage documentation standards. Exposure to automation and CI/CD frameworks within Databricks.