Databricks Engineer

As our Senior Databricks Engineer, you will operate within an Agile delivery environment, working closely with the Data Product Manager and Data Architect to ensure your team maintains a pipeline of delivery against the Backlog; providing vital insight from our wide-ranging dataset to support executive and operational decision making that will underpin sustained growth of BFS business units domestically and internationally.

You will have an active leadership role in determining and developing the shape of your teams solution delivery against business requirements, as well as helping to inform and input into the wider technical architecture and strategy.

This is very much a hands-on role, where the majority of your time will be spent actively developing solutions. You will also have management responsibilities for a small team of Data Engineers, who you will coach, support and organise to ensure we sustain a predictable pipeline of delivery, whilst ensuring all appropriate governance and best practice is adhered to., * Understand the business / product strategy and supporting goals with the purpose of ensuring data interpretation aligns

• Provide technical leadership on how to break down initiatives into appropriately sized features, epics and stories that balance value and risk. Take a leadership role on setting standards, driving quality and consistency in solution delivery
• Work closely with the Data Architect to collaborate on Design of our data architecture and interpret into a build plan
• Lead the build and maintenance of scalable data pipelines and ETL processes to support data integration and analytics from a diverse range of data sources, Cloud storage, databases and APIs
• Deliver large-scale data processing workflows (ingestion, cleansing, transformation, validation, storage) using best practice tools and techniques
• Collaborate with the BI Product Owner, analysts, and other business stakeholders to understand data requirements and deliver solutions that meet business needs
• Optimize and tune data processing systems for performance, reliability, and scalability
• Implement data quality and validation processes to ensure the accuracy and integrity of data throughout the pipelines
• Operate an agile CI/CD environment within Azure DevOps, collaborating on Sprint cycles, code deployment, version control, and development practices
• Develop and maintain data models, schemas, and documentation
• Monitor and troubleshoot data pipeline issues, ensuring timely resolution
• Stay up-to-date with the latest industry trends and technologies in data engineering and recommend improvements to existing systems or processes as appropriate
• Ensure adherence to BFS Governance processes
• Provide line management and technical leadership to a small team of Data Engineers
• Work closely with the Data Architect to collaborate on Design of our data architecture and interpret into a build plan
• Lead the build and maintenance of scalable data pipelines and ETL processes to support data integration and analytics from a diverse range of data sources, Cloud storage, databases and APIs
• Deliver large-scale data processing workflows (ingestion, cleansing, transformation, validation, storage) using best practice tools and techniques

• Significant years of Databricks experience, including Unity Catalog
• Terraform, defining, deploying, and managing cloud infrastructure as code
• Proficiency in programming languages such as Python, Spark, SQL
• Strong experience with SQL databases
• Expertise in data pipeline and workflow management tools (e.g., Apache Airflow, ADF)
• Experience with cloud platforms (Azure preferred) and related data services
• Excellent problem-solving skills and attention to detail
• Inclusive and curious, continuously seeks to build knowledge and understanding
• Strong communication and collaboration skills.
• Experience of Waterfall and Agile delivery methodologies

Bibby Financial Services have an exciting opportunity available for a reliable Senior Databricks Engineer to join our team. You will join us on a full time, permanent basisand in return, you will receive a competitive salary of £65,000 - £75,000 per annum., * Private healthcare for you and your family

• Company car allowance
• Company pension scheme
• Wide range of flexible benefits, such as gym membership, technology, or health assessments
• Access to an online wellbeing centre
• Range of discounts from many businesses
• 25 days holiday, which increases with servic,e and options to buy or sell more
• Electric Vehicle/Plug-in Hybrid Vehicle (EV/PHEV) scheme

We've supported small and medium-sized enterprises (SMEs) since 1982, and today we support more than 9000 businesses worldwide. We are proud to help businesses, both big and small to grow and thrive in domestic and international markets. We're in the business of relationships. Whether working together or supporting our clients, that's what makes us stand out from the crowd. We know our value lies in our brilliant people, It is fast-paced and varied, and we can handle it. We trust each other, take ownership, work flexibly, have fun and deliver for our clients every time. We empower our colleagues to be themselves and grow in the way they want - we believe in them and celebrate their success. There's no place quite like BFS and we're proud of that. It's all down to our colleagues - they make us the business with which every SME wants to partner. If you would like to join us, please click 'Apply' today to be considered as our Senior Databricks Engineer - we would love to hear from you! This role will be hybrid and can be based in any of our UK offices. We're absolutely committed to being a truly inclusive place to work, where everyone has an equal opportunity to reach their true potential. Let us know if you need adjustments to support you through any stage of the recruitment process., Bibby Financial Services (BFS) is part of Bibby Line Group, a family run business formed of a diverse group of international businesses, operating around the world, employing around 1,400 colleagues, and managing over £1billion of funds. BFS was formed over 40 years ago, and since this time we have been supporting our colleagues and customers to help them thrive and grow. We not only serve our customers, but we also contribute to our local communities and play our part in creating a sustainable future for generations to come. We're one team connecting and collaborating with the world around us to work towards better outcomes for all. This isn't just work - it's something we really care about. It's this belief in a wider purpose of business that drives us forward and enables us to positively connect with our customers, and the communities in which will live and work . What do we do? We're a highly customer-centred business partner, providing access to flexible finance and working capital solutions that enable SMEs to thrive and grow. As the backbone of our communities, we are here to be a constant, professional and reliable partner for SMEs in need of flexible financial solutions. We offer trustworthy, professional financial solutions through partnerships with SMEs. We provide our customers high quality, competitive, access to flexible and tailored finance in any season and for every business stage. Everything we do is underpinned by our business values: Be innovative, Work together, Trust each other, Do the right thing and Be better. Why work for BFS? We're in the business of relationships because we know that it's together we grow. Whether working together or supporting our clients, that's what makes us stand out from the crowd. We know our value lies in our brilliant people, we trust each other, take ownership, work flexibly, have fun and deliver for our clients every time. We empower our colleagues to be themselves and grow in the way they want. There's no place quite like BFS and we're proud of that. It's all down to our colleagues - they make us the business with which every SME wants to partner. How we make a difference: Our Project Compass Project Compass is our important long-term commitment to be a sustainable and resilient business, supporting us to better deliver on our purpose by focusing our efforts on four areas that are important to us: Environment, Community, People and Customer. Our Project Compass: Where we are now At Bibby, when we say we are committed to something, we really mean it. We don't just talk the talk here, we walk the walk too. In just 12 short months since we launched our 'Project Compass' we have achieved so much already and we still want to go further, together. We need talented people like you to join our team to help us to make a meaningful difference to our people, our customers and to help us to achieve our shared goal of being 'A family business, known for creating a better future together'.