> Markdown version of [/jobs/ext/3192866-software-engineer-senior-consultant-ii-hybrid](https://www.wearedevelopers.com/jobs/ext/3192866-software-engineer-senior-consultant-ii-hybrid). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Software Engineer Senior Consultant II - (Hybrid) - **Company:** Allstate - **Location:** Belfast, UK - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Artificial Intelligence, Data Analysis, Automation of Tests, Big Data, Cloud Computing, Software Documentation, Software Quality, Code Review, Continuous Integration, Data Architecture, Information Engineering, Data Governance, Data Infrastructure, Data Integration, Extract Transform Load (ETL), Data Systems, Data Warehousing, Decision Support Systems, Distributed Computing Environment, Distributed Systems, Apache Hive, Meta-Data Management, Scrum Methodology, Standard Sql, DataOps, Data Streaming, Privacy Controls, Data Logging, Data Processing, Cloud Platform System, Apache Spark, Git, Microsoft Fabric, Containerization, Pyspark, Deployment Automation, Data Analytics, Apache Kafka, Data Management, Data Pipelines - **Published:** September 4, 2026 - **Apply:** https://uk.indeed.com/viewjob?jk=5d578d5292583ec5 ## About the Role * All applicants must demonstrate they have a legal right to work in the UK for employment at Allstate. Allstate is not providing sponsorship for this vacancy. * A minimum of 2 years of experience in Data Engineering and Big Data technologies. * Deep expertise in Apache Spark (PySpark/Spark SQL). * Experience with Microsoft Fabric, OneLake, Lakehouse Architecture, Data Warehouse, and building Semantic Models. * Strong SQL and data modeling skills. * Experience building and optimizing ETL/ELT pipelines. Desirable Skills: * Strong proficiency in Java and experience building distributed applications and services. * Experience with Kafka or similar event-streaming platforms. * Understanding of Data Governance, Data Quality, Security, and Privacy controls. * Ability to mentor junior engineers and provide technical leadership without direct people management responsibilities. * Knowledge of streaming and batch data processing patterns. * Experience with observability solutions including logging, monitoring, tracing, and performance analysis. * Experience working in cloud-based analytics and data platform environments. * Experience with CI/CD, Git-based development, testing, and deployment automation. * Experience working in Agile delivery environments. ## Description Allstate's Data & Analytics Technology organization is seeking a highly skilled and hands-on Senior Data Engineer to design, build, optimize, and support scalable, reliable, and high-performing data platforms and pipelines. This role is focused on developing enterprise data solutions that enable analytics, reporting, AI/ML, and data-driven decision making across the organization. The ideal candidate possesses strong expertise in modern data engineering practices, Apache Spark, cloud-based data platforms, and lakehouse architectures. As a senior individual contributor, the candidate will provide technical leadership, mentor junior engineers, drive engineering best practices, and contribute directly to the design and implementation of enterprise-scale data solutions. The role requires close collaboration with Analytics Engineers, Data Scientists, Product Managers, Platform Engineers, Security teams, and business stakeholders to deliver trusted, scalable, and analytics-ready data products., * Design, develop, and maintain scalable batch and streaming data pipelines using Apache Spark, cloud-native technologies, and modern data platforms. * Build and optimize ETL/ELT frameworks to ingest, transform, validate, and curate data from diverse enterprise source systems. * Develop and implement data solutions leveraging Microsoft Fabric, OneLake, Lakehouse, Data Warehouse, and Semantic Model capabilities. * Design data models, storage strategies, and data processing architectures that support enterprise analytics and reporting requirements. * Partner with Analytics Engineers, BI teams, Data Scientists, and business stakeholders to deliver trusted, reusable, and analytics-ready datasets. * Drive adoption of engineering best practices including code quality, peer reviews, automated testing, CI/CD, and documentation standards. * Ensure data quality, governance, lineage, metadata management, security, privacy, and compliance requirements are incorporated into data solutions. * Monitor and optimize performance, scalability, reliability, and cost efficiency of distributed data processing workloads. * Develop reusable frameworks, libraries, templates, and engineering patterns that improve development productivity and consistency. * Implement and support CI/CD pipelines, Infrastructure-as-Code principles, and deployment automation for data engineering solutions. * Troubleshoot complex data integration, performance, and operational issues across platforms and pipelines. * Collaborate with platform, infrastructure, architecture, and security teams to build resilient, scalable, and secure enterprise data solutions. * Create and maintain technical documentation, architecture diagrams, operational runbooks, and implementation guides. * Participate in technical design sessions, architecture reviews, and engineering discussions while providing recommendations and solutions. * Mentor and guide junior and mid-level data engineers through technical coaching, code reviews, design reviews, and knowledge sharing. * Evaluate and adopt emerging technologies, engineering practices, and AI-assisted development capabilities to improve delivery velocity and quality. * Support production operations through monitoring, root cause analysis, incident resolution, and continuous improvement initiatives. * Contribute to Agile delivery processes including sprint planning, estimation, backlog refinement, and execution of engineering commitments. * Promote DataOps practices, automation, observability, and engineering excellence across development teams. * Serve as a technical subject matter expert for modern data platforms, Spark processing, Microsoft Fabric, and lakehouse architectures. ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [The 12 Best Jobs for Software Engineers](https://www.wearedevelopers.com/magazine/401-the-12-best-jobs-for-software-engineers) - [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries) - [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london)