> Markdown version of [/jobs/ext/1620497-databricks-senior-developer](https://www.wearedevelopers.com/jobs/ext/1620497-databricks-senior-developer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Databricks Senior Developer - **Company:** CRISIL Limited - **Location:** London, UK (Remote available) - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Airflow, Automation of Tests, Microsoft Azure, Cloud Computing, Continuous Integration, Data Architecture, Information Engineering, Data Governance, Data Infrastructure, Extract Transform Load (ETL), Data Systems, DevOps, Apache Hive, Python (Programming Language), Performance Tuning, SQL Databases, Systems Architecture, Workflow Management Systems, Data Processing, Data Storage Technologies, Cloud Platform System, Apache Spark, Pyspark, Data Management, Cloud Migration, Cloud Integration, Software Coding, Code Restructuring, Software Version Control, Data Pipelines, Serverless Computing, Databricks - **Published:** July 2, 2026 - **Apply:** https://www.gradsouthwest.com/jobs/apply/ats-redirect/?id=2854468113-2 ## About the Role + 5 to 7 years of data engineering experience, with at least 3+ years of hands-on Databricks experience in production environments. + Migration Expertise + Proven experience in migrating on-premises data systems to cloud platforms. + Strong proficiency in Python, PySpark, and SQL. + Experience in converting legacy SQL dialects to Spark SQL is a plus. + Familiarity with cloud computing concepts, preferably within Azure ecosystems. + Experience integrating cloud functions or runtime services with data platforms is desirable. + Solid understanding of CI/CD, data quality frameworks, and orchestration tools such as Databricks Workflows, Airflow, or dbt. ## Description Crisil Integral IQ delivers solutions and analytics to top financial institutions, driving strategic transformation, risk optimization, and operational excellence. Our offerings across research, risk, lending, analytics and operations have empowered clients to navigate complex markets, mitigate risks and unlock new opportunities. Our domain expertise, innovative solutions, and future-ready technologies such as AI and data science give clients the confidence to accelerate growth and achieve sustainable competitive advantage. Our globally diverse workforce operates in the Americas, Asia-Pacific, Europe, Australia and the Middle East. We are looking for a Databricks Senior Developer who will join our Team supporting one of the biggest investment banks., Design and Develop Data Pipelines + Build scalable, high-performance data pipelines using Databricks, PySpark, and SQL. + Develop and optimize ETL/ELT workflows for batch and streaming data processing. Lead Cloud Migration Initiatives + Drive and support the migration of on-premises data platforms to cloud environments (preferably Azure). + Refactor legacy data pipelines and ensure seamless transition to modern cloud-based architectures. + Convert and optimize legacy SQL dialects into Spark SQL for improved performance and scalability. Data Platform Engineering + Develop and manage data solutions within the Databricks Lakehouse architecture. + Ensure reliability, scalability, and performance of data processing workloads in production environments. Collaboration & Stakeholder Engagement + Work closely with data architects, analysts, and business stakeholders to understand data requirements and deliver solutions. + Provide technical guidance and mentorship to junior engineers. Data Quality & Governance + Implement data quality frameworks, validation rules, and monitoring mechanisms. + Ensure data accuracy, consistency, and compliance across pipelines. CI/CD and DevOps Practices + Design and maintain CI/CD pipelines for data engineering workflows. + Apply best practices for version control, automated testing, and deployment. Workflow Orchestration + Design, implement, and maintain orchestration pipelines using tools such as: Databricks Workflows, Apache Airflow, dbt. + Monitor job performance and troubleshoot failures proactively. Cloud Integration & Optimization + Integrate Databricks with Azure services (e.g., Data Factory, Azure Functions, storage accounts). + Optimize cluster configurations, job execution, and cost utilization. Performance Tuning & Optimization + Analyze and optimize queries, Spark jobs, and data models for efficiency and reduced processing time. + Implement best practices for partitioning, caching, and data storage formats. Documentation & Best Practices + Maintain clear technical documentation for pipelines, workflows, and system architecture. + Promote coding standards, reusable components, and best engineering practices. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform](https://www.wearedevelopers.com/videos/100018-from-devops-to-scaled-devops-how-we-re-rebuilding-continuous-delivery-as-a-platform) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers)