Senior Data Engineer Databricks

” Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL. “ Build and optimize data processing workflows using Apache Spark and Delta Lake. “ Develop reliable data ingestion, transformation, validation, and enrichment processes across multiple data sources. “ Work with large and complex datasets and optimize pipelines for performance, scalability, and cost efficiency. “ Build reusable data products and data services that support analytics, investment workflows, and AI applications. “ Implement data quality, validation, reconciliation, and monitoring processes to ensure data accuracy and reliability. “ Contribute to data architecture and lakehouse design decisions. “ Work with data governance, metadata, lineage, and access-control requirements. “ Collaborate with Data Architects, Product Managers, Software Engineers, Data Scientists, and business stakeholders to translate requirements into scalable data solutions. “ Integrate data from APIs, databases, files, and third-party data providers. “ Develop and maintain automated testing and data-quality frameworks. “ Support CI/CD practices and automated deployment of data engineering workloads. “ Troubleshoot complex data and pipeline issues and continuously improve platform reliability. “ Contribute to the evolution of Rimes AI-ready data infrastructure and intelligence fabric., “ Improve the quality, accessibility, and usability of investment data. “ Create governed and reusable data products. “ Improve data-processing performance and operational efficiency. “ Strengthen the foundation required for AI-powered investment workflows. “ Help transform complex capital-markets data into trusted, actionable intelligence.

” 6 years of experience in data engineering or a related discipline. “ Strong hands-on experience with Databricks. “ Strong programming experience with PySpark and Python. “ Advanced SQL skills, including query optimization and performance tuning. “ Strong experience building production-grade data pipelines and ETL/ELT workflows. “ Experience with Delta Lake / Delta tables and modern lakehouse architectures. “ Strong understanding of data modeling, data integration, and distributed data processing. “ Experience working with large-scale datasets in a cloud environment. “ Experience with data quality, validation, monitoring, and troubleshooting. “ Strong understanding of software engineering practices, including version control, testing, and CI/CD. “ Excellent problem-solving and communication skills. Preferred Qualifications “ Experience with Databricks Unity Catalog and data governance. “ Experience with Delta Sharing or data-sharing technologies. “ Experience with Databricks Workflows, Jobs, or similar orchestration capabilities. “ Experience with AWS or Azure cloud platforms. “ Experience with technologies such as AWS Glue, S3, Azure Data Factory, ADLS, or similar cloud data services. “ Experience with Snowflake or other modern cloud data platforms. “ Experience working with REST APIs and third-party data providers. “ Experience with Terraform or infrastructure-as-code. “ Experience implementing data lineage, metadata management, and data governance. “ Financial services, investment management, capital markets, or market-data experience.