ETL Developer

Experteer Overview In this role you will design and optimize large-scale data platforms for global markets. You will build real-time and batch data pipelines, focusing on PySpark, Kafka, and Hadoop ecosystems, while advancing Databricks Lakehouse architectures. You will work closely with quants, risk teams, and product owners to deliver governed, high-performance data solutions for regulatory, trading, and analytics workloads. The opportunity centers on shaping scalable, compliant data platforms that support mission-critical financial functions. You will mentor engineers and contribute to strong engineering practices. Compensation / Benefits * Design, develop, and optimize PySpark ETL pipelines on on-prem Hadoop clusters and cloud environments * Build high-volume ingestion frameworks using Kafka for real-time data * Tuning and managing Hadoop components: HDFS, YARN, MapReduce/Tez, Oozie/Airflow * Develop high-performance Hive data models for regulatory reporting and risk processing * Architect and implement Bronze/Silver/Gold layer modeling in Databricks Lakehouse * Apply Delta Lake best practices (file management, Z-Ordering, CDF, schema evolution, ACID) * Create reusable ingestion, cleansing, transformation, and consumption frameworks across Lakehouse layers * Enable governance, lineage, and auditability using cataloging tools (Unity Catalog or equivalent) * Collaborate with quants, product owners, risk tech, and business users; participate in agile ceremonies * Mentor junior engineers and promote strong engineering practices across teams Tasks * 10-13 years of hands-on Big Data engineering experience * Expert in PySpark (optimizations, partitioning, broadcasting) * Expert in Kafka (producer/consumer design, schema registry, streaming ETLs) * Strong Hadoop ecosystem knowledge (HDFS, YARN, MapReduce/Tez, Oozie/Airflow) * Advanced Hive skills (query tuning, TEZ, partitioning) * Extensive Databricks Lakehouse experience (Bronze/Silver/Gold, Delta Lake optimizations) * Experience with data quality frameworks on Lakehouse and handling structured/unstructured data * Experience in Global Markets, Risk, Treasury, Trade Surveillance, or Regulatory Reporting * Strong SQL on TB/PB-scale datasets * Experience with CI/CD practices (Git, Jenkins, Bitbucket) * Familiarity with governance/catalog tools for lineage and auditability * Experience with on-prem and cloud big data platforms Key requirements * Discretionary annual incentive * Medical coverage * Parental leaves * Commuter benefits * Certification & training reimbursement * Vacation & holidays

Jenkins, and implement Bronze/Silver/Gold layer modeling in Databricks Lakehouse * Apply Delta Lake best practices (file management, Z-Ordering, CDF, schema evolution, ACID) * Create reusable ingestion, cleansing, transformation, and consumption frameworks across Lakehouse layers * Enable governance, lineage, and auditability using cataloging tools (Unity Catalog or equivalent) * Collaborate with quants, product owners, risk tech, and business users; participate in agile ceremonies * Mentor junior engineers and promote strong engineering practices across teams Tasks * 10-13 years of hands-on Big Data engineering experience * Expert in PySpark (optimizations, partitioning, broadcasting) * Expert in Kafka (producer/consumer design, schema registry, streaming ETLs) * Strong Hadoop ecosystem knowledge (HDFS, YARN, MapReduce/Tez, Oozie/Airflow) * Advanced Hive skills (query tuning, TEZ, partitioning) * Extensive Databricks Lakehouse experience (Bronze/Silver/Gold, Delta Lake optimizations) * Experience with data quality frameworks on Lakehouse and handling structured/unstructured data * Experience in Global Markets, Risk, Treasury, Trade Surveillance, or Regulatory Reporting * Strong SQL on TB/PB-scale datasets * Experience with CI/CD practices (Git, Jenkins, Bitbucket) * Familiarity with governance/catalog tools for lineage and auditability * Experience with on-prem and cloud big data platforms Key requirements * Discretionary annual incentive * Medical coverage * Parental leaves * Commuter benefits * Certification & training reimbursement * Vacation & holidays