Software Engineer II - Databricks

• We design and implement batch and streaming data pipelines using Databricks, Spark, Delta Lake, and orchestrators such as Workflows, Airflow, and ADF.
• We develop and optimize Spark jobs and SQL transformations for performance, reliability, and cost efficiency.
• We build and maintain curated data models, data quality checks, and automated testing.
• We implement CI/CD for notebooks and code using Git-based workflows and automate deployments across environments.
• We manage and tune Databricks clusters, jobs, and configurations, monitor production workloads, and resolve incidents.
• We integrate multiple data sources, including cloud storage, relational databases, APIs, and event streams, and implement robust ingestion patterns.
• We apply data governance and security best practices, including access controls, secrets management, lineage and metadata, and auditing.
• We create clear documentation for pipelines, data contracts, and operational runbooks.
• We leverage enterprise-authorized AI coding assist tools to improve code quality, delivery speed, and productivity while validating outputs through peer review, automated testing, and secure coding standards.
• We apply knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.

Technologies:

• AI
• Airflow
• AWS
• Azure
• CI/CD
• Cloud
• Databricks
• ETL
• GCP
• Git
• Kafka
• Python
• PySpark
• SQL
• Scala
• Security
• Spark
• Unity
• GameDev

More:

We are partnering directly with JPMorganChase for this Software Engineer II role within our Corporate Investment Bank Payments Technology team. We are a global leader in financial services, providing strategic advice and products to corporations, governments, wealthy individuals, and institutional investors. Our Commercial & Investment Bank operates across banking, markets, securities services, and payments in more than 100 countries. We value trusted long-term partnerships, diversity and inclusion, and we provide reasonable accommodations for applicants and employees with religious practices and beliefs, as well as mental health or physical disability needs.

• We have experience building data pipelines on Databricks and/or Apache Spark in production.
• We have strong coding skills in Python (PySpark) and SQL; Scala is a plus.
• We have hands-on experience with Delta Lake, including MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE, and VACUUM.
• We have experience with orchestration and scheduling tools such as Databricks Workflows, Airflow, and Azure Data Factory.
• We are familiar with cloud data platforms and storage such as AWS, Azure, GCP, S3, ADLS, and GCS.
• We have a solid understanding of data engineering fundamentals, including data modeling, ETL/ELT patterns, reliability, observability, and performance tuning.
• We have hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment and can critically evaluate and validate AI-generated outputs.
• We understand responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs and outputs, and adherence to resiliency and security expectations.
• Preferred: We have experience with streaming technologies such as Structured Streaming, Kafka, Event Hubs, or Kinesis.
• Preferred: We have experience implementing data quality frameworks such as Great Expectations or DQ and data testing in CI.
• Preferred: We have exposure to Unity Catalog or similar tools for governance and fine-grained permissions.