Associate, Data Engineer

We are seeking an Associate Data Engineer to join the Trade Data Repository (TDR) team within Market Surveillance Technology. The role will support the development, enhancement, monitoring, and modernization of a critical surveillance data platform responsible for ingesting, validating, transforming, and distributing trade and order data to surveillance systems and analytical applications. TDR currently operates on Azure SQL Server and Azure Blob Storage and is actively migrating to an Azure Databricks Lakehouse architecture utilizing Bronze, Silver, and Gold data layers.

The successful candidate will work closely with senior engineers, business analysts, surveillance stakeholders, Data Governance teams, and upstream application teams to build reliable and scalable data pipelines supporting regulatory and compliance objectives.

Role Objectives: Delivery, Data Engineering & Development

• Develop and maintain SQL-based ingestion, transformation, and reconciliation processes within the TDR platform.
• Design and optimize complex SQL queries, stored procedures, views, functions, and ETL processes.
• Build and support Azure Data Factory (ADF) data pipelines for ingestion of structured and semi-structured data.
• Develop Databricks notebooks using PySpark, Spark SQL, and Python.
• Participate in migration of TDR workloads from Azure SQL Server to Azure Databricks Lakehouse.
• Support implementation of Bronze (Pre-Stage), Silver (Stage), and Gold (Outbound) data layers.

Azure Platform Development

• Develop integrations involving:
• Azure SQL Database
• Azure Blob Storage / ADLS Gen2
• Azure Data Factory
• Azure Databricks
• Delta Lake
• Build reusable ingestion and transformation frameworks.
• Support batch processing, incremental loads, and historical data processing.
• Implement metadata-driven pipeline designs and parameterized frameworks.

Data Quality & Governance

• Implement data quality validations and reconciliation controls.
• Develop source-to-target validation checks and record-count reconciliations.
• Support KDE (Key Data Element) controls and exception management processes.
• Participate in root-cause analysis of data quality issues.
• Assist in maintaining auditability, lineage, and data governance controls required for regulatory compliance.

Production Support & Monitoring

• Monitor daily processing and data loads.
• Diagnose and resolve data pipeline failures.
• Support incident management, defect remediation, and change implementation.
• Perform performance tuning for SQL and Databricks workloads.
• Partner with business and support teams during production incidents.

Regulatory & Surveillance Data Processing

• Support ingestion and processing of:
• Equities trades and orders
• Futures trades and orders
• Fixed Income trades
• FX trades
• Interest Rate Swaps
• Employee trades
• Surveillance alerts and analytics outputs
• Ensure data delivered to surveillance platforms is accurate, timely, and complete.

SDLC, Agile & DevOps

• Participate in Agile delivery processes including sprint planning, daily stand-ups, backlog refinement, retrospectives, and release planning.
• Utilize Jira for user story management, defect tracking, sprint execution, and workflow management.
• Utilize Confluence for technical documentation, architecture diagrams, knowledge sharing, operational procedures, and project collaboration.
• Perform unit testing and support SIT, UAT, regression testing, and production validation activities.
• Support CI/CD deployment processes across Development, QA, UAT, and Production environments.
• Utilize Git and Azure DevOps repositories for source control, branch management, code reviews, and release management.
• Collaborate with developers, business analysts, QA teams, and stakeholders throughout the SDLC.
• Prepare and maintain technical documentation, source-to-target mappings, support procedures, and operational run-books.
• Participate in production support, incident management, root cause analysis, and continuous process improvement initiatives.

• Bachelor’s degree in Computer Science, Information Systems, Engineering, Data Analytics, or related discipline., * 2-5 years of experience in Data Engineering, ETL Development, or Data Warehousing.
• Experience developing solutions in Microsoft Azure environments.
• Experience working with financial services, capital markets, risk, compliance, or surveillance data preferred.

Required Technical Skills

SQL & Database Development

• Advanced SQL development
• Stored Procedures
• Views
• Performance tuning
• Query optimization
• Data modeling
• Data reconciliation

Azure Data Engineering

• Azure SQL Database
• Azure Data Factory (ADF)
• Azure Blob Storage / ADLS Gen2
• Azure Databricks
• Delta Lake
• PySpark
• Spark SQL
• Python

Data Integration

• ETL/ELT Design
• Batch Processing
• Incremental Processing
• Source-to-Target Mapping
• Data Validation
• Data Reconciliation

DevOps

• Git
• Azure DevOps
• CI/CD concepts
• Release Management

Preferred Qualifications

• Experience with Market Surveillance, Trade Surveillance, or Regulatory Reporting.
• Familiarity with Actimize, Quantexa, Compliance Analytics, or surveillance ecosystems.
• Experience implementing Delta Lake architectures.
• Exposure to Unity Catalog and data governance frameworks.
• Understanding of financial products including Equities, Fixed Income, Futures, FX, and Derivatives.
• Experience working within regulated financial institutions.

The anticipated salary range for this role is between $104,000.00 and $170,000.00. The specific salary offered to an applicant will be based on their individual qualifications, experiences, and an analysis of the current compensation paid in their geography and the market for similar roles at the time of hire. The role may also be eligible for an annual discretionary incentive award. In addition to cash compensation, SMBC offers a competitive portfolio of benefits to its employees.

SMBC Group is a top-tier global financial group. Headquartered in Tokyo and with a 400-year history, SMBC Group offers a diverse range of financial services, including banking, leasing, securities, credit cards, and consumer finance. The Group has more than 130 offices and 80,000 employees worldwide in nearly 40 countries. Sumitomo Mitsui Financial Group, Inc. (SMFG) is the holding company of SMBC Group, which is one of the three largest banking groups in Japan. SMFG’s shares trade on the Tokyo, Nagoya, and New York (NYSE: SMFG) stock exchanges.

In the Americas, SMBC Group has a presence in the US, Canada, Mexico, Brazil, Chile, Colombia, and Peru. Backed by the capital strength of SMBC Group and the value of its relationships in Asia, the Group offers a range of commercial and investment banking services to its corporate, institutional, and municipal clients. It connects a diverse client base to local markets and the organization’s extensive global network. The Group’s operating companies in the Americas include Sumitomo Mitsui Banking Corp. (SMBC), SMBC Nikko Securities America, Inc., SMBC Capital Markets, Inc., SMBC MANUBANK, JRI America, Inc., SMBC Leasing and Finance, Inc., Banco Sumitomo Mitsui Brasileiro S.A., and Sumitomo Mitsui Finance and Leasing Co., Ltd.