Sr Data Software Engineer

Note: The strongest candidates are likely to come from organizations with mature AWS data lake ecosystems, such as large financial services, healthcare, retail, or cloud-native technology companies that have recently adopted Apache Iceberg for their data platform modernization efforts. Must-have skills: Hands-on data engineer with strong Python skills and experience building large-scale Apache Spark pipelines on AWS. Deep platform fluency across the modern data lake stack,S3, Glue Data Catalog, Athena, EMR, and Redshift, with a focus on reliable ingestion, transformation, and validation at scale. Project: The client has a central data platform built on AWS Glue Data Catalog. They’ve developed a fine-grained data access control (FGAC) system that is now being adopted across various data-producing teams. As part of this adoption, each application team must review their data pipelines to identify queries affected by the new access controls and modify the queries they own accordingly. Delta-to-Iceberg Migration

• Design and build a migration feature/framework to convert tables from Delta Lake format to Apache Iceberg at scale.
• Leverage deep understanding of Iceberg internals (metadata layers, manifests, snapshots, partitioning, schema evolution) to ensure correct and efficient migration
• Validate data integrity, schema fidelity, and partitioning strategy parity between source Delta tables and migrated Iceberg tables
• Optimize migration processes for performance and reliability across large volumes of tables and data
• Identify and resolve edge cases, compatibility issues, or performance bottlenecks during migration at scale

FGAC (Fine-Grained Access Control)

• Ensure Row-Level Security (RLS) and Column-Level Security (CLS) policies are correctly preserved or re-implemented on migrated Iceberg tables
• Integrate migration workflows with AWS Lake Formation to maintain governance and access control continuity
• Collaborate with security/governance teams to validate that FGAC policies enforce correctly post-migration
• Support GDC (Global Data Catalog / Governed Data Catalog) integration to ensure migrated tables remain properly cataloged and governed
• Build validation/reconciliation tooling to confirm access control parity between pre- and post-migration states

General Responsibilities

• Document migration architecture, access control mapping logic, and operational runbooks
• Collaborate with data platform, governance, and security teams throughout the migration lifecycle
• Monitor and troubleshoot production issues related to migrated tables and access enforcement
• Contribute to best practices and reusable tooling for future Delta-to-Iceberg migration phases

• Strong, hands-on knowledge of Apache Iceberg internals (table format, metadata management, snapshots, partitioning, schema evolution)
• Proven experience performing Delta-to-Iceberg migrations at scale
• Experience with AWS Lake Formation for access control and data governance
• Practical experience implementing or preserving Row-Level Security (RLS) and Column-Level Security (CLS)
• Experience with GDC (Global/Governed Data Catalog) or equivalent data cataloging/governance systems
• Solid understanding of distributed data processing and large-scale data migration challenges
• Strong SQL skills and experience with table format internals (Delta Lake, Iceberg, or similar), * Experience with Spark for large-scale data processing and migration tooling
• Familiarity with AWS data platform services (S3, Glue, Athena)
• Experience with data governance frameworks and compliance-driven access control requirements
• Prior experience building reusable migration frameworks or tooling for table format conversions
• Scripting/automation experience (Python, Scala) for migration validation and tooling