Iceberg DBA / Lakehouse Operations Engineer

We are seeking a highly skilled Iceberg DBA / Lakehouse Operations Engineer to own the reliability, performance, and operational integrity of the Iceberg data layer powering enterprise analytics and business-critical applications.

This role operates in a large-scale, multi-engine Lakehouse environment, supporting workloads across Spark, Hive, and Impala, and plays a key role in enterprise data modernization initiatives (Hive and Teradata * Iceberg).

The ideal candidate brings deep expertise in Iceberg table operations, metadata management, and query performance optimization, ensuring consistent, high-performance data access across platforms in a cloud-based environment.

This role is critical to ensuring data accuracy and performance-any degradation directly impacts downstream reporting, analytics, and business-critical decision-making.

Key Responsibilities:

Iceberg Data Layer Ownership & Operations

· Own day-to-day operations of Apache Iceberg tables supporting multiple enterprise applications

· Ensure data reliability, consistency, and availability across all Lakehouse workloads

· Maintain operational integrity for datasets at multi-terabyte to petabyte scale

Advanced Table Management & Optimization

· Execute advanced Iceberg table maintenance and optimization strategies:

o Compaction (minor/major) and small file mitigation

o Snapshot expiration and metadata compaction to control metadata growth

o Orphan file cleanup (vacuum) to maintain storage efficiency

· Optimize data layout and performance through:

o File size tuning and distribution strategies

o Partition evolution and pruning optimization

o Clustering and ordering techniques (e.g., Z-ordering or similar patterns)

Data Modeling Standards & Lakehouse Design Alignment

· Support and enforce data modeling best practices aligned with:

o Normalized data structures (3NF) for source-aligned datasets

o Medallion architecture (Bronze / Silver / Gold layers) for curated data flows

· Ensure Iceberg table design aligns with:

o Data ingestion patterns (raw vs curated layers)

o Downstream consumption and performance requirements

· Assist in structuring datasets to balance:

o Data integrity and normalization

o Query performance and analytical efficiency

· Work with data engineering teams to ensure consistent implementation of layered data architecture across multiple applications

Multi-Engine Query Performance & Consistency

· Ensure consistent and performant query behavior across:

o Spark (CDE)

o Hive / Impala (CDW)

· Troubleshoot and resolve:

o Query performance bottlenecks

o Metadata inconsistencies across engines

o Inefficient execution plans and scan patterns

Hive & Teradata Modernization Support

· Play a key role in enterprise data platform modernization (Hive and Teradata * Iceberg)

· Support:

o Schema alignment and data type mapping

o Data validation and reconciliation

· Troubleshoot migration-related issues and ensure post-migration stability and performance

Metadata & Data Lifecycle Management

· Manage Iceberg metadata to ensure:

o Efficient scaling and performance

o Consistent table state across engines

· Execute lifecycle operations:

o Data retention and archival policies

o Snapshot lifecycle management and cleanup

o Time-travel optimization and maintenance

Production Support, Incident Resolution & On-Call

· Provide L2/L3 support for data-related production issues across Iceberg-based Lakehouse workloads

· Participate in on-call rotation to support critical data platforms and ensure timely response to incidents

· Respond to and resolve P1/P2 production incidents within defined SLAs, minimizing impact to downstream applications and reporting

· Troubleshoot:

o Data inconsistencies and reporting discrepancies

o Query failures and performance degradation

· Perform root cause analysis (RCA) and implement preventive measures to avoid recurring issues

· Collaborate with platform and application teams during incident triage and resolution

Security & Data Governance Support

· Support fine-grained access control using:

o Ranger policies and RBAC

· Own and ensure data validation, reconciliation, and accuracy between source and Iceberg datasets

· Ensure secure and compliant access to data across applications

· 4-6 years of experience in Big Data / Data Operations / DBA roles

· Minimum 1+ year of experience with Apache Iceberg or similar table formats (Hive/Delta/Hudi)

· 4+ years of experience with Cloudera ecosystem (CDP)

· Hands-on experience with:

o Iceberg table operations and maintenance

o Spark SQL, Hive, or Impala

· Experience in:

o Production support and incident handling

o Monitoring, troubleshooting, and operational support

· Apply established data modeling and Lakehouse standards in day-to-day operations

· Support:

o Table structuring

o Partition alignment with ingestion patterns

· Assist in maintaining consistency of datasets across Bronze/Silver/Gold layers

Required Skills

· Strong hands-on experience with Apache Iceberg and/or Hive-based data lakes

· Understanding of data modeling concepts (normal forms) and modern Lakehouse patterns (Medallion architecture)

· Expertise in:

o Table-level optimization and performance tuning

o Large-scale data management (TB/PB scale)

· Experience with:

o Spark SQL, Hive, Impala, NiFI, Trino

· Strong understanding of:

o Partitioning strategies

o File formats (Parquet/ORC)

o Distributed query processing

Preferred Skills

· Experience with:

o Hive-to-Iceberg or Teradata-to-Iceberg migration

o Cloudera CDP (CDE/CDW)

· Familiarity with:

o Cloud platforms (AWS, Azure)

· Scripting/automation (Python, Shell)

What You’ll Work On

· Enterprise-scale Iceberg Lakehouse platform supporting multiple applications

· Large-scale data modernization initiatives

· Performance optimization and stability of mission-critical analytical workloads