Iceberg DBA / Lakehouse Operations Engineer_remote in dallas,TX

We have an immediate opening for a Iceberg DBA / Lakehouse Operations Engineer Experience with strong hands-on experience to support a long-term engagement with our client. Must collaborate effectively with cross-functional teams to deliver high-quality solutions., We are seeking a highly skilled Iceberg DBA / Lakehouse Operations Engineer to manage the reliability, performance, and operational integrity of the Apache Iceberg data layer supporting enterprise analytics and business-critical applications.

The role involves managing a large-scale, multi-engine Lakehouse environment supporting workloads across Spark, Hive, and Impala. The engineer will play a key role in enterprise data modernization initiatives, particularly migrations from Hive and Teradata to Apache Iceberg.

The ideal candidate will have strong hands-on experience with Iceberg table operations, metadata management, data modeling, and query performance optimization in cloud-based environments.

This role is critical to maintaining data accuracy, availability, and performance, as any degradation can directly impact downstream reporting, analytics, and business-critical decision-making.

Key Responsibilities

1. Iceberg Data Layer Ownership & Operations * Own day-to-day operations and administration of Apache Iceberg tables supporting multiple enterprise applications. * Ensure data reliability, consistency, and availability across Lakehouse workloads. * Maintain operational integrity of datasets at multi-terabyte to petabyte scale. * Monitor table health, metadata growth, and storage utilization. * Ensure consistent data access and operational stability across environments.

2. Advanced Table Management & Optimization * Perform advanced Iceberg table maintenance and optimization activities, including: + Minor and major compaction. + Small file mitigation and optimization. + Snapshot expiration and metadata compaction. + Orphan file cleanup and storage management. * Optimize data layout and query performance through file sizing, distribution, and partitioning strategies. * Implement partition evolution and partition pruning techniques. * Apply clustering and data ordering strategies, such as Z-ordering or equivalent approaches. * Monitor and optimize table maintenance operations to improve query performance and storage efficiency.

3. Data Modeling & Lakehouse Architecture * Support and enforce data modeling best practices, including: + Third Normal Form (3NF) for source-aligned datasets. + Medallion architecture (Bronze, Silver, and Gold) for curated data pipelines. * Ensure Iceberg table designs align with ingestion patterns, downstream consumption, and performance requirements. * Balance data integrity, normalization, and analytical query performance. * Collaborate with data engineering teams to ensure consistent implementation of layered data architecture across applications.

4. Multi-Engine Query Performance & Consistency * Ensure consistent and optimized query performance across: + Apache Spark / Spark SQL (CDE). + Hive. + Impala (CDW). * Troubleshoot query performance bottlenecks, metadata inconsistencies, and inefficient execution plans. * Analyze query execution plans, scan patterns, and data access behavior. * Ensure consistent table metadata and query results across multiple processing engines.

5. Hive & Teradata Modernization Support * Support enterprise data modernization initiatives involving Hive-to-Iceberg and Teradata-to-Iceberg migrations. * Assist with schema alignment, data type mapping, and table structure conversion. * Perform data validation, reconciliation, and accuracy checks between source and target datasets. * Troubleshoot migration-related issues and ensure post-migration stability and performance. * Collaborate with data engineering teams to optimize migrated workloads.

6. Metadata & Data Lifecycle Management * Manage Iceberg metadata to ensure scalability, consistency, and performance. * Maintain consistent table states and metadata across multiple query engines. * Implement data retention and archival policies. * Manage snapshot lifecycle operations, expiration, and cleanup. * Optimize time-travel capabilities and metadata maintenance. * Monitor metadata growth and prevent performance degradation.

7. Production Support & Incident Management * Provide L2/L3 operational support for production issues across Iceberg-based Lakehouse workloads. * Participate in on-call rotations to support critical data platforms. * Respond to and resolve P1/P2 incidents within defined SLAs. * Troubleshoot data inconsistencies, reporting discrepancies, query failures, and performance degradation. * Perform Root Cause Analysis (RCA) and implement permanent corrective and preventive actions. * Collaborate with platform, infrastructure, and application teams during incident triage and resolution.

8. Security & Data Governance * Support fine-grained data access controls using Apache Ranger policies and Role-Based Access Control (RBAC). * Ensure data validation, reconciliation, and accuracy between source systems and Iceberg datasets. * Maintain secure and compliant data access across applications and processing environments. * Collaborate with security and governance teams to enforce enterprise data access policies.

• Strong hands-on experience with Apache Iceberg and/or Hive-based data lake environments.
• Strong understanding of Iceberg table architecture, metadata management, and table maintenance operations.
• Experience with table-level optimization, compaction, snapshot management, and performance tuning.
• Understanding of data modeling concepts, including normalization and 3NF.
• Knowledge of modern Lakehouse architecture and Medallion architecture (Bronze/Silver/Gold).
• Hands-on experience with Spark SQL, Hive, and Impala.
• Working knowledge of NiFi and Trino.
• Strong understanding of partitioning strategies, partition evolution, and partition pruning.
• Experience with columnar file formats such as Parquet and ORC.
• Understanding of distributed query processing and large-scale data management.
• Experience supporting multi-terabyte to petabyte-scale data platforms.

Preferred Skills

• Experience with Hive-to-Iceberg and/or Teradata-to-Iceberg migrations.
• Experience with Cloudera CDP, particularly CDE and CDW.
• Familiarity with cloud platforms such as AWS and Azure.
• Experience with Apache Ranger, data governance, and access management.
• Scripting and automation experience using Python and Shell.
• Experience supporting enterprise-scale Lakehouse modernization initiatives.