Lead Data Engineer

• Lead onshore and offshore data engineering teams across client organizations, implementation partners, and multiple vendors.
• Drive end-to-end migration of legacy data warehouse workloads from IBM Netezza to Databricks.
• Convert business requirements and architecture designs into technical specifications, development plans, and engineering deliverables.
• Design, develop, and optimize scalable ETL/ELT pipelines, data transformations, and workflows using Databricks, Spark, SQL, and Python/PySpark.
• Modernize legacy SQL queries, stored procedures, scripts, and data processing logic for execution on Databricks.
• Implement data quality checks, automated testing, performance tuning, and coding standards to ensure reliable and efficient data pipelines.
• Coordinate with architects, project managers, business stakeholders, and vendor teams to manage technical dependencies and delivery milestones.
• Mentor junior and mid-level engineers, conduct code reviews, and provide technical guidance.
• Support deployment activities, production troubleshooting, incident resolution, and ongoing pipeline maintenance.
• Lead data validation, source-to-target reconciliation, migration cutover, and post-deployment stabilization.
• Leverage Azure and Google Cloud Platform cloud services to support data integration, storage, and modernization initiatives.

We are seeking an experienced Lead Data Engineer with strong hands-on expertise in Databricks, Apache Spark, SQL, and Python/PySpark to lead enterprise data modernization initiatives, particularly migrating legacy data warehouse platforms such as IBM Netezza to Databricks. The ideal candidate will combine technical leadership, large-scale data engineering expertise, and experience managing distributed onshore and offshore teams., * 8+ years of experience in data engineering, data warehousing, ETL/ELT development, or analytics engineering.

• 2+ years of experience leading onshore/offshore teams and coordinating distributed engineering groups.
• Strong hands-on expertise in Databricks (mandatory), Apache Spark, SQL, Python, and PySpark.
• Experience with IBM Netezza or comparable enterprise data warehouse platforms.
• Proven experience in data warehouse migration, cloud data platform modernization, or legacy system transformation.
• Strong understanding of data pipelines, Lakehouse architecture, distributed processing, and data modeling.
• Familiarity with Azure and Google Cloud Platform (Google Cloud Platform).
• Excellent communication, technical leadership, problem-solving, and troubleshooting skills.

Preferred Skills:

• Experience migrating Netezza SQL workloads and legacy ETL processes to Databricks.
• Knowledge of Spark optimization, partitioning, caching, job performance tuning, and scalable data processing.
• Familiarity with Databricks workflows, Delta Lake, and modern Lakehouse design patterns.
• Experience with migration testing, data reconciliation, production deployment, and cutover planning.
• Ability to manage multiple vendors, mentor engineers, and deliver projects in a collaborative, onsite environment.

Important: Hands-on Databricks expertise is a mandatory requirement for this role.