> Markdown version of [/jobs/ext/643919-lead-big-data-engineer](https://www.wearedevelopers.com/jobs/ext/643919-lead-big-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Big Data Engineer - **Company:** TEK INC - **Location:** United States (Remote available) - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Airflow, Amazon Web Services, Microsoft Azure, Big Data, Cloud Computing, Computer Programming, Information Engineering, Data Infrastructure, Data Integration, Data Systems, Query Languages, Monitoring of Systems, Data Intelligence, Python (Programming Language), Oracle (Applications), Performance Tuning, DataOps, SQL Databases, Data Streaming, Transaction Data, Data Processing, Google Cloud, Data Ingestion, Apache Spark, Multi-Cloud, Change Data Capture, Infrastructure as Code (IaC), Event Driven Architecture, Data Lakes, Pyspark, Debezium, Apache Kafka, Data Management, Terraform, Stream Processing, Data Pipelines, Confluent, Databricks - **Published:** June 25, 2026 - **Apply:** https://www.dice.com/job-detail/5211e56f-e893-4062-bbf5-7216a9eebd71 ## About the Role * 12+ years of experience in Data Engineering, Big Data Engineering, or Data Platform Engineering. * Proven experience leading engineering teams and driving large-scale data initiatives. * Strong experience designing both streaming and batch data processing systems. Technical Skills Streaming & Event-Driven Systems * Strong expertise in streaming data architectures and event-driven systems. * Hands-on experience with: * Apache Kafka * Confluent Platform / Confluent Cloud Deep understanding of: * Kafka Topics and Partitions * Schema Design * Schema Registry * Kafka Connect * ksqlDB Data Processing & Lakehouse Platforms * Strong hands-on experience with: * Databricks * Apache Spark (PySpark and/or Scala) * Delta Lake Experience designing and implementing Lakehouse architectures. Programming & Query Languages * Advanced proficiency in: * Python * SQL Scala experience is a plus. CDC & Data Integration * Strong understanding of Change Data Capture (CDC) patterns and implementation. Cloud Platforms * Experience with one or more major cloud providers: * Microsoft Azure * Amazon Web Services (AWS) * Google Cloud Platform (Google Cloud Platform) Data Engineering Best Practices * Data modeling * Performance tuning and optimization * Security and governance controls, * Debezium * Kafka Connect ecosystem * Oracle GoldenGate * SQL-based CDC tools Experience with orchestration tools: * Apache Airflow * Databricks Workflows Infrastructure as Code (IaC): * Terraform CI/CD pipelines for data platforms. Experience with monitoring and observability platforms. Exposure to multi-cloud data platform strategies. ## Description We are seeking a highly skilled Lead Big Data Engineer to join our team and work closely with our client in designing, building, and modernizing enterprise-scale data platforms. The ideal candidate will have deep expertise in streaming architectures, event-driven systems, real-time data processing, and modern cloud-based data platforms. This role requires a strong technical leader who can drive architecture decisions, mentor engineering teams, and deliver scalable, secure, and high-performance data solutions., Leadership & Architecture * Define and drive data platform architecture, engineering standards, and best practices. * Lead and mentor a team of data engineers, providing technical guidance and support. * Drive technical decision-making and ensure alignment with business objectives. * Collaborate with Architects, Product Owners, and cross-functional engineering teams. Data Platform Engineering * Design and implement end-to-end data ingestion frameworks. * Build and optimize real-time and batch data pipelines. * Develop scalable event-driven architectures using Kafka and related technologies. * Design and implement Change Data Capture (CDC) solutions for transactional data replication. * Develop and maintain modern data platforms leveraging Databricks and Lakehouse architectures. * Support cloud migration and data platform modernization initiatives. Performance, Reliability & Governance * Ensure solutions meet requirements for scalability, reliability, security, and governance. * Implement data modeling and performance optimization strategies. * Support monitoring, troubleshooting, and continuous improvement of data platforms. * Enable data-driven decision-making through reliable and accessible data solutions., * Lead the design and evolution of a modern enterprise data platform. * Work with cutting-edge technologies including Kafka, Confluent Cloud, Databricks, Spark, and Delta Lake. * Drive real-time data processing and event-driven architecture initiatives. * Influence technical strategy and architecture decisions. * Collaborate with talented teams on large-scale cloud modernization programs. * Make a direct impact on business outcomes through data-driven innovation. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) ## Related Articles - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Best US AI Conferences for CTOs in 2026: Build vs. Buy, Vendor Evaluation, and Peer Intelligence](https://www.wearedevelopers.com/magazine/736-best-us-ai-conferences-for-ctos-in-2026-build-vs-buy-vendor-evaluation-and-peer-intelligence) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)