> Markdown version of [/jobs/ext/188917-streaming-data](https://www.wearedevelopers.com/jobs/ext/188917-streaming-data). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Streaming Data - **Company:** OpenKyber LLC - **Location:** United States (Remote available) - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Java (Programming Language), Airflow, Amazon Web Services, Business Analytics Applications, Apache HTTP Server, Microsoft Azure, Big Data, Cloud Computing, Computer Programming, Data Architecture, Information Engineering, Data Governance, Data Systems, Apache Hive, Python (Programming Language), Machine Learning, Performance Tuning, Query Optimization, Cloud Services, SQL Databases, Data Streaming, Workflow Management Systems, Google Cloud, Snowflake, Apache Spark, Indexer, Data Lakes, Infrastructure Automation Frameworks, Collibra, Apache Kafka, Spark Streaming, Data Lakehouse, Data Pipelines, Databricks - **Published:** May 19, 2026 - **Apply:** https://www.indeed.com/viewjob?jk=1bf44e9b8dd895de ## About the Role * 10+ years of experience in Data Engineering or Big Data Engineering . * Strong expertise with Snowflake and Databricks Lakehouse platform . * Hands-on experience with Apache Spark (PySpark / Spark SQL) . * Experience working with Apache Iceberg or modern table formats . * Advanced knowledge of SQL performance tuning and query optimization . * Experience designing data lake / lakehouse architectures . * Strong programming experience in Python, Scala, or Java . * Experience with workflow orchestration tools (Airflow, Prefect, or similar). * Knowledge of cloud platforms such as Amazon Web Services , Microsoft Azure , or Google Cloud . * Strong understanding of data modeling, partitioning, indexing, and storage optimization. Preferred Qualifications * Experience with data lakehouse architecture and open table formats . * Knowledge of streaming data pipelines using Kafka or Spark Streaming. * Experience with CI/CD pipelines and infrastructure-as-code tools . * Strong leadership and mentoring experience. * Experience supporting enterprise-scale analytics platforms . Nice to Have * Experience with data governance tools. * Knowledge of machine learning data pipelines. * Certifications in cloud platforms or data engineering technologies. ## Description We are seeking a highly skilled Senior Lead Data Engineer with strong experience in modern data platforms including Snowflake , Databricks , Apache Iceberg , and Apache Spark . The ideal candidate will lead the design, development, and optimization of scalable data pipelines and analytics platforms while ensuring high performance for large-scale SQL workloads . This role requires strong expertise in data architecture, performance tuning, and big data technologies to support enterprise-level analytics and data-driven decision-making., * Design and implement scalable data pipelines and data lakehouse architectures using Snowflake, Databricks, and Apache Iceberg. * Lead the development and optimization of Spark-based ETL/ELT pipelines for large-scale data processing. * Optimize complex SQL workloads for performance, cost efficiency, and scalability. * Build and maintain high-performance data models supporting analytics, reporting, and machine learning workloads. * Implement data governance, security, and data quality frameworks. * Collaborate with data scientists, analysts, and business stakeholders to deliver reliable data solutions. * Perform performance tuning for distributed processing frameworks such as Spark and Databricks. * Guide engineering teams on best practices for data architecture, pipeline orchestration, and cloud data platforms . * Monitor and troubleshoot data pipeline performance and reliability issues. * Mentor junior data engineers and lead technical design discussions. ## Related Videos - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Optimizing Discovery: PostgreSQL's Role in Transforming GetYourGuide's Search](https://www.wearedevelopers.com/videos/1647-optimizing-discovery-postgresql-s-role-in-transforming-getyourguide-s-search) - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Why and when should we consider Stream Processing frameworks in our solutions](https://www.wearedevelopers.com/videos/1085-why-and-when-should-we-consider-stream-processing-frameworks-in-our-solutions) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) ## Related Articles - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [7 Cloud Computing Trends Coming in 2025 for Developers](https://www.wearedevelopers.com/magazine/412-7-cloud-computing-trends-coming-in-2025-for-developers)