> Markdown version of [/jobs/ext/2425301-hadoop-hive-python-developer](https://www.wearedevelopers.com/jobs/ext/2425301-hadoop-hive-python-developer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Hadoop Hive Python Developer - **Company:** DTEL Engineering & Consultants Inc - **Location:** Charlotte, United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Airflow, Data Analysis, Big Data, Cloud Computing, Continuous Integration, Data Architecture, Data Governance, Data Infrastructure, Extract Transform Load (ETL), Data Systems, Distributed Systems, Apache Hadoop, Hadoop Distributed File System, MapReduce, Apache Hive, Python (Programming Language), Apache Oozie, Scrum Methodology, Query Optimization, SQL Databases, Data Streaming, Unstructured Data, Workflow Management Systems, Cloud Platform System, Apache Yarn, Delivery Pipeline, Git, Data Lakes, Pyspark, Real Time Data, Apache Kafka, Bitbucket, Tez (Software), Jenkins, Databricks - **Published:** August 28, 2026 - **Apply:** https://www.dice.com/job-detail/b82fa9ed-539a-42fb-9fcd-e6961ac33498 ## About the Role Must Have Technical/Functional Skills Primary skills: Hadoop, Hive, Python, PySpark, Apache Kafka, Hadoop Ecosystem, Hive, Databricks Lakehouse Architecture, Delta Lake, Bronze/Silver/Gold Data Modeling, Big Data ETL Pipeline Development, SQL, Real-time Data Ingestion Frameworks, Data Governance & Cataloging, CI/CD Tools Git, Jenkins, Bitbucket, Workflow Orchestration, and Cloud & On-Prem Big Data Platforms. Roles & Responsibilities Seeking a Senior Big Data Engineer with 9-14 years of experience specializing in Hadoop, Python, Hive PySpark, Kafka, and strong experience designing data solutions for large-scale financial systems. In addition, the candidate must possess advanced expertise in Databricks Lakehouse architecture, particularly around Bronze/Silver/Gold layer data modeling, Delta Lake optimizations, and building reliable, scalable pipelines for regulatory, risk, trading, and analytics workloads., * 9-14 years of hands-on experience in Big Data Engineering. * Expert skills in: o PySpark data frame optimizations, partitioning, broadcast strategies, distributed computing. o Kafka producer/consumer design, schema registry, streaming ETLs. o Hadoop ecosystem HDFS, YARN, MapReduce/Tez, Oozie/Airflow. o Hive advanced query tuning, TEZ optimization, partition/bucket management. * Extensive hands-on experience with Databricks Lakehouse, including: o Bronze/Silver/Gold layer modeling o Delta Lake optimizations o Data quality frameworks on Lakehouse o Structured & unstructured data handling * Experience in Global Markets, Risk, Treasury, Trade Surveillance, or Regulatory Reporting. * Strong SQL knowledge with experience working on massive datasets (TB/PB scale). * Experience with CI/CD practices Git, Jenkins, Bitbucket, build pipelines. ## Description Big Data Platform Engineering * Design, develop, and optimize PySpark-based ETL pipelines running on onprem Hadoop clusters and cloud environments. * Build highvolume ingestion frameworks using Kafka for real-time and near-real-time trading and market data. * Develop, tune, and manage Hadoop ecosystem components HDFS, YARN, MapReduce, Tez, Oozie/Airflow. * Build high-performance, optimized Hive data models for regulatory reporting, trade lifecycle, and market risk processing. Databricks Lakehouse & Delta Framework * Architect and implement Bronze/Silver/Gold layer modeling patterns within the Databricks Lakehouse. * Apply Delta Lake best practices including: o optimized file management o Z-Ordering o Delta Change Data Feed (CDF) o schema evolution & enforcement o ACID transaction handling * Build reusable frameworks for ingestion, cleansing, transformation, and consumption of data across Lakehouse layers. * Enable governance, lineage, and auditability using Unity Catalog or equivalent cataloging tools. Collaboration, Leadership & Delivery * Collaborate closely with quants, product owners, architects, risk tech, and business users. * Participate in agile ceremonies sprint planning, refinement, design reviews. * Mentor junior engineers and contribute to building strong engineering practices across tech teams. ## Related Videos - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [The Road to MLOps: How Verivox Transitioned to AWS](https://www.wearedevelopers.com/videos/1050-the-road-to-mlops-how-verivox-transitioned-to-aws) - [Enjoying SQL data pipelines with dbt](https://www.wearedevelopers.com/videos/823-enjoying-sql-data-pipelines-with-dbt) - [Enterprise-Cloud-Native - Fast-Paced Development & Deployment in a Highly Secure Banking Environment](https://www.wearedevelopers.com/videos/671-enterprise-cloud-native-fast-paced-development-deployment-in-a-highly-secure-banking-environment) ## Related Articles - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Software Engineer Salary London](https://www.wearedevelopers.com/magazine/252-software-engineer-salary-london)