Cloudera Platform Engineer

We are seeking an experienced Senior Data Engineer is responsible for designing, building, and maintaining scalable data pipelines and platforms that support enterprise data integration, analytics, and decision-making. This role plays a critical part in managing the full data lifecycle-from ingestion and transformation to storage and delivery-ensuring data quality, integrity, and accessibility across the organization.

The ideal candidate will have strong experience with the Cloudera Data Platform (CDP), modern data engineering tools, and DevOps-enabled data practices. This individual will collaborate closely with cross-functional teams to enable efficient data flow, optimize performance, and support data governance initiatives.

Position Responsibilities

· Design, develop, and maintain robust, scalable data pipelines to ingest, process, and transform structured and unstructured data from multiple sources.

· Implement and manage data workflows within Cloudera Data Platform (CDP), ensuring high availability, performance, and reliability.

· Perform data extraction, transformation, and loading (ETL/ELT) activities, ensuring adherence to data quality and governance standards.

· Monitor, troubleshoot, and optimize data pipelines and processing systems for performance, scalability, and efficiency.

· Integrate data from diverse sources and formats, resolving data flow, consistency, and content issues.

· Utilize modern data engineering tools and frameworks such as PySpark, pandas, dbt, and SQL-based solutions for data processing and transformation.

· Collaborate with DevOps teams to implement and maintain CI/CD pipelines for automated deployment and testing of data solutions.

· Support data platform operations, including system monitoring, logging, and long-term maintenance.

· Work within Agile frameworks (Scrum/Kanban) to deliver incremental data solutions aligned with business priorities.

· Partner with data analysts, data scientists, and business stakeholders to understand data requirements and deliver high-quality data assets.

· 5+ years of experience in data engineering or application/data development, with strong proficiency in Python.

· 5+ years of experience with data integration and ingestion tools such as Apache NiFi.

· Hands-on experience with Cloudera Data Platform (CDP) for building and managing data pipelines.

· Advanced knowledge of SQL and experience working with relational databases such as Microsoft SQL Server.

· Strong experience with distributed data and computing frameworks, including Hadoop, Spark, Hive, HBase, Kafka, and MapReduce.

· Proficiency in data transformation and processing frameworks such as PySpark, pandas, and dbt.

· Experience with CI/CD pipelines and DevOps practices for data platform management and deployment.

· Solid understanding of data quality, data governance, and data lifecycle management principles.

· Experience working in Agile environments using Scrum and/or Kanban methodologies.

· Strong working knowledge of UNIX/Linux systems, including shell scripting and command-line tools.

Preferred Qualifications:

· Experience with large-scale, enterprise data platforms and cloud-based data ecosystems.

· Familiarity with data security, compliance, and governance frameworks.