Lead Data Engineer

We are seeking a highly experienced Senior Data Engineer with 12+ years of experience in designing, developing, and maintaining scalable data platforms and data pipelines. The ideal candidate will have strong hands-on expertise in Python, SQL, Spark, Databricks, cloud technologies, ETL/ELT, data warehousing, and data integration . Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines for enterprise applications.
• Develop complex data transformation and processing workflows using Python, SQL, and Apache Spark .
• Build and optimize data pipelines using Databricks and PySpark .
• Design and implement data solutions on AWS, Azure, or Google Cloud Platform cloud platforms.
• Develop data ingestion processes from databases, APIs, files, and other structured/unstructured sources.
• Implement data quality, validation, reconciliation, and monitoring processes.
• Design and optimize data warehouses, data lakes, and lakehouse architectures .
• Work with relational and NoSQL databases to support data engineering requirements.
• Perform performance tuning and optimization of SQL queries, Spark jobs, and data pipelines.
• Implement orchestration using Apache Airflow, Azure Data Factory, AWS Glue, or similar tools .
• Develop reusable frameworks and components for data ingestion and transformation.
• Integrate data engineering solutions with CI/CD pipelines and DevOps processes.
• Implement data security, access controls, encryption, and governance best practices.
• Troubleshoot production data pipelines and resolve data processing issues.
• Collaborate with data architects, analysts, data scientists, application developers, and business stakeholders.
• Participate in Agile/Scrum ceremonies and contribute to technical design and architecture discussions.
• Mentor junior and mid-level data engineers and provide technical guidance.

• 12+ years of experience in Data Engineering, ETL, or related data technologies.
• Strong hands-on experience with Python and SQL .
• Extensive experience with Apache Spark / PySpark .
• Strong experience with Databricks and Delta Lake.
• Experience building enterprise-scale ETL/ELT pipelines .
• Strong knowledge of data warehousing concepts, dimensional modeling, and data lake architectures.
• Experience with one or more cloud platforms: AWS, Azure, or Google Cloud Platform .
• Experience with databases such as SQL Server, Oracle, PostgreSQL, MySQL, Snowflake, or similar .
• Experience with workflow orchestration tools such as Airflow, Azure Data Factory, AWS Glue, or similar .
• Strong understanding of batch and near-real-time data processing.
• Experience with Git, CI/CD, Jenkins, Azure DevOps, or GitHub Actions .
• Strong understanding of data quality, data validation, and performance optimization.
• Excellent analytical, troubleshooting, and communication skills.

Preferred Skills

• Experience with Azure Data Lake, AWS S3, ADLS, or Google Cloud Storage .
• Experience with Snowflake or other modern cloud data warehouses.
• Knowledge of Kafka, Event Hubs, or other streaming technologies .
• Experience with Terraform or Infrastructure as Code .
• Knowledge of Docker and Kubernetes.
• Experience implementing data governance, metadata management, and lineage .
• Experience working with large-scale distributed data environments.
• Knowledge of Medallion Architecture (Bronze, Silver, Gold) .
• Experience with real-time/streaming data pipelines.
• Experience in designing highly available and fault-tolerant data platforms.

For applications and inquiries, contact:hirings@openkyber.com

• $144,000-210,000 per year Cargill’s size and scale allows us to make a positive impact in the world. Our purpose is to nourish the world in a safe, responsible and sustainable way. Cargill is a family comp…

• 15 days ago +