Lead Data Engineer

We are seeking a highly experienced Senior AWS Data Engineer with AI/GenAI experience to design, develop, and support scalable cloud-based data platforms and intelligent data solutions. The ideal candidate should have extensive experience in AWS data services, Python, SQL, ETL/ELT pipelines, data lakes, data warehouses, and AI/ML integrations . The candidate should be comfortable working independently, collaborating with architects and business stakeholders, and taking ownership of enterprise-scale data engineering initiatives. Key Responsibilities:

• Design and develop scalable data pipelines and ETL/ELT solutions using AWS services.
• Build and maintain data ingestion, transformation, and processing frameworks.
• Develop solutions using AWS Glue, S3, Lambda, Redshift, Athena, EMR, and Step Functions .
• Develop reusable data engineering components using Python .
• Write complex and optimized SQL queries for data transformation and analytics.
• Design and implement data lake and data warehouse architectures .
• Build batch and real-time data processing pipelines.
• Implement data quality, validation, reconciliation, and monitoring processes.
• Integrate structured and semi-structured data from multiple enterprise sources.
• Develop scalable data models for reporting, analytics, and AI applications.
• Work with AI/ML and GenAI teams to prepare high-quality datasets for AI use cases.
• Integrate AI/LLM capabilities into existing data engineering workflows where applicable.
• Develop and maintain AI-enabled data pipelines and analytics solutions.
• Support model training, feature engineering, data preparation, and inference pipelines.
• Troubleshoot production data pipeline failures and perform root-cause analysis.
• Optimize data processing jobs, SQL queries, storage, and AWS infrastructure for performance and cost.
• Implement CI/CD practices for data engineering applications.
• Collaborate with Data Scientists, ML Engineers, Cloud Architects, Business Analysts, and stakeholders.

AWS Technologies

• Amazon S3
• AWS Glue
• AWS Lambda
• Amazon Redshift
• Amazon Athena
• Amazon EMR
• AWS Step Functions
• Amazon Kinesis
• AWS Lake Formation
• AWS IAM
• CloudWatch
• AWS Secrets Manager

• Advanced Python
• Advanced SQL
• ETL / ELT
• Data Pipelines
• Data Lakes
• Data Warehousing
• Data Modeling
• Apache Spark / PySpark
• Apache Airflow
• Kafka / Kinesis
• Snowflake / Redshift
• Data Quality & Governance

AI / GenAI Skills

• Experience integrating Generative AI / LLM solutions with enterprise data platforms.
• Experience working with OpenAI, Azure OpenAI, Anthropic Claude, Amazon Bedrock, or Google Gemini .
• Knowledge of Prompt Engineering and LLM APIs.
• Experience preparing enterprise data for RAG (Retrieval-Augmented Generation) applications.
• Experience integrating vector databases and embeddings is a plus.
• Understanding of AI/ML data pipelines and model lifecycle.
• Ability to use AI-assisted development tools to improve engineering productivity.

For applications and inquiries, contact:hirings@openkyber.com

• $144,000-210,000 per year Cargill’s size and scale allows us to make a positive impact in the world. Our purpose is to nourish the world in a safe, responsible and sustainable way. Cargill is a family comp…

• 15 days ago +