Data Engineer with Python Spark Databricks and AIML

As a Computational Engineering Expert, you will leverage your specialized knowledge to enhance the training of next-generation AI systems. Your contributions will directly influenc…

• 1 month ago

We are seeking an experienced Data Engineer with expertise in Python, Spark, Databricks, and AI/ML technologies to build scalable data platforms and Generative AI solutions. The ideal candidate should have experience developing data pipelines, LLM applications, AI agent workflows, and cloud-based data engineering solutions. Key Responsibilities Design and build scalable data pipelines using Python, Spark, Databricks, and Delta Lake. Develop and manage data solutions on Azure, Snowflake, and PostgreSQL. Build and optimize LLM/GenAI pipelines, including data ingestion, prompt engineering, fine-tuning, and deployment. Develop AI workflows using LangChain, LangGraph, AI Agents, and RAG architectures. Integrate with OpenAI, Azure OpenAI, Hugging Face, and other LLM APIs. Collaborate with data scientists, AI engineers, and business teams. Ensure data quality, governance, security, and performance across enterprise data platforms. Required Skills 7 12 years of Data Engineering experience. Strong expertise in Python, PySpark, and Apache Spark. Hands-on experience with Databricks and Delta Lake. Experience with Snowflake and PostgreSQL. Experience building LLM/GenAI applications and pipelines. Strong knowledge of LangChain, LangGraph, AI Agents, and RAG (Retrieval-Augmented Generation). Experience with OpenAI, Azure OpenAI, or Hugging Face APIs. Good understanding of Azure Cloud architecture and scalable data solutions. Strong analytical, problem-solving, and communication skills. Preferred Skills Telecom domain experience. Experience with Model Context Protocol (MCP). AI Agent orchestration. Docker. CI/CD pipelines. Azure or GCP cloud platforms. Top Skills Python & PySpark Databricks & Apache Spark LLM/GenAI (LangChain, LangGraph, RAG, OpenAI/Azure OpenAI) Snowflake & Delta Lake Azure Cloud Nice to Have Telecom industry experience MCP (Model Context Protocol) Docker & CI/CD GCP Cloud Experience