AI Data Engineer

· Pipeline Engineering: Design, build, and maintain production-level data pipelines to deploy and operationalize ML and LLM workflows.

· LLM & RAG Integration: Implement Retrieval-Augmented Generation (RAG) frameworks using libraries like LangChain or LlamaIndex to query structured and unstructured data sources.

· API & System Integration: Integrate LLM APIs (e.g., OpenAI, Anthropic, or open-source models) into data processing workflows.

· Performance Optimization: Optimize distributed workloads, data processing engines, and pipeline latency for real-time and batch execution.

· CI/CD & DevOps: Build and maintain CI/CD pipelines to deploy data and AI workflows using relevant SDKs and automation tools.

· Data Quality & Validation: Implement strict schema validation rules and data quality checks to ensure reliable pipeline execution.

· AI Evaluation & Quality Control: Monitor and measure output quality using key metrics such as retrieval quality, answer correctness, and faithfulness.

· Experience: Proven experience as a Data Engineer building production-grade ETL/ELT data pipelines.

· LLM / AI Concepts: Minimum working knowledge of ML concepts, LLM architectures, vector databases, and RAG frameworks (e.g., LangChain, LlamaIndex).

· API Integration: Hands-on experience integrating third-party or self-hosted LLM APIs into data pipelines.

· Distributed Computing: Experience optimizing distributed data processing workloads (e.g., PySpark, Spark, Databricks, Ray, or Cloud-native processing services).

· CI/CD & Automation: Solid understanding of CI/CD pipeline implementation, deployment SDKs, and containerization (e.g., Docker, Kubernetes, GitHub Actions, Jenkins).

· Schema & Data Quality: Expertise in enforcing schema validation rules, data contracts, and pipeline performance optimization.

· Evaluation Metrics: Familiarity with AI/RAG evaluation metrics (e.g., retrieval precision, answer correctness, context relevance, faithfulness).