Sr. Data Engineer (AI + AWS)

We are seeking a highly skilled Data Engineer with expertise in AI-enabled data platforms, AWS cloud services, Python, PySpark, and Kubernetes to design, develop, and optimize scalable data pipelines and machine learning data infrastructure. The ideal candidate will have experience building cloud-native data solutions, processing large-scale datasets, and supporting AI/ML workloads in AWS environments., * Design, build, and maintain scalable ETL/ELT data pipelines using Python and PySpark.

• Develop cloud-native data solutions utilizing AWS services such as S3, EMR, Glue, Lambda, Redshift, Athena, ECS/EKS, IAM, CloudWatch, and Step Functions.
• Build and optimize data ingestion frameworks for structured, semi-structured, and streaming data.
• Collaborate with Data Scientists and AI Engineers to prepare, transform, and deliver high-quality datasets for AI/ML model training and inference.
• Deploy and manage containerized data applications using Kubernetes (EKS) and Docker.
• Develop data processing workflows using Spark and optimize performance for large-scale distributed processing.
• Design data lakes and modern data architectures following AWS best practices.
• Implement data quality checks, monitoring, logging, and alerting mechanisms.
• Optimize SQL queries and data models for analytical workloads.
• Build CI/CD pipelines for automated deployment of data engineering solutions.
• Ensure data governance, security, compliance, and access controls across cloud environments.
• Troubleshoot production issues and provide performance tuning for distributed data systems.
• Work closely with cross-functional teams in Agile/Scrum environments.

• Bachelor’s or Master’s degree in Computer Science, Information Technology, Data Engineering, or related field.
• 10+ years of Data Engineering experience.
• Strong programming experience in Python.
• Hands-on expertise with PySpark and Apache Spark.
• Strong experience with AWS Cloud services.
• Experience with Kubernetes (EKS) and Docker.
• Strong SQL skills and experience with relational databases.
• Experience building scalable ETL/ELT pipelines.
• Familiarity with Git and CI/CD practices.
• Excellent analytical, debugging, and problem-solving skills.

Required Technical Skills:

• Cloud: AWS (S3, Glue, EMR, Lambda, Redshift, Athena, ECS/EKS, IAM, CloudWatch, Step Functions)
• Programming: Python
• Big Data: PySpark, Apache Spark
• Containers: Kubernetes, Docker
• Databases: PostgreSQL, MySQL, SQL Server, Redshift
• Data Storage: Data Lake, Data Warehouse
• Version Control: Git
• Operating Systems: Linux
• Methodology: Agile/Scrum

AI/ML Experience:

• Support AI/ML data pipelines and feature engineering.
• Prepare datasets for model training and inference.
• Experience integrating ML workflows into cloud-based data platforms.
• Familiarity with LLMs, Generative AI, Vector Databases, or Retrieval-Augmented Generation (RAG) is a plus.
• Experience with AWS AI services such as Amazon SageMaker, Bedrock, or Amazon OpenSearch is preferred., * Experience with Apache Airflow or AWS Managed Workflows (MWAA).
• Knowledge of Kafka or Kinesis for streaming data.
• Experience with Delta Lake, Iceberg, or Apache Hudi.
• Infrastructure-as-Code experience using Terraform or CloudFormation.
• AWS certifications (Solutions Architect, Data Engineer, or Machine Learning Specialty) are highly desirable.