Store AI Agent - Onshore Data Engineer (Python, Pytorch, NodeJS)

Databricks Platform cloud platforms Docker & Kubernetes data modeling mlops DevOps & CI/CD PySpark/Python

• We are seeking a highly skilled Data Engineer with strong expertise in Databricks, Python, and PyTorch to design, develop, and optimize scalable data and machine learning pipelines. The ideal candidate will be responsible for building cloud-native data platforms, enabling advanced analytics, and supporting AI/ML initiatives through robust data engineering practices.
• This role requires hands-on experience in large-scale data processing, feature engineering, model deployment support, and performance optimization across modern data ecosystems., Data Engineering & Platform Development:
• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.
• Build and optimize batch and real-time data ingestion frameworks from diverse data sources.
• Develop ETL/ELT solutions to support enterprise data warehousing, analytics, and AI/ML workloads.
• Implement data quality checks, validation frameworks, monitoring, and observability solutions.
• Optimize Databricks jobs, notebooks, clusters, and workflows for performance and cost efficiency.
• Design and maintain Delta Lake architectures, data models, and metadata management solutions.

Machine Learning Data Enablement:

• Collaborate with Data Scientists and ML Engineers to prepare training datasets and feature stores.
• Support development and deployment of deep learning solutions using PyTorch.
• Develop scalable feature engineering pipelines and data transformation frameworks.
• Enable model training, experimentation, and inference infrastructure on Databricks.
• Integrate ML workflows with MLOps pipelines and CI/CD processes.

Cloud & DevOps:

• Implement DataOps and MLOps best practices.
• Automate deployments using CI/CD pipelines.
• Work with cloud-native services for storage, security, monitoring, and orchestration.
• Support infrastructure-as-code and environment automation initiatives.

Collaboration & Governance:

• Partner with business, analytics, and data science teams to understand requirements and translate them into technical solutions.
• Ensure compliance with data governance, privacy, and security standards.
• Participate in architecture reviews and technology evaluations.

Technical Skills:

• Strong experience with Databricks Platform
• Expertise in Python
• Strong hands-on experience with PySpark
• Experience with PyTorch for machine learning and deep learning workloads
• Knowledge of Delta Lake, Unity Catalog, and Databricks Workflows
• Experience with ETL/ELT pipeline development
• Strong SQL and data modeling skills
• Experience with distributed computing and big data architectures
• Knowledge of REST APIs and data integration patterns
• Git version control and CI/CD tools

Cloud Platforms:

Experience with one or more:

• Microsoft Azure (preferred)
• AWS
• Google Cloud Platform (GCP)

Data Technologies:

• Spark
• Kafka/Event Streaming
• Azure Data Factory or equivalent integration tools
• Data Lake architectures
• Data Warehousing solutions
• Feature Stores

Preferred Skills:

• Experience with GenAI and LLM data pipelines.
• Knowledge of MLflow and MLOps frameworks.
• Hands-on experience with Docker and Kubernetes.
• Experience with real-time streaming architectures.
• Familiarity with vector databases and AI platforms.
• Certifications in Databricks or Cloud platforms., * Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or related field.
• 5+ years of experience in Data Engineering.
• 3+ years of experience with Databricks and cloud-based data platforms.
• Experience supporting machine learning and AI use cases using PyTorch.

Nice-to-Have Experience:

• Retail, QSR, Restaurant Technology, Digital Commerce, or Customer Analytics domains.
• Large-scale enterprise data modernization initiatives.
• AI/ML and advanced analytics implementations.

The pay range that the employer in good faith reasonably expects to pay for this position is $39.30/hour - $61.40/hour. Our benefits include medical, dental, vision and retirement benefits. Applications will be accepted on an ongoing basis.