Senior Data Engineer Azure Databricks / PySpark
TALENT Software Services
Minnetonka, MN, United States
14 days ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.thejobnetwork.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
4 years minimum
Working hours
Regular working hours
Job source
Tech stack
Application Programming Interfaces (APIs)
Artificial Intelligence
Application Performance Management
Microsoft Azure
Big Data
Cloud Computing
Continuous Integration
Information Engineering
Data Governance
Extract Transform Load (ETL)
Data Security
Data Systems
+22 more
DevOps
Github
Python (Programming Language)
Performance Tuning
Cloud Services
Standard Sql
Azure Data Lake
Data Processing
Azure Data Factory
Large Language Models
Software Security
Agentic-AI
Data Lakes
Pyspark
Kubernetes
Azure AKS
Graphql
Claude
Api Design
Restful APIs
Data Pipelines
Databricks
Job description
- Primary Role: Data Engineer / API Role with AI Builder
- Cloud Platform: Azure Cloud
- Role Type: Individual Contributor
Core Technical Skills
- Azure Databricks
- PySpark
- Scala
- Azure Data Factory (ADF)
- Azure Data Lake Storage Gen2 (ADLS Gen2)
- Delta Lake
- Unity Catalog
- Python
- SQL
- GitHub
- CI/CD
Data Engineering Responsibilities
- Design, develop, and optimize scalable data pipelines.
- Build cloud-based data solutions using Azure.
- Develop ETL/ELT data processing workflows.
- Work with large-scale datasets in enterprise environments.
- Perform data modeling and performance tuning.
- Implement and maintain Delta Lake solutions.
- Work with Databricks, ADF, and PySpark for data engineering workloads.
- Develop end-to-end data solutions independently.
- Apply data governance and security best practices., * Own end-to-end solution delivery.
- Analyze and solve complex data engineering problems.
- Collaborate with cross-functional teams and business stakeholders.
- Build scalable and reliable data pipelines.
- Optimize data processing and API performance.
- Follow enterprise standards for governance, security, and deployment.
Requirements
API Development Skills
- Strong hands-on experience building REST or GraphQL APIs.
- Understanding of:
- API design and contracts
- API security
- Error handling
- API performance optimization
- Experience deploying APIs using Kubernetes, preferably Azure Kubernetes Service (AKS).
AI / ML Skills
- Exposure to AI/LLM technologies.
- Experience or exposure to agentic frameworks.
- Familiarity with:
- Databricks AI/ML
- Codex
- Claude
- Other LLM/AI development frameworks
Cloud & DevOps
- Strong hands-on experience with Azure Cloud.
- Experience with CI/CD practices.
- GitHub experience.
- Knowledge of cloud-native data platforms.
- Understanding of Kubernetes/AKS deployment.
Data Governance & Security
- Strong understanding of data governance.
- Knowledge of data security and access control.
- Experience with Unity Catalog is preferred.
- Ability to implement secure and governed data solutions.
Essential Skills
- Databricks
- Azure Data Factory (ADF)
- PySpark
- Azure Cloud
- ADLS Gen2
- Delta Lake
Desirable Skills
- Scala
- REST APIs / GraphQL
- Kubernetes / AKS
- AI/LLM
- Agentic frameworks
- Databricks AI/ML
- CI/CD
- Unity Catalog
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.thejobnetwork.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦