> Markdown version of [/jobs/ext/4585466-senior-data-engineer-azure-databricks-pyspark](https://www.wearedevelopers.com/jobs/ext/4585466-senior-data-engineer-azure-databricks-pyspark). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer Azure Databricks / PySpark - **Company:** TALENT Software Services - **Location:** Minnetonka, MN, United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Application Performance Management, Microsoft Azure, Big Data, Cloud Computing, Continuous Integration, Information Engineering, Data Governance, Extract Transform Load (ETL), Data Security, Data Systems, DevOps, Github, Python (Programming Language), Performance Tuning, Cloud Services, Standard Sql, Azure Data Lake, Data Processing, Azure Data Factory, Large Language Models, Software Security, Agentic-AI, Data Lakes, Pyspark, Kubernetes, Azure AKS, Graphql, Claude, Api Design, Restful APIs, Data Pipelines, Databricks - **Published:** November 5, 2026 - **Apply:** https://www.thejobnetwork.com/job/3e1bc18c-ba4e-40cb-b08a-40857bff6a8b/senior-data-engineer-azure-databricks-pyspark ## About the Role API Development Skills * Strong hands-on experience building REST or GraphQL APIs. * Understanding of: + API design and contracts + API security + Error handling + API performance optimization * Experience deploying APIs using Kubernetes, preferably Azure Kubernetes Service (AKS). AI / ML Skills * Exposure to AI/LLM technologies. * Experience or exposure to agentic frameworks. * Familiarity with: + Databricks AI/ML + Codex + Claude + Other LLM/AI development frameworks Cloud & DevOps * Strong hands-on experience with Azure Cloud. * Experience with CI/CD practices. * GitHub experience. * Knowledge of cloud-native data platforms. * Understanding of Kubernetes/AKS deployment. Data Governance & Security * Strong understanding of data governance. * Knowledge of data security and access control. * Experience with Unity Catalog is preferred. * Ability to implement secure and governed data solutions. Essential Skills * Databricks * Azure Data Factory (ADF) * PySpark * Azure Cloud * ADLS Gen2 * Delta Lake Desirable Skills * Scala * REST APIs / GraphQL * Kubernetes / AKS * AI/LLM * Agentic frameworks * Databricks AI/ML * CI/CD * Unity Catalog ## Description * Primary Role: Data Engineer / API Role with AI Builder * Cloud Platform: Azure Cloud * Role Type: Individual Contributor Core Technical Skills * Azure Databricks * PySpark * Scala * Azure Data Factory (ADF) * Azure Data Lake Storage Gen2 (ADLS Gen2) * Delta Lake * Unity Catalog * Python * SQL * GitHub * CI/CD Data Engineering Responsibilities * Design, develop, and optimize scalable data pipelines. * Build cloud-based data solutions using Azure. * Develop ETL/ELT data processing workflows. * Work with large-scale datasets in enterprise environments. * Perform data modeling and performance tuning. * Implement and maintain Delta Lake solutions. * Work with Databricks, ADF, and PySpark for data engineering workloads. * Develop end-to-end data solutions independently. * Apply data governance and security best practices., * Own end-to-end solution delivery. * Analyze and solve complex data engineering problems. * Collaborate with cross-functional teams and business stakeholders. * Build scalable and reliable data pipelines. * Optimize data processing and API performance. * Follow enterprise standards for governance, security, and deployment.