Senior Data Engineer Azure Databricks / PySpark

TALENT Software Services
Minnetonka, MN, United States
14 days ago
Apply on www.thejobnetwork.com
Prepare application

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
4 years minimum
Working hours
Regular working hours

Tech stack

Application Programming Interfaces (APIs) Artificial Intelligence Application Performance Management Microsoft Azure Big Data Cloud Computing Continuous Integration Information Engineering Data Governance Extract Transform Load (ETL) Data Security Data Systems
+22 more
DevOps Github Python (Programming Language) Performance Tuning Cloud Services Standard Sql Azure Data Lake Data Processing Azure Data Factory Large Language Models Software Security Agentic-AI Data Lakes Pyspark Kubernetes Azure AKS Graphql Claude Api Design Restful APIs Data Pipelines Databricks

Job description

  • Primary Role: Data Engineer / API Role with AI Builder
  • Cloud Platform: Azure Cloud
  • Role Type: Individual Contributor

Core Technical Skills

  • Azure Databricks
  • PySpark
  • Scala
  • Azure Data Factory (ADF)
  • Azure Data Lake Storage Gen2 (ADLS Gen2)
  • Delta Lake
  • Unity Catalog
  • Python
  • SQL
  • GitHub
  • CI/CD

Data Engineering Responsibilities

  • Design, develop, and optimize scalable data pipelines.
  • Build cloud-based data solutions using Azure.
  • Develop ETL/ELT data processing workflows.
  • Work with large-scale datasets in enterprise environments.
  • Perform data modeling and performance tuning.
  • Implement and maintain Delta Lake solutions.
  • Work with Databricks, ADF, and PySpark for data engineering workloads.
  • Develop end-to-end data solutions independently.
  • Apply data governance and security best practices., * Own end-to-end solution delivery.
  • Analyze and solve complex data engineering problems.
  • Collaborate with cross-functional teams and business stakeholders.
  • Build scalable and reliable data pipelines.
  • Optimize data processing and API performance.
  • Follow enterprise standards for governance, security, and deployment.

Requirements

API Development Skills

  • Strong hands-on experience building REST or GraphQL APIs.
  • Understanding of:
  • API design and contracts
  • API security
  • Error handling
  • API performance optimization
  • Experience deploying APIs using Kubernetes, preferably Azure Kubernetes Service (AKS).

AI / ML Skills

  • Exposure to AI/LLM technologies.
  • Experience or exposure to agentic frameworks.
  • Familiarity with:
  • Databricks AI/ML
  • Codex
  • Claude
  • Other LLM/AI development frameworks

Cloud & DevOps

  • Strong hands-on experience with Azure Cloud.
  • Experience with CI/CD practices.
  • GitHub experience.
  • Knowledge of cloud-native data platforms.
  • Understanding of Kubernetes/AKS deployment.

Data Governance & Security

  • Strong understanding of data governance.
  • Knowledge of data security and access control.
  • Experience with Unity Catalog is preferred.
  • Ability to implement secure and governed data solutions.

Essential Skills

  • Databricks
  • Azure Data Factory (ADF)
  • PySpark
  • Azure Cloud
  • ADLS Gen2
  • Delta Lake

Desirable Skills

  • Scala
  • REST APIs / GraphQL
  • Kubernetes / AKS
  • AI/LLM
  • Agentic frameworks
  • Databricks AI/ML
  • CI/CD
  • Unity Catalog

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.thejobnetwork.com
Prepare application

Good distractions

Loading talks and stories from around this role…