Data Engineer

Intersources Inc.
yesterday

Role details

Contract type
Permanent contract
Employment type
Full-time (> 32 hours)
Working hours
Regular working hours
Languages
English
Experience level
Intermediate

Job location

Tech stack

API
Data analysis
Azure
Cloud Computing
Information Systems
Continuous Integration
Information Engineering
Data Governance
DevOps
Monitoring of Systems
Identity and Access Management
Python
Log Analysis
Microsoft SQL Server
Performance Tuning
Role-Based Access Control
SQL Databases
Systems Integration
Enterprise Data Management
Azure
Data Lake
PySpark
Information Technology
Terraform
Splunk
Azure
Key Vault
Databricks

Job description

  • Administration & Configuration: Administer, configure, and manage Databricks workspaces, clusters, jobs, and notebooks to support enterprise data and analytics workloads.
  • Security & Access Management: Implement and maintain user access controls, security policies, and workspace permissions aligned with enterprise and regulatory standards.
  • Performance Optimization: Monitor and optimize cluster performance, troubleshoot execution issues, and ensure efficient resource utilization and cost management.
  • Integrations: Manage integrations with Azure cloud services (ADF, Synapse, Data Lake, Key Vault, etc.) and enterprise data sources such as SQL Server.
  • Governance & Compliance: Ensure adherence to data governance, financial services compliance, and security best practices.
  • Automation & DevOps: Automate environment provisioning, monitoring, and operational tasks using Terraform, CI/CD pipelines, or APIs.
  • Collaboration: Partner with data engineering, analytics, and security teams to support business-critical workloads.
  • Documentation & Standards: Develop and maintain detailed documentation, SOPs, and architectural standards for Databricks management.

Requirements

  • 6+ years of experience in Data Engineering / Cloud Administration, with at least 2+ years in Databricks administration.

  • Strong expertise in Azure Databricks and Azure ecosystem (Azure Data Factory, Synapse, Data Lake, Key Vault, etc.).

  • Hands-on experience managing Databricks clusters, jobs, notebooks, and workspace configurations.

  • Proficiency in SQL, Python, and/or PySpark for troubleshooting and validation.

  • Strong knowledge of identity and access management (IAM), RBAC, and workspace security policies.

  • Experience with Terraform, CI/CD pipelines, and DevOps practices for automation.

  • Understanding of data governance, compliance, and security controls in the financial services industry.

  • Excellent problem-solving and stakeholder communication skills. Preferred Qualifications: -

  • Experience working in financial services or banking domain.

  • Familiarity with Databricks Unity Catalog for governance and security.

  • Azure certifications (e.g., DP-203: Azure Data Engineer Associate, AZ-305: Solutions Architect).

  • Knowledge of monitoring tools such as Azure Monitor, Log Analytics, or Splunk. Education: -

  • Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field.

Apply for this position