Databricks SME (Remote)

• Implement and manage linked services, mount points, and catalog configuration within Databricks and VA’s VHA Data Lake.
• Develop and optimize PySpark (Python/Scala) jobs, notebooks, and workflows in Databricks.
• Build ETL/ELT pipelines using Azure Databricks Lakehouse features.
• Design and implement Spark-based data processing for analytics workloads.
• Optimize Databricks Spark jobs for performance, cost, and scalability (partitioning, caching, tuning, etc.).
• Collaborate with data scientists, analysts, infrastructure specialists and business SME to deliver production-ready data solutions.
• Experience with Delta Lake and Synapse.
• Ensure data quality, governance, and security best practices.
• Act as subject matter experts to the workgroups from start to finish in their migration to the cloud
• Help workgroups set up their resources and migrate their work loads
• Support automated data warehouse access provisioning for eligible users, ensuring seamless integration with VA Cloud Data Warehouse (CDW) systems.
• Provide white-glove customer success services, including onboarding sessions, office hours, rapid async support, and creation of reusable templates, guides, and best-practice documentation.
• Collaborate with Customer Success, Business Analysts, and Platform Engineering teams to refine intake processes, improve user satisfaction, and standardize workspace deployment.
• Develop migration guidance and self-service resources to accelerate platform adoption and reduce user friction when transitioning from legacy data systems.
• Troubleshoot user issues related to data access, workspace configuration, pipelines, cataloging, or permissions, escalating to engineering teams when needed.

• Bachelor’s degree in Information Technology or a related field. (or commensurate experience)
• 12+ years of experience in business analysis, project management, or a similar role.
• Strong experience with Databricks and Azure data services, including workspace administration, catalog creation, linked services, storage mounts, and access configuration.
• Proficiency in data engineering fundamentals such as ETL/ELT pipeline development, data modeling, and managing structured and unstructured datasets.
• Ability to automate provisioning workflows and infrastructure using scripting or infrastructure-as-code tools (Python, PowerShell, Bash, Terraform, ARM, or Bicep).
• Strong troubleshooting and problem-solving abilities for diagnosing platform, data access, and pipeline issues and collaborating across teams for resolution.
• Effective communication skills with the ability to document technical processes, create reusable templates, and support users through onboarding, office hours, and white-glove assistance.

Preferred Skills and Experience:

• Experience with Data Analysis, Data Curation, Data Visualization (PowerBi), Python, Azure Synapse/Azure Data Factory, Azure Data Lake Storage, SQL Server.
• Certifications: Azure Databricks and Spark for Data Engineers, PySpark and SQL, Microsoft Azure Cert, SQL, SQLlite, Python Pandas, Sybase/SAP PowerDesigner, Databricks DAB, Databricks Unity Catalog, and Databricks lakflow pipelines
• Exposure to Starburst, PowerBI, SSRS, SSIS and Unity Catalog
• Familiarity with modern data lakehouse architectures, distributed data systems, and cloud data engineering practices.
• Strong understanding of Azure data services, including storage, compute, identity and access management, and DevOps workflows.
• Experience with workflow automation, CI/CD pipelines, or infrastructure-as-code (e.g., Terraform, ARM, Bicep).
• Background in user-facing or customer-success-adjacent work such as onboarding, training, communications, or technical support.
• Ability to translate business needs into technical requirements and develop scalable solutions that improve platform efficiency.
• Knowledge of enterprise data systems common to healthcare or federal environments, especially data governance and data-security considerations.
• Experience with tools such as Jira, ServiceNow, or SharePoint for intake, change management, and support processes.

Clearance Required

• Ability to obtain and maintain a suitability/Public Trust

USD $175,000.00 - USD $195,000.00 /Yr.