> Markdown version of [/jobs/ext/1402330-vp-data-engineer](https://www.wearedevelopers.com/jobs/ext/1402330-vp-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # VP, Data Engineer - **Company:** SMBC, L.C. - **Location:** Charlotte, NC, United States (Remote available) - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Business Logic, Audit Trail, Automation of Tests, Microsoft Azure, Big Data, Cloud Computing, Code Review, Information Systems, Continuous Integration, Information Engineering, Data Governance, Extract Transform Load (ETL), Data Retention, Data Security, Data Visualization, Database Queries, Distributed Computing Environment, Github, Apache Hive, Python (Programming Language), Meta-Data Management, Performance Tuning, Scrum Methodology, Role-Based Access Control, Power BI, Cloud Services, Azure Data Lake, Anaplan, Software Deployment, SQL Databases, Tableau (Software), Unstructured Data, Management of Software Versions, Data Ingestion, Azure Data Factory, Data Lakes, Pyspark, Information Technology, Data Lineage, Collibra, Optimization Algorithms, Deployment Automation, Azure Synapse Analytics, Software Version Control, Data Pipelines, Serverless Computing, Databricks - **Published:** July 23, 2026 - **Apply:** https://dejobs.org/x/x/2B0E5F01A34147BF93C8E650A5DEDBEE/job/ ## About the Role We are seeking an experienced Vice President - Data Engineer with 10-15 years of hands-on experience to lead the design, development, and optimization of scalable, cloud-native data platforms. This role requires deep technical expertise in Azure, Databricks, PySpark, Python, and SQL, supporting enterprise data pipelines for regulatory, compliance, and analytics workloads., * Master's degree in Computer Science, Engineering, Information Systems, or a related field. * 10-15 years of hands-on experience in data engineering, preferably within financial services or other regulated industries Required Technical Expertise * Azure Databricks (clusters, jobs, notebooks, Delta Lake, performance tuning) * PySpark (RDDs, DataFrames, Spark SQL, optimization techniques) * Azure Cloud Services: * Azure Data Factory (ADF) * ADLS Gen2 * Azure Synapse * Azure Functions * Azure DevOps / GitHub * Python for data engineering and automation workflows * SQL (complex queries, performance optimization, large-scale datasets) Preferred / Nice-to-Have Skills * Experience designing and supporting enterprise-scale ETL/ELT pipelines. * Strong understanding of Delta Lake, medallion architecture (Bronze/Silver/Gold), and distributed data processing. * Familiarity with data governance, security, encryption, and RBAC in cloud-native environments. * Experience with CI/CD best practices and automated deployment pipelines. * Exposure to BI and visualization tools such as Power BI or Tableau. * Familiarity with Anaplan or other enterprise planning platforms is a plus, particularly in supporting downstream financial analytics or planning use cases. * Exposure to or hands-on experience with AI agents, intelligent automation, or GenAI-enabled data workflows is a strong plus. * Excellent analytical, communication, and cross-functional collaboration skills. ## Description * Own the architecture, design, and implementation of end-to-end ETL/ELT workflows using Azure Data Factory (ADF) and Azure Databricks for regulatory and compliance-driven data ingestion and transformation. * Integrate, standardize, and normalize structured and unstructured data from multiple internal and external sources while enforcing strict data quality and governance controls. * Build secure, auditable, and high-performance data pipelines supporting large-scale, sensitive financial datasets. * Automate ingestion, transformation, and validation processes to enable near real-time analytics and regulatory reporting. * Design and implement efficient storage formats, partitioning, and optimization strategies for fast data access and retrieval. * Utilize Databricks and Delta Lake for distributed processing, ACID-compliant storage, versioning, and time-travel capabilities. * Enforce data retention, archiving, and purging policies aligned with global regulatory and compliance requirements. * Drive the migration of legacy application logic into modern Azure Databricks, Data Lake, and SQL-based architectures. * Maintain comprehensive data lineage, metadata management, and audit trails using Azure Purview or equivalent frameworks. * Partner with data governance, risk, and compliance teams to define data standards, access controls, and security requirements. * Implement and manage CI/CD pipelines using GitHub and GitHub Actions, enabling automated testing, version control, and reliable deployments. * Review code, enforce engineering best practices, and support production deployments and operational stability. * Participate in Agile/Scrum ceremonies, including sprint planning, design reviews, and regulatory or audit engagements. * Provide technical leadership and mentorship to data engineers, setting standards and best practices across the organization. ## Related Videos - [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data) - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail) - [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning) - [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) ## Related Articles - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [What Are The Top Skills Required For Azure Developers?](https://www.wearedevelopers.com/magazine/77-what-are-the-top-skills-required-for-azure-developers) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Data Analyst Salary in the UK](https://www.wearedevelopers.com/magazine/278-data-analyst-salary-in-the-uk) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)