> Markdown version of [/jobs/ext/2462639-manager-data-engineer](https://www.wearedevelopers.com/jobs/ext/2462639-manager-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Manager Data Engineer

- **Company:** Penn Foster
- **Location:** United States (Remote available)
- **Experience:** Experienced
- **Contract:** Permanent contract
- **Skills:** Artificial Intelligence, Data Analysis, Automation of Tests, Microsoft Azure, Cloud Computing Security, Cluster Analysis, Code Review, Information Systems, Continuous Integration, Data Architecture, Information Engineering, Data Governance, Data Infrastructure, Dataspaces, Data Systems, Dimensional Modeling, Python (Programming Language), Machine Learning, Meta-Data Management, Operational Data Store, Role-Based Access Control, Power BI, Software Tools, Azure Active Directory, Azure Data Lake, Software Engineering, SQL Databases, Tableau (Software), Enterprise Data Management, Data Ingestion, Apache Spark, IT Architecture, Caching, Generative AI, Sap Business Objects, Git, Data Lakes, Pyspark, Infrastructure Automation Frameworks, Information Technology, Data Lineage, Deployment Automation, Data Analytics, Machine Learning Operations, Api Design, Cloud Optimization, Serverless Computing, Databricks
- **Published:** August 10, 2026
- **Apply:** https://www.indeed.com/viewjob?jk=0f3bcc219aec63d0

## About the Role

* Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field, or equivalent practical experience.
 * 8+ years of experience in Data Engineering, Analytics Engineering, or Data Platform Engineering.
 * 3+ years of experience leading technical initiatives and mentoring engineering teams.
 * Demonstrated success developing engineers while driving engineering excellence and technical standards.
 * Strong communication and collaboration skills with the ability to influence both technical and business stakeholders., * Expert knowledge of the Databricks Lakehouse Platform, including:
 * Apache Spark (PySpark)
 * Delta Lake
 * Unity Catalog
 * Databricks Workflows & Jobs
 * Repos
 * SQL Warehouses
 * Cluster Policies
 * Serverless Compute
 * MLflow
 * Lakehouse Monitoring
 * Auto Loader
 * Delta Live Tables / Lakeflow
 * Expert Knowledge in SQL and Python.
 * Strong Experience working with BI tools such as PowerBI, Tableau or Business Objects
 * Strong Microsoft Azure experience, including ADLS Gen2, Microsoft Entra ID (Azure AD), RBAC, networking, and cloud security.
 * Experience implementing Medallion Architecture, dimensional modeling, and domain-oriented data products.
 * Deep understanding of Spark optimization techniques, including Adaptive Query Execution, partitioning, caching, Photon, Liquid Clustering, and Delta optimization.
 * Experience implementing CI/CD pipelines, Git-based development workflows, Infrastructure as Code, and automated testing.
 * Strong understanding of data governance, metadata management, data quality, observability, security, and compliance.

Preferred

 * Experience implementing Databricks Genie, Genie Spaces, and Genie Ontologies.
 * Experience designing semantic models and AI-ready data products.
 * Experience supporting Power BI, Tableau, or other enterprise BI platforms.
 * Experience with machine learning platforms, MLOps, or Generative AI applications.
 * Experience with dbt, Great Expectations, or similar modern data engineering tools.
 * Databricks Certified Data Engineer Professional and/or Microsoft Azure certifications.

## Description

Penn Foster Group is seeking an experienced Lead Data Engineer to help shape the future of our enterprise data platform and AI strategy. This is a highly visible technical leadership role responsible for architecting, building, and evolving our modern cloud data platform while establishing engineering standards, mentoring a growing team, and driving innovation across the organization.
As a Lead Data Engineer, you will serve as the technical leader for our Databricks Lakehouse platform, providing hands-on leadership across data ingestion, transformation, governance, analytics enablement, and AI-ready data products. You will partner closely with Business Intelligence, Product, Platform Engineering, Security, Data Governance, and MLOps teams to build trusted, scalable, and governed data solutions that power enterprise analytics, machine learning, and generative AI.
This role combines deep technical expertise with leadership, mentoring, and strategic influence. You'll help establish the long-term technical direction of Penn Foster Group's data ecosystem while coaching and developing engineers, driving engineering excellence, and implementing modern data engineering best practices., Technical Leadership

 * Provide technical leadership for the Data Engineering team through architecture guidance, code reviews, mentoring, and engineering best practices.
 * Mentor and develop junior and mid-level Data Engineers, fostering technical growth, knowledge sharing, and continuous learning.
 * Establish engineering standards for software development, testing, CI/CD, documentation, observability, and operational excellence.
 * Lead technical design discussions, evaluate architectural tradeoffs, and drive adoption of modern engineering practices.
 * Champion a culture of quality, collaboration, innovation, and continuous improvement.

Data Platform & Databricks Leadership

 * Serve as the technical lead for Penn Foster Group's Databricks Lakehouse platform.
 * Design, build, and support scalable enterprise data pipelines using SQL, Python, Apache Spark, and Databricks.
 * Define best practices for Databricks development, including Workflows, Repos, Jobs, notebooks, reusable Python libraries, Git integration, cluster policies, SQL Warehouses, and deployment automation.
 * Design and optimize Delta Lake architectures using Medallion patterns, Delta optimization, Liquid Clustering, and Photon.
 * Build reusable ingestion frameworks supporting batch, streaming, CDC, and API-based integration patterns.
 * Optimize Spark workloads for performance, scalability, reliability, and cloud cost efficiency.

AI & Databricks Genie Enablement

 * Partner with Analytics and business stakeholders to develop trusted semantic data products that power Databricks Genie.
 * Design and maintain Genie Spaces and Genie Ontologies that accurately represent business entities, relationships, metrics, and terminology.
 * Establish best practices for semantic modeling, governed metrics, business metadata, and AI-ready datasets.
 * Evaluate and implement emerging Databricks AI capabilities to improve self-service analytics and business productivity.
 * Collaborate with Data Science and MLOps teams to enable machine learning, generative AI, and advanced analytics initiatives.

Data Architecture, Governance & Operational Excellence

 * Design scalable Lakehouse architectures supporting analytics, reporting, AI, and operational data products.
 * Partner with Data Governance teams to implement enterprise governance using Unity Catalog, including data lineage, fine-grained security, metadata management, and access controls.
 * Champion automated testing, monitoring, observability, data quality, and production reliability.
 * Serve as the technical escalation point for complex production issues and lead root cause analysis and continuous improvement efforts.
 * Drive platform modernization initiatives while balancing delivery, scalability, maintainability, and operational excellence.

Cross-Functional Partnership

 * Collaborate with Business Intelligence, Product, Platform Engineering, Security, and MLOps teams to deliver trusted, scalable data products.
 * Translate business requirements into modern technical solutions that support enterprise reporting, analytics, AI, and strategic decision-making.
 * Contribute to technical roadmaps, platform strategy, and the continued evolution of Penn Foster Group's Data & Analytics capabilities., * Role Assessment: Candidates will be required to complete a role-specific assessment as the first step in the hiring process. Assessment results will be used to evaluate alignment with the position and determine next steps in the process.
 * Recorded Video Interviews: All video interviews conducted during the hiring process will be recorded to support internal hiring consistency and process improvement.
 * Background Checks: Employment with Penn Foster Group is contingent upon successfully completing applicable pre-employment screening requirements, which may include verification of employment history, education, and criminal background, where permitted by law.
 * Identity Verification (Form I-9): As part of our onboarding process, all hires must complete employment eligibility verification in compliance with federal law. This includes remote Form I-9 verification, which may require an in-person identity verification step with an authorized representative.
 * On-Camera Work Environment: We operate in a highly collaborative, remote, on-camera culture when working remotely. Employees are expected to be on camera during meetings unless extenuating circumstances apply.


## Related Videos

- [PySpark - Combining Machine Learning & Big Data](https://www.wearedevelopers.com/videos/44-pyspark-combining-machine-learning-big-data)
- [HTTP headers that make your website go faster](https://www.wearedevelopers.com/videos/1676-http-headers-that-make-your-website-go-faster)
- [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94)
- [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works)
- [Data Science in Retail](https://www.wearedevelopers.com/videos/586-data-science-in-retail)
- [Empowering Retail Through Applied Machine Learning](https://www.wearedevelopers.com/videos/976-empowering-retail-through-applied-machine-learning)

## Related Articles

- [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk)
- [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know)
- [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)
- [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers)