> Markdown version of [/jobs/ext/2737388-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/2737388-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data Engineer - **Company:** The University of Texas - **Location:** Austin, TX, United States (Remote available) - **Experience:** Expert - **Salary:** $125,000.0 - $143,712.0 - **Contract:** Temporary to permanent - **Skills:** Agile Methodology, Artificial Intelligence, Airflow, Amazon Web Services, Amazon S3, Data Analysis, Application Integration Architecture, JIRA, Big Data, Cloud Computing, Cloud Database, Information Systems, Data Architecture, Data Validation, Information Engineering, Data Governance, Data Hub, Data Infrastructure, Extract Transform Load (ETL), Data Transformation, Data Systems, Software Debugging, Distributed Computing Environment, Distributed Data Store, Github, Python (Programming Language), Machine Learning, Performance Tuning, Scrum Methodology, DataOps, SQL Stored Procedures, SQL Databases, Data Streaming, Systems Integration, Parquet, Enterprise Software Applications, Cloud Platform System, Test-Driven Development (TDD), Data Ingestion, Delivery Pipeline, Apache Spark, Data Lakes, Pyspark, Information Technology, Apache Flink, Data Analytics, Apache Kafka, Data Management, Machine Learning Operations, Terraform, Data Pipelines, Databricks - **Published:** September 5, 2026 - **Apply:** https://jobs.localjobnetwork.com/apply/add/85670971/1 ## About the Role Must be authorized to work in the United States on a full-time basis for any employer without sponsorship., * Bachelor's orMaster's degree in Computer Science, Information Systems, Engineering, or equivalent professional experience. * 5+ years of experience designing, implementing, andoptimizingcomplex, production-grade datapipelinesor enterprise-scale data platforms. * 5 years of experience in cloud-based data engineering using Databricks and Amazon Web Services (AWS) (e.g., Glue, S3, Lambda, Redshift). * 3+ years of experience managing or leading teams of data and/or software engineers, including mentorship, performance management, and project delivery. * Expertisein Python,PySpark, and SQL, withstrongunderstanding of data modeling, stored procedures, and scalable data transformations. * Proven experience architecting and implementing ETL/ELT solutions across relational, non-relational, andlakehouseenvironments (e.g., Delta Lake, Parquet, or Iceberg). * Experience designing and managing CI/CD pipelines andinfrastructureascode(IaC) using tools such as Databricks Repos, CDK, Terraform, or GitHub Actions. * Demonstrated knowledge of test-driven development (TDD) and data quality frameworks, ensuring reliability and reproducibility across data workflows. * Deep understanding of data governance, security, and compliance standards in cloud environments. * Excellent analytical, problem-solving, and debugging skills across distributed data systems. * Proven ability to communicate complex technical concepts clearly to both technical and non-technical audiences. * Experience supervising, mentoring, and guiding junior team members on technical and professional development. Equivalent combination of relevant education and experience may be substituted as appropriate., * 8+ years of experience in Data Engineering or related fields, including 5+ years of hands-on experience building andoptimizingdata pipelines on Databricks or similar large-scale data platforms. * Proven experience implementinglakehousearchitectures leveraging Databricks Delta Lake, Delta Live Tables, and Unity Catalog for governance and scalability. * Experience designing AI-ready data platforms and integrating machine learning pipelines using tools such asMLflowor model registry frameworks. * 3+ years of experience managing or leading cross-functional technical teams, fostering collaboration between Data Engineering, Analytics, and AI/ML teams. * 5+ years of experience with Agile software development methodologies and project tracking systems such as JIRA. * Expertisein distributed data processing and streaming frameworks, such as Apache Spark, Kafka, Flink, or Airflow, for orchestration and automation. * Strong familiarity with data observability, cost optimization, and performance tuning in Databricks and cloud-nativearchitecture. * Professional certifications such as Databricks Certified Data Engineer Professional or AWS Solutions Architector AWSData Analytics Specialty are highly desirable. * Demonstrated ability to introducenew technologiesand best practices to modernize existing data environments and promote AI/analytics maturity across the organization. * Passion for continuous learning and staying current with emerging technologies in data engineering, AI integration, and Databricks ecosystem advancements., A criminal history background check will be required for finalist(s) under consideration for this position., * E-Verify Poster (English and Spanish) [PDF] * Right to Work Poster (English) [PDF] * Right to Work Poster (Spanish) [PDF] ## Description This is a fixed term position that is expected to continue for a 1-year limited term from start date with a possibility for extension. Flexible work arrangements are available for this position, including the ability to work 100% remotely. Remote work for individuals who reside outside Texas but within the United States and its territories will be considered and requires Central Office approval. This position provides life/work balance with typically a 40-hour work week and travel limited to training (e.g., conferences/courses). Enterprise Technology is dedicated to supporting the mission of the University of Texas at Austin of unlocking potential and preparing future leaders of the state. Your skills will make a difference. You'll be working for a university that is internationally recognized for research and the work you do will make a difference in the lives of our students, faculty and staff. If you're the type of person that wants to know your work has meaning and impact, you'll like working for our campus., The Lead Data Engineer for the UT Data Hub improves university outcomes and advances the UT mission to transform lives for the benefit of society by increasing the useability and value of institutional data. You will lead senior data engineers and data engineers to create complex data pipelines within UT's cloud data ecosystem in support of academic and administrative needs. In collaboration with our team of data professionals, you will help build and run a modern data hub to enable advanced data-driven decision making for UT. You will leverage your creativity to solve complex technical problems and build effective relationships through open communication within the team and outside partners. ResponsibilitiesTechnical Leadership: * Design, architect, and deliver production-grade, scalable data pipelines and AI-ready data platforms using Databricks,AWScloud-native services and modern data engineering frameworks. * Lead end-to-end implementation oflakehousedata pipelines, ensuring performance, reliability, and cost efficiency. * Championindustrybest practices for data engineering. * Conduct andparticipatein peer code reviews tomaintaincode quality and consistency across the team. * Proactivelyidentifyand resolve bottlenecks in data ingestion, transformation, and orchestration processes using Databricks Delta Live Tables, Spark optimizationtechniques, and workflow automation. * Implement systems for data quality, observability, governance, and compliance using tools such as Unity Catalog, Delta Lake, and data validation frameworks. * Lead technical knowledge-sharing sessions on topics such as AI/ML integration, datalakehousearchitecture, and emerging data technologies. Project Management: * Define project milestones, timelines, and deliverables for data and AI initiatives, ensuringtimelyand high-quality outcomes. * Collaborate with both internal and external stakeholderssuch as data architects, system architects, business users, Agile teammembers, and other D2I internal groups. * Manage project priorities, sprint planning, and team workloads while balancing innovation with delivery. * Communicate risks, dependencies, and resource constraints effectively, and develop mitigation plans for on-time project delivery. Team Management and Leadership: * Supervise and mentor a team of Data Engineers (2-5 individuals) working on cloud, Databricks, and AI pipeline initiatives. * Foster a culture of continuous learning, experimentation, and technical excellence, encouraging engineers to explore AI and automation use cases. * Participate in recruiting, onboarding, and developing data engineering talent with strong Databricks and AI skillsets. * Conduct performance reviews, set development goals, and create individualized growth plans for team members. * Encourage collaboration across Data, AI/ML, Analytics, and Infrastructure teams to drive cross-functional success. Communication: * Provide regular updates on project progress, technical challenges, andprojectmilestones to both technical and business stakeholders. * Translate complex technical concepts related to Databricks, AI, and data architecture into clear narratives for non-technical audiences. * Foster a transparent communication culture and provide actionable feedback to promote a growth mindset. * Ensure all data engineering processes, architectures, and standards are well-documented for reuse, governance, andknowledgecontinuity. Innovation and Other Duties: * Stay current with advancements in AI, data engineering, and Databricks ecosystem, evaluating new tools and frameworks for potential adoption. * Pilot and promote innovative solutions such as AI-assisted data quality checks, data observability automation, and intelligent pipeline optimization. * Perform other duties as assigned, contributing to the organization's data-driven and AI-enabled transformation., The retirement plan for this position is Teacher Retirement System of Texas (TRS), subject to the position being at least 20 hours per week and at least 135 days in length., Employees may be required to report violations of law under Title IX and the Jeanne Clery Disclosure of Campus Security Policy and Crime Statistics Act (Clery Act). If this position is identified a Campus Security Authority (Clery Act), you will be notified and provided resources for reporting. Responsible employees under Title IX are defined and outlined in HOP-3031. ## Related Videos - [Innovating Developer Tools with AI: Insights from GitHub Next](https://www.wearedevelopers.com/videos/1268-innovating-developer-tools-with-ai-insights-from-github-next) - [Improving quality with Agentic AI with Rovo Dev and Xray](https://www.wearedevelopers.com/videos/2005-improving-quality-with-agentic-ai-with-rovo-dev-and-xray) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Bringing AI Model Testing and Prompt Management to Your Codebase with GitHub Models](https://www.wearedevelopers.com/videos/1536-bringing-ai-model-testing-and-prompt-management-to-your-codebase-with-github-models) - [Collaboration Quantified: Lessons from Open Source Developer Networks](https://www.wearedevelopers.com/videos/1422-collaboration-quantified-lessons-from-open-source-developer-networks) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Coffee with Developers - Maria Apazoglou - Making AI understandable for all in production](https://www.wearedevelopers.com/magazine/475-coffee-with-developers-maria-apazoglou-making-ai-understandable-for-all-in-production) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)