> Markdown version of [/jobs/ext/3604220-lead-data-engineer](https://www.wearedevelopers.com/jobs/ext/3604220-lead-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Lead Data Engineer - **Company:** ElasticSearch Inc. - **Location:** San Francisco, CA, United States (Remote available) - **Experience:** Expert - **Salary:** $113,000.0 - $179,000.0 - **Contract:** Permanent contract - **Skills:** Artificial Intelligence, Airflow, Apache HTTP Server, Computing Platforms, Bash Shell, Cloud Computing, Databases, Information Engineering, Data Governance, Data Infrastructure, Data Security, Data Warehousing, Dimensional Modeling, Elasticsearch, Python (Programming Language), Online Analytical Processing, Online Transaction Processing, Standard Sql, Software Deployment, SQL Databases, Systems Integration, Unstructured Data, Data Processing, Google Cloud, Apache Spark, Data Layers, Apache Flink, Apache Kafka, Terraform, Data Pipelines, User Identification, Programming Languages - **Published:** October 7, 2026 - **Apply:** https://www.builtincolorado.com/auth/login?destination=/job/lead-data-engineer/11532399 ## About the Role * 8+ years of experience in data engineering or related roles * Experience with Apache Airflow for orchestrating data workflows * Experience with Spark, Flink, Kafka, and dbt Labs for data processing and transformation * Understanding of Apache Iceberg for data table formats and data warehouse and lakehouse strategies * Skilled in using Terraform for infrastructure-as-code deployments * Proficient in programming languages, particularly Bash, Python, and SQL * Knowledge of database systems, including OLAP, OLTP, Document, and Vector databases * Experience with Google Cloud Platform (GCP) for cloud resource management * Bonus: Proven experience with data security and access controls for SOX Compliant data * Bonus: Experience and knowledge of the Elastic stack ## Description Lead Data Engineer responsible for designing, maintaining, and improving data pipelines and platform infrastructure across ingestion, streaming, transformation, and delivery. The role uses Airflow, Spark, Iceberg, Kafka, Flink, dbt, Terraform, and GCP. Responsibilities also include defining data contracts, documenting systems, optimizing performance, reviewing code, supporting access controls and compliance, and collaborating with stakeholders and teammates. The summary above was generated by AI Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale - unleashing the potential of businesses and people. The Elastic Search AI Platform, used by more than 50% of the Fortune 500, brings together the precision of search and the intelligence of AI to enable everyone to accelerate the results that matter. By taking advantage of all structured and unstructured data - securing and protecting private information more effectively - Elastic's complete, cloud-based solutions for search, security, and observability help organizations deliver on the promise of AI. What is The Role We're looking for a Lead Data Engineer to join our Data Engineering & Architecture team. In this role, you'll help grow a Data Platform that makes it easier for teams across our company to work with data confidently and effectively. Your work will directly support our business goals not just by keeping data flowing, but by turning it into something people can actually use and trust. This is a great opportunity for someone who enjoys both the technical craft of data engineering and the human side of it: collaborating with partners, sharing knowledge, and helping others get more out of the tools and systems you build together. What You Will Be Doing You'll design, maintain, and improve the pipelines and infrastructure that power our data platform. Day to day, that means working across the full data lifecycle - from ingestion and streaming to transformation and delivery - using tools like Apache Airflow, Spark, Iceberg, Kafka, Flink, and dbt Labs. Beyond the technical work, you'll spend meaningful time with stakeholders to understand what they need and help define how data should flow across systems. That includes co-creating data contracts, writing clear documentation, and guiding teammates on how to get the most out of the platform. A few other things you'll take on: * Monitor and refine pipeline performance and refine processes to keep things running smoothly. * Use Terraform to provision and manage resources on Google Cloud Platform (GCP). * Help steward the platform by reviewing code and ensuring good practices and standard processes are followed. * Process access requests and support compliance with platform policies., Leads a data engineering team and supervises one Data Engineer while designing, maintaining, and optimizing reliable data pipelines and infrastructure. The role integrates internal and external data sources, supports SaaS analytics dashboards and connected platforms, improves data-store performance, and ensures infrastructure quality, security, and uptime. Requires proficiency in Python, databases, distributed processing, orchestration, AWS, and machine learning. Top Skills: Amazon EmrAmazon KinesisAmazon S3Apache AirflowSparkAws EcsDockerLookerPandasPostgresPythonRedshiftScalaSnowflakeSQL Fullbay Lead Data Engineer 5 Days Ago In-Office or Remote Senior level Senior level Automotive * Software * Transportation Own Fullbay's Snowflake data platform end to end, including dimensional modeling, certified metric definitions, semantic layers, identity resolution, transformations, governance, security, cost management, source ingestion, and AI-agent enablement. Build tested SQL and transformation code, establish company-wide data standards, document systems thoroughly, and direct an architecture consultant. Provide guidance for self-service data use while serving as the final authority on definitions, access, and platform design. Top Skills: Change Data CaptureContinuous IntegrationSalesforceSnowflakeSQL Nexcess Lead Data Engineer 9 Days Ago Remote United States Senior level Senior level Software Leads and mentors the Data Engineering team while remaining hands-on in designing, building, and maintaining scalable data pipelines, warehouses, integrations, and infrastructure. Establishes architecture, reliability, quality, and performance standards; guides technical delivery; and resolves complex issues. Partners with Finance and business stakeholders to translate data needs into practical solutions, sets technical direction, evaluates technologies, and communicates strategy, risks, and recommendations to leadership. Top Skills: AirflowAWSAzureCi/CdCloud Data PlatformsData PipelinesData WarehousesDbtEltETLGoogle Cloud PlatformInfrastructure As CodeJavaKafkaPythonScalaSparkSQL What you need to know about the Colorado Tech Scene With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation. Key Facts About Colorado Tech * Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey) * Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3 * Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech * Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook) * Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures * Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute