> Markdown version of [/jobs/ext/2836029-data-engineer](https://www.wearedevelopers.com/jobs/ext/2836029-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Data Engineer - **Company:** Ai, Inc - **Location:** San Francisco, CA, United States - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Query Performance, A/B Testing, Application Programming Interfaces (APIs), Airflow, Amazon Web Services, Apache HTTP Server, Automation of Tests, Microsoft Azure, BigQuery, Databases, Continuous Integration, Data as a Services, Information Engineering, Data Infrastructure, Extract Transform Load (ETL), Data Systems, Data Warehousing, Database Queries, Dimensional Modeling, Python (Programming Language), Performance Tuning, Standard Sql, DataOps, Data Streaming, Data Processing, Data Storage Technologies, Data Ingestion, Delivery Pipeline, Snowflake, Grafana, Git, Cloudformation, Event Driven Architecture, Containerization, Data Lakes, Kubernetes, Performance Monitor, Integration Frameworks, Apache Kafka, Terraform, Stream Analytics, Software Version Control, Data Pipelines, Docker, Amazon Redshift, Databricks - **Published:** September 10, 2026 - **Apply:** https://startup.jobs/senior-data-engineer-way-7225891 ## About the Role * Minimum 5+ years of hands-on data engineering experience with increasing responsibility * Preferred 7+ years in data engineering, analytics engineering, or data platform roles * Proven track record of building data systems from scratch or leading data infrastructure transformations * Experience working as a solo data engineer or in small, autonomous data teams Required Technical Skills * Python proficiency required - demonstrated experience building data pipelines, ETL frameworks, and automation scripts * SQL expertise - advanced knowledge of complex queries, performance optimization, and data modeling * Strong experience with cloud platforms (AWS, GCP, or Azure) and their data services * Proficiency with data lake technologies (Delta Lake, Apache Iceberg, or Apache Hudi) * Experience with data orchestration tools (Apache Airflow, Prefect, Dagster, or similar) * Knowledge of streaming data technologies (Apache Kafka, Kinesis, Pub/Sub) * Familiarity with data warehouse technologies (Snowflake, BigQuery, Redshift, Databricks) * Understanding of containerization (Docker, Kubernetes) and infrastructure as code * Experience with version control systems (Git) and collaborative development workflows, * Background in real-time analytics and event-driven architectures * Previous experience in startup or fast-paced environments * Understanding of data privacy regulations (GDPR, CCPA) and security best practices * Experience with performance monitoring and observability tools * Knowledge of dimensional modeling and data warehouse design patterns ## Description We are seeking an experienced Senior Data Engineer to establish and lead our data infrastructure as an early member of our data team. This role will be responsible for building our entire data platform from the ground up, implementing a comprehensive data lake and pipeline architecture, and establishing a culture of data-driven decision-making throughout our organization. The ideal candidate will serve as both a technical leader and data advocate, driving automation excellence while flexing into data science and analytics responsibilities as needed., * Design and implement a comprehensive data lake architecture using modern cloud-native technologies * Build scalable ETL/ELT pipelines for real-time and batch data processing across all data sources * Establish data ingestion frameworks to collect data from application APIs, third-party services, and databases * Architect automated data quality monitoring, validation, and alerting systems * Create robust data warehousing solutions optimized for analytics and business intelligence * Implement DataOps practices with automated testing and deployment pipelines (CI/CD for data) Data Engineering & Analysis * Develop and maintain Python-based data processing frameworks and utilities * Build an automated data pipeline orchestration using Apache Airflow or similar tools * Create streaming data processing solutions using Apache Kafka, Kinesis, or Pub/Sub * Implement infrastructure as code for all data platform components (Terraform, CloudFormation) * Establish feature stores and data models that support both operational and analytical workloads * Optimize data storage costs and query performance across the entire platform * Collaborate with product and business teams to identify key metrics, KPIs, and analytical requirements * Build automated reporting dashboards and self-service business intelligence tools * Support predictive modeling initiatives and A/B testing frameworks ## Related Videos - [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered) - [How a Small Team Shrank a Microsoft Monorepo by 94%](https://www.wearedevelopers.com/videos/1236-how-a-small-team-shrank-a-microsoft-monorepo-by-94) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Docker build without Docker](https://www.wearedevelopers.com/videos/100114-docker-build-without-docker) - [Git for Code Reviews](https://www.wearedevelopers.com/videos/429-git-for-code-reviews) - [Modern Data Architectures need Software Engineering](https://www.wearedevelopers.com/videos/1030-modern-data-architectures-need-software-engineering) ## Related Articles - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [How to Become an AI Engineer](https://www.wearedevelopers.com/magazine/331-how-to-become-an-ai-engineer) - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [Navigating the AI Shift](https://www.wearedevelopers.com/magazine/629-navigating-the-ai-shift) - [Top-Paying Tech Jobs (with Salaries)](https://www.wearedevelopers.com/magazine/372-top-paying-tech-jobs-with-salaries)