> Markdown version of [/jobs/ext/3397360-senior-data-engineer](https://www.wearedevelopers.com/jobs/ext/3397360-senior-data-engineer). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Senior Data Engineer - **Company:** ALPACA - **Location:** New York, NY, United States (Remote available) - **Experience:** Expert - **Contract:** Permanent contract - **Skills:** Application Programming Interfaces (APIs), Artificial Intelligence, Airflow, Data Analysis, Apache HTTP Server, Cloud Database, Cloud Engineering, Customer Data Management, Data as a Services, Information Engineering, Data Infrastructure, Extract Transform Load (ETL), Data Security, DevOps, Distributed Systems, Python (Programming Language), Octopus Deploy, Open Source Technology, Cloud Services, Ansible, Cloudera, Runbook, SQL Databases, Data Streaming, Google Cloud, Build Server, Build Management, Core Data, Debezium, Kubernetes, Low Latency, Apache Kafka, Data Management, Presto, Terraform, Looker Analytics, Data Pipelines, Docker - **Published:** September 10, 2026 - **Apply:** https://www.manhattanjobs.com/job.asp?id=3385115808&tx=ZT2928THD&pt=1&aff=0B19D771-A501-4A5E-8338-2A822B784D54&utm_source=Job%20Feed&utm_medium=textkernel&utm_campaign=DE&utm_term=0B19D771-A501-4A5E-8338-2A822B784D54 ## About the Role * 5+ years of experience in Data Engineering, including 2+ years building and operating scalable, low-latency data platforms handling > 100M events/day. * Strong hands-on experience running data infrastructure on Kubernetes, with cloud-native tooling like Docker and Helm. * Production experience with IaC: Terraform, Ansible, and ArgoCD (or equivalents). * Deep knowledge of distributed systems (storage, transactions, and query processing) with hands-on experience operating open-source query engines like Trino or Presto. * Strong experience with object storage and open table formats, specifically Apache Iceberg. * Experience with streaming and CDC systems: Kafka, Redpanda, and Debezium. * Hands-on experience with orchestration frameworks (Airflow) and ELT tools (Airbyte). * Strong working knowledge of Python and SQL for building pipelines and platform tooling. * Experience with Google Cloud Platform and its data services (GCS, Cloud Build, Cloud SQL, Dataproc, etc); or related experience with other cloud services. * Ability to thrive in a fast-paced startup environment and adapt infrastructure to rapidly changing needs. Nice to Haves: * Experience with semantic/metrics layers (Cube, dbt, Looker). * Familiarity with transformation frameworks (dbt). * Familiarity with reverse ETL tooling (Hightouch) * Familiarity with data catalog and lineage tooling (OpenMetadata, Datahub) * Experience with data access control and governance frameworks (Apache Ranger) ## Description We're searching for passionate individuals eager to contribute to Alpaca's rapid growth. If you align with our core values-Stay Curious, Have Empathy, and Be Accountable-and are ready to make a significant impact, we encourage you to apply. Your Role: We are seeking a Senior Data Engineer to help design and build the next generation of our Data Platform as we continue to scale to larger customers and new jurisdictions. At Alpaca, Data Engineering encompasses financial transactions, customer data, API logs, system metrics, augmented data, and third-party systems that impact decision making for both internal and external stakeholders. We process hundreds of millions of events daily, and this number continues to grow as we onboard new customers and products. We prioritize open-source technologies in our data stack while leveraging Google Cloud Platform (GCP) as the foundation for our data infrastructure. This spans batch and stream ingestion, transformation, and consumption layers for BI/Reporting, AI/agent interfaces (MCP), and external third-party sinks. We also oversee data experimentation, cataloging, and monitoring/alerting systems. Our team is 100% distributed and remote. Responsibilities: * Design, build, and evolve the core data platform infrastructure e.g., distributed query engines, orchestration, warehousing, cataloging, and more. * Own our lakehouse infrastructure as code, managing deployments through Terraform and Ansible on Kubernetes. * Build and maintain low-latency streaming and CDC ingestion pipelines, as well as batch ingestion paths landing in Iceberg. * Develop and scale our BI landscape so downstream teams and agents get performant, self-serve access to lakehouse data. * Enforce platform reliability best practices, including monitoring and alerting, on-call rotations, incident response, maintenance windows, runbooks, and SLAs. * Partner with DevOps, Analytics Engineering, and other stakeholders to close infrastructure gaps and support new data requirements. ## Related Videos - [Alibaba Big Data and Machine Learning Technology](https://www.wearedevelopers.com/videos/37-alibaba-big-data-and-machine-learning-technology) - [Docker Compose: Rediscovered](https://www.wearedevelopers.com/videos/1978-docker-compose-rediscovered) - [Shifting Stress to Progress— Understanding DevOps to do DevOps Better](https://www.wearedevelopers.com/videos/268-shifting-stress-to-progress-understanding-devops-to-do-devops-better) - [From Messy Queries to Scalable Systems - How Data Engineering actually works](https://www.wearedevelopers.com/videos/100203-from-messy-queries-to-scalable-systems-how-data-engineering-actually-works) - [Docker build without Docker](https://www.wearedevelopers.com/videos/100114-docker-build-without-docker) - [From DevOps to Scaled DevOps: How We’re Rebuilding Continuous Delivery as a Platform](https://www.wearedevelopers.com/videos/100018-from-devops-to-scaled-devops-how-we-re-rebuilding-continuous-delivery-as-a-platform) ## Related Articles - [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story) - [How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again](https://www.wearedevelopers.com/magazine/751-how-we-built-a-worry-free-system-that-runs-for-10-years-and-what-we-d-do-again) - [Top Big Data Technologies That You Need to Know](https://www.wearedevelopers.com/magazine/108-top-big-data-technologies-that-you-need-to-know) - [Highest Paying Tech Companies for Developers](https://www.wearedevelopers.com/magazine/220-highest-paying-tech-companies-for-developers) - [Data Engineer Salary UK](https://www.wearedevelopers.com/magazine/253-data-engineer-salary-uk) - [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)