Data Pipeline Engineer
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Tech stack
+10 more
Job description
We are seeking a Data Pipeline Engineer to help build and manage the imagery data pipeline behind perception model development for a longstanding Department of War client. The platform and pipeline are new, and much of the design, process, and tooling is still being decided. The engineer will do hands-on pipeline work, provide technical oversight of the integrator and vendors building parts of the system, and act as a technical point of contact between the government team, model developers, and end users., The program runs much like a startup. We need someone who takes ownership of problems, not someone who completes assigned tasks and stops there. When the engineer spots an issue, such as a recurring data quality problem or a broken handoff between teams, we expect them to bring together the platformβs users, the integrator, and the government team, work out a solution, and see it through with minimal oversight. The right candidate is comfortable with ambiguity and willing to put structure in place where none exists yet., n \n
-
Build, operate, and improve the pipelines that ingest, curate, label, and deliver EO/IR imagery and metadata to perception model developers, primarily on Databricks with supporting custom software. \n
-
Monitor data quality across the pipeline, and trace and fix problems with ingest, formats, labeling, metadata, and lineage. \n
-
Provide technical oversight of integrator and vendor work: review designs, code, and deliverables; push back on weak technical approaches; and give the government team an independent assessment of quality and risk. \n
-
Find problems and gaps in the pipeline or platform without waiting for them to be assigned, and lead the effort with the government team, integrator, users, and model developers to resolve them. \n
-
Turn operational needs from non-technical subject matter experts into data requirements and pipeline changes, and guide junior integrator engineers in implementing them. \n
-
Coordinate with stakeholders across multiple military services, vendors, other SETA contractors, and the model development community. \n
-
Document pipeline architecture, data standards, processes, and runbooks while the platformβs conventions are still being set. \n
-
Write clear technical summaries, status updates, and issue papers for government leads, and over time contribute to material for senior leaders. \n
-
Recommend improvements to tools, processes, and technical approaches as the platform matures and adds new sensor modalities. \n
Requirements
Prior management experience is not required. This position is meant for a strong engineer who wants to grow into technical leadership. As the platform matures and the engineer earns the clientβs trust, the role will shift toward overseeing vendor work, planning pipeline development, and coordinating across programs and with senior stakeholders., * Bachelorβs degree in computer science, data science, software engineering, information systems, engineering, or a related technical field. \n
-
Five or more years of hands-on experience in data engineering, MLOps, software engineering, or a closely related technical role. \n
-
Working proficiency with a standard data engineering stack: Python, SQL, distributed processing such as Spark, data modeling, data versioning, and pipeline orchestration. \n
-
Experience building or operating data pipelines that support ML model development, including data curation, preparation, and labeling workflows. \n
-
The technical depth to evaluate vendor architectures, code, and deliverables independently and to hold oneβs own with experienced engineers. \n
-
A demonstrated record of taking ownership: identifying problems outside oneβs assigned scope and driving them to resolution. \n
-
Ability to explain technical issues clearly, in writing and in person, to audiences ranging from non-technical users to senior engineers. \n
-
Comfort working on a new platform where requirements are incomplete and priorities shift. \n
-
Ability to work on-site in Arlington, Virginia, with travel up to 5 percent. \n, n \n
-
Experience with Databricks, such as Unity Catalog, Delta Lake, Workflows, or MLflow. \n
-
Experience working with EO/IR imagery, full-motion video, or other sensor data. Familiarity with additional modalities such as SAR or LiDAR is a plus. \n
-
Experience with computer vision or perception model development (detection, classification, tracking), and an understanding of what model developers need from training and evaluation data. \n
-
Experience with data labeling and annotation tools and workflows. \n
-
Experience with cloud platforms, particularly AWS GovCloud. \n
-
Experience supporting Department of War, federal, or other complex multi-stakeholder environments. \n
-
Familiarity with Agile/Scrum delivery, backlog management, and writing requirements or user stories in tools such as Jira or Confluence.
Benefits & conditions
n
-
Experience overseeing contractor or vendor technical work, or leading a small technical team or effort. \n
-
Experience at a startup or on a team standing up a new platform or capability. \n
\n
\n
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦