Technical Lead/Sr SW Engineer - Infrastructure Telemetry & Capacity Planning Platform
BayOne
San Francisco, CA, United States
1 day ago
- Discuss this with your agent
- Open in Claude
- Open in ChatGPT
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Role details
Contract type
Contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
10 years minimum
Compensation
$197,600
Working hours
Regular working hours
Job source
Tech stack
Java (Programming Language)
Agile Methodology
Amazon Web Services
Big Data
Code Coverage
Software Quality
Continuous Integration
Data Stores
Cursor
DevOps
Distributed Computing Environment
Amazon DynamoDB
+17 more
Python (Programming Language)
MongoDB
NoSQL
Prometheus
Software Engineering
Data Streaming
Parquet
Data Logging
Graphics Processing Unit (GPU)
System Availability
Grafana
Apache Spark
Kubernetes
Apache Flink
Apache Kafka
Data Pipelines
Docker
Job description
We are seeking an experienced Technical Lead contractor to deliver hands-on software development for our Infrastructure Telemetry and Capacity Planning platform. This engagement covers the collection, streaming, processing, and persistence of telemetry data from various sources that contains utilization of servers and GPUs across our fleet, enabling efficient querying and dashboarding for capacity planning. The ideal contractor hits the ground running, engages actively in architecture design discussions, and drives execution with minimal ramp time.
- Scope of Engagement - The contractor will serve as an active technical contributor and platform specialist working closely with cross-functional teams, focused on the design, development, and delivery of an end-to-end telemetry pipeline that turns raw server and GPU metrics into a reliable, queryable capacity planning backbone.
- Telemetry Pipeline Design & Architecture - Participate in end-to-end solution design for streaming telemetry ingestion from servers (utilization, memory, thermal, power, and workload metrics), contributing to data flow models, schema design, and error-handling strategies. Recommend appropriate patterns (streaming vs. batch, event-driven ingestion, EMR-based processing) based on data volume, cardinality, and latency requirements for capacity planning use cases.
- Development & Implementation - Develop production-grade ingestion and processing pipelines following team coding standards, peer review practices, and CI/CD workflows. Build streaming collectors/agents for server telemetry, implement processing jobs on EMR (Spark), and persist processed data in NoSQL stores like Trino-queryable Lakehouse formats (e.g., Parquet). Write adequate test coverage and instrument pipelines with logging and monitoring for operational observability.
Requirements
- 10 years of software development experience, with demonstrated technical execution on complex, large-scale data or infrastructure platforms
- Strong proficiency in Java and/or Python; hands-on experience with Spark and distributed data processing
- Deep, hands-on experience with AWS EMR (or Spark) for large-scale data processing pipelines
- Experience with streaming data ingestion technologies (e.g., Kafka, Flink) for high-volume telemetry data.
- Solid experience with NoSQL data stores (e.g., DynamoDB, MongoDB) OR Lakehouse/query engines such as Trino, including schema and partitioning strategies for efficient querying at scale
- Experience collecting and processing infrastructure telemetry using tools such as Prometheus, Grafana.
- Proven expertise in cloud-native architecture - high availability, extensibility; AWS preferred
- Familiarity with containerized deployment environments (Kubernetes, Docker) and CI/CD pipelines
- Strong experience using AI-assisted development tools (e.g., Cursor, Windsurf, or equivalent) to accelerate delivery and maintain code quality
- Strong analytical and problem-solving skills with the ability to operate independently in an ambiguous environment, * Working knowledge of capacity planning concepts - utilization forecasting, resource trending, and translating telemetry into actionable capacity signals
- Experience building dashboards and visualization layers for operational and reporting.
- Experience with Agile development methodologies and DevOps practices.
Apply for this position
This job is hosted externally. Click below to view the full posting and apply.
Apply on www.dice.com
Prepare application
- Draft this with your agent
- Open in Claude
- Open in ChatGPT
Good distractions
Loading talks and stories from around this roleβ¦