Technical Lead/Sr SW Engineer - Infrastructure Telemetry & Capacity Planning Platform

BayOne
San Francisco, CA, United States
1 day ago
Apply on www.dice.com
Prepare application

Role details

Contract type
Contract
Employment type
Full-time (> 32 hours)
Experience level
Expert
Experience required
10 years minimum
Compensation
$197,600
Working hours
Regular working hours
Job source

Tech stack

Java (Programming Language) Agile Methodology Amazon Web Services Big Data Code Coverage Software Quality Continuous Integration Data Stores Cursor DevOps Distributed Computing Environment Amazon DynamoDB
+17 more
Python (Programming Language) MongoDB NoSQL Prometheus Software Engineering Data Streaming Parquet Data Logging Graphics Processing Unit (GPU) System Availability Grafana Apache Spark Kubernetes Apache Flink Apache Kafka Data Pipelines Docker

Job description

We are seeking an experienced Technical Lead contractor to deliver hands-on software development for our Infrastructure Telemetry and Capacity Planning platform. This engagement covers the collection, streaming, processing, and persistence of telemetry data from various sources that contains utilization of servers and GPUs across our fleet, enabling efficient querying and dashboarding for capacity planning. The ideal contractor hits the ground running, engages actively in architecture design discussions, and drives execution with minimal ramp time.

  • Scope of Engagement - The contractor will serve as an active technical contributor and platform specialist working closely with cross-functional teams, focused on the design, development, and delivery of an end-to-end telemetry pipeline that turns raw server and GPU metrics into a reliable, queryable capacity planning backbone.
  • Telemetry Pipeline Design & Architecture - Participate in end-to-end solution design for streaming telemetry ingestion from servers (utilization, memory, thermal, power, and workload metrics), contributing to data flow models, schema design, and error-handling strategies. Recommend appropriate patterns (streaming vs. batch, event-driven ingestion, EMR-based processing) based on data volume, cardinality, and latency requirements for capacity planning use cases.
  • Development & Implementation - Develop production-grade ingestion and processing pipelines following team coding standards, peer review practices, and CI/CD workflows. Build streaming collectors/agents for server telemetry, implement processing jobs on EMR (Spark), and persist processed data in NoSQL stores like Trino-queryable Lakehouse formats (e.g., Parquet). Write adequate test coverage and instrument pipelines with logging and monitoring for operational observability.

Requirements

  • 10 years of software development experience, with demonstrated technical execution on complex, large-scale data or infrastructure platforms
  • Strong proficiency in Java and/or Python; hands-on experience with Spark and distributed data processing
  • Deep, hands-on experience with AWS EMR (or Spark) for large-scale data processing pipelines
  • Experience with streaming data ingestion technologies (e.g., Kafka, Flink) for high-volume telemetry data.
  • Solid experience with NoSQL data stores (e.g., DynamoDB, MongoDB) OR Lakehouse/query engines such as Trino, including schema and partitioning strategies for efficient querying at scale
  • Experience collecting and processing infrastructure telemetry using tools such as Prometheus, Grafana.
  • Proven expertise in cloud-native architecture - high availability, extensibility; AWS preferred
  • Familiarity with containerized deployment environments (Kubernetes, Docker) and CI/CD pipelines
  • Strong experience using AI-assisted development tools (e.g., Cursor, Windsurf, or equivalent) to accelerate delivery and maintain code quality
  • Strong analytical and problem-solving skills with the ability to operate independently in an ambiguous environment, * Working knowledge of capacity planning concepts - utilization forecasting, resource trending, and translating telemetry into actionable capacity signals
  • Experience building dashboards and visualization layers for operational and reporting.
  • Experience with Agile development methodologies and DevOps practices.

Apply for this position

This job is hosted externally. Click below to view the full posting and apply.

Apply on www.dice.com
Prepare application

Good distractions

Loading talks and stories from around this role…