Software Engineer, Sr. -Remote

Along with our partner, we are looking for a Senior Software Engineer to take validated engineering and turn it into production software that customers can trust while operating on hardware worth tens of millions of dollars. This is a systems engineering and safety-focused software role, centered on failure handling, safe defaults, graceful degradation, observability, operational documentation, installation, upgrades, and rollback., You will own production software end to end, working closely with a small, senior team and maintaining a high standard of evidence and reproducibility. The role is fully remote, with daily collaboration across the team and occasional evening work during hardware validation campaigns. Native/Bilingual English proficiency in written, spoken, and read English is required., * Own production software from prototype through deployment, hardening, and ongoing operation.

• Design and implement software with failure modes, safe defaults, graceful degradation, and operational recovery in mind.

• Develop and maintain long-running Python services and daemons.

• Work with concurrency, subprocesses, signal handling, systemd, packaging, and software distribution.

• Diagnose and troubleshoot Linux systems, including process and signal behavior, resource control, kernel, and driver-layer issues.

• Operate productively on remote bare-metal Linux machines using Secure Shell (SSH) without a graphical interface.

• Develop software that operates on physical systems, infrastructure, devices, or hardware.

• Perform failure-mode analysis and address identified reliability and safety gaps.

• Build regression test coverage for production failure scenarios.

• Develop observability and structured logging that allow operators to reconstruct software behavior and the reasons behind it.

• Build packaging, installation, upgrade, and rollback processes suitable for customer environments.

• Document software behavior, operations, failure modes, and technical processes for customer engineering teams.

• Participate in architecture, validation, and technical problem-solving activities.

• Support occasional evening work during hardware validation campaigns.

• Work with reproducible evidence and maintain records of the conditions under which results are measured.

• 7+ years of experience shipping and personally owning production software that other people depend on.

• Deep production Python experience, including long-running services and daemons, concurrency, subprocess and signal handling, systemd, packaging and distribution, and structured logging.

• Proven experience taking software from prototype to production, including hardening, test coverage, failure-mode analysis, observability, safe defaults, graceful degradation, and operational documentation.

• Strong Linux fundamentals, including process and signal semantics, systemd, resource control, and kernel and driver-layer troubleshooting.

• Ability to work on remote bare-metal machines over Secure Shell (SSH) without a graphical interface.

• Experience with software operating on physical systems, such as device-level software, drivers, embedded systems, telemetry, industrial or process control, robotics, or data center and accelerator infrastructure.

• Native/Bilingual English proficiency, including strong written English suitable for co-authoring technical documentation for a customer’s engineering team.

• Ability to maintain at least six hours of daily overlap with United States Mountain Time (Phoenix, UTC-7).

• Willingness to work as a full-time, dedicated team member and fully integrate with the team.

Preferred Qualifications

• Experience with Rust, particularly performance-sensitive production components.

• Experience with real-time or latency-sensitive systems where missed deadlines are correctness failures.

• Experience in safety-critical software, including industrial control, medical devices, automotive, or aerospace.

• Experience with data center or accelerator infrastructure.

• Experience packaging and deploying software into environments outside your direct control, including installers, agents, restricted or air-gapped networks, and customer change-control processes.

• Production on-call ownership for systems with a meaningful blast radius.

• Experience with Go.

• Experience with Kubernetes-based deployment tooling.

• Eligibility to obtain a United States visitor visa for occasional customer-site and industry-conference travel.

Skills & Competencies

• Production Python and systems engineering

• Linux systems engineering

• Distributed or infrastructure software

• Long-running services and daemons

• Concurrency and subprocess management

• Signal handling and systemd

• Packaging and software distribution

• Structured logging and observability

• Failure-mode analysis

• Regression testing

• Safe defaults and graceful degradation

• Production hardening

• Linux troubleshooting

• Kernel and driver-layer troubleshooting

• Physical-system and hardware-integrated software

• Technical documentation

• Operational reliability and safety engineering

• Reproducibility and evidence-based engineering

• Independent ownership

• Direct technical communication

Education & Experience

Education:

• No specific degree is required.

• An electrical engineering degree is explicitly not required.

Experience:

• 7+ years shipping and personally owning production software.

• Relevant experience with software operating on physical systems is required.

• Experience taking software from prototype through production hardening is required.

Work Arrangement & Schedule

• Work Arrangement: Fully remote

• Employment Type: Full-time, dedicated team member ⓘ View potential timezone restrictions

• Time Zone: At least 6 hours of daily overlap with United States Mountain Time (Phoenix, UTC-7), * Native/Bilingual English proficiency in reading, writing, and speaking is required.

• All work is covered by a confidentiality and intellectual property assignment agreement executed directly with AIONTRACKS before the first day.

• The role requires full-time dedication and integration within the partner’s team.

• The following are not required: machine learning modeling or research, data science, front-end development, an electrical engineering degree, and prior energy-sector experience.