How We Built a Worry-Free System That Runs for 10+ Years – And What We’d Do Again

The Problem: Big Science, Tiny Teams

Between 2012 and 2016, we built a control system for high-throughput tomography beamlines at PETRA III (DESY, Hamburg) and FRM II (Munich) – facilities where researchers scan materials at synchrotron and neutron sources to understand everything from aircraft composites to bone structure. When scientists from around the world arrive at DESY for their 3-day beamtime slot to scan materials at the molecular level, they expect 24/7 reliability.

Our system controlled everything from motors and detectors to metadata pipelines and experiment schedulers — all orchestrated via web interfaces and backend services.

The challenge? Two developers. Heterogeneous hardware stack. Zero tolerance for downtime during precious beamtime (costing thousands per hour). And a demand to revolutionize how we collect, store, and provide access to experimental data.

The result? A system that’s been running in production for 10+ years, supported hundreds of experiments, and contributed to dozens of scientific publications. Since reaching maturity in 2016, it’s required essentially zero maintenance — the definition of “worry-free.”

Here’s how we did it, what we’d do again, and what we learned.

System Impact at a Glance

Note on metrics: uptime, data volume, and incident counts are based on internal ops logs and storage monitoring.

Why Java? And Why It Worked

Back in 2012 – 2013, choosing Java in a scientific facility was controversial. Everyone was writing C++. Today they’re writing Python. We went a different way. Java sometimes gets dismissed as “slow” – but we found the opposite to be true when properly tuned. One of our core components – StatusServer – consistently responded in under 1ms, even under heavy load. Why did that matter? Because in distributed systems, latency compounds. Handling 1000 sequential requests in under a second (instead of 10+) meant we could keep up with hardware and operator demands in real time.

The numbers: Professional JMH benchmarks showed 370+ million operations per second for single-value retrieval, and 1.7–6 million ops/sec for full snapshot queries across 100 concurrent data streams. In practical terms, this meant 1000 sequential requests completed in under a second instead of 10+, keeping experiment control responsive during peak monitoring loads.

How did we achieve this? We went low-level: sun.misc.Unsafe for direct memory access, bypassing JVM safety checks where performance mattered most; ConcurrentSkipListSet for lock-free time-ordered storage with O(log n) lookups – critical for range queries over time-series data; AtomicReferenceArray with direct array access via Unsafe.arrayBaseOffset and arrayIndexScale for CPU cache-line optimization; Carefully tuned G1GC (-XX:MaxGCPauseMillis=10) to keep garbage collection pauses under 5ms even during heavy data ingestion from dozens of simultaneous instrument sources.

The ConcurrentSkipListSet choice was the key: we needed both fast concurrent writes from instrument data streams AND efficient time-range queries for historical snapshots. A hash map would have been faster for point lookups but couldn’t provide the ordered range semantics we required.

Using sun.misc.Unsafe was controversial even then – it’s unsupported API that breaks JVM guarantees. But for a component handling millions of operations per second while maintaining sub-millisecond latency, the performance gains justified the risk. We isolated this into the Snapshot class, heavily tested it, and could replace it if needed. We matched native C++ in reliability and beat it in performance where it counted. View the complete implementation with benchmarks on GitHub → https://github.com/scientific-software-hub/status-server

Java’s stability also mattered: when the facility upgraded from Debian 6 to 7, 8, 9 and all the way to today’s 12, teams using native C++ spent weeks debugging segfaults and rebuilding their stacks with new toolchains. Our Java components? We recompiled two JNI bindings and kept running.

Here’s our dead simple single executable fat jar, build by Maven:

<plugin>  
  <groupId>org.apache.maven.plugins</groupId>  
  <artifactId>maven-assembly-plugin</artifactId>  
  <configuration>  
    <archive>  
     <manifest>  
       <mainClass>org.desy.myserver.Main</mainClass>  
     </manifest>  
    </archive>  
    <descriptorRefs>  
      <descriptorRef>jar-with-dependencies</descriptorRef>  
    </descriptorRefs>  
   </configuration>  
</plugin>

This gave us:

• Immutable artifacts
• Zero dependency hell
• Instant rollbacks (just swap the JAR)
• Consistent behavior across test/prod environments

We implemented our high-level components (like metadata collection, status monitoring, data routing) in Java, and only used JNI for truly hardware-near elements, still distributed as jar files with native code on board. This decision also paid off in terms of maintainability. Java lets us build reusable components, package them into fat jars, and deploy them cleanly. We didn’t worry about compiler flags, memory corruption, or distro-specific libraries. It just worked.

In fact, this packaging approach — building a single fat executable JAR — offered many of the same benefits we now associate with Docker containers. Back in the day, we had a self-contained, immutable binary that included everything it needed to run: dependencies, config, even JNI stubs. You could drop it onto any Debian box with a JVM and it would behave the same, every time.

Today, developers take this for granted with containers, but the JVM gave us that superpower before containers went mainstream. It meant one-click rollbacks (just replace the JAR), seamless upgrades, and confidence that what passed in the test would behave identically in production.

A Couple of The Architecture Decisions That Mattered

Trade-off 1: In-Memory vs. Persistent Storage for StatusServer

❌ Rejected: Writing status to disk for durability
✅ Chose: RAM-only with eventual persistence to NeXus
Why: The experiment itself was the source of truth. StatusServer was an ephemeral monitoring layer. Disk I/O would have violated our <1ms SLA.

Trade-off 2: Monolith vs. Microservices (before it was called that)

❌ Rejected: Single Java process handling everything
✅ Chose: Separate components and servers per responsibility
Why: Isolation of failures and independent deployment cycles. If one of the low priority components crashed, the experiment itself wasn’t affected.

What We Got Right (And Why It Still Matters)

Our “meta-project” pattern in 2012 essentially invented monorepo-style dependency management before Bazel existed. Our isolation strategy mirrored what would later be called “microservices” - a component with its own storage and API. Our comprehensive data collection matched what modern observability platforms now advocate.

We didn’t stumble into these patterns by accident – we were deliberately looking beyond our immediate domain. While scientific computing conferences like ICALEPCS and NOBUGS provided valuable insights into domain-specific challenges, we also actively tracked what was happening in mainstream software engineering through books, industry conferences, and engineering blogs from companies operating at web scale.

This cross-pollination mattered. Scientific computing and mainstream software engineering were often on different timelines – practices that were cutting-edge at Netflix or Google in 2011 wouldn’t appear in research infrastructure for years. By actively bringing those ideas into our system early, we benefited from the hard-won lessons of teams running similar challenges (high availability, zero-downtime deployments, distributed systems) but at different scales and in different contexts. The key was staying open-minded and avoiding domain silos.

The constraints were real—two developers, zero downtime tolerance, mission-critical uptime—but the solutions came from combining domain expertise with broader industry practices. That’s how the best engineering emerges: not from staying in one community’s echo chamber, but from actively seeking knowledge across boundaries.

CI/CD Before CI/CD Was Cool

Our CI/CD pipeline in 2013 (before GitHub Actions existed):

1. Developer commits → Git hook triggers TeamCity
2. Maven builds component + runs integration tests
3. Meta-project aggregates components into beamline-specific bundles
4. Automated deployment to test environment
5. Manual promotion to production with one-click rollback

The key innovation was the meta-project pattern: a single POM file that defined which component versions composed each beamline’s deployment. Changing a version? Edit one line, and TeamCity would rebuild everything in under 5 minutes.

Here’s a snippet from the pom.xml:

<properties>  
  <status.server.version>2.3.1</status.server.version>  
  <dataformat.server.version>1.8.0</dataformat.server.version>  
</properties>

<dependencies>  
  <dependency>  
    <groupId>org.desy</groupId>  
    <artifactId>status-server</artifactId>  
    <version>${status.server.version}</version>  
  </dependency>  
  <dependency>  
    <groupId>org.desy</groupId>  
    <artifactId>dataformat-server</artifactId>  
    <version>${dataformat.server.version}</version>  
  </dependency>  
</dependencies>

We knew from the start that we couldn’t afford downtime — or broken updates. But in 2013, there were no GitHub Actions, nor GitLab CI, nor containers. So we rolled our own with TeamCity and Maven.

This gave us:

• Deployment frequency: multiple times per week
• Time-to-production: under 5 minutes for beamline-specific bundles
• Rollback rate: near-instant — just promote the previous build

That last point was critical. If something broke, we always had a stable fallback version ready. And yes — it saved us more than once.

By the numbers:

• Deployment frequency: 3-5x per week to test, 1-2x per week to production
• Time-to-production: 4 minutes average for full beamline bundle rebuild
• Rollback time: <60 seconds (just swap JARs + restart services)
• Failed deployments requiring rollback: ~8% (industry average: 15-25%)

In many ways, this mirrored what modern teams now achieve with Infrastructure as Code (IaC) practices. Our meta-projects defined infrastructure-like concerns — versioning, environment-specific configs, reproducible rollouts — all codified in XML and driven by Maven. Much like Terraform or Pulumi today, our TeamCity jobs were effectively declarative pipelines. Change a version in pom.xml, and the whole environment would shift accordingly, reproducibly, and revertibly.

Production and Test Were Separated — For Real

We took pride in having a truly isolated test environment. No shared configs, no backdoors into prod, no shortcuts. That made testing real.

Concretely, this meant:

• Separate database instances
• Isolated network segments (test couldn’t accidentally talk to prod hardware)
• Independent authentication (different SSH keys, different service accounts)
• Beamline-specific test configs (simulated experiments, not live beam)

It also made onboarding easier: you could break things in test, iterate fast, and promote confidently. Our TeamCity meta-project handled the rest.

If we were building this today, we’d achieve the same isolation using Kubernetes namespaces with network policies. Each environment would get its own namespace (beamline-test, beamline-prod), with NetworkPolicy resources enforcing that test pods can’t accidentally reach production services or hardware endpoints. The principle remains identical — strict boundaries prevent “just this once” shortcuts that inevitably cause outages — but the tooling has evolved. Back then, we used separate VLANs and firewall rules; today, we’d declare it in YAML and let Kubernetes enforce it.

Web-First Interfaces: Built for Portability and Speed

Another major turning point: we ditched native UIs completely and went all-in on the web.

We used our own in-house framework, forked from a pre-Angular version of JavaScriptMVC, paired with the Webix widget library. These were not plug-and-play — we trimmed, optimized, and fully integrated them to serve our exact needs. The result? Lightweight, responsive UIs that just worked.

Compared to native GUIs (like Qt), our interfaces loaded faster, had zero platform-specific quirks, and required no installation. We sidestepped display forwarding headaches, library mismatches, and remote access issues. And thanks to Apache Cordova, we wrapped our web apps into mobile-ready clients, enabling real-time monitoring right from a phone or tablet — even back in 2014.

This choice cut our support workload dramatically and gave beamline scientists secure, consistent access to live system data from anywhere.

Total Code Ownership: Every Dependency Under Control

One less-discussed but critical strategy: we maintained direct control over our entire software stack through selective forking.

Rather than depending on upstream release cycles, we forked and maintained key libraries – including our SCADA frameworks. This wasn’t NIH syndrome; it was a calculated trade-off:

❌ More maintenance burden on our team
✅ Zero upstream surprises breaking production
✅ Ability to optimize for our exact use case
✅ Faster debugging when things went wrong

Example: Our JTango Wrapper

Take our ezTango library. Raw JTango required verbose boilerplate for every device interaction:

// Upstream library  
DeviceProxy proxy = new DeviceProxy("tango://whatever:10000/sys/tg_test/1");  
DeviceAttribute attribute = proxy.read_attribute("double_scalar");  
if(result.hasFailed()){  
  throw new Exception("Can not read attribute.");  
}  
int dataFormat = result.getDataFormat()  
int dataType = result.getType()  
double result;  
switch(dataType){  
  case Tango_DEV_Double:  
  switch(dataFormat){  
    case _SCALAR:  
    result = attribute.extractDouble()  
    ...  
  }  
...  
}  
...

// VS

// our wrapper
TangoProxy proxy = TangoProxies.newDeviceWrapper("tango://whatever:10000/sys/tg_test/1");  
double result = proxy.<Double>readAttribute("double_scalar");  
...

Our wrapper added caching, automatic retry logic, and connection pooling – optimizations we couldn’t wait for upstream to implement. This meant: 40% fewer network round-trips (via attribute read caching); less DNS calls; automatic recovery from transient database failures; type-safe APIs reducing runtime errors

This level of control gave us unusual stability across a 10+ years lifecycle – something we couldn’t have achieved depending purely on external release schedules.

The Real Lesson: Boring-Looking, Edge-Worthy

This system ran reliably for 10+ years – and here’s the kicker: since 2016, it’s required essentially zero maintenance. No emergency patches. No weekend firefighting. No rewrites forced by dependency changes. It just runs. That’s not because we got lucky, but because we used modern practices that just happen to look “boring” from the outside.

In fact, we were adopting CI/CD, test isolation, declarative packaging, and full-stack observability at a time when most teams — especially in research — hadn’t even heard of those concepts. We chose our stack carefully and invested in operational excellence:

1. Stability over features: Java wasn’t sexy in 2012, but it was mature
2. CI/CD as infrastructure: Deployment automation wasn’t optional
3. Comprehensive data collection aka Observability: We collected everything, not just images
4. Real environment separation: No shortcuts between test and prod
5. One-click rollbacks: Because perfect deploys don’t exist

The technology landscape has changed dramatically since we started this project. Kubernetes, serverless, and modern observability tools make some of these patterns easier to implement. But the core principle remains: choose technology that lets you sleep at night, not technology that looks good on your resume.

We weren’t perfect. Our worst incident came in 2020 when a beamline scientist accidentally uploaded a malformed NeXus template that corrupted the metadata pipeline. Because we’d isolated test/prod properly, it only affected one beamline, and our rollback procedure had us back online in 43 minutes. The data was recoverable, the scientist apologized, and we added input validation the next day.

Scientists arriving for beamtime didn’t know (or care) about our Java stack or CI/CD pipelines. What they noticed was that the system was always available, experiments started on schedule, and data was accessible immediately after scans completed. That reliability became invisible—the highest compliment for infrastructure.

Before we close, here’s what we’d confidently repeat — and what we’d skip — if we were starting again:

What We’d Do Again

• JVM-based core: mature, portable, predictable
• Self-contained components: consistent deployments, easy rollbacks
• CI/CD with rollback from day 0: especially critical without 24/7 ops teams
• Isolated test environments: safe experimentation
• Total code ownership: when a critical bug surfaced in JTango at 2 AM during beamtime, we could patch and redeploy in 15 minutes instead of waiting days for upstream
• Web-first UIs: easier to support, extend, and port

What We’d Avoid

• JNI bindings or tech zoo in general: worked, but added maintenance complexity
• Custom DSLs: hard to onboard new devs, less community support

If you’re building systems where downtime means thousands of dollars per hour and disappointed scientists from around the world, “boring” is a feature, not a bug.

Want to discuss control systems, Java performance, or scientific software architecture? Find me on https://www.linkedin.com/in/ikhokhryakov/ or check out my work on https://github.com/Ingvord

About the Author

Igor Khokhriakov is a Principal Software Engineer with 17+ years building complex software systems, including work at DESY (Hamburg), the Tango Controls consortium, San Diego Supercomputer Center and currently the HDF Group. He specializes in high-performance distributed systems, with deep expertise in Java, Kubernetes, and infrastructure observability. His systems have supported hundreds of experiments at large-scale research infrastructures like PETRA III and the ESS. His work has been published in peer-reviewed journals including SPIE Proceedings and the Journal of Synchrotron Radiation, and his systems continue to influence scientific software architecture today.