> Markdown version of [/videos/231-using-non-functional-testing-to-guide-user-interface-backend-services-voice-interface-and-media-development?t=232](https://www.wearedevelopers.com/videos/231-using-non-functional-testing-to-guide-user-interface-backend-services-voice-interface-and-media-development?t=232). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Using non-functional testing to guide user interface, backend services, voice interface, and media development

Want to deliver a five-star user experience? Discover how double-purposing functional automated tests for non-functional metrics safeguards performance across UI, voice, and media.

- **Speakers:** Brien Colwell
- **Event:** World Congress 2021
- **Published:** June 28, 2021
- **Duration:** 47:37
- **URL:** https://www.wearedevelopers.com/videos/231-using-non-functional-testing-to-guide-user-interface-backend-services-voice-interface-and-media-development

## Summary

Transitioning an application from fundamentally functional to delivering a five-star user experience requires moving beyond binary pass/fail correctness. Non-functional testing evaluates performance, reliability, and usability through actions, scores, and established baselines. By building a regression safety net, cross-functional project teams—combining product engineering, data analytics, and testing—can track these critical metrics over time. This approach ensures that product updates consistently improve rather than degrade the end-user experience, relying on statistical analysis to flag performance regressions before they impact users.

Implementing this framework involves gathering vast amounts of raw data. While instrumenting application code provides granular system statistics, instrumenting the synthetic test environment using tools like Appium allows QA teams to capture rich datasets without bloating the production app. This enables the collection of complete session videos and network traces without performance overhead. A highly effective technique in this process is decoupling the "region of interest" generated by automated test scripts from the subsequent visual analytics. This separation allows data teams to extract pixel-perfect measurements for complex metrics like First Contentful Paint without being bottlenecked by the fuzziness of automation framework execution times.

The true power of this methodology emerges when applied to complex, data-heavy features like gaming, voice, and video streaming. For gaming applications, monitoring CPU usage and frames per second against hardware-specific baselines prevents battery drain and poor playability. Voice interfaces benefit heavily from automated synthetic tests that evaluate response latency across diverse locales and phrase permutations. Meanwhile, streaming media testing focuses on identifying loading latency and visual artifacts like blockiness or sudden resolution switches using reference-free quality measures. Ultimately, organizations can maximize their QA investments by double-purposing existing functional automated tests to seamlessly capture non-functional regression metrics directly within CI/CD pipelines and live production environments.

**Keywords:** non-functional testing, regression safety net, synthetic test environments, appium automation testing, first contentful paint metrics, voice interface latency testing, streaming media quality analysis, test environment instrumentation, CI/CD pipeline regression tracking, user perceived quality, visual analytics measurement, application performance baseline, mobile game FPS monitoring, automated test data collection, network traffic burst analysis, reference-free video quality

## Chapters

1. **Differences between functional and non-functional testing** (00:02) — Non-functional testing focuses on measuring performance, quality, and reliability against target baselines rather than simple pass-fail outcomes.
1. **Creating a safety net to catch non-functional regressions** (01:52) — Establishing a framework centered on data collection creates a continuous safety net for identifying and addressing regressions.
1. **Integrating testing, data, and engineering skills for project teams** (03:52) — Combining the expertise of functional testing, data analysis, and product engineering enables an effective non-functional testing practice.
1. **Implementing a continuous workflow to track non-functional regressions** (05:13) — Identifying hotspots, measuring against established baselines, and tracking changes across builds ensures consistent product improvement.
1. **Collecting raw performance data by instrumenting application logic components** (09:41) — Inserting start and stop markers inside the application logic generates precise telemetry for analyzing performance hotspots.
1. **Instrumenting test environments to gather non-functional metrics synthetically** (13:07) — Capturing external signals like video, network traces, and system statistics avoids modifying application code while generating rich datasets.
1. **Extracting standardized performance metrics from raw test data** (16:26) — Combining application events and visual analytics yields critical insights like first contentful paint and application connection efficiency.
1. **Defining regions of interest to measure page load accuracy** (20:03) — Separating input commands from visual analytics establishes pixel-perfect measurements of user actions within automated testing frameworks.
1. **Structuring project teams to manage non-functional testing goals** (24:59) — Engineering sets the architecture, data teams extract measurements, and testing teams incorporate metric collection into existing functional runs.
1. **Tracking performance baselines and regressions in gaming applications** (28:17) — Monitoring frames per second and processor usage across different devices ensures consistent playability during continuous deployment.
1. **Benchmarking voice application response times across different locales** (32:04) — Synthetically testing voice inputs helps measure application response intervals and identify regressions against industry benchmarks.
1. **Measuring video playback quality and stream resolution transitions** (34:53) — Tracking buffering animations and visual artifacts provides objective visibility into the perceived quality of on-demand media.
1. **Resources for mastering automated non-functional software testing techniques** (37:19) — Teams can access specialized training, developer documentation, and synthetic testing platforms to improve product performance checks.
1. **Tools and strategies for practical non-functional testing workflows** (38:54) — Integrating test frameworks, monitoring production networks, and adopting appropriate instrumentation simplifies comprehensive performance measurements over time.

## Related Moments

- [Adopting modern end-to-end testing for automated deployment pipelines](https://www.wearedevelopers.com/videos/1837-why-testing-matters-in-ai-luise-freese-and-elio-struyf) (from "Why Testing Matters in AI  - Luise Freese and Elio Struyf")
- [Bridging the gap between fantasy testing and real-world conditions](https://www.wearedevelopers.com/videos/1672-your-testing-strategy-is-broken-lets-fix-it) (from "Your Testing Strategy is broken - lets fix it!")
- [The software testing timeline and regression checking pitfalls](https://www.wearedevelopers.com/videos/87-excellent-software-testing) (from "Excellent Software Testing")
- [Key strategies for continuous frontend performance testing](https://www.wearedevelopers.com/videos/719-measure-and-improve-frontend-performance-by-using-test-automation) (from "Measure and improve frontend performance by using test automation")
- [Live load testing functional applications using concurrent tooling](https://www.wearedevelopers.com/videos/814-serverless-architectures-with-spring-cloud-functions-and-knative) (from "Serverless Architectures with Spring Cloud Functions and Knative")
- [Audience questions on visual testing execution and framework differences](https://www.wearedevelopers.com/videos/540-let-s-get-visual-visual-testing-in-your-project) (from "Let's get visual - Visual testing in your project")

## Related Articles

- [Benefits of Using JMeter For Performance Testing](https://www.wearedevelopers.com/magazine/96-benefits-of-using-jmeter-for-performance-testing)
- [Integration And E2E Testing: Are They Complementary or Interchangeable?](https://www.wearedevelopers.com/magazine/86-integration-and-e2e-testing-are-they-complementary-or-interchangeable)
- [The 8 Best Code Testing Tools](https://www.wearedevelopers.com/magazine/402-the-8-best-code-testing-tools)
- [How to create a test plan for software testing](https://www.wearedevelopers.com/magazine/56-how-to-create-a-test-plan-for-software-testing)

## Related Jobs

- [Bank Platform Transformation - Test Execution Leader - Sr Associate](https://www.wearedevelopers.com/jobs/ext/2407777-bank-platform-transformation-test-execution-leader-sr-associate) at **PwC**
- [Bank Platform Transformation - Test Execution Leader - Sr Associate](https://www.wearedevelopers.com/jobs/ext/2391699-bank-platform-transformation-test-execution-leader-sr-associate) at **PwC**
- [Bank Platform Transformation - Test Execution Leader - Sr Associate](https://www.wearedevelopers.com/jobs/ext/2628569-bank-platform-transformation-test-execution-leader-sr-associate) at **PwC**
- [Bank Platform Transformation - Test Execution Leader - Sr Associate](https://www.wearedevelopers.com/jobs/ext/2619436-bank-platform-transformation-test-execution-leader-sr-associate) at **PwC**
- [Bank Platform Transformation - Test Execution Leader - Sr Associate](https://www.wearedevelopers.com/jobs/ext/2433232-bank-platform-transformation-test-execution-leader-sr-associate) at **PwC**
- [Bank Platform Transformation - Test Execution Leader - Sr Associate](https://www.wearedevelopers.com/jobs/ext/2575332-bank-platform-transformation-test-execution-leader-sr-associate) at **PwC**