> Markdown version of [/videos/806-leveraging-real-time-data-in-fsis?t=535](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis?t=535). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Leveraging Real time data in FSIs

Traditional SQL schemas throttle high-throughput financial systems under heavy loads. Discover how intelligent NoSQL aggregation drops real-time transaction latency to sub-millisecond ranges.

- **Speakers:** Tim Faulkes
- **Event:** WeAreDevelopers LIVE
- **Published:** November 10, 2023
- **Duration:** 41:51
- **URL:** https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis

## Summary

The demand for real-time data processing in financial services is accelerating, driven largely by the need to feed extensive historical contexts into complex AI and fraud detection models. Strict service-level agreements, such as executing credit card validation within a 120-millisecond window, require loading tens of thousands of transaction records almost instantaneously. Under massive concurrent loads, traditional relational databases like PostgreSQL often struggle to maintain consistent 99th-percentile execution times. Relying on highly normalized SQL schemas forces systems into a scatter-gather approach across indexes, multiplying disk reads and creating an exponential "hockey stick" latency curve that fundamentally limits high-throughput decision-making.

Overcoming these bottlenecks requires a strategic architectural shift toward NoSQL data aggregation techniques. By abandoning traditional normalization in favor of unstructured, self-describing records, developers can nest associated entities directly inside primary objects or intelligently bucket event data—such as grouping an entire day of transactions into a single record. This model eliminates complex joins and foreign key lookups, dramatically accelerating retrieval. Furthermore, the CAP theorem dictates that distributed systems must balance consistency, availability, and partition tolerance. While many NoSQL databases default to high availability at the cost of accuracy, localized tools like Aerospike can be configured to guarantee the strict consistency needed for financial workloads while maintaining fault tolerance.

Evaluating database topologies using custom Java-based simulation reveals the stark operational differences between RDBMS and optimized NoSQL engines. While an in-memory relational cache performs adequately at lower volumes, performance degrades heavily once data sets exceed hardware limits and hit disk storage. Migrating to an optimized key-value approach allows systems to perform parallel batch reads directly from solid-state drives, effectively dropping average response times from distinct system delays to sub-millisecond ranges. Ultimately, no single database acts as a universal remedy; modern system architects must actively decouple application logic from persistence layers and match specific data modeling patterns directly to workload latency constraints.

**Keywords:** real-time fraud detection, FSI data architecture, nosql data modeling, 99th percentile latency, cap theorem partition tolerance, aerospike database deployment, relational normalization bottlenecks, transaction bucketing patterns, postgresql cache limitations, java database simulation, ssd parallel batch reads, redis memory constraints, high-throughput decisioning

## Chapters

1. **Modern application stacks and real-time data requirements** (00:28) — How developers structure scalable microservice architectures while historically underestimating database performance in real-time use cases.
1. **High-speed data use cases in financial services** (03:04) — Faster dataset ingestion gives fraud, risk, and compliance algorithms more time to execute sophisticated real-time decisions.
1. **Measuring database latency through high percentiles** (05:24) — Why analyzing the ninety-ninth latency percentile matters far more than average response times for high-volume database performance.
1. **Terminology differences between relational and NoSQL databases** (08:55) — Comparing how structured SQL systems and unstructured NoSQL stores handle schemas, namespaces, and records.
1. **Normalization constraints versus nested data aggregations** (12:11) — How embedding related entity sets inside a single NoSQL object eliminates complex database joins and index reads.
1. **Handling NoSQL entity associations with batch processing** (15:26) — Retrieving multiple independent objects efficiently by performing parallel cluster batch reads across separated node distributions.
1. **Meeting CAP theorem consistency rules in distributed clusters** (16:54) — Why distributed databases must choose between availability and system consistency during node network partitions.
1. **Boosting scale and lowering latency natively with NoSQL** (19:19) — Large-scale fraud and identity platforms achieve microsecond latencies by shifting workloads from caching tiers to memory-optimized platforms.
1. **Defining rules and constraints for credit card fraud validation** (21:11) — Setting tight millisecond execution windows to scan vast user transaction histories against deep learning fraud models.
1. **Live benchmarking database transaction throughput and latency** (23:51) — Testing thread concurrency limits by pumping simulated credit swipe data into a relational testing container.
1. **Integrating a Redis database connection into Java applications** (25:38) — Extending a custom Spring application test harness by injecting and configuring a new in-memory NoSQL interface.
1. **Analyzing relational database bottlenecks during heavy SQL loads** (30:20) — Relational cache warmup requirements cause dramatic latency spikes that quickly violate strict real-time query deadlines.
1. **Improving NoSQL performance by bucketing transaction data points** (34:41) — Restructuring event histories into distinct time-based bins minimizes scatter-gather index scans and parallelizes drive reads.
1. **Final takeaways on database selection and community resources** (40:12) — Exploring custom database architectures is necessary because there is no universal engine suited for every enterprise data problem.

## Related Moments

- [Q&A on database vendor lock-in and alternative architectural choices](https://www.wearedevelopers.com/videos/746-swapping-low-latency-data-storage-under-high-load) (from "Swapping Low Latency Data Storage Under High Load")
- [Comparing PostgreSQL and DynamoDB for read-heavy storage](https://www.wearedevelopers.com/videos/746-swapping-low-latency-data-storage-under-high-load) (from "Swapping Low Latency Data Storage Under High Load")
- [Understanding fast, accurate, and big data store trade-offs](https://www.wearedevelopers.com/videos/430-make-your-data-fabulous) (from "Make Your Data FABulous")
- [Recognizing architectural drivers pushing event streaming system adoption](https://www.wearedevelopers.com/videos/538-event-messaging-and-streaming-with-apache-pulsar) (from "Event Messaging and Streaming with Apache Pulsar")
- [Audience questions on databases, freeze modes, and monitoring](https://www.wearedevelopers.com/videos/269-serverless-on-cloud) (from "Serverless on Cloud")
- [Evaluating NoSQL databases and data conversion bottlenecks](https://www.wearedevelopers.com/videos/626-in-memory-computing-the-big-picture) (from "In-Memory Computing - The Big Picture")

## Related Articles

- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)](https://www.wearedevelopers.com/magazine/745-why-event-driven-architecture-isn-t-about-speed-and-when-you-actually-need-it)
- [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)
- [What does the history of data storage tell us about the future?](https://www.wearedevelopers.com/magazine/495-what-does-the-history-of-data-storage-tell-us-about-the-future)

## Related Jobs

- [Lead Software Engineer - Data Engineering](https://www.wearedevelopers.com/jobs/ext/2000968-lead-software-engineer-data-engineering) at **Dynatrace**
- [Principal Software Engineer, Database Infrastructure](https://www.wearedevelopers.com/jobs/ext/1465908-principal-software-engineer-database-infrastructure) at **GitHub**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/353953-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/319507-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Staff Software Engineer, Database Infrastructure](https://www.wearedevelopers.com/jobs/ext/1470125-staff-software-engineer-database-infrastructure) at **GitHub**