Volatile data lakes in the cloud

In a world where managers want to compare multiple KPIs here and now, the ability to quickly get the right data in front of the end user is critical. However, frequent data updates cause other records to become outdated or unnecessary. Extracting interesting information from such an append-only data lake requires complex SQL queries that include windowing functions and/or join operations. As the system grows, queries become longer and longer, and time is money. This begs the question of how to cost-effectively update or delete unwanted records from the data lake to exclude them from queries. Based on my own project experience as an architect, I will show which tools in the AWS cloud can be used to answer this question. I will also share best practices for designing serverless reporting systems in the cloud.