> Markdown version of [/videos/79-hate-organising-your-photos-try-it-with-5-terabytes](https://www.wearedevelopers.com/videos/79-hate-organising-your-photos-try-it-with-5-terabytes). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Hate organising your photos? Try it with 5 Terabytes

How do you store and index 5TB of RAW photos without blowing your cloud budget? Discover a hybrid Raspberry Pi and AWS architecture costing under $15 a month.

- **Speakers:** Arto Liukkonen
- **Event:** WeAreDevelopers LIVE
- **Published:** November 30, 2020
- **Duration:** 30:25
- **URL:** https://www.wearedevelopers.com/videos/79-hate-organising-your-photos-try-it-with-5-terabytes

## Summary

Managing an expanding repository of RAW photography files demands more than consumer cloud services can economically provide. Faced with storing over five terabytes of client photos, traditional options like Google Drive or standard Amazon S3 quickly exceed acceptable monthly budgets. To solve this, a hybrid local-to-cloud architecture leverages a home Raspberry Pi to extract lightweight JPEG previews and EXIF data directly from RAW files. The heavy RAW files are then routed to Amazon S3 Glacier Deep Archive, yielding a durable disaster recovery solution for just $5 a month, while the smaller thumbnails sync to a standard S3 bucket for active processing.

Building a searchable gallery out of raw storage requires an intelligent indexing strategy. Processing these thumbnails through AWS Rekognition enables automated object and skin recognition, allowing for deep content filtering by specific subjects or individual faces. However, working with managed AI services often surfaces operational constraints. Because Rekognition enforces strict operations-per-second quotas that easily buckle under bulk uploads, throttling data ingestion via `s3cmd` instead of the standard AWS CLI became a necessary workaround to avoid continuous throttle errors. Additionally, while serverless functions offer scalable compute, native binary incompatibilities—like running EXIF extraction tools in an AWS Lambda layer—prove that offloading initial media extraction to local hardware remains a pragmatic architectural choice.

Extracting metadata once and caching the results in Amazon DynamoDB ensures AI processing costs remain a one-time expense, keeping the ongoing application overhead below $10 per month. The fully custom frontend, built on AWS Amplify and React, replaces generic folder navigation with a dynamic, tag-driven gallery. Looking forward, this decoupled metadata approach allows the architecture to naturally extend into a client-facing ecosystem, paving the way for automated portfolio updates, password-protected client galleries, and integrated e-commerce workflows.

**Keywords:** aws s3 glacier deep archive, cloud cost optimization, serverless image processing, aws rekognition tagging, aws lambda environment constraints, raspberry pi hardware offloading, exif data extraction, dynamodb metadata caching, cloud api rate throttling, s3 lifecycle policies, raw image processing, disaster recovery storage, aws amplify react frontend, ai automated gallery indexing

## Chapters

1. **Managing massive volumes of raw digital camera photography** (00:17) — A massive collection of raw photography files creates a need for an accessible and scalable backup solution.
1. **Evaluating commercial cloud storage costs for large archives** (03:49) — Comparing commercial storage options reveals that standard tiers are too expensive for multi-terabyte archives.
1. **Utilizing deep archive storage for affordable digital backups** (07:44) — Archiving raw files in deep storage provides incredibly cheap capacity with acceptable retrieval delays.
1. **Extracting preview images and metadata from raw files** (12:09) — Generating smaller thumbnails and copying metadata enables a browsing interface without fetching massive raw files.
1. **Using a local hardware device for image processing** (14:17) — A local script handles image resizing and synchronization to avoid native binary issues in serverless functions.
1. **Implementing automated object and face recognition for images** (17:26) — Integrating artificial intelligence services allows automatic categorization of occurrences and specific individuals across thousands of photos.
1. **Handling service rate limits and cloud execution costs** (20:58) — Throttling upload speeds prevents errors from strict quotas while free tiers help mitigate processing expenses.
1. **Demonstrating the final architecture and web interface workflow** (23:54) — Connecting local processing to cloud services powers a secure web application for browsing tagged photography galleries.
1. **Integrating the backup gallery with a portfolio website** (27:41) — Tagged image metadata automatically populates a public portfolio site using cloud storage triggers and serverless functions.
1. **Summarizing the total monthly cost of the serverless architecture** (29:04) — A combination of deep archiving and serverless compute reduces massive data storage expenses to mere dollars a month.

## Related Moments

- [Architecting common use cases via serverless design](https://www.wearedevelopers.com/videos/34-serverless-past-present-and-future) (from "Serverless: Past, Present and Future")
- [Integrating cloud capabilities in a hybrid architecture](https://www.wearedevelopers.com/videos/1206-single-server-global-reach-running-a-worldwide-marketplace-on-bare-metal-in-a-cloud-dominated-world) (from "Single Server, Global Reach: Running a Worldwide Marketplace on Bare Metal in a Cloud-Dominated World")
- [Automating data collection and managing crowdsourced training image sets](https://www.wearedevelopers.com/videos/154-machine-learning-for-software-developers-and-knitters) (from "Machine Learning for Software Developers (and Knitters)")
- [Optimizing software architectures to minimize infrastructure resource drain](https://www.wearedevelopers.com/videos/988-times-of-climate-crisis-how-and-why-sustainable-software-is-a-must) (from "Times of (climate) crisis - How and why sustainable software is a must!")
- [Integrating advanced cloud functionality into on-premise applications](https://www.wearedevelopers.com/videos/186-effective-java-strategies-and-architectures-for-clouds) (from "Effective Java Strategies and Architectures for Clouds")
- [Struggling with ungoverned data lakes and massive storage costs](https://www.wearedevelopers.com/videos/1622-data-governance-in-the-era-of-ai) (from "Data Governance in the Era of AI")

## Related Articles

- [Quick trick: working with mobile phone data in images to create thumbnails and add meta information](https://www.wearedevelopers.com/magazine/498-quick-trick-working-with-mobile-phone-data-in-images-to-create-thumbnails-and-add-meta-information)
- [Dev Digest 121 - AI goes offline](https://www.wearedevelopers.com/magazine/456-dev-digest-121-ai-goes-offline)
- [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)
- [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix)

## Related Jobs

- [Senior Software Engineer, Data](https://www.wearedevelopers.com/jobs/48273-senior-software-engineer-data) at **Sportradar Media Services GmbH**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/353953-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/381484-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Senior Engineer, Infrastructure Platform](https://www.wearedevelopers.com/jobs/ext/328836-senior-engineer-infrastructure-platform) at **Intercom, Inc.**
- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/639235-remote-senior-full-stack-engineer) at **Edge Impulse**
- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/643147-remote-senior-full-stack-engineer) at **Edge Impulse**