WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Apr 18, 2023

Celery on AWS ECS - the art of background tasks & continuous deployment

Jan Giacomelli

Frequent AWS ECS deployments are silently killing your Celery tasks. Stop dropping background jobs during scaling events. Discover the exact infrastructure tweaks to bulletproof your continuous deployment pipeline.

Pause
Mute Enter Fullscreen
#1 about 3 min

Understanding the need for background task processing in applications

Long-running or retry-dependent operations require asynchronous background tasks to avoid blocking users.

#2 about 2 min

Core concepts of Celery and message broker integration

Celery reads background tasks from a message broker and executes them on workers.

#3 about 2 min

Deploying Celery workers on AWS ECS Fargate containers

Fargate provides serverless compute environments that simplify scaling and deploying worker resources.

#4 about 2 min

Configuring ECS container lifecycles for graceful worker shutdown

Setting appropriate stop timeouts and minimum healthy percentages allows workers to finish processing.

#5 about 3 min

Impact of continuous deployments on long running background tasks

Frequent deployments often interrupt long tasks, causing delayed processing or lost data.

#6 about 4 min

Configuring late acknowledgment and visibility timeouts in Celery

Late acknowledgment ensures interrupted jobs are retried safely after a defined visibility timeout.

#7 about 2 min

Preventing task duplication by avoiding Celery countdown settings

Storing delayed tasks in worker memory risks data loss and duplicated execution during interruptions.

#8 about 3 min

Rejecting tasks on worker loss to prevent data loss

Configuring tasks to reject upon unexpected worker exits ensures at-least-once processing guarantees.

#9 about 3 min

Limiting prefetch multipliers to avoid task latency during deployments

Setting the prefetch multiplier to one prevents unstarted tasks from getting trapped in terminating workers.

#10 about 4 min

Remapping termination signals for proper cold shutdowns on ECS

Mapping SIGTERM to SIGQUIT forces cold shutdowns, allowing uncompleted tasks to properly retry.

#11 about 4 min

Maintaining short target processing times and ensuring task idempotency

Tasks must run shorter than typical deployment windows and use exponential backoff to handle external failures safely.

#12 about 5 min

Structuring large workloads with parallel fan out task patterns

Splitting massive data sets into individual worker jobs prevents lengthy processing times and timeout failures.

#13 about 3 min

Processing sequential data chunks recursively using task batching patterns

Processing a subset of data and spawning a subsequent task via cursors keeps individual runtime predictably short.

#14 about 3 min

Preventing duplicate execution logic using Redis distributed task locking

Task locking ensures strict at-most-once or at-least-once task execution when continuous scaling events occur.

#15 about 5 min

Reviewing Python code examples for batching and locking strategies

Code demonstrations show how decorators and cursors implement reliable job submission and concurrency control.

#16 about 3 min

Insights on the underlying career journey and tech evolution

A transition from electrical engineering to systems software highlights the value of deep infrastructure knowledge.

#17 about 2 min

Handling task interruptions when using Django database result backends

Interruptions inherently disrupt asynchronous workflows under frequent deployment cycles regardless of the specific backend.

#18 about 2 min

Evaluating formal AWS certifications versus raw practical engineering experience

Enterprise environments value architecture certificates, but empirical systems knowledge often translates better to solving complex deployment problems.

#19 about 1 min

Running lightweight Python scripts natively using AWS Lambda functions

Leveraging serverless capabilities provides an accessible environment to launch cloud code without provisioning full machine clusters.

#20 about 3 min

Scaling boundaries and performance optimizations for large worker pools

Combining task patterns maximizes processing throughput without hitting typical cloud infrastructure constraint ceilings.

#21 about 3 min

Troubleshooting sudden worker termination events and memory segmentation faults

Connecting external monitoring integrations easily exposes out-of-memory errors before falling back to manual log debug streams.

#22 about 5 min

Exploring the specific workplace responsibilities of staff software engineers

Driving architectural best practices across data science and infrastructure teams characterizes the staff engineering role in modern startups.

Matching moments

2:20 min

Exploring auto-scaling benefits inside event-driven cloud ecosystems

Developersteve · LIVE

1:55 min

Best practices for implementing durable hyperscale serverless systems

Marco Plaul Marco Plaul +1 · World Congress 2023

2:11 min

Executing massive cloud network migrations while maintaining live systems

Dana Lawson Dana Lawson +1 · World Congress 2026 Europe

10:34 min

Challenging best practices to fix queue bottlenecks

Ivan Milanov Ivan Milanov · Europe 2026 Virtual

3:38 min

Building scalable backends with a modular monolith

Josip Stuhli Josip Stuhli · World Congress 2026 Europe

1:59 min

Emphasizing statelessness, caching, and delayed processing architectures

Josip Stuhli Josip Stuhli · World Congress 2023

More from WeAreDevelopers LIVE

WeAreDevelopers LIVE - Is CSS Catching Up to Flash? thumbnail
1:03:33
10 days ago

WeAreDevelopers LIVE - Is CSS Catching Up to Flash?

Chris Heilmann, Daniel Cranney, Bramus Van Damme

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along thumbnail
1:21:17
17 days ago

WeAreDevelopers LIVE - Project Chopin: Make Your Team and Agents Play Along

Chris Heilmann, Daniel Cranney, Krzysztof Cieślak

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak thumbnail
4:23
19 days ago

Fake or News - AI Toothbrushes, Carrier Pigeon Tech & Life on Mars - Krzysztof Cieślak

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Open Source Agent Toolkits thumbnail
1:00:38
24 days ago

WeAreDevelopers LIVE - Open Source Agent Toolkits

Chris Heilmann, Daniel Cranney, Eddie Jaoude

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude thumbnail
6:16
26 days ago

Fake or News - AI Music Detectors, Translating Ancient Languages & Tiny Heads in Teslas - Eddie Jaoude

Chris Heilmann, Daniel Cranney

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas thumbnail
5:46
about 1 month ago

Fake or News: AI Agents Hire Each Other, Spider-Man Ads in Cars & RAM Thieves - Marc Plogas

Chris Heilmann, Daniel Cranney, Marc Plogas

Related videos

From 0 to 1.000.000: How to build a serverless raffle service for hyperscale thumbnail
30:03
World Congress 2023 • almost 3 years ago

From 0 to 1.000.000: How to build a serverless raffle service for hyperscale

Marco Plaul, Martin Sakowski

Reliable scalability: How Amazon.com scales on AWS thumbnail
22:24
World Congress 2024 • about 2 years ago

Reliable scalability: How Amazon.com scales on AWS

Florian Mair

CI/CD Patterns and Antipatterns - Things your Pipeline Should (Not) Do thumbnail
29:36
World Congress 2024 • about 2 years ago

CI/CD Patterns and Antipatterns - Things your Pipeline Should (Not) Do

Raniz

Cloud as the new mainframe: why the cloud hype does not reflect the dev reality thumbnail
23:00
World Congress 2023 • almost 3 years ago

Cloud as the new mainframe: why the cloud hype does not reflect the dev reality

Gerta Sheganaku, Waldemar Hummer

Shipping Faster with Less: Render on Cloud Hosting, AI Workloads, and the Future of DevOps thumbnail
35:19
Coffee With Developers • 4 months ago

Shipping Faster with Less: Render on Cloud Hosting, AI Workloads, and the Future of DevOps

Hazal Mestci, Raph Terrier

Containers in the cloud - State of the Art in 2022 thumbnail
28:03
World Congress 2022 • over 4 years ago

Containers in the cloud - State of the Art in 2022

Federico Fregosi

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.