WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
WeAreDevelopers LIVE • Apr 18, 2023

Celery on AWS ECS - the art of background tasks & continuous deployment

Jan Giacomelli

Frequent AWS ECS deployments are silently killing your Celery tasks. Stop dropping background jobs during scaling events. Discover the exact infrastructure tweaks to bulletproof your continuous deployment pipeline.

Pause
Mute Enter Fullscreen
#1 about 3 min

Understanding the need for background task processing in applications

Long-running or retry-dependent operations require asynchronous background tasks to avoid blocking users.

#2 about 2 min

Core concepts of Celery and message broker integration

Celery reads background tasks from a message broker and executes them on workers.

#3 about 2 min

Deploying Celery workers on AWS ECS Fargate containers

Fargate provides serverless compute environments that simplify scaling and deploying worker resources.

#4 about 2 min

Configuring ECS container lifecycles for graceful worker shutdown

Setting appropriate stop timeouts and minimum healthy percentages allows workers to finish processing.

#5 about 3 min

Impact of continuous deployments on long running background tasks

Frequent deployments often interrupt long tasks, causing delayed processing or lost data.

#6 about 4 min

Configuring late acknowledgment and visibility timeouts in Celery

Late acknowledgment ensures interrupted jobs are retried safely after a defined visibility timeout.

#7 about 2 min

Preventing task duplication by avoiding Celery countdown settings

Storing delayed tasks in worker memory risks data loss and duplicated execution during interruptions.

#8 about 3 min

Rejecting tasks on worker loss to prevent data loss

Configuring tasks to reject upon unexpected worker exits ensures at-least-once processing guarantees.

#9 about 3 min

Limiting prefetch multipliers to avoid task latency during deployments

Setting the prefetch multiplier to one prevents unstarted tasks from getting trapped in terminating workers.

#10 about 4 min

Remapping termination signals for proper cold shutdowns on ECS

Mapping SIGTERM to SIGQUIT forces cold shutdowns, allowing uncompleted tasks to properly retry.

#11 about 4 min

Maintaining short target processing times and ensuring task idempotency

Tasks must run shorter than typical deployment windows and use exponential backoff to handle external failures safely.

#12 about 5 min

Structuring large workloads with parallel fan out task patterns

Splitting massive data sets into individual worker jobs prevents lengthy processing times and timeout failures.

#13 about 3 min

Processing sequential data chunks recursively using task batching patterns

Processing a subset of data and spawning a subsequent task via cursors keeps individual runtime predictably short.

#14 about 3 min

Preventing duplicate execution logic using Redis distributed task locking

Task locking ensures strict at-most-once or at-least-once task execution when continuous scaling events occur.

#15 about 5 min

Reviewing Python code examples for batching and locking strategies

Code demonstrations show how decorators and cursors implement reliable job submission and concurrency control.

#16 about 3 min

Insights on the underlying career journey and tech evolution

A transition from electrical engineering to systems software highlights the value of deep infrastructure knowledge.

#17 about 2 min

Handling task interruptions when using Django database result backends

Interruptions inherently disrupt asynchronous workflows under frequent deployment cycles regardless of the specific backend.

#18 about 2 min

Evaluating formal AWS certifications versus raw practical engineering experience

Enterprise environments value architecture certificates, but empirical systems knowledge often translates better to solving complex deployment problems.

#19 about 1 min

Running lightweight Python scripts natively using AWS Lambda functions

Leveraging serverless capabilities provides an accessible environment to launch cloud code without provisioning full machine clusters.

#20 about 3 min

Scaling boundaries and performance optimizations for large worker pools

Combining task patterns maximizes processing throughput without hitting typical cloud infrastructure constraint ceilings.

#21 about 3 min

Troubleshooting sudden worker termination events and memory segmentation faults

Connecting external monitoring integrations easily exposes out-of-memory errors before falling back to manual log debug streams.

#22 about 5 min

Exploring the specific workplace responsibilities of staff software engineers

Driving architectural best practices across data science and infrastructure teams characterizes the staff engineering role in modern startups.

Matching moments

2:20 min

Exploring auto-scaling benefits inside event-driven cloud ecosystems

Developersteve · LIVE

1:55 min

Best practices for implementing durable hyperscale serverless systems

Marco Plaul Marco Plaul +1 · WWC 2023

2:11 min

Executing massive cloud network migrations while maintaining live systems

Dana Lawson Dana Lawson +1 · WWC Europe 2026

10:34 min

Challenging best practices to fix queue bottlenecks

Ivan Milanov Ivan Milanov · Europe 2026 Virtual

3:38 min

Building scalable backends with a modular monolith

Josip Stuhli Josip Stuhli · WWC Europe 2026

1:59 min

Emphasizing statelessness, caching, and delayed processing architectures

Josip Stuhli Josip Stuhli · WWC 2023

More from WeAreDevelopers LIVE

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz thumbnail
5:43
5 days ago

Aliens in Quarantine, Python Deprecates Loops, and AI Buys Old Books - Zbyszek Tenerowicz

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Node and Package Security thumbnail
1:08:30
5 days ago

WeAreDevelopers LIVE - Node and Package Security

Chris Heilmann, Daniel Cranney, Zbyszek Tenerowicz

WeAreDevelopers LIVE - Streaming HTML thumbnail
1:10:26
12 days ago

WeAreDevelopers LIVE - Streaming HTML

Chris Heilmann, Daniel Cranney, Jad Joubran

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe thumbnail
1:02:34
19 days ago

WeAreDevelopers LIVE - CODE100 at WeAreDevelopers Europe

Chris Heilmann, Daniel Cranney, Justin Halsall, Werner Jeschko, Nicole Jeschko, Mirco Tomić

WeAreDevelopers LIVE - Web Performance in an AI World thumbnail
1:07:59
26 days ago

WeAreDevelopers LIVE - Web Performance in an AI World

Chris Heilmann, Daniel Cranney

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko thumbnail
5:55
about 2 months ago

FIFA Hack, Midjourney Does Ultrasounds, and Trampoline Traffic Lights - Nicole Jeschko and Werner Jeschko

Chris Heilmann, Daniel Cranney

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Boring Failover: Predictable Region Recovery Across 5,000 Microservices

Garvit Kataria, Sahil Sabharwal

Garvit Kataria
Sahil Sabharwal
Open session

World Congress 2026 North America

When Logging Becomes The Outage: Escaping the ECS Logging Trap

Rahul Tanniru

Senior Vice President Of Software Engineering, Jp Morgan Chase

Rahul Tanniru
Open session

World Congress 2026 North America

From Static Rules to Reasoning Platforms: Scaling Intelligent Canary Delivery in 2026

Daniel Oh

Senior Principal Developer Advocate

Daniel Oh
Open session

World Congress 2026 North America

Run your agents in Kubernetes: Build once, deploy anywhere. But really?

Michal Salanci

Senior Systems Engineer at ESET Cybersecurity

Michal Salanci
Open session

World Congress 2026 North America

rm -rf: Horror Stories From Unsandboxed AI Agents (and How Docker Fixes This)

Rishab Kumar

Staff Developer Evangelist @ Twilio

Rishab Kumar
Open session

World Congress 2026 North America

Context Engineering Kung Fu

Carl Lapierre

Tech Lead and AI Engineer at Osedea

Carl Lapierre

Related videos

From 0 to 1.000.000: How to build a serverless raffle service for hyperscale thumbnail
30:03
WWC 2023 • almost 3 years ago

From 0 to 1.000.000: How to build a serverless raffle service for hyperscale

Marco Plaul, Martin Sakowski

Reliable scalability: How Amazon.com scales on AWS thumbnail
22:24
WWC 2024 • almost 2 years ago

Reliable scalability: How Amazon.com scales on AWS

Florian Mair

CI/CD Patterns and Antipatterns - Things your Pipeline Should (Not) Do thumbnail
29:36
WWC 2024 • almost 2 years ago

CI/CD Patterns and Antipatterns - Things your Pipeline Should (Not) Do

Raniz

Cloud as the new mainframe: why the cloud hype does not reflect the dev reality thumbnail
23:00
WWC 2023 • almost 3 years ago

Cloud as the new mainframe: why the cloud hype does not reflect the dev reality

Gerta Sheganaku, Waldemar Hummer

Shipping Faster with Less: Render on Cloud Hosting, AI Workloads, and the Future of DevOps thumbnail
35:19
Coffee With Developers • 3 months ago

Shipping Faster with Less: Render on Cloud Hosting, AI Workloads, and the Future of DevOps

Hazal Mestci, Raph Terrier

Containers in the cloud - State of the Art in 2022 thumbnail
28:03
WWC 2022 • about 4 years ago

Containers in the cloud - State of the Art in 2022

Federico Fregosi

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.