Quickstart Task Queues with Celery and Python

What is Celery ?
Features of Celery
Installation and Quickstart
Frequently Asked questions
Conclusion

What is Celery ?

Celery is a open-source and distributed task queue written in Python. It's used to schedule and run background tasks without blocking the main process. A quick example use case is : Consider you have an application that sends bulk marketing emails to your users. With Celery, you can just schedule a background task to send emails to your users and the main process can continue to serve other requests.

Before moving any forward, we need to know these three main components of Celery :

Tasks : These are the units of work that are executed by Celery. Tasks are defined as Python functions or methods and decorated with the @app.task decorator. Read more about Celery tasks here.
Workers : These are the celery processes that will execute the tasks. Workers are started by the celery worker command and they run in background as daemons by default, listening for tasks to be sent to them by the message broker.
Message Broker : This is a message queue that Celery uses to send and receive tasks. It acts as a communication channel between the workers and the main application. While celery supports various message brokers, the most popular ones are Redis and RabbitMQ.

Features of Celery

Easy to use : A Quickstart is all you need, celery is very easy to use.
Asynchronous : Celery is async in nature, you can run/schedule multiple tasks in parallel.
Distributed : Celery can be used to run tasks on multiple machines. You can create multiple celery workers accross different machines, and they can pick up tasks from the "common message broker" and execute them.
Reliable : Celery can recover from failures and continue to process tasks. In addition to this, celery also provides with various handlers to handle failure cases within defined tasks as well.
Scalable : Celery is horizontally scalable. You can add more workers to process more tasks without disrupting existing workers.
Flexible : Celery can be used with any python web framework. It has an impressive community support for Django and Flask pacakges. It also has support for other frameworks like Pyramid, Bottle, Tornado, etc.

In a typical Celery setup, the main application sends tasks to the message broker, which in turn sends them to the workers for execution. The workers execute the tasks in background parallely, results are then stored in Celery's result backend. This allows the main application to offload long-running tasks and continue handling requests while the tasks are being executed in the background.

Celery is widely used in web applications, data processing pipelines, and other distributed systems to run tasks asynchronously and improve the performance and scalability of the application.

Installation and Quickstart

Celery can be installed using python's pip package manager. You can install it using the following command :
```
pip install celery
```
Create a Python script for your task. For example, let's create a task that subtracts two numbers and retrusn the result. Create a file named app.py and add the following code :
```
from celery import Celery

app = Celery('my_app', broker='amqp://guest@localhost//')

@app.task
def subtract(x, y):
    return x - y
```
The Celery class is used to create a Celery application instance. Here, my_app is the name of our application. The second argument, amqp://guest@localhost/ is the message broker that will be used by the application. Celery supports various message brokers, but we will be using RabbitMQ as our message broker.
- Read more about RabbitMQ here.
- Read more about Celery's supported message brokers here.
Now, we need to start a celery worker. Run the Celery worker by using the following command:
```
celery -A app worker --loglevel=info
```
This will start a celery worker that will listen for tasks on the message broker. The -A flag is used to specify the module where the celery app is defined. The worker flag is used to start a worker process. The --loglevel flag is used to specify the log level of the worker process. You can use debug, info, warning, error, critical as the log level.
Now, we can run the task by importing the subtract function from the app module and calling it. In a separate terminal, start a Python interpreter and import the task:
```
from tasks import subtract
```
Call the task and print the result. Note : This task will be executed in the background and the result will be returned immediately.
```
result = subtract.delay(10-5)
print(result.get())
```

You should see the output 5 printed in the terminal. This is the result of the task that was executed in the background.

Frequently Asked questions

Can I run Celery on Windows ?

Yes! Celery can be installed on any operating system. Steps to install celery remain same across all operating systems, you can install it via pip

Does Celery needs a Python web framework to run ?

No! Celery can be used without web frameworks as well. In this tutorial, we've used Celery with a simple Python script without any frameworks.

What is backend in Celery ?

Celery has a concept of Result backend, i.e. a place where the results of tasks will be stored. This can be a database or a cache, the most widely used ones are Redis and MongoDB. Celery supports various backends, read about them here.

What is a broker in Celery ?

Celery uses a message broker to communicate between the client and the worker. Where client can be your webapp from which you are triggering the task and worker is the background celery process that will execute the task. To facilitate this communication, Celery uses a message broker. Celery supports various brokers, read about them here.

Is celery suitable for production ?

Yes! Celery is used by companies of all scale in production. It's distributed, scalable and has tons of feature to accomodate simple-complex needs.

Which companies use Celery ?

Celery is used by companies like Pinterest, Instagram, Dropbox, etc.

Conclusion

That's it! You have successfully set up and run a task using Celery. You can now use Celery to run tasks asynchronously in your Python application.

In this tutorial, we've just scratched the surface of Celery. Celery has a lot of features like task priorities, retries, error/success/retry handlers, databases as result backends and lot more stuff! You can read more about Celery here.

We'll soon be posting in-depth articles on various advanced celery topics. I hope you enjoyed this tutorial. If you have any questions or suggestions, feel free to leave a comment below. Thanks for reading!

Table of contents