Celery Result Backends : Options, Configurations & Best Practices

Introduction

While working with Celery for async task execution, you cannot by default access the status and result of a scheduled/running or completed task. It is often a requirement to store the result, status and other task details for later retrieval or to do further processing on it.

Celery has built-in support for "Result Backends". Celery result backend allows you to store your task details and results in various backends, allowing for flexibility and scalability as per your requirements. In this blog post, we will explore the supported result backends by Celery, their configurations and best practices to use them.

Supported Result Backends by Celery

Celery has built-in support for loads of result backends, you can choose as per your use case and development environment. Here's the complete list of result backends supported by Celery.

• rpc
• database
• redis
• cache
• mongodb
• cassandra/astraDB
• elasticsearch
• ironcache
• couchbase
• arangodb
• couchdb
• cosmosdbsql (experimental)
• filesystem
• consul
• azureblockblob
• s3
• gcs
• dynamodb

You can also find the same list on official Celery documentation here.

Custom Result Backends in Celery

While we've mentioned all the supported result backends above, if you have a different requirement and need a custom result backend you can implement one on your own using celery's BaseBackend class, here.

Let's move on to each backend now, exploring how to configure them, and their best practices.

RPC (Remote Procedure Call) Result Backend

The RPC backend behind the scenes uses AMQP to send task results and state updates back as messages.

How to configure RPC result backend ?

To configure RPC backend, you have to prefix the backend URI with rpc:// and you must specify an AMQP broker as well. Let's see an example :

celery_app = Celery('celery_app', backend='rpc://', broker='amqp://guest@localhost//')

When to use RPC result backend ?

• RPC backend is best whenever you require a low latency, and high throughput in your task results.

Database Result Backend

The database backend allows you to use any relational database supported by SQLAlchemy as your celery result backend. This must be obvious, but Database result backend uses SQLAlchemy behind the scenes.

How to configure Database result backend ?

To configure Database backend, you have to prefix the SQLAlchemy backend URI with db+ and you must specify an AMQP broker as well. Let's see an example :

# Example using SQLite DB
celery_app = Celery('celery_app', backend='db+sqlite:///result_db.sqlite', broker='amqp://guest@localhost//')

# Example using MySQL
celery_app = Celery('celery_app', backend='db+mysql://username:password@host:port/database_name', broker='amqp://guest@localhost//')

# Example using PostgreSQL
celery_app = Celery('celery_app', backend='db+postgresql://username:password@host:port/database_name', broker='amqp://guest@localhost//')

You can find the complete list of databases supported by SQLAlchemy here.

When to use Database result backend ?

• Database result backend should be used whenever you want to speed up the read operations on your task results.
• Database result backends makes it easier to optimize query performance by indexing, sharding etc methods.

Redis Result Backend

Redis is a in-memory database, that can be used as Celery result backend as well. Actually, redis can be used both as your broker and backend in Celery.

How to configure Redis result backend ?

To use Redis as your result backend, you have to prefix the backend URI with redis://. Let's see an example :

# Example using Redis for backend
celery_app = Celery('celery_app', backend='redis://localhost', broker='amqp://guest@localhost//')

# Example using Redis for both backend and broker
celery_app = Celery('celery_app', backend='redis://localhost', broker='redis://localhost')

When to use Redis result backend ?

• Ideal for high-performance and real-time applications because of Redis' fast query time.
• Easy to scale, as redis can be scaled horizontally.
• Less services to manage, as Redis can be used both for backend and broker.

Memcached(Cache) Result Backend

Memcached is another in-memory key-value storage, it can be used to store the results as well.

How to configure Memcached result backend ?

To use Memcached as your result backend, you have to prefix the backend URI with cache+memcached://. Let's see an example :

celery_app = Celery('celery_app', backend='cache+memcached://127.0.0.1:11211/', broker='amqp://guest@localhost//')

When to use Memcached result backend ?

• Ideal for cases where fast-access is a priority, and data persistence is not needed.
• Easy to scale, as Memcached can be scaled horizontally.

MongoDB Result Backend

MongoDB is a NoSQL database, it can be used to store the task results in Celery.

How to configure MongoDB result backend ?

To use MongoDB as your result backend, you have to prefix the backend URI with mongodb://. Let's see an example :

celery_app = Celery('celery_app', backend='mongodb://localhost:27017/database_name', broker='amqp://guest@localhost//')

When to use MongoDB result backend ?

• Ideal if you require a flexible result schema as MongoDB doesn't enforce any schema on the data.
• Ideal when you want to optimize your query-performance as MongoDB enables all db benefits like indexing, sharding etc.
• Easy to scale, as MongoDB is horizontally scalable as well.

DynamoDB Result Backend

DynamoDB is a fully managed NoSQL database similar to MongoDB. It is managed by AWS. It can be used to store the task results in Celery.

How to configure DynamoDB result backend ?

To use DynamoDB as your result backend, you have to prefix the backend URI with dynamodb://. Let's see an example :

celery_app = Celery('celery_app', backend='dynamodb://<uri>', broker='amqp://guest@localhost//')

When to use DynamoDB result backend ?

• Ideal if your application relies on a serverless architecture.
• Super low latency if your application is deployed on AWS.
• Easy to scale, and reliable as it's managed by AWS.

Cassandra Result Backend

Cassandra is a highly scalable, distributed NoSQL database, it can be used to store the task results in Celery.

How to configure Cassandra result backend ?

To use Cassandra as your result backend, you have to prefix the backend URI with cassandra://. Let's see an example :

celery_app = Celery('celery_app', backend='cassandra://localhost:9042/keyspacename', broker='amqp://guest@localhost//')

When to use Cassandra result backend ?

• Suitable for use-cases where high write throughputs are needed.
• Easy to scale as Cassandra is horizontally scalable.

Elasticsearch Result Backend

Celery result backend has built-in support for Elasticsearch, a distributed search and analytics engine.

How to configure Elasticsearch result backend ?

To use elasticsearch as your result backend, you have to prefix the backend URI with elasticsearch://. Let's see an example :

celery_app = Celery('celery_app', backend='elasticsearch://localhost:9200/index_name', broker='amqp://guest@localhost//')

When to use Elasticsearch result backend ?

• Suitable for applications where you want to build reporting, analytics or dashboards on top of Celery task results.
• Ideal for real-time data indexing and retrieval.
• Ideal for high frequency of data ingestion, and faster query processing.
• Scalable as well using Elasticsearch's shadring and replication settings.

Elasticsearch Result Backend

Celery result backend has built-in support for Elasticsearch, a distributed search and analytics engine.

How to configure Elasticsearch result backend ?

To use elasticsearch as your result backend, you have to prefix the backend URI with elasticsearch://. Let's see an example :

celery_app = Celery('celery_app', backend='elasticsearch://localhost:9200/index_name', broker='amqp://guest@localhost//')

When to use Elasticsearch result backend ?

• Suitable for applications where you want to build reporting, analytics or dashboards on top of Celery task results.
• Ideal for real-time data indexing and retrieval.
• Ideal for high frequency of data ingestion, and faster query processing.
• Scalable as well using Elasticsearch's shadring and replication settings.

IronCache Result Backend

IronCache is another caching engine similar to Memcached, but it's a managed service.

How to configure IronCache result backend ?

To use IronCache as your result backend, you have to prefix the backend URI with ironcache://. Let's see an example :

celery_app = Celery('celery_app', backend='ironcache://project_id:token@/cache_name', broker='amqp://guest@localhost//')

When to use IronCache result backend ?

• When you prefer a managed caching solution.
• Ideal to scale because of IronCache's distributed caching.

Couchbase Result Backend

Couchbase is another high-performance NoSQL datanase similar to MongoDB.

How to configure Couchbase result backend ?

To use Couchbase as your result backend, you have to prefix the backend URI with couchbase://. Let's see an example :

celery_app = Celery('celery_app', backend='couchbase://localhost/bucket_name', broker='amqp://guest@localhost//')

When to use Couchbase result backend ?

• Ideal for low-latency data access, and storage.
• Easy to scale as Couchbase can scale horizontally.
• Query performance can be improved with optimal couchbase bucket configurations.

ArangoDB Result Backend

ArangoDB is a multi-model database, it can be used as Celery result backend too.

How to configure ArangoDB result backend ?

To use ArangoDB as your result backend, you have to prefix the backend URI with arangodb://. Let's see an example :

celery_app = Celery('celery_app', backend='arangodb://localhost:8529/db_name', broker='amqp://guest@localhost//')

When to use ArangoDB result backend ?

• Ideal if your use-case requires use of a multi-model db, for example using graph, document, kv pairs all at once.
• Ideal if you plan on using different schema definitions for different task results.
• Easy to scale as ArangoDB can scale both vertically and horizontally.
• Query performance can be improved with indexing and other db optimization techniques.

CouchDB Result Backend

Apache CouchDB is a distributed, multi-master NoSQL database.

How to configure CouchDB result backend ?

To use CouchDB as your result backend, you have to prefix the backend URI with couchdb://. Let's see an example :

celery_app = Celery('celery_app', backend='couchdb://localhost:5984/db_name', broker='amqp://guest@localhost//')

When to use CouchDB result backend ?

• Ideal if you plan on using different schema definitions for different task results.
• Easy to scale as CouchDB can scale both vertically and horizontally.
• Query performance can be improved with indexing and other db optimization techniques.

Azure CosmosDB Result Backend (Experimental)

Azure CosmosDB is a NoSQL database that allows querying data via SQL queries. It's a managed service by Microsoft Azure.

How to configure CosmosDB result backend ?

To use CosmosDB as your result backend, you have to prefix the backend URI with cosmosdbsql://. Let's see an example :

celery_app = Celery('celery_app', backend='cosmosdbsql://<connection-string>', broker='amqp://guest@localhost//')

When to use CosmosDB result backend ?

• Ideal if your use-case requires use of a multi-model db, for example using graph, document, kv pairs all at once.
• Ideal if you plan on using different schema definitions for different task results.
• Query performance can be improved with indexing and other db optimization techniques.

Filesystem Result Backend

Celery has built-in support to store task results as files in your local filesystem. Celery uses a shared directory on the filesystem to keep the task results.

How to configure Filesystem result backend ?

To use Filesystem as your result backend, you have to prefix the backend URI with file://. Let's see an example :

celery_app = Celery('celery_app', backend='file:///shared/results', broker='amqp://guest@localhost//')

When to use Filesystem result backend ?

• Only for small and POC deployments, it's not recommended to use filesystem as your result backend for high traffic applications.
• Ideal for development and debugging projects.

Consul Result Backend

Consul is a distributed key-value store that can be used as a result backend in Celery.

How to configure Consul result backend ?

To use Consul as your result backend, you have to prefix the backend URI with consul://. Let's see an example :

celery_app = Celery('celery_app', backend='consul://localhost:8500', broker='amqp://guest@localhost//')

When to use Consul result backend ?

• Easy to scale because of Consul's distributed nature.

Azure Block Blob Result Backend

Azure Block Blob is an Object storage service managed by Microsoft Azure to store files as blobs. It can be used as a Result backend too, very much similar like Filesystem result backend, but remote and infinitely scalable.

How to configure Azure Block Blob result backend ?

To use Azure Block Blob as your result backend, you have to prefix the backend URI with azureblockblob://. Let's see an example :

celery_app = Celery('celery_app', backend='azureblockblob://account_name:account_key@container', broker='amqp://guest@localhost//')

When to use Azure Block Blob result backend ?

• Super Low latency for applications deployed on Azure.
• Ideal for scalable, durable cloud storage backed result backend.
• Cost effective solution for large amounts of data.

Amazon S3 Result Backend

Amazon S3 is an Object storage service similar to Microsoft Azure Block Blob to store files as blobs. S3 is a managed service by AWS.

How to configure Amazon S3 result backend ?

To use Amazon S3 as your result backend, you have to prefix the backend URI with s3://. Let's see an example :

celery_app = Celery('celery_app', backend='s3://aws_access_key_id:aws_secret_access_key@bucket_name', broker='amqp://guest@localhost//')

When to use Amazon S3 result backend ?

• Super Low latency for applications deployed on AWS.
• Ideal for scalable, durable cloud storage backed result backend.
• Cost effective solution for large amounts of data.

Google Cloud Storage (GCS) Result Backend

Google Cloud Storage (GCS) is an Object storage service similar to Microsoft Azure Block Blob to store files as blobs. GCS is a managed service by Google Cloud.

How to configure Google Cloud Storage (GCS) result backend ?

To use Google Cloud Storage (GCS) as your result backend, you have to prefix the backend URI with gcs://. Let's see an example :

celery_app = Celery('celery_app', backend='gcs://your_project_id:your_keyfile@bucket_name', broker='amqp://guest@localhost//')

When to use Google Cloud Storage (GCS) result backend ?

• Super Low latency for applications deployed on Google Cloud.
• Ideal for scalable, durable cloud storage backed result backend.
• Cost effective solution for large amounts of data.

Conclusion

And that's all! In this post we discussed all the supported Celery result backends. Always remember, choosing the result backend for Celery depends on various factors like your use case, whether the backend is scalable and reliable or not, cost-factor etc. In this post, we mentioned all type of backends.