# Celery > This is a list of external blog posts, tutorials, and slides related to --- # Community Resources {#community} This is a list of external blog posts, tutorials, and slides related to Celery. If you have a link that\'s missing from this list, please contact the mailing-list or submit a patch. ::: {.contents local=""} ::: ## Resources {#community-resources} ### Who\'s using Celery {#res-using-celery} ### Wiki {#res-wiki} ### Celery questions on Stack Overflow {#res-stackoverflow} ### Mailing-list Archive: celery-users {#res-mailing-list-archive} ## News[]{#res-irc-logs} {#community-news} This section has moved to the Celery homepage: --- # Copyright *Celery User Manual* by Ask Solem Copyright © 2009-2016, Ask Solem. All rights reserved. This material may be copied or distributed only subject to the terms and conditions set forth in the [Creative Commons Attribution-ShareAlike 4.0 International]{.title-ref} \<\>\`\_ license. You may share and adapt the material, even for commercial purposes, but you must give the original author credit. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same license or a license compatible to this one. ::: note ::: title Note ::: While the *Celery* documentation is offered under the Creative Commons *Attribution-ShareAlike 4.0 International* license the Celery *software* is offered under the [BSD License (3 Clause)](http://www.opensource.org/licenses/BSD-3-Clause) ::: --- # First steps with Django {#django-first-steps} ## Using Celery with Django ::: note ::: title Note ::: Previous versions of Celery required a separate library to work with Django, but since 3.1 this is no longer the case. Django is supported out of the box now so this document only contains a basic way to integrate Celery and Django. You\'ll use the same API as non-Django users so you\'re recommended to read the `first-steps`{.interpreted-text role="ref"} tutorial first and come back to this tutorial. When you have a working example you can continue to the `next-steps`{.interpreted-text role="ref"} guide. ::: ::: note ::: title Note ::: Celery 5.5.x supports Django 2.2 LTS or newer versions. Please use Celery 5.2.x for versions older than Django 2.2 or Celery 4.4.x if your Django version is older than 1.11. ::: To use Celery with your Django project you must first define an instance of the Celery library (called an \"app\") If you have a modern Django project layout like: - proj/ - manage.py - proj/ - __init__.py - settings.py - urls.py then the recommended way is to create a new [proj/proj/celery.py]{.title-ref} module that defines the Celery instance: file : [proj/proj/celery.py]{.title-ref} ::: literalinclude ../../examples/django/proj/celery.py ::: Then you need to import this app in your `proj/proj/__init__.py`{.interpreted-text role="file"} module. This ensures that the app is loaded when Django starts so that the `@shared_task` decorator (mentioned later) will use it: `proj/proj/__init__.py`{.interpreted-text role="file"}: ::: literalinclude ../../examples/django/proj/\_\_init\_\_.py ::: Note that this example project layout is suitable for larger projects, for simple projects you may use a single contained module that defines both the app and tasks, like in the `tut-celery`{.interpreted-text role="ref"} tutorial. Let\'s break down what happens in the first module, first, we set the default `DJANGO_SETTINGS_MODULE`{.interpreted-text role="envvar"} environment variable for the `celery`{.interpreted-text role="program"} command-line program: ``` python os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings') ``` You don\'t need this line, but it saves you from always passing in the settings module to the `celery` program. It must always come before creating the app instances, as is what we do next: ``` python app = Celery('proj') ``` This is our instance of the library, you can have many instances but there\'s probably no reason for that when using Django. We also add the Django settings module as a configuration source for Celery. This means that you don\'t have to use multiple configuration files, and instead configure Celery directly from the Django settings; but you can also separate them if wanted. ``` python app.config_from_object('django.conf:settings', namespace='CELERY') ``` The uppercase name-space means that all `Celery configuration options `{.interpreted-text role="ref"} must be specified in uppercase instead of lowercase, and start with `CELERY_`, so for example the `task_always_eager`{.interpreted-text role="setting"} setting becomes `CELERY_TASK_ALWAYS_EAGER`, and the `broker_url`{.interpreted-text role="setting"} setting becomes `CELERY_BROKER_URL`. This also applies to the workers settings, for instance, the `worker_concurrency`{.interpreted-text role="setting"} setting becomes `CELERY_WORKER_CONCURRENCY`. For example, a Django project\'s configuration file might include: ``` {.python caption="settings.py"} ... # Celery Configuration Options CELERY_TIMEZONE = "Australia/Tasmania" CELERY_TASK_TRACK_STARTED = True CELERY_TASK_TIME_LIMIT = 30 * 60 ``` You can pass the settings object directly instead, but using a string is better since then the worker doesn\'t have to serialize the object. The `CELERY_` namespace is also optional, but recommended (to prevent overlap with other Django settings). Next, a common practice for reusable apps is to define all tasks in a separate `tasks.py` module, and Celery does have a way to auto-discover these modules: ``` python app.autodiscover_tasks() ``` With the line above Celery will automatically discover tasks from all of your installed apps, following the `tasks.py` convention: - app1/ - tasks.py - models.py - app2/ - tasks.py - models.py This way you don\'t have to manually add the individual modules to the `CELERY_IMPORTS `{.interpreted-text role="setting"} setting. Finally, the `debug_task` example is a task that dumps its own request information. This is using the new `bind=True` task option introduced in Celery 3.1 to easily refer to the current task instance. ### Using the `@shared_task` decorator The tasks you write will probably live in reusable apps, and reusable apps cannot depend on the project itself, so you also cannot import your app instance directly. The `@shared_task` decorator lets you create tasks without having any concrete app instance: `demoapp/tasks.py`{.interpreted-text role="file"}: ::: literalinclude ../../examples/django/demoapp/tasks.py ::: ::: seealso You can find the full source code for the Django example project at: ::: ### Trigger tasks at the end of the database transaction A common pitfall with Django is triggering a task immediately and not wait until the end of the database transaction, which means that the Celery task may run before all changes are persisted to the database. For example: ``` python # views.py def create_user(request): # Note: simplified example, use a form to validate input user = User.objects.create(username=request.POST['username']) send_email.delay(user.pk) return HttpResponse('User created') # task.py @shared_task def send_email(user_pk): user = User.objects.get(pk=user_pk) # send email ... ``` In this case, the `send_email` task could start before the view has committed the transaction to the database, and therefore the task may not be able to find the user. A common solution is to use Django\'s [on_commit](https://docs.djangoproject.com/en/stable/topics/db/transactions/#django.db.transaction.on_commit) hook to trigger the task after the transaction has been committed: ``` diff - send_email.delay(user.pk) + transaction.on_commit(lambda: send_email.delay(user.pk)) ``` ::: versionadded 5.4 ::: Since this is such a common pattern, Celery 5.4 introduced a handy shortcut for this, using a `~celery.contrib.django.task.DjangoTask`{.interpreted-text role="class"}. Instead of calling `~celery.app.task.Task.delay`{.interpreted-text role="meth"}, you should call `~celery.contrib.django.task.DjangoTask.delay_on_commit`{.interpreted-text role="meth"}: ``` diff - send_email.delay(user.pk) + send_email.delay_on_commit(user.pk) ``` This API takes care of wrapping the call into the [on_commit](https://docs.djangoproject.com/en/stable/topics/db/transactions/#django.db.transaction.on_commit) hook for you. In rare cases where you want to trigger a task without waiting, the existing `~celery.app.task.Task.delay`{.interpreted-text role="meth"} API is still available. One key difference compared to the `delay` method, is that `delay_on_commit` will NOT return the task ID back to the caller. The task is not sent to the broker when you call the method, only when the Django transaction finishes. If you need the task ID, best to stick to `~celery.app.task.Task.delay`{.interpreted-text role="meth"}. This task class should be used automatically if you\'ve follow the setup steps above. However, if your app `uses a custom task base class `{.interpreted-text role="ref"}, you\'ll need inherit from `~celery.contrib.django.task.DjangoTask`{.interpreted-text role="class"} instead of `~celery.app.task.Task`{.interpreted-text role="class"} to get this behaviour. ### Django Connection pool From Django 5.1+ there is built-in support for database connection pooling. If you enable it in Django `DATABASES` settings Celery will automatically handle connection pool closing in worker processes via `close_pool` database backend method as [sharing connections across processes is not possible.](https://github.com/psycopg/psycopg/issues/544#issuecomment-1500886864) You can find more about Connection pool at [Django docs.](https://docs.djangoproject.com/en/dev/ref/databases/#connection-pool) ## Extensions ### `django-celery-results` - Using the Django ORM/Cache as a result backend {#django-celery-results} The `django-celery-results`{.interpreted-text role="pypi"} extension provides result backends using either the Django ORM, or the Django Cache framework. To use this with your project you need to follow these steps: 1. Install the `django-celery-results`{.interpreted-text role="pypi"} library: > ``` console > $ pip install django-celery-results > ``` 2. Add `django_celery_results` to `INSTALLED_APPS` in your Django project\'s `settings.py`{.interpreted-text role="file"}: INSTALLED_APPS = ( ..., 'django_celery_results', ) Note that there is no dash in the module name, only underscores. 3. Create the Celery database tables by performing a database migrations: > ``` console > $ python manage.py migrate django_celery_results > ``` 4. Configure Celery to use the `django-celery-results`{.interpreted-text role="pypi"} backend. > Assuming you are using Django\'s `settings.py`{.interpreted-text > role="file"} to also configure Celery, add the following settings: > > ``` python > CELERY_RESULT_BACKEND = 'django-db' > ``` > > When using the cache backend, you can specify a cache defined > within Django\'s CACHES setting. > > ``` python > CELERY_RESULT_BACKEND = 'django-cache' > > # pick which cache from the CACHES setting. > CELERY_CACHE_BACKEND = 'default' > > # django setting. > CACHES = { > 'default': { > 'BACKEND': 'django.core.cache.backends.db.DatabaseCache', > 'LOCATION': 'my_cache_table', > } > } > ``` > > For additional configuration options, view the > `conf-result-backend`{.interpreted-text role="ref"} reference. ### `django-celery-beat` - Database-backed Periodic Tasks with Admin interface. See `beat-custom-schedulers`{.interpreted-text role="ref"} for more information. ## Starting the worker process In a production environment you\'ll want to run the worker in the background as a daemon - see `daemonizing`{.interpreted-text role="ref"} - but for testing and development it is useful to be able to start a worker instance by using the `celery worker`{.interpreted-text role="program"} manage command, much as you\'d use Django\'s `manage.py runserver`{.interpreted-text role="command"}: ``` console $ celery -A proj worker -l INFO ``` For a complete listing of the command-line options available, use the help command: ``` console $ celery --help ``` ## Where to go from here If you want to learn more you should continue to the `Next Steps `{.interpreted-text role="ref"} tutorial, and after that you can study the `User Guide `{.interpreted-text role="ref"}. --- # Django Release : Date : ::: {.toctree maxdepth="2"} first-steps-with-django ::: --- # Frequently Asked Questions {#faq} ::: {.contents local=""} ::: ## General {#faq-general} ### What kinds of things should I use Celery for? {#faq-when-to-use} **Answer:** [Queue everything and delight everyone](https://decafbad.com/blog/2008/07/04/queue-everything-and-delight-everyone) is a good article describing why you\'d use a queue in a web context. These are some common use cases: - Running something in the background. For example, to finish the web request as soon as possible, then update the users page incrementally. This gives the user the impression of good performance and \"snappiness\", even though the real work might actually take some time. - Running something after the web request has finished. - Making sure something is done, by executing it asynchronously and using retries. - Scheduling periodic work. And to some degree: - Distributed computing. - Parallel execution. ## Misconceptions {#faq-misconceptions} ### Does Celery really consist of 50.000 lines of code? {#faq-loc} **Answer:** No, this and similarly large numbers have been reported at various locations. The numbers as of this writing are: > - core: 7,141 lines of code. > - tests: 14,209 lines. > - backends, contrib, compat utilities: 9,032 lines. Lines of code isn\'t a useful metric, so even if Celery did consist of 50k lines of code you wouldn\'t be able to draw any conclusions from such a number. ### Does Celery have many dependencies? A common criticism is that Celery uses too many dependencies. The rationale behind such a fear is hard to imagine, especially considering code reuse as the established way to combat complexity in modern software development, and that the cost of adding dependencies is very low now that package managers like pip and PyPI makes the hassle of installing and maintaining dependencies a thing of the past. Celery has replaced several dependencies along the way, and the current list of dependencies are: #### celery - `kombu`{.interpreted-text role="pypi"} Kombu is part of the Celery ecosystem and is the library used to send and receive messages. It\'s also the library that enables us to support many different message brokers. It\'s also used by the OpenStack project, and many others, validating the choice to separate it from the Celery code-base. - `billiard`{.interpreted-text role="pypi"} Billiard is a fork of the Python multiprocessing module containing many performance and stability improvements. It\'s an eventual goal that these improvements will be merged back into Python one day. It\'s also used for compatibility with older Python versions that don\'t come with the multiprocessing module. #### kombu Kombu depends on the following packages: - `amqp`{.interpreted-text role="pypi"} The underlying pure-Python amqp client implementation. AMQP being the default broker this is a natural dependency. ::: note ::: title Note ::: To handle the dependencies for popular configuration choices Celery defines a number of \"bundle\" packages, see `bundles`{.interpreted-text role="ref"}. ::: ### Is Celery heavy-weight? {#faq-heavyweight} Celery poses very little overhead both in memory footprint and performance. But please note that the default configuration isn\'t optimized for time nor space, see the `guide-optimizing`{.interpreted-text role="ref"} guide for more information. ### Is Celery dependent on pickle? {#faq-serialization-is-a-choice} **Answer:** No, Celery can support any serialization scheme. We have built-in support for JSON, YAML, Pickle, and msgpack. Every task is associated with a content type, so you can even send one task using pickle, another using JSON. The default serialization support used to be pickle, but since 4.0 the default is now JSON. If you require sending complex Python objects as task arguments, you can use pickle as the serialization format, but see notes in `security-serializers`{.interpreted-text role="ref"}. If you need to communicate with other languages you should use a serialization format suited to that task, which pretty much means any serializer that\'s not pickle. You can set a global default serializer, the default serializer for a particular Task, or even what serializer to use when sending a single task instance. ### Is Celery for Django only? {#faq-is-celery-for-django-only} **Answer:** No, you can use Celery with any framework, web or otherwise. ### Do I have to use AMQP/RabbitMQ? {#faq-is-celery-for-rabbitmq-only} **Answer**: No, although using RabbitMQ is recommended you can also use Redis, SQS, or Qpid. See `brokers`{.interpreted-text role="ref"} for more information. Redis as a broker won\'t perform as well as an AMQP broker, but the combination RabbitMQ as broker and Redis as a result store is commonly used. If you have strict reliability requirements you\'re encouraged to use RabbitMQ or another AMQP broker. Some transports also use polling, so they\'re likely to consume more resources. However, if you for some reason aren\'t able to use AMQP, feel free to use these alternatives. They will probably work fine for most use cases, and note that the above points are not specific to Celery; If using Redis/database as a queue worked fine for you before, it probably will now. You can always upgrade later if you need to. ### Is Celery multilingual? {#faq-is-celery-multilingual} **Answer:** Yes. `~celery.bin.worker`{.interpreted-text role="mod"} is an implementation of Celery in Python. If the language has an AMQP client, there shouldn\'t be much work to create a worker in your language. A Celery worker is just a program connecting to the broker to process messages. Also, there\'s another way to be language-independent, and that\'s to use REST tasks, instead of your tasks being functions, they\'re URLs. With this information you can even create simple web servers that enable preloading of code. Simply expose an endpoint that performs an operation, and create a task that just performs an HTTP request to that endpoint. You can also use [Flower\'s](https://flower.readthedocs.io) [REST API](https://flower.readthedocs.io/en/latest/api.html#post--api-task-async-apply-(.+)) to invoke tasks. ## Troubleshooting {#faq-troubleshooting} ### MySQL is throwing deadlock errors, what can I do? {#faq-mysql-deadlocks} **Answer:** MySQL has default isolation level set to [REPEATABLE-READ]{.title-ref}, if you don\'t really need that, set it to [READ-COMMITTED]{.title-ref}. You can do that by adding the following to your `my.cnf`{.interpreted-text role="file"}: [mysqld] transaction-isolation = READ-COMMITTED For more information about InnoDB's transaction model see [MySQL - The InnoDB Transaction Model and Locking](https://dev.mysql.com/doc/refman/5.1/en/innodb-transaction-model.html) in the MySQL user manual. (Thanks to Honza Kral and Anton Tsigularov for this solution) ### The worker isn\'t doing anything, just hanging {#faq-worker-hanging} **Answer:** See [MySQL is throwing deadlock errors, what can I do?](#mysql-is-throwing-deadlock-errors-what-can-i-do), or [Why is Task.delay/apply\\\*/the worker just hanging?](). ### Task results aren\'t reliably returning {#faq-results-unreliable} **Answer:** If you\'re using the database backend for results, and in particular using MySQL, see [MySQL is throwing deadlock errors, what can I do?](#mysql-is-throwing-deadlock-errors-what-can-i-do). ### Why is Task.delay/apply\*/the worker just hanging? {#faq-publish-hanging} **Answer:** There\'s a bug in some AMQP clients that\'ll make it hang if it\'s not able to authenticate the current user, the password doesn\'t match or the user doesn\'t have access to the virtual host specified. Be sure to check your broker logs (for RabbitMQ that\'s `/var/log/rabbitmq/rabbit.log`{.interpreted-text role="file"} on most systems), it usually contains a message describing the reason. ### Does it work on FreeBSD? {#faq-worker-on-freebsd} **Answer:** Depends; When using the RabbitMQ (AMQP) and Redis transports it should work out of the box. For other transports the compatibility prefork pool is used and requires a working POSIX semaphore implementation, this is enabled in FreeBSD by default since FreeBSD 8.x. For older version of FreeBSD, you have to enable POSIX semaphores in the kernel and manually recompile billiard. Luckily, Viktor Petersson has written a tutorial to get you started with Celery on FreeBSD here: ### I\'m having [IntegrityError: Duplicate Key]{.title-ref} errors. Why? {#faq-duplicate-key-errors} **Answer:** See [MySQL is throwing deadlock errors, what can I do?](#mysql-is-throwing-deadlock-errors-what-can-i-do). Thanks to `@howsthedotcom`{.interpreted-text role="github_user"}. ### Why aren\'t my tasks processed? {#faq-worker-stops-processing} **Answer:** With RabbitMQ you can see how many consumers are currently receiving tasks by running the following command: ``` console $ rabbitmqctl list_queues -p name messages consumers Listing queues ... celery 2891 2 ``` This shows that there\'s 2891 messages waiting to be processed in the task queue, and there are two consumers processing them. One reason that the queue is never emptied could be that you have a stale worker process taking the messages hostage. This could happen if the worker wasn\'t properly shut down. When a message is received by a worker the broker waits for it to be acknowledged before marking the message as processed. The broker won\'t re-send that message to another consumer until the consumer is shut down properly. If you hit this problem you have to kill all workers manually and restart them: ``` console $ pkill 'celery worker' $ # - If you don't have pkill use: $ # ps auxww | awk '/celery worker/ {print $2}' | xargs kill ``` You may have to wait a while until all workers have finished executing tasks. If it\'s still hanging after a long time you can kill them by force with: ``` console $ pkill -9 'celery worker' $ # - If you don't have pkill use: $ # ps auxww | awk '/celery worker/ {print $2}' | xargs kill -9 ``` ### Why won\'t my Task run? {#faq-task-does-not-run} **Answer:** There might be syntax errors preventing the tasks module being imported. You can find out if Celery is able to run the task by executing the task manually: ``` python >>> from myapp.tasks import MyPeriodicTask >>> MyPeriodicTask.delay() ``` Watch the workers log file to see if it\'s able to find the task, or if some other error is happening. ### Why won\'t my periodic task run? {#faq-periodic-task-does-not-run} **Answer:** See [Why won\'t my Task run?](#why-wont-my-task-run). ### How do I purge all waiting tasks? {#faq-purge-the-queue} **Answer:** You can use the `celery purge` command to purge all configured task queues: ``` console $ celery -A proj purge ``` or programmatically: ``` pycon >>> from proj.celery import app >>> app.control.purge() 1753 ``` If you only want to purge messages from a specific queue you have to use the AMQP API or the `celery amqp`{.interpreted-text role="program"} utility: ``` console $ celery -A proj amqp queue.purge ``` The number 1753 is the number of messages deleted. You can also start the worker with the `--purge `{.interpreted-text role="option"} option enabled to purge messages when the worker starts. ### I\'ve purged messages, but there are still messages left in the queue? {#faq-messages-left-after-purge} **Answer:** Tasks are acknowledged (removed from the queue) as soon as they\'re actually executed. After the worker has received a task, it will take some time until it\'s actually executed, especially if there are a lot of tasks already waiting for execution. Messages that aren\'t acknowledged are held on to by the worker until it closes the connection to the broker (AMQP server). When that connection is closed (e.g., because the worker was stopped) the tasks will be re-sent by the broker to the next available worker (or the same worker when it has been restarted), so to properly purge the queue of waiting tasks you have to stop all the workers, and then purge the tasks using `celery.control.purge`{.interpreted-text role="func"}. ## Results {#faq-results} ### How do I get the result of a task if I have the ID that points there? {#faq-get-result-by-task-id} **Answer**: Use \`task.AsyncResult\`: ``` pycon >>> result = my_task.AsyncResult(task_id) >>> result.get() ``` This will give you a `~celery.result.AsyncResult`{.interpreted-text role="class"} instance using the tasks current result backend. If you need to specify a custom result backend, or you want to use the current application\'s default backend you can use `@AsyncResult`{.interpreted-text role="class"}: ``` pycon >>> result = app.AsyncResult(task_id) >>> result.get() ``` ## Security {#faq-security} ### Isn\'t using [pickle]{.title-ref} a security concern? **Answer**: Indeed, since Celery 4.0 the default serializer is now JSON to make sure people are choosing serializers consciously and aware of this concern. It\'s essential that you protect against unauthorized access to your broker, databases and other services transmitting pickled data. Note that this isn\'t just something you should be aware of with Celery, for example also Django uses pickle for its cache client. For the task messages you can set the `task_serializer`{.interpreted-text role="setting"} setting to \"json\" or \"yaml\" instead of pickle. Similarly for task results you can set `result_serializer`{.interpreted-text role="setting"}. For more details of the formats used and the lookup order when checking what format to use for a task see `calling-serializers`{.interpreted-text role="ref"} ### Can messages be encrypted? **Answer**: Some AMQP brokers supports using SSL (including RabbitMQ). You can enable this using the `broker_use_ssl`{.interpreted-text role="setting"} setting. It\'s also possible to add additional encryption and security to messages, if you have a need for this then you should contact the `mailing-list`{.interpreted-text role="ref"}. ### Is it safe to run `celery worker`{.interpreted-text role="program"} as root? **Answer**: No! We\'re not currently aware of any security issues, but it would be incredibly naive to assume that they don\'t exist, so running the Celery services (`celery worker`{.interpreted-text role="program"}, `celery beat`{.interpreted-text role="program"}, `celeryev`{.interpreted-text role="program"}, etc) as an unprivileged user is recommended. ## Brokers {#faq-brokers} ### Why is RabbitMQ crashing? **Answer:** RabbitMQ will crash if it runs out of memory. This will be fixed in a future release of RabbitMQ. please refer to the RabbitMQ FAQ: ::: note ::: title Note ::: This is no longer the case, RabbitMQ versions 2.0 and above includes a new persister, that\'s tolerant to out of memory errors. RabbitMQ 2.1 or higher is recommended for Celery. If you\'re still running an older version of RabbitMQ and experience crashes, then please upgrade! ::: Misconfiguration of Celery can eventually lead to a crash on older version of RabbitMQ. Even if it doesn\'t crash, this can still consume a lot of resources, so it\'s important that you\'re aware of the common pitfalls. - Events. Running `~celery.bin.worker`{.interpreted-text role="mod"} with the `-E `{.interpreted-text role="option"} option will send messages for events happening inside of the worker. Events should only be enabled if you have an active monitor consuming them, or if you purge the event queue periodically. - AMQP backend results. When running with the AMQP result backend, every task result will be sent as a message. If you don\'t collect these results, they will build up and RabbitMQ will eventually run out of memory. This result backend is now deprecated so you shouldn\'t be using it. Use either the RPC backend for rpc-style calls, or a persistent backend if you need multi-consumer access to results. Results expire after 1 day by default. It may be a good idea to lower this value by configuring the `result_expires`{.interpreted-text role="setting"} setting. If you don\'t use the results for a task, make sure you set the [ignore_result]{.title-ref} option: ``` python @app.task(ignore_result=True) def mytask(): pass class MyTask(Task): ignore_result = True ``` ### Can I use Celery with ActiveMQ/STOMP? {#faq-use-celery-with-stomp} **Answer**: No. It used to be supported by `Carrot`{.interpreted-text role="pypi"} (our old messaging library) but isn\'t currently supported in `Kombu`{.interpreted-text role="pypi"} (our new messaging library). ### What features aren\'t supported when not using an AMQP broker? {#faq-non-amqp-missing-features} This is an incomplete list of features not available when using the virtual transports: > - Remote control commands (supported only by Redis). > > - Monitoring with events may not work in all virtual transports. > > - > > The [header]{.title-ref} and [fanout]{.title-ref} exchange types > > : ([fanout]{.title-ref} is supported by Redis). ## Tasks {#faq-tasks} ### How can I reuse the same connection when calling tasks? {#faq-tasks-connection-reuse} **Answer**: See the `broker_pool_limit`{.interpreted-text role="setting"} setting. The connection pool is enabled by default since version 2.5. ### `sudo`{.interpreted-text role="command"} in a `subprocess`{.interpreted-text role="mod"} returns `None`{.interpreted-text role="const"} {#faq-sudo-subprocess} There\'s a `sudo`{.interpreted-text role="command"} configuration option that makes it illegal for process without a tty to run `sudo`{.interpreted-text role="command"}: ``` text Defaults requiretty ``` If you have this configuration in your `/etc/sudoers`{.interpreted-text role="file"} file then tasks won\'t be able to call `sudo`{.interpreted-text role="command"} when the worker is running as a daemon. If you want to enable that, then you need to remove the line from `/etc/sudoers`{.interpreted-text role="file"}. See: ### Why do workers delete tasks from the queue if they\'re unable to process them? {#faq-deletes-unknown-tasks} **Answer**: The worker rejects unknown tasks, messages with encoding errors and messages that don\'t contain the proper fields (as per the task message protocol). If it didn\'t reject them they could be redelivered again and again, causing a loop. Recent versions of RabbitMQ has the ability to configure a dead-letter queue for exchange, so that rejected messages is moved there. ### Can I call a task by name? {#faq-execute-task-by-name} **Answer**: Yes, use `@send_task`{.interpreted-text role="meth"}. You can also call a task by name, from any language, using an AMQP client: ``` python >>> app.send_task('tasks.add', args=[2, 2], kwargs={}) ``` To use `chain`, `chord` or `group` with tasks called by name, use the `@Celery.signature`{.interpreted-text role="meth"} method: ``` python >>> chain( ... app.signature('tasks.add', args=[2, 2], kwargs={}), ... app.signature('tasks.add', args=[1, 1], kwargs={}) ... ).apply_async() ``` ### Can I get the task id of the current task? {#faq-get-current-task-id} **Answer**: Yes, the current id and more is available in the task request: @app.task(bind=True) def mytask(self): cache.set(self.request.id, "Running") For more information see `task-request-info`{.interpreted-text role="ref"}. If you don\'t have a reference to the task instance you can use `app.current_task <@current_task>`{.interpreted-text role="attr"}: ``` python >>> app.current_task.request.id ``` But note that this will be any task, be it one executed by the worker, or a task called directly by that task, or a task called eagerly. To get the current task being worked on specifically, use `app.current_worker_task <@current_worker_task>`{.interpreted-text role="attr"}: ``` python >>> app.current_worker_task.request.id ``` ::: note ::: title Note ::: Both `~@current_task`{.interpreted-text role="attr"}, and `~@current_worker_task`{.interpreted-text role="attr"} can be `None`{.interpreted-text role="const"}. ::: ### Can I specify a custom task_id? {#faq-custom-task-ids} **Answer**: Yes, use the [task_id]{.title-ref} argument to `Task.apply_async`{.interpreted-text role="meth"}: ``` pycon >>> task.apply_async(args, kwargs, task_id='…') ``` ### Can I use decorators with tasks? **Answer**: Yes, but please see note in the sidebar at `task-basics`{.interpreted-text role="ref"}. ### Can I use natural task ids? {#faq-natural-task-ids} **Answer**: Yes, but make sure it\'s unique, as the behavior for two tasks existing with the same id is undefined. The world will probably not explode, but they can definitely overwrite each others results. ### Can I run a task once another task has finished? {#faq-task-callbacks} **Answer**: Yes, you can safely launch a task inside a task. A common pattern is to add callbacks to tasks: ``` python from celery.utils.log import get_task_logger logger = get_task_logger(__name__) @app.task def add(x, y): return x + y @app.task(ignore_result=True) def log_result(result): logger.info("log_result got: %r", result) ``` Invocation: ``` pycon >>> (add.s(2, 2) | log_result.s()).delay() ``` See `userguide/canvas`{.interpreted-text role="doc"} for more information. ### Can I cancel the execution of a task? {#faq-cancel-task} **Answer**: Yes, Use `result.revoke() `{.interpreted-text role="meth"}: ``` pycon >>> result = add.apply_async(args=[2, 2], countdown=120) >>> result.revoke() ``` or if you only have the task id: ``` pycon >>> from proj.celery import app >>> app.control.revoke(task_id) ``` The latter also support passing a list of task-ids as argument. ### Why aren\'t my remote control commands received by all workers? {#faq-node-not-receiving-broadcast-commands} **Answer**: To receive broadcast remote control commands, every worker node creates a unique queue name, based on the nodename of the worker. If you have more than one worker with the same host name, the control commands will be received in round-robin between them. To work around this you can explicitly set the nodename for every worker using the `-n `{.interpreted-text role="option"} argument to `~celery.bin.worker`{.interpreted-text role="mod"}: ``` console $ celery -A proj worker -n worker1@%h $ celery -A proj worker -n worker2@%h ``` where `%h` expands into the current hostname. ### Can I send some tasks to only some servers? {#faq-task-routing} **Answer:** Yes, you can route tasks to one or more workers, using different message routing topologies, and a worker instance can bind to multiple queues. See `userguide/routing`{.interpreted-text role="doc"} for more information. ### Can I disable prefetching of tasks? {#faq-disable-prefetch} **Answer**: Maybe! The AMQP term \"prefetch\" is confusing, as it\'s only used to describe the task prefetching *limit*. There\'s no actual prefetching involved. Disabling the prefetch limits is possible, but that means the worker will consume as many tasks as it can, as fast as possible. You can use the `--disable-prefetch `{.interpreted-text role="option"} flag (or set `worker_disable_prefetch`{.interpreted-text role="setting"} to `True`) so that a worker only fetches a task when one of its processes is free. This feature is currently only supported when using Redis as the broker. A discussion on prefetch limits, and configuration settings for a worker that only reserves one task at a time is found here: `optimizing-prefetch-limit`{.interpreted-text role="ref"}. ### Can I change the interval of a periodic task at runtime? {#faq-change-periodic-task-interval-at-runtime} **Answer**: Yes, you can use the Django database scheduler, or you can create a new schedule subclass and override `~celery.schedules.schedule.is_due`{.interpreted-text role="meth"}: ``` python from celery.schedules import schedule class my_schedule(schedule): def is_due(self, last_run_at): return run_now, next_time_to_check ``` ### Does Celery support task priorities? {#faq-task-priorities} **Answer**: Yes, RabbitMQ supports priorities since version 3.5.0, and the Redis transport emulates priority support. You can also prioritize work by routing high priority tasks to different workers. In the real world this usually works better than per message priorities. You can use this in combination with rate limiting, and per message priorities to achieve a responsive system. ### Should I use retry or acks_late? {#faq-acks_late-vs-retry} **Answer**: Depends. It\'s not necessarily one or the other, you may want to use both. [Task.retry]{.title-ref} is used to retry tasks, notably for expected errors that is catch-able with the `try`{.interpreted-text role="keyword"} block. The AMQP transaction isn\'t used for these errors: **if the task raises an exception it\'s still acknowledged!** The [acks_late]{.title-ref} setting would be used when you need the task to be executed again if the worker (for some reason) crashes mid-execution. It\'s important to note that the worker isn\'t known to crash, and if it does it\'s usually an unrecoverable error that requires human intervention (bug in the worker, or task code). In an ideal world you could safely retry any task that\'s failed, but this is rarely the case. Imagine the following task: ``` python @app.task def process_upload(filename, tmpfile): # Increment a file count stored in a database increment_file_counter() add_file_metadata_to_db(filename, tmpfile) copy_file_to_destination(filename, tmpfile) ``` If this crashed in the middle of copying the file to its destination the world would contain incomplete state. This isn\'t a critical scenario of course, but you can probably imagine something far more sinister. So for ease of programming we have less reliability; It\'s a good default, users who require it and know what they are doing can still enable acks_late (and in the future hopefully use manual acknowledgment). In addition [Task.retry]{.title-ref} has features not available in AMQP transactions: delay between retries, max retries, etc. So use retry for Python errors, and if your task is idempotent combine that with [acks_late]{.title-ref} if that level of reliability is required. ### Can I schedule tasks to execute at a specific time? {#faq-schedule-at-specific-time} **Answer**: Yes. You can use the [eta]{.title-ref} argument of `Task.apply_async`{.interpreted-text role="meth"}. Note that using distant [eta]{.title-ref} times is not recommended, and in such case `periodic tasks`{.interpreted-text role="ref"} should be preferred. See `calling-eta`{.interpreted-text role="ref"} for more details. ### Can I safely shut down the worker? {#faq-safe-worker-shutdown} **Answer**: Yes, use the `TERM`{.interpreted-text role="sig"} signal. This will tell the worker to finish all currently executing jobs and shut down as soon as possible. No tasks should be lost even with experimental transports as long as the shutdown completes. You should never stop `~celery.bin.worker`{.interpreted-text role="mod"} with the `KILL`{.interpreted-text role="sig"} signal (`kill -9`), unless you\'ve tried `TERM`{.interpreted-text role="sig"} a few times and waited a few minutes to let it get a chance to shut down. Also make sure you kill the main worker process only, not any of its child processes. You can direct a kill signal to a specific child process if you know the process is currently executing a task the worker shutdown is depending on, but this also means that a `WorkerLostError` state will be set for the task so the task won\'t run again. Identifying the type of process is easier if you have installed the `setproctitle`{.interpreted-text role="pypi"} module: ``` console $ pip install setproctitle ``` With this library installed you\'ll be able to see the type of process in `ps`{.interpreted-text role="command"} listings, but the worker must be restarted for this to take effect. ::: seealso `worker-stopping`{.interpreted-text role="ref"} ::: ### Can I run the worker in the background on \[platform\]? {#faq-daemonizing} **Answer**: Yes, please see `daemonizing`{.interpreted-text role="ref"}. ## Django {#faq-django} ### What purpose does the database tables created by `django-celery-beat` have? {#faq-django-beat-database-tables} When the database-backed schedule is used the periodic task schedule is taken from the `PeriodicTask` model, there are also several other helper tables (`IntervalSchedule`, `CrontabSchedule`, `PeriodicTasks`). ### What purpose does the database tables created by `django-celery-results` have? {#faq-django-result-database-tables} The Django database result backend extension requires two extra models: `TaskResult` and `GroupResult`. ## Windows {#faq-windows} ### Does Celery support Windows? {#faq-windows-worker-embedded-beat} **Answer**: No. Since Celery 4.x, Windows is no longer supported due to lack of resources. But it may still work and we are happy to accept patches. --- # Using Google Pub/Sub {#broker-gcpubsub} ::: versionadded 5.5 ::: ## Installation {#broker-gcpubsub-installation} For the Google Pub/Sub support you have to install additional dependencies. You can install both Celery and these dependencies in one go using the `celery[gcpubsub]` `bundle `{.interpreted-text role="ref"}: ``` console $ pip install "celery[gcpubsub]" ``` ## Configuration {#broker-gcpubsub-configuration} You have to specify gcpubsub and google project in the broker URL: broker_url = 'gcpubsub://projects/project-id' where the URL format is: ``` text gcpubsub://projects/project-id ``` Please note that you must prefix the project-id with [projects/]{.title-ref} in the URL. The login credentials will be your regular GCP credentials set in the environment. ## Options ### Resource expiry The default settings are built to be as simple cost effective and intuitive as possible and to \"just work\". The pubsub messages and subscriptions are set to expire after 24 hours, and can be set by configuring the `expiration_seconds`{.interpreted-text role="setting"} setting: expiration_seconds = 86400 ::: seealso An overview of Google Cloud Pub/Sub settings can be found here: > ::: ### Ack Deadline Seconds {#gcpubsub-ack_deadline_seconds} The [ack_deadline_seconds]{.title-ref} defines the number of seconds pub/sub infra shall wait for the worker to acknowledge the task before the message is redelivered to another worker. This option is set via the `broker_transport_options`{.interpreted-text role="setting"} setting: broker_transport_options = {'ack_deadline_seconds': 60} # 1 minute. The default visibility timeout is 240 seconds, and the worker takes care for automatically extending all pending messages it has. ::: seealso An overview of Pub/Sub deadline can be found here: > ::: ### Polling Interval The polling interval decides the number of seconds to sleep between unsuccessful polls. This value can be either an int or a float. By default the value is *0.1 seconds*. However it doesn\'t mean that the worker will bomb the Pub/Sub API every 0.1 seconds when there\'s no more messages to read, since it will be blocked by a blocking call to the Pub/Sub API, which will only return when there\'s a new message to read or after 10 seconds. The polling interval can be set via the `broker_transport_options`{.interpreted-text role="setting"} setting: broker_transport_options = {'polling_interval': 0.3} Very frequent polling intervals can cause *busy loops*, resulting in the worker using a lot of CPU time. If you need sub-millisecond precision you should consider using another transport, like [RabbitMQ \]{.title-ref}, or [Redis \]{.title-ref}. ### Queue Prefix By default Celery will assign [kombu-]{.title-ref} prefix to the queue names, If you have other services using Pub/Sub you can configure it do so using the `broker_transport_options`{.interpreted-text role="setting"} setting: broker_transport_options = {'queue_name_prefix': 'kombu-'} ### Results {#gcpubsub-results-configuration} Google Cloud Storage (GCS) could be a good candidate to store the results. See `gcs`{.interpreted-text role="ref"} for more information. ## Caveats - When using celery flower, an \--inspect-timeout=10 option is required to detect workers state correctly. - GCP Subscriptions idle subscriptions (no queued messages) are configured to removal after 24hrs. This aims at reducing costs. - Queued and unacked messages are set to auto cleanup after 24 hrs. Same reason as above. - Channel queue size is approximation, and may not be accurate. The reason is that the Pub/Sub API does not provide a way to get the exact number of messages in a subscription. - Orphan (no subscriptions) Pub/Sub topics aren\'t being auto removed!! Since GCP introduces a hard limit of 10k topics per project, it is recommended to remove orphan topics manually in a periodic manner. - Max message size is limited to 10MB, as a workaround you can use GCS Backend to store the message in GCS and pass the GCS URL to the task. --- # Backends and Brokers {#brokers} Release : Date : Celery supports several message transport alternatives. ## Broker Instructions {#broker_toc} ::: {.toctree maxdepth="1"} rabbitmq redis sqs kafka gcpubsub ::: ## Broker Overview This is comparison table of the different transports supports, more information can be found in the documentation for each individual transport (see `broker_toc`{.interpreted-text role="ref"}). --------------- -------------- ---------------- -------------------- **Name** **Status** **Monitoring** **Remote Control** *RabbitMQ* Stable Yes Yes *Redis* Stable Yes Yes *Amazon SQS* Stable No No *Zookeeper* Experimental No No *Kafka* Experimental No No *GC PubSub* Experimental Yes Yes --------------- -------------- ---------------- -------------------- Experimental brokers may be functional but they don\'t have dedicated maintainers. Missing monitor support means that the transport doesn\'t implement events, and as such Flower, [celery events]{.title-ref}, [celerymon]{.title-ref} and other event-based monitoring tools won\'t work. Remote control means the ability to inspect and manage workers at runtime using the [celery inspect]{.title-ref} and [celery control]{.title-ref} commands (and other tools using the remote control API). ## Summaries *Note: This section is not comprehensive of backends and brokers.* Celery has the ability to communicate and store with many different backends (Result Stores) and brokers (Message Transports). ### Redis Redis can be both a backend and a broker. **As a Broker:** Redis works well for rapid transport of small messages. Large messages can congest the system. `See documentation for details `{.interpreted-text role="ref"} **As a Backend:** Redis is a super fast K/V store, making it very efficient for fetching the results of a task call. As with the design of Redis, you do have to consider the limit memory available to store your data, and how you handle data persistence. If result persistence is important, consider using another DB for your backend. ### RabbitMQ RabbitMQ is a broker. **As a Broker:** RabbitMQ handles larger messages better than Redis, however if many messages are coming in very quickly, scaling can become a concern and Redis or SQS should be considered unless RabbitMQ is running at very large scale. `See documentation for details `{.interpreted-text role="ref"} **As a Backend:** RabbitMQ can store results via `rpc://` backend. This backend creates separate temporary queue for each client. *Note: RabbitMQ (as the broker) and Redis (as the backend) are very commonly used together. If more guaranteed long-term persistence is needed from the result store, consider using PostgreSQL or MySQL (through SQLAlchemy), Cassandra, or a custom defined backend.* ### SQS SQS is a broker. If you already integrate tightly with AWS, and are familiar with SQS, it presents a great option as a broker. It is extremely scalable and completely managed, and manages task delegation similarly to RabbitMQ. It does lack some of the features of the RabbitMQ broker such as `worker remote control commands`. `See documentation for details `{.interpreted-text role="ref"} ### SQLAlchemy SQLAlchemy is a backend. It allows Celery to interface with MySQL, PostgreSQL, SQlite, and more. It is an ORM, and is the way Celery can use a SQL DB as a result backend. `See documentation for details `{.interpreted-text role="ref"} ### GCPubSub Google Cloud Pub/Sub is a broker. If you already integrate tightly with Google Cloud, and are familiar with Pub/Sub, it presents a great option as a broker. It is extremely scalable and completely managed, and manages task delegation similarly to RabbitMQ. `See documentation for details `{.interpreted-text role="ref"} --- # Using Kafka {#broker-kafka} ## Configuration {#broker-Kafka-installation} For celeryconfig.py: ``` python import os task_serializer = 'json' broker_transport_options = { # "allow_create_topics": True, } broker_connection_retry_on_startup = True # For using SQLAlchemy as the backend # result_backend = 'db+postgresql://postgres:example@localhost/postgres' broker_transport_options.update({ "security_protocol": "SASL_SSL", "sasl_mechanism": "SCRAM-SHA-512", }) sasl_username = os.environ["SASL_USERNAME"] sasl_password = os.environ["SASL_PASSWORD"] broker_url = f"confluentkafka://{sasl_username}:{sasl_password}@broker:9094" broker_transport_options.update({ "kafka_admin_config": { "sasl.username": sasl_username, "sasl.password": sasl_password, }, "kafka_common_config": { "sasl.username": sasl_username, "sasl.password": sasl_password, "security.protocol": "SASL_SSL", "sasl.mechanism": "SCRAM-SHA-512", "bootstrap_servers": "broker:9094", } }) ``` Please note that \"allow_create_topics\" is needed if the topic does not exist yet but is not necessary otherwise. For tasks.py: ``` python from celery import Celery app = Celery('tasks') app.config_from_object('celeryconfig') @app.task def add(x, y): return x + y ``` ## Auth See above. The SASL username and password are passed in as environment variables. ## Further Info Celery queues get routed to Kafka topics. For example, if a queue is named \"add_queue\", then a topic named \"add_queue\" will be created/used in Kafka. For canvas, when using a backend that supports it, the typical mechanisms like chain, group, and chord seem to work. ## Limitations Currently, using Kafka as a broker means that only one worker can be used. See . --- # Using RabbitMQ {#broker-rabbitmq} ::: {.contents local=""} ::: ## Installation & Configuration RabbitMQ is the default broker so it doesn\'t require any additional dependencies or initial configuration, other than the URL location of the broker instance you want to use: ``` python broker_url = 'amqp://myuser:mypassword@localhost:5672/myvhost' ``` For a description of broker URLs and a full list of the various broker configuration options available to Celery, see `conf-broker-settings`{.interpreted-text role="ref"}, and see below for setting up the username, password and vhost. ## Installing the RabbitMQ Server {#installing-rabbitmq} See [Downloading and Installing RabbitMQ](https://www.rabbitmq.com/download.html) over at RabbitMQ\'s website. For macOS see [Installing RabbitMQ on macOS](#installing-rabbitmq-on-macos). ::: note ::: title Note ::: If you\'re getting [nodedown]{.title-ref} errors after installing and using `rabbitmqctl`{.interpreted-text role="command"} then this blog post can help you identify the source of the problem: > ::: ### Setting up RabbitMQ {#rabbitmq-configuration} To use Celery we need to create a RabbitMQ user, a virtual host and allow that user access to that virtual host: ``` console $ sudo rabbitmqctl add_user myuser mypassword ``` ``` console $ sudo rabbitmqctl add_vhost myvhost ``` ``` console $ sudo rabbitmqctl set_user_tags myuser mytag ``` ``` console $ sudo rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*" ``` Substitute in appropriate values for `myuser`, `mypassword` and `myvhost` above. See the RabbitMQ [Admin Guide](https://www.rabbitmq.com/admin-guide.html) for more information about [access control](https://www.rabbitmq.com/access-control.html). ### Installing RabbitMQ on macOS {#rabbitmq-macOS-installation} The easiest way to install RabbitMQ on macOS is using [Homebrew](https://github.com/mxcl/homebrew/) the new and shiny package management system for macOS. First, install Homebrew using the one-line command provided by the [Homebrew documentation](https://github.com/Homebrew/homebrew/wiki/Installation): ``` console /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` Finally, we can install RabbitMQ using `brew`{.interpreted-text role="command"}: ``` console $ brew install rabbitmq ``` ::: {#rabbitmq-macOS-system-hostname} After you\'ve installed RabbitMQ with `brew`{.interpreted-text role="command"} you need to add the following to your path to be able to start and stop the broker: add it to the start-up file for your shell (e.g., `.bash_profile`{.interpreted-text role="file"} or `.profile`{.interpreted-text role="file"}). ::: ``` bash PATH=$PATH:/usr/local/sbin ``` #### Configuring the system host name If you\'re using a DHCP server that\'s giving you a random host name, you need to permanently configure the host name. This is because RabbitMQ uses the host name to communicate with nodes. Use the `scutil`{.interpreted-text role="command"} command to permanently set your host name: ``` console $ sudo scutil --set HostName myhost.local ``` Then add that host name to `/etc/hosts`{.interpreted-text role="file"} so it\'s possible to resolve it back into an IP address: 127.0.0.1 localhost myhost myhost.local If you start the `rabbitmq-server`{.interpreted-text role="command"}, your rabbit node should now be [rabbit@myhost]{.title-ref}, as verified by `rabbitmqctl`{.interpreted-text role="command"}: ``` console $ sudo rabbitmqctl status Status of node rabbit@myhost ... [{running_applications,[{rabbit,"RabbitMQ","1.7.1"}, {mnesia,"MNESIA CXC 138 12","4.4.12"}, {os_mon,"CPO CXC 138 46","2.2.4"}, {sasl,"SASL CXC 138 11","2.1.8"}, {stdlib,"ERTS CXC 138 10","1.16.4"}, {kernel,"ERTS CXC 138 10","2.13.4"}]}, {nodes,[rabbit@myhost]}, {running_nodes,[rabbit@myhost]}] ...done. ``` This is especially important if your DHCP server gives you a host name starting with an IP address, (e.g., [23.10.112.31.comcast.net]{.title-ref}). In this case RabbitMQ will try to use \`\`: an illegal host name. #### Starting/Stopping the RabbitMQ server {#rabbitmq-macOS-start-stop} To start the server: ``` console $ sudo rabbitmq-server ``` you can also run it in the background by adding the `-detached` option (note: only one dash): ``` console $ sudo rabbitmq-server -detached ``` Never use `kill`{.interpreted-text role="command"} (`kill(1)`{.interpreted-text role="manpage"}) to stop the RabbitMQ server, but rather use the `rabbitmqctl`{.interpreted-text role="command"} command: ``` console $ sudo rabbitmqctl stop ``` When the server is running, you can continue reading [Setting up RabbitMQ](#setting-up-rabbitmq). ## Using Quorum Queues ::: versionadded 5.5 ::: ::: warning ::: title Warning ::: Quorum Queues require disabling global QoS which means some features won\'t work as expected. See [limitations](#limitations) for details. ::: Celery supports [Quorum Queues](https://www.rabbitmq.com/docs/quorum-queues) by setting the `x-queue-type` header to `quorum` like so: ``` python from kombu import Queue task_queues = [Queue('my-queue', queue_arguments={'x-queue-type': 'quorum'})] broker_transport_options = {"confirm_publish": True} ``` If you\'d like to change the type of the default queue, set the `task_default_queue_type`{.interpreted-text role="setting"} setting to `quorum`. Another way to configure [Quorum Queues](https://www.rabbitmq.com/docs/quorum-queues) is by relying on default settings and using `task_routes`: ``` python task_default_queue_type = "quorum" task_default_exchange_type = "topic" task_default_queue = "my-queue" broker_transport_options = {"confirm_publish": True} task_routes = { "*": { "routing_key": "my-queue", }, } ``` Celery automatically detects if quorum queues are used using the `worker_detect_quorum_queues`{.interpreted-text role="setting"} setting. We recommend to keep the default behavior turned on. To migrate from classic mirrored queues to quorum queues, please refer to RabbitMQ\'s [documentation](https://www.rabbitmq.com/blog/2023/03/02/quorum-queues-migration) on the subject. ### Limitations Disabling global QoS means that the the per-channel QoS is now static. This means that some Celery features won\'t work when using Quorum Queues. Autoscaling relies on increasing and decreasing the prefetch count whenever a new process is instantiated or terminated so it won\'t work when Quorum Queues are detected. Similarly, the `worker_enable_prefetch_count_reduction`{.interpreted-text role="setting"} setting will be a no-op even when set to `True` when Quorum Queues are detected. In addition, `ETA/Countdown `{.interpreted-text role="ref"} will block the worker when received until the ETA arrives since we can no longer increase the prefetch count and fetch another task from the queue. In order to properly schedule ETA/Countdown tasks we automatically detect if quorum queues are used and in case they are, Celery automatically enables `Native Delayed Delivery `{.interpreted-text role="ref"}. ### Native Delayed Delivery Since tasks with ETA/Countdown will block the worker until they are scheduled for execution, we need to use RabbitMQ\'s native capabilities to schedule the execution of tasks. The design is borrowed from NServiceBus. If you are interested in the implementation details, refer to their [documentation](https://docs.particular.net/transports/rabbitmq/delayed-delivery). Native Delayed Delivery is automatically enabled when quorum queues are detected. By default the Native Delayed Delivery queues are quorum queues. If you\'d like to change them to classic queues you can set the `broker_native_delayed_delivery_queue_type`{.interpreted-text role="setting"} to classic. --- # Using Redis {#broker-redis} ## Installation {#broker-redis-installation} For the Redis support you have to install additional dependencies. You can install both Celery and these dependencies in one go using the `celery[redis]` `bundle `{.interpreted-text role="ref"}: ``` console $ pip install -U "celery[redis]" ``` ## Configuration {#broker-redis-configuration} Configuration is easy, just configure the location of your Redis database: ``` python app.conf.broker_url = 'redis://localhost:6379/0' ``` Where the URL is in the format of: ``` text redis://:password@hostname:port/db_number ``` all fields after the scheme are optional, and will default to `localhost` on port 6379, using database 0. If redis credential provider should be used, the URL needs to be in the following format: ``` text redis://@hostname:port/db_number?credential_provider=mymodule.myfile.myclass ``` If a Unix socket connection should be used, the URL needs to be in the format: ``` text redis+socket:///path/to/redis.sock ``` Specifying a different database number when using a Unix socket is possible by adding the `virtual_host` parameter to the URL: ``` text redis+socket:///path/to/redis.sock?virtual_host=db_number ``` It is also easy to connect directly to a list of Redis Sentinel: ``` python app.conf.broker_url = 'sentinel://localhost:26379;sentinel://localhost:26380;sentinel://localhost:26381' app.conf.broker_transport_options = { 'master_name': "cluster1" } ``` Additional options can be passed to the Sentinel client using `sentinel_kwargs`: ``` python app.conf.broker_transport_options = { 'sentinel_kwargs': { 'password': "password" } } ``` ### Visibility Timeout {#redis-visibility_timeout} The visibility timeout defines the number of seconds to wait for the worker to acknowledge the task before the message is redelivered to another worker. Be sure to see `redis-caveats`{.interpreted-text role="ref"} below. This option is set via the `broker_transport_options`{.interpreted-text role="setting"} setting: ``` python app.conf.broker_transport_options = {'visibility_timeout': 3600} # 1 hour. ``` The default visibility timeout for Redis is 1 hour. ### Results {#redis-results-configuration} If you also want to store the state and return values of tasks in Redis, you should configure these settings: app.conf.result_backend = 'redis://localhost:6379/0' For a complete list of options supported by the Redis result backend, see `conf-redis-result-backend`{.interpreted-text role="ref"}. If you are using Sentinel, you should specify the master_name using the `result_backend_transport_options`{.interpreted-text role="setting"} setting: ``` python app.conf.result_backend_transport_options = {'master_name': "mymaster"} ``` #### Global keyprefix {#redis-result-backend-global-keyprefix} The global key prefix will be prepended to all keys used for the result backend, which can be useful when a redis database is shared by different users. By default, no prefix is prepended. To configure the global keyprefix for the Redis result backend, use the `global_keyprefix` key under `result_backend_transport_options`{.interpreted-text role="setting"}: ``` python app.conf.result_backend_transport_options = { 'global_keyprefix': 'my_prefix_' } ``` #### Connection timeouts {#redis-result-backend-timeout} To configure the connection timeouts for the Redis result backend, use the `retry_policy` key under `result_backend_transport_options`{.interpreted-text role="setting"}: ``` python app.conf.result_backend_transport_options = { 'retry_policy': { 'timeout': 5.0 } } ``` See `~kombu.utils.functional.retry_over_time`{.interpreted-text role="func"} for the possible retry policy options. ## Serverless {#redis-serverless} Celery supports utilizing a remote serverless Redis, which can significantly reduce the operational overhead and cost, making it a favorable choice in microservice architectures or environments where minimizing operational expenses is crucial. Serverless Redis provides the necessary functionalities without the need for manual setup, configuration, and management, thus aligning well with the principles of automation and scalability that Celery promotes. ### Upstash [Upstash](http://upstash.com/?code=celery) offers a serverless Redis database service, providing a seamless solution for Celery users looking to leverage serverless architectures. Upstash\'s serverless Redis service is designed with an eventual consistency model and durable storage, facilitated through a multi-tier storage architecture. Integration with Celery is straightforward as demonstrated in an [example provided by Upstash](https://github.com/upstash/examples/tree/main/examples/using-celery). ### Dragonfly [Dragonfly](https://www.dragonflydb.io/) is a drop-in Redis replacement that cuts costs and boosts performance. Designed to fully utilize the power of modern cloud hardware and deliver on the data demands of modern applications, Dragonfly frees developers from the limits of traditional in-memory data stores. ## Caveats {#redis-caveats} ### Visibility timeout If a task isn\'t acknowledged within the `redis-visibility_timeout`{.interpreted-text role="ref"} the task will be redelivered to another worker and executed. This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact if that happens it will be executed again, and again in a loop. To remediate that, you can increase the visibility timeout to match the time of the longest ETA you\'re planning to use. However, this is not recommended as it may have negative impact on the reliability. Celery will redeliver messages at worker shutdown, so having a long visibility timeout will only delay the redelivery of \'lost\' tasks in the event of a power failure or forcefully terminated workers. Broker is not a database, so if you are in need of scheduling tasks for a more distant future, database-backed periodic task might be a better choice. Periodic tasks won\'t be affected by the visibility timeout, as this is a concept separate from ETA/countdown. You can increase this timeout by configuring all of the following options with the same name (required to set all of them): ``` python app.conf.broker_transport_options = {'visibility_timeout': 43200} app.conf.result_backend_transport_options = {'visibility_timeout': 43200} app.conf.visibility_timeout = 43200 ``` The value must be an int describing the number of seconds. Note: If multiple applications are sharing the same Broker, with different settings, the \_[shortest]() value will be used. This include if the value is not set, and the default is sent ### Soft Shutdown During `shutdown `{.interpreted-text role="ref"}, the worker will attempt to re-queue any unacknowledged messages with `task_acks_late`{.interpreted-text role="setting"} enabled. However, if the worker is terminated forcefully (`cold shutdown `{.interpreted-text role="ref"}), the worker might not be able to re-queue the tasks on time, and they will not be consumed again until the `redis-visibility_timeout`{.interpreted-text role="ref"} has passed. This creates a problem when the `redis-visibility_timeout`{.interpreted-text role="ref"} is very high and a worker needs to shut down just after it has received a task. If the task is not re-queued in such case, it will need to wait for the long visibility timeout to pass before it can be consumed again, leading to potentially very long delays in tasks execution. The `soft shutdown `{.interpreted-text role="ref"} introduces a time-limited warm shutdown phase just before the `cold shutdown `{.interpreted-text role="ref"}. This time window significantly increases the chances of re-queuing the tasks during shutdown which mitigates the problem of long visibility timeouts. To enable the `soft shutdown `{.interpreted-text role="ref"}, set the `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} to a value greater than 0. The value must be an float describing the number of seconds. During this time, the worker will continue to process the running tasks until the timeout expires, after which the `cold shutdown `{.interpreted-text role="ref"} will be initiated automatically to terminate the worker gracefully. If the `REMAP_SIGTERM `{.interpreted-text role="ref"} is configured to SIGQUIT in the environment variables, and the `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} is set, the worker will initiate the `soft shutdown `{.interpreted-text role="ref"} when it receives the `TERM`{.interpreted-text role="sig"} signal (*and* the `QUIT`{.interpreted-text role="sig"} signal). ### Key eviction Redis may evict keys from the database in some situations If you experience an error like: ``` text InconsistencyError: Probably the key ('_kombu.binding.celery') has been removed from the Redis database. ``` then you may want to configure the `redis-server`{.interpreted-text role="command"} to not evict keys by setting in the redis configuration file: - the `maxmemory` option - the `maxmemory-policy` option to `noeviction` or `allkeys-lru` See Redis server documentation about Eviction Policies for details: > ### Group result ordering {#redis-group-result-ordering} Versions of Celery up to and including 4.4.6 used an unsorted list to store result objects for groups in the Redis backend. This can cause those results to be be returned in a different order to their associated tasks in the original group instantiation. Celery 4.4.7 introduced an opt-in behaviour which fixes this issue and ensures that group results are returned in the same order the tasks were defined, matching the behaviour of other backends. In Celery 5.0 this behaviour was changed to be opt-out. The behaviour is controlled by the [result_chord_ordered]{.title-ref} configuration option which may be set like so: ``` python # Specifying this for workers running Celery 4.4.6 or earlier has no effect app.conf.result_backend_transport_options = { 'result_chord_ordered': True # or False } ``` This is an incompatible change in the runtime behaviour of workers sharing the same Redis backend for result storage, so all workers must follow either the new or old behaviour to avoid breakage. For clusters with some workers running Celery 4.4.6 or earlier, this means that workers running 4.4.7 need no special configuration and workers running 5.0 or later must have [result_chord_ordered]{.title-ref} set to [False]{.title-ref}. For clusters with no workers running 4.4.6 or earlier but some workers running 4.4.7, it is recommended that [result_chord_ordered]{.title-ref} be set to [True]{.title-ref} for all workers to ease future migration. Migration between behaviours will disrupt results currently held in the Redis backend and cause breakage if downstream tasks are run by migrated workers - plan accordingly. --- # Using Amazon SQS {#broker-sqs} ## Installation {#broker-sqs-installation} For the Amazon SQS support you have to install additional dependencies. You can install both Celery and these dependencies in one go using the `celery[sqs]` `bundle `{.interpreted-text role="ref"}: ``` console $ pip install "celery[sqs]" ``` ## Configuration {#broker-sqs-configuration} You have to specify SQS in the broker URL: broker_url = 'sqs://ABCDEFGHIJKLMNOPQRST:ZYXK7NiynGlTogH8Nj+P9nlE73sq3@' where the URL format is: ``` text sqs://aws_access_key_id:aws_secret_access_key@ ``` Please note that you must remember to include the `@` sign at the end and encode the password so it can always be parsed correctly. For example: ``` python from kombu.utils.url import safequote aws_access_key = safequote("ABCDEFGHIJKLMNOPQRST") aws_secret_key = safequote("ZYXK7NiynG/TogH8Nj+P9nlE73sq3") broker_url = "sqs://{aws_access_key}:{aws_secret_key}@".format( aws_access_key=aws_access_key, aws_secret_key=aws_secret_key, ) ``` ::: warning ::: title Warning ::: Don\'t use this setup option with django\'s `debug=True`. It may lead to security issues within deployed django apps. In debug mode django shows environment variables and the SQS URL may be exposed to the internet including your AWS access and secret keys. Please turn off debug mode on your deployed django application or consider a setup option described below. ::: The login credentials can also be set using the environment variables `AWS_ACCESS_KEY_ID`{.interpreted-text role="envvar"} and `AWS_SECRET_ACCESS_KEY`{.interpreted-text role="envvar"}, in that case the broker URL may only be `sqs://`. If you are using IAM roles on instances, you can set the BROKER_URL to: `sqs://` and kombu will attempt to retrieve access tokens from the instance metadata. ## Options ### Region The default region is `us-east-1` but you can select another region by configuring the `broker_transport_options`{.interpreted-text role="setting"} setting: broker_transport_options = {'region': 'eu-west-1'} ::: seealso An overview of Amazon Web Services regions can be found here: > ::: ### Visibility Timeout {#sqs-visibility-timeout} The visibility timeout defines the number of seconds to wait for the worker to acknowledge the task before the message is redelivered to another worker. Also see caveats below. This option is set via the `broker_transport_options`{.interpreted-text role="setting"} setting: broker_transport_options = {'visibility_timeout': 3600} # 1 hour. The default visibility timeout is 30 minutes. This option is used when creating the SQS queue and has no effect if using `predefined queues `{.interpreted-text role="ref"}. ### Polling Interval The polling interval decides the number of seconds to sleep between unsuccessful polls. This value can be either an int or a float. By default the value is *one second*: this means the worker will sleep for one second when there\'s no more messages to read. You must note that **more frequent polling is also more expensive, so increasing the polling interval can save you money**. The polling interval can be set via the `broker_transport_options`{.interpreted-text role="setting"} setting: broker_transport_options = {'polling_interval': 0.3} Very frequent polling intervals can cause *busy loops*, resulting in the worker using a lot of CPU time. If you need sub-millisecond precision you should consider using another transport, like [RabbitMQ \]{.title-ref}, or [Redis \]{.title-ref}. ### Long Polling [SQS Long Polling](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) is enabled by default and the `WaitTimeSeconds` parameter of [ReceiveMessage](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html) operation is set to 10 seconds. The value of `WaitTimeSeconds` parameter can be set via the `broker_transport_options`{.interpreted-text role="setting"} setting: broker_transport_options = {'wait_time_seconds': 15} Valid values are 0 to 20. Note that newly created queues themselves (also if created by Celery) will have the default value of 0 set for the \"Receive Message Wait Time\" queue property. ### Queue Prefix By default Celery won\'t assign any prefix to the queue names, If you have other services using SQS you can configure it do so using the `broker_transport_options`{.interpreted-text role="setting"} setting: broker_transport_options = {'queue_name_prefix': 'celery-'} ### Predefined Queues If you want Celery to use a set of predefined queues in AWS, and to never attempt to list SQS queues, nor attempt to create or delete them, pass a map of queue names to URLs using the `predefined_queues`{.interpreted-text role="setting"} setting: broker_transport_options = { 'predefined_queues': { 'my-q': { 'url': 'https://ap-southeast-2.queue.amazonaws.com/123456/my-q', 'access_key_id': 'xxx', 'secret_access_key': 'xxx', } } } ::: warning ::: title Warning ::: **Important:** When using `predefined_queues`, do NOT use URL-encoded credentials (`safequote`) for the `access_key_id` and `secret_access_key` values. URL encoding should only be applied to credentials in the broker URL. Using URL-encoded credentials in `predefined_queues` will cause signature mismatch errors like: \"The request signature we calculated does not match the signature you provided.\" ::: **Correct example combining broker URL and predefined queues:** ``` python import os from kombu.utils.url import safequote from celery import Celery # Raw credentials from environment AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID") AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY") # URL-encode ONLY for broker URL aws_access_key_encoded = safequote(AWS_ACCESS_KEY_ID) aws_secret_key_encoded = safequote(AWS_SECRET_ACCESS_KEY) # Use encoded credentials in broker URL broker_url = f"sqs://{aws_access_key_encoded}:{aws_secret_key_encoded}@" celery_app = Celery("tasks", broker=broker_url) celery_app.conf.broker_transport_options = { "region": "us-east-1", "predefined_queues": { "my-queue": { "url": "https://sqs.us-east-1.amazonaws.com/123456/my-queue", # Use RAW credentials here (NOT encoded) "access_key_id": AWS_ACCESS_KEY_ID, "secret_access_key": AWS_SECRET_ACCESS_KEY, }, }, } ``` When using this option, the visibility timeout should be set in the SQS queue (in AWS) rather than via the `visibility timeout `{.interpreted-text role="ref"} option. ### Back-off policy Back-off policy is using SQS visibility timeout mechanism altering the time difference between task retries. The mechanism changes message specific `visibility timeout` from queue `Default visibility timeout` to policy configured timeout. The number of retries is managed by SQS (specifically by the `ApproximateReceiveCount` message attribute) and no further action is required by the user. Configuring the queues and backoff policy: broker_transport_options = { 'predefined_queues': { 'my-q': { 'url': 'https://ap-southeast-2.queue.amazonaws.com/123456/my-q', 'access_key_id': 'xxx', 'secret_access_key': 'xxx', 'backoff_policy': {1: 10, 2: 20, 3: 40, 4: 80, 5: 320, 6: 640}, 'backoff_tasks': ['svc.tasks.tasks.task1'] } } } `backoff_policy` dictionary where key is number of retries, and value is delay seconds between retries (i.e SQS visibility timeout) `backoff_tasks` list of task names to apply the above policy The above policy: ---------------------------------- ------------------------------------ **Attempt** **Delay** `2nd attempt` 20 seconds `3rd attempt` 40 seconds `4th attempt` 80 seconds `5th attempt` 320 seconds `6th attempt` 640 seconds ---------------------------------- ------------------------------------ ### STS token authentication AWS STS authentication is supported by using the `sts_role_arn` and `sts_token_timeout` broker transport options. `sts_role_arn` is the assumed IAM role ARN we use to authorize our access to SQS. `sts_token_timeout` is the token timeout, defaults (and minimum) to 900 seconds. After the mentioned period, a new token will be created: broker_transport_options = { 'predefined_queues': { 'my-q': { 'url': 'https://ap-southeast-2.queue.amazonaws.com/123456/my-q', 'access_key_id': 'xxx', 'secret_access_key': 'xxx', 'backoff_policy': {1: 10, 2: 20, 3: 40, 4: 80, 5: 320, 6: 640}, 'backoff_tasks': ['svc.tasks.tasks.task1'] } }, 'sts_role_arn': 'arn:aws:iam:::role/STSTest', # optional 'sts_token_timeout': 900 # optional } ## Caveats {#sqs-caveats} - If a task isn\'t acknowledged within the `visibility_timeout`, the task will be redelivered to another worker and executed. This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact if that happens it will be executed again, and again in a loop. So you have to increase the visibility timeout to match the time of the longest ETA you\'re planning to use. Note that Celery will redeliver messages at worker shutdown, so having a long visibility timeout will only delay the redelivery of \'lost\' tasks in the event of a power failure or forcefully terminated workers. Periodic tasks won\'t be affected by the visibility timeout, as it is a concept separate from ETA/countdown. The maximum visibility timeout supported by AWS as of this writing is 12 hours (43200 seconds): broker_transport_options = {'visibility_timeout': 43200} - SQS doesn\'t yet support worker remote control commands. - SQS doesn\'t yet support events, and so cannot be used with `celery events`{.interpreted-text role="program"}, `celerymon`{.interpreted-text role="program"}, or the Django Admin monitor. - With FIFO queues it might be necessary to set additional message properties such as `MessageGroupId` and `MessageDeduplicationId` when publishing a message. Message properties can be passed as keyword arguments to `~celery.app.task.Task.apply_async`{.interpreted-text role="meth"}: ``` python message_properties = { 'MessageGroupId': '', 'MessageDeduplicationId': '' } task.apply_async(**message_properties) ``` - During `shutdown `{.interpreted-text role="ref"}, the worker will attempt to re-queue any unacknowledged messages with `task_acks_late`{.interpreted-text role="setting"} enabled. However, if the worker is terminated forcefully (`cold shutdown `{.interpreted-text role="ref"}), the worker might not be able to re-queue the tasks on time, and they will not be consumed again until the `sqs-visibility-timeout`{.interpreted-text role="ref"} has passed. This creates a problem when the `sqs-visibility-timeout`{.interpreted-text role="ref"} is very high and a worker needs to shut down just after it has received a task. If the task is not re-queued in such case, it will need to wait for the long visibility timeout to pass before it can be consumed again, leading to potentially very long delays in tasks execution. The `soft shutdown `{.interpreted-text role="ref"} introduces a time-limited warm shutdown phase just before the `cold shutdown `{.interpreted-text role="ref"}. This time window significantly increases the chances of re-queuing the tasks during shutdown which mitigates the problem of long visibility timeouts. To enable the `soft shutdown `{.interpreted-text role="ref"}, set the `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} to a value greater than 0. The value must be an float describing the number of seconds. During this time, the worker will continue to process the running tasks until the timeout expires, after which the `cold shutdown `{.interpreted-text role="ref"} will be initiated automatically to terminate the worker gracefully. If the `REMAP_SIGTERM `{.interpreted-text role="ref"} is configured to SIGQUIT in the environment variables, and the `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} is set, the worker will initiate the `soft shutdown `{.interpreted-text role="ref"} when it receives the `TERM`{.interpreted-text role="sig"} signal (*and* the `QUIT`{.interpreted-text role="sig"} signal). ### Results {#sqs-results-configuration} Multiple products in the Amazon Web Services family could be a good candidate to store or publish results with, but there\'s no such result backend included at this point. ::: warning ::: title Warning ::: Don\'t use the `amqp` result backend with SQS. It will create one queue for every task, and the queues will not be collected. This could cost you money that would be better spent contributing an AWS result store backend back to Celery :) ::: --- # First Steps with Celery[]{#tut-celery} {#first-steps} Celery is a task queue with batteries included. It\'s easy to use so that you can get started without learning the full complexities of the problem it solves. It\'s designed around best practices so that your product can scale and integrate with other languages, and it comes with the tools and support you need to run such a system in production. In this tutorial you\'ll learn the absolute basics of using Celery. Learn about: - Choosing and installing a message transport (broker). - Installing Celery and creating your first task. - Starting the worker and calling tasks. - Keeping track of tasks as they transition through different states, and inspecting return values. Celery may seem daunting at first - but don\'t worry - this tutorial will get you started in no time. It\'s deliberately kept simple, so as to not confuse you with advanced features. After you have finished this tutorial, it\'s a good idea to browse the rest of the documentation. For example the `next-steps`{.interpreted-text role="ref"} tutorial will showcase Celery\'s capabilities. ::: {.contents local=""} ::: ## Choosing a Broker {#celerytut-broker} Celery requires a solution to send and receive messages; usually this comes in the form of a separate service called a *message broker*. There are several choices available, including: ### RabbitMQ [RabbitMQ](http://www.rabbitmq.com/) is feature-complete, stable, durable and easy to install. It\'s an excellent choice for a production environment. Detailed information about using RabbitMQ with Celery: > `broker-rabbitmq`{.interpreted-text role="ref"} If you\'re using Ubuntu or Debian install RabbitMQ by executing this command: ``` console $ sudo apt-get install rabbitmq-server ``` Or, if you want to run it on Docker execute this: ``` console $ docker run -d -p 5672:5672 rabbitmq ``` When the command completes, the broker will already be running in the background, ready to move messages for you: `Starting rabbitmq-server: SUCCESS`. Don\'t worry if you\'re not running Ubuntu or Debian, you can go to this website to find similarly simple installation instructions for other platforms, including Microsoft Windows: > ### Redis [Redis](https://redis.io/) is also feature-complete, but is more susceptible to data loss in the event of abrupt termination or power failures. Detailed information about using Redis: `broker-redis`{.interpreted-text role="ref"} If you want to run it on Docker execute this: ``` console $ docker run -d -p 6379:6379 redis ``` ### Other brokers In addition to the above, there are other experimental transport implementations to choose from, including `Amazon SQS `{.interpreted-text role="ref"}. See `broker-overview`{.interpreted-text role="ref"} for a full list. ## Installing Celery {#celerytut-installation} Celery is on the Python Package Index (PyPI), so it can be installed with standard Python tools like `pip`: ``` console $ pip install celery ``` ## Application The first thing you need is a Celery instance. We call this the *Celery application* or just *app* for short. As this instance is used as the entry-point for everything you want to do in Celery, like creating tasks and managing workers, it must be possible for other modules to import it. In this tutorial we keep everything contained in a single module, but for larger projects you want to create a `dedicated module `{.interpreted-text role="ref"}. Let\'s create the file `tasks.py`{.interpreted-text role="file"}: ``` python from celery import Celery app = Celery('tasks', broker='pyamqp://guest@localhost//') @app.task def add(x, y): return x + y ``` The first argument to `~celery.app.Celery`{.interpreted-text role="class"} is the name of the current module. This is only needed so that names can be automatically generated when the tasks are defined in the [\_\_main\_\_]{.title-ref} module. The second argument is the broker keyword argument, specifying the URL of the message broker you want to use. Here we are using RabbitMQ (also the default option). See `celerytut-broker`{.interpreted-text role="ref"} above for more choices \--for RabbitMQ you can use `amqp://localhost`, or for Redis you can use `redis://localhost`. You defined a single task, called `add`, returning the sum of two numbers. ## Running the Celery worker server {#celerytut-running-the-worker} You can now run the worker by executing our program with the `worker` argument: ``` console $ celery -A tasks worker --loglevel=INFO ``` ::: note ::: title Note ::: See the `celerytut-troubleshooting`{.interpreted-text role="ref"} section if the worker doesn\'t start. ::: In production you\'ll want to run the worker in the background as a daemon. To do this you need to use the tools provided by your platform, or something like [supervisord](http://supervisord.org) (see `daemonizing`{.interpreted-text role="ref"} for more information). For a complete listing of the command-line options available, do: ``` console $ celery worker --help ``` There are also several other commands available, and help is also available: ``` console $ celery --help ``` ## Calling the task {#celerytut-calling} To call our task you can use the `~@Task.delay`{.interpreted-text role="meth"} method. This is a handy shortcut to the `~@Task.apply_async`{.interpreted-text role="meth"} method that gives greater control of the task execution (see `guide-calling`{.interpreted-text role="ref"}): >>> from tasks import add >>> add.delay(4, 4) The task has now been processed by the worker you started earlier. You can verify this by looking at the worker\'s console output. Calling a task returns an `~@AsyncResult`{.interpreted-text role="class"} instance. This can be used to check the state of the task, wait for the task to finish, or get its return value (or if the task failed, to get the exception and traceback). Results are not enabled by default. In order to do remote procedure calls or keep track of task results in a database, you will need to configure Celery to use a result backend. This is described in the next section. ## Keeping Results {#celerytut-keeping-results} If you want to keep track of the tasks\' states, Celery needs to store or send the states somewhere. There are several built-in result backends to choose from: [SQLAlchemy](http://www.sqlalchemy.org/)/[Django](http://djangoproject.com) ORM, [MongoDB](http://www.mongodb.org), [Memcached](http://memcached.org), [Redis](https://redis.io/), `RPC `{.interpreted-text role="ref"} ([RabbitMQ](http://www.rabbitmq.com/)/AMQP), and \-- or you can define your own. For this example we use the [rpc]{.title-ref} result backend, that sends states back as transient messages. The backend is specified via the `backend` argument to `@Celery`{.interpreted-text role="class"}, (or via the `result_backend`{.interpreted-text role="setting"} setting if you choose to use a configuration module). So, you can modify this line in the [tasks.py]{.title-ref} file to enable the [rpc://]{.title-ref} backend: ``` python app = Celery('tasks', backend='rpc://', broker='pyamqp://') ``` Or if you want to use Redis as the result backend, but still use RabbitMQ as the message broker (a popular combination): ``` python app = Celery('tasks', backend='redis://localhost', broker='pyamqp://') ``` To read more about result backends please see `task-result-backends`{.interpreted-text role="ref"}. Now with the result backend configured, restart the worker, close the current python session and import the `tasks` module again to put the changes into effect. This time you\'ll hold on to the `~@AsyncResult`{.interpreted-text role="class"} instance returned when you call a task: ``` pycon >>> from tasks import add # close and reopen to get updated 'app' >>> result = add.delay(4, 4) ``` The `~@AsyncResult.ready`{.interpreted-text role="meth"} method returns whether the task has finished processing or not: ``` pycon >>> result.ready() False ``` You can wait for the result to complete, but this is rarely used since it turns the asynchronous call into a synchronous one: ``` pycon >>> result.get(timeout=1) 8 ``` In case the task raised an exception, `~@AsyncResult.get`{.interpreted-text role="meth"} will re-raise the exception, but you can override this by specifying the `propagate` argument: ``` pycon >>> result.get(propagate=False) ``` If the task raised an exception, you can also gain access to the original traceback: ``` pycon >>> result.traceback ``` ::: warning ::: title Warning ::: Backends use resources to store and transmit results. To ensure that resources are released, you must eventually call `~@AsyncResult.get`{.interpreted-text role="meth"} or `~@AsyncResult.forget`{.interpreted-text role="meth"} on EVERY `~@AsyncResult`{.interpreted-text role="class"} instance returned after calling a task. ::: See `celery.result`{.interpreted-text role="mod"} for the complete result object reference. ## Configuration {#celerytut-configuration} Celery, like a consumer appliance, doesn\'t need much configuration to operate. It has an input and an output. The input must be connected to a broker, and the output can be optionally connected to a result backend. However, if you look closely at the back, there\'s a lid revealing loads of sliders, dials, and buttons: this is the configuration. The default configuration should be good enough for most use cases, but there are many options that can be configured to make Celery work exactly as needed. Reading about the options available is a good idea to familiarize yourself with what can be configured. You can read about the options in the `configuration`{.interpreted-text role="ref"} reference. The configuration can be set on the app directly or by using a dedicated configuration module. As an example you can configure the default serializer used for serializing task payloads by changing the `task_serializer`{.interpreted-text role="setting"} setting: ``` python app.conf.task_serializer = 'json' ``` If you\'re configuring many settings at once you can use `update`: ``` python app.conf.update( task_serializer='json', accept_content=['json'], # Ignore other content result_serializer='json', timezone='Europe/Oslo', enable_utc=True, ) ``` For larger projects, a dedicated configuration module is recommended. Hard coding periodic task intervals and task routing options is discouraged. It is much better to keep these in a centralized location. This is especially true for libraries, as it enables users to control how their tasks behave. A centralized configuration will also allow your SysAdmin to make simple changes in the event of system trouble. You can tell your Celery instance to use a configuration module by calling the `@config_from_object`{.interpreted-text role="meth"} method: ``` python app.config_from_object('celeryconfig') ``` This module is often called \"`celeryconfig`\", but you can use any module name. In the above case, a module named `celeryconfig.py` must be available to load from the current directory or on the Python path. It could look something like this: `celeryconfig.py`{.interpreted-text role="file"}: ``` python broker_url = 'pyamqp://' result_backend = 'rpc://' task_serializer = 'json' result_serializer = 'json' accept_content = ['json'] timezone = 'Europe/Oslo' enable_utc = True ``` To verify that your configuration file works properly and doesn\'t contain any syntax errors, you can try to import it: ``` console $ python -m celeryconfig ``` For a complete reference of configuration options, see `configuration`{.interpreted-text role="ref"}. To demonstrate the power of configuration files, this is how you\'d route a misbehaving task to a dedicated queue: `celeryconfig.py`{.interpreted-text role="file"}: ``` python task_routes = { 'tasks.add': 'low-priority', } ``` Or instead of routing it you could rate limit the task instead, so that only 10 tasks of this type can be processed in a minute (10/m): `celeryconfig.py`{.interpreted-text role="file"}: ``` python task_annotations = { 'tasks.add': {'rate_limit': '10/m'} } ``` If you\'re using RabbitMQ or Redis as the broker then you can also direct the workers to set a new rate limit for the task at runtime: ``` console $ celery -A tasks control rate_limit tasks.add 10/m worker@example.com: OK new rate limit set successfully ``` See `guide-routing`{.interpreted-text role="ref"} to read more about task routing, and the `task_annotations`{.interpreted-text role="setting"} setting for more about annotations, or `guide-monitoring`{.interpreted-text role="ref"} for more about remote control commands and how to monitor what your workers are doing. ## Where to go from here If you want to learn more you should continue to the `Next Steps `{.interpreted-text role="ref"} tutorial, and after that you can read the `User Guide `{.interpreted-text role="ref"}. ## Troubleshooting {#celerytut-troubleshooting} There\'s also a troubleshooting section in the `faq`{.interpreted-text role="ref"}. ### Worker doesn\'t start: Permission Error - If you\'re using Debian, Ubuntu or other Debian-based distributions: > Debian recently renamed the `/dev/shm`{.interpreted-text > role="file"} special file to `/run/shm`{.interpreted-text > role="file"}. > > A simple workaround is to create a symbolic link: > > ``` console > # ln -s /run/shm /dev/shm > ``` - Others: > If you provide any of the > `--pidfile `{.interpreted-text > role="option"}, > `--logfile `{.interpreted-text > role="option"} or > `--statedb `{.interpreted-text > role="option"} arguments, then you must make sure that they point > to a file or directory that\'s writable and readable by the user > starting the worker. ### Result backend doesn\'t work or tasks are always in `PENDING` state All tasks are `PENDING`{.interpreted-text role="state"} by default, so the state would\'ve been better named \"unknown\". Celery doesn\'t update the state when a task is sent, and any task with no history is assumed to be pending (you know the task id, after all). 1) Make sure that the task doesn\'t have `ignore_result` enabled. > Enabling this option will force the worker to skip updating > states. 2) Make sure the `task_ignore_result`{.interpreted-text role="setting"} setting isn\'t enabled. 3) Make sure that you don\'t have any old workers still running. > It\'s easy to start multiple workers by accident, so make sure > that the previous worker is properly shut down before you start a > new one. > > An old worker that isn\'t configured with the expected result > backend may be running and is hijacking the tasks. > > The `--pidfile `{.interpreted-text > role="option"} argument can be set to an absolute path to make > sure this doesn\'t happen. 4) Make sure the client is configured with the right backend. > If, for some reason, the client is configured to use a different > backend than the worker, you won\'t be able to receive the result. > Make sure the backend is configured correctly: > > ``` pycon > >>> result = task.delay() > >>> print(result.backend) > ``` --- # Getting Started Release : Date : ::: {.toctree maxdepth="2"} introduction backends-and-brokers/index first-steps-with-celery next-steps resources ::: --- # Introduction to Celery {#intro} ::: {.contents local="" depth="1"} ::: ## What\'s a Task Queue? Task queues are used as a mechanism to distribute work across threads or machines. A task queue\'s input is a unit of work called a task. Dedicated worker processes constantly monitor task queues for new work to perform. Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task the client adds a message to the queue, the broker then delivers that message to a worker. A Celery system can consist of multiple workers and brokers, giving way to high availability and horizontal scaling. Celery is written in Python, but the protocol can be implemented in any language. In addition to Python there\'s [node-celery](https://github.com/mher/node-celery) for Node.js, a [PHP client](https://github.com/gjedeer/celery-php), [gocelery](https://github.com/gocelery/gocelery), [gopher-celery](https://github.com/marselester/gopher-celery) for Go, and [rusty-celery](https://github.com/rusty-celery/rusty-celery) for Rust. Language interoperability can also be achieved exposing an HTTP endpoint and having a task that requests it (webhooks). ## What do I need? ::: {.sidebar subtitle="Celery version 5.5.x runs on:"} **Version Requirements: Celery version 5.5.x runs on:** - Python ❨3.8, 3.9, 3.10, 3.11, 3.12, 3.13❩ - PyPy3.9+ ❨v7.3.12+❩ If you\'re running an older version of Python, you need to be running an older version of Celery: - Python 3.7: Celery 5.2 or earlier. - Python 3.6: Celery 5.1 or earlier. - Python 2.7: Celery 4.x series. - Python 2.6: Celery series 3.1 or earlier. - Python 2.5: Celery series 3.0 or earlier. - Python 2.4: Celery series 2.2 or earlier.. Celery is a project with minimal funding, so we don\'t support Microsoft Windows. Please don\'t open any issues related to that platform. ::: *Celery* requires a message transport to send and receive messages. The RabbitMQ and Redis broker transports are feature complete, but there\'s also support for a myriad of other experimental solutions, including using SQLite for local development. *Celery* can run on a single machine, on multiple machines, or even across data centers. ## Get Started If this is the first time you\'re trying to use Celery, or if you haven\'t kept up with development in the 3.1 version and are coming from previous versions, then you should read our getting started tutorials: - `first-steps`{.interpreted-text role="ref"} - `next-steps`{.interpreted-text role="ref"} ## Celery is... ::: topic **\\** - **Simple** > Celery is easy to use and maintain, and it *doesn\'t need > configuration files*. > > It has an active, friendly community you can talk to for support, > including a > [mailing-list](https://groups.google.com/group/celery-users) and > an `IRC channel `{.interpreted-text role="ref"}. > > Here\'s one of the simplest applications you can make: > > ``` python > from celery import Celery > > app = Celery('hello', broker='amqp://guest@localhost//') > > @app.task > def hello(): > return 'hello world' > ``` - **Highly Available** > Workers and clients will automatically retry in the event of > connection loss or failure, and some brokers support HA in way of > *Primary/Primary* or *Primary/Replica* replication. - **Fast** > A single Celery process can process millions of tasks a minute, > with sub-millisecond round-trip latency (using RabbitMQ, > librabbitmq, and optimized settings). - **Flexible** > Almost every part of *Celery* can be extended or used on its own, > Custom pool implementations, serializers, compression schemes, > logging, schedulers, consumers, producers, broker transports, and > much more. ::: ::: topic **It supports** ::: {.hlist columns="2"} - **Brokers** > - `RabbitMQ `{.interpreted-text role="ref"}, > `Redis `{.interpreted-text role="ref"}, > - `Amazon SQS `{.interpreted-text role="ref"}, and > more... - **Concurrency** > - prefork (multiprocessing), > - [Eventlet](http://eventlet.net/), [gevent](http://gevent.org/) > - thread (multithreaded) > - [solo]{.title-ref} (single threaded) - **Result Stores** > - AMQP, Redis > - Memcached, > - SQLAlchemy, Django ORM > - Apache Cassandra, Elasticsearch, Riak > - MongoDB, CouchDB, Couchbase, ArangoDB > - Amazon DynamoDB, Amazon S3 > - Microsoft Azure Block Blob, Microsoft Azure Cosmos DB > - Google Cloud Storage > - File system - **Serialization** > - *pickle*, *json*, *yaml*, *msgpack*. > - *zlib*, *bzip2* compression. > - Cryptographic message signing. ::: ::: ## Features ::: topic **\\** ::: {.hlist columns="2"} - **Monitoring** > A stream of monitoring events is emitted by workers and is used by > built-in and external tools to tell you what your cluster is doing > \-- in real-time. > > `Read more… `{.interpreted-text role="ref"}. - **Work-flows** > Simple and complex work-flows can be composed using a set of > powerful primitives we call the \"canvas\", including grouping, > chaining, chunking, and more. > > `Read more… `{.interpreted-text role="ref"}. - **Time & Rate Limits** > You can control how many tasks can be executed per > second/minute/hour, or how long a task can be allowed to run, and > this can be set as a default, for a specific worker or > individually for each task type. > > `Read more… `{.interpreted-text role="ref"}. - **Scheduling** > You can specify the time to run a task in seconds or a > `~datetime.datetime`{.interpreted-text role="class"}, or you can > use periodic tasks for recurring events based on a simple > interval, or Crontab expressions supporting minute, hour, day of > week, day of month, and month of year. > > `Read more… `{.interpreted-text role="ref"}. - **Resource Leak Protection** > The > `--max-tasks-per-child `{.interpreted-text > role="option"} option is used for user tasks leaking resources, > like memory or file descriptors, that are simply out of your > control. > > `Read more… `{.interpreted-text > role="ref"}. - **User Components** > Each worker component can be customized, and additional components > can be defined by the user. The worker is built up using > \"bootsteps\" --- a dependency graph enabling fine grained control > of the worker\'s internals. ::: ::: ## Framework Integration Celery is easy to integrate with web frameworks, some of them even have integration packages: > ---------------------------------------------------------------------- ------------------------------------ > [Pyramid](http://docs.pylonsproject.org/en/latest/docs/pyramid.html) `pyramid_celery`{.interpreted-text > role="pypi"} > > [Pylons](http://pylonshq.com/) `celery-pylons`{.interpreted-text > role="pypi"} > > [Flask](http://flask.pocoo.org/) not needed > > [web2py](http://web2py.com/) `web2py-celery`{.interpreted-text > role="pypi"} > > [Tornado](http://www.tornadoweb.org/) `tornado-celery`{.interpreted-text > role="pypi"} > > [Tryton](http://www.tryton.org/) `celery_tryton`{.interpreted-text > role="pypi"} > ---------------------------------------------------------------------- ------------------------------------ For [Django](https://djangoproject.com/) see `django-first-steps`{.interpreted-text role="ref"}. The integration packages aren\'t strictly necessary, but they can make development easier, and sometimes they add important hooks like closing database connections at `fork(2)`{.interpreted-text role="manpage"}. ## Quick Jump ::: topic **I want to ⟶** ::: {.hlist columns="2"} - `get the return value of a task `{.interpreted-text role="ref"} - `use logging from my task `{.interpreted-text role="ref"} - `learn about best practices `{.interpreted-text role="ref"} - `create a custom task base class `{.interpreted-text role="ref"} - `add a callback to a group of tasks `{.interpreted-text role="ref"} - `split a task into several chunks `{.interpreted-text role="ref"} - `optimize the worker `{.interpreted-text role="ref"} - `see a list of built-in task states `{.interpreted-text role="ref"} - `create custom task states `{.interpreted-text role="ref"} - `set a custom task name `{.interpreted-text role="ref"} - `track when a task starts `{.interpreted-text role="ref"} - `retry a task when it fails `{.interpreted-text role="ref"} - `get the id of the current task `{.interpreted-text role="ref"} - `know what queue a task was delivered to `{.interpreted-text role="ref"} - `see a list of running workers `{.interpreted-text role="ref"} - `purge all messages `{.interpreted-text role="ref"} - `inspect what the workers are doing `{.interpreted-text role="ref"} - `see what tasks a worker has registered `{.interpreted-text role="ref"} - `migrate tasks to a new broker `{.interpreted-text role="ref"} - `see a list of event message types `{.interpreted-text role="ref"} - `contribute to Celery `{.interpreted-text role="ref"} - `learn about available configuration settings `{.interpreted-text role="ref"} - `get a list of people and companies using Celery `{.interpreted-text role="ref"} - `write my own remote control command `{.interpreted-text role="ref"} - `change worker queues at runtime `{.interpreted-text role="ref"} ::: ::: ::: topic **Jump to ⟶** ::: {.hlist columns="4"} - `Brokers `{.interpreted-text role="ref"} - `Applications `{.interpreted-text role="ref"} - `Tasks `{.interpreted-text role="ref"} - `Calling `{.interpreted-text role="ref"} - `Workers `{.interpreted-text role="ref"} - `Daemonizing `{.interpreted-text role="ref"} - `Monitoring `{.interpreted-text role="ref"} - `Optimizing `{.interpreted-text role="ref"} - `Security `{.interpreted-text role="ref"} - `Routing `{.interpreted-text role="ref"} - `Configuration `{.interpreted-text role="ref"} - `Django `{.interpreted-text role="ref"} - `Contributing `{.interpreted-text role="ref"} - `Signals `{.interpreted-text role="ref"} - `FAQ `{.interpreted-text role="ref"} - `API Reference `{.interpreted-text role="ref"} ::: ::: ## Installation {#celery-installation} You can install Celery either via the Python Package Index (PyPI) or from source. To install using `pip`{.interpreted-text role="command"}: ``` console $ pip install -U Celery ``` ### Bundles Celery also defines a group of bundles that can be used to install Celery and the dependencies for a given feature. You can specify these in your requirements or on the `pip`{.interpreted-text role="command"} command-line by using brackets. Multiple bundles can be specified by separating them by commas. ``` console $ pip install "celery[librabbitmq]" $ pip install "celery[librabbitmq,redis,auth,msgpack]" ``` The following bundles are available: #### Serializers `celery[auth]` : for using the `auth` security serializer. `celery[msgpack]` : for using the msgpack serializer. `celery[yaml]` : for using the yaml serializer. #### Concurrency `celery[eventlet]` : for using the `eventlet`{.interpreted-text role="pypi"} pool. `celery[gevent]` : for using the `gevent`{.interpreted-text role="pypi"} pool. #### Transports and Backends `celery[librabbitmq]` : for using the librabbitmq C library. `celery[redis]` : for using Redis as a message transport or as a result backend. `celery[sqs]` : for using Amazon SQS as a message transport (*experimental*). `celery[tblib]` : for using the `task_remote_tracebacks`{.interpreted-text role="setting"} feature. `celery[memcache]` : for using Memcached as a result backend (using `pylibmc`{.interpreted-text role="pypi"}) `celery[pymemcache]` : for using Memcached as a result backend (pure-Python implementation). `celery[cassandra]` : for using Apache Cassandra/Astra DB as a result backend with DataStax driver. `celery[couchbase]` : for using Couchbase as a result backend. `celery[arangodb]` : for using ArangoDB as a result backend. `celery[elasticsearch]` : for using Elasticsearch as a result backend. `celery[riak]` : for using Riak as a result backend. `celery[dynamodb]` : for using AWS DynamoDB as a result backend. `celery[zookeeper]` : for using Zookeeper as a message transport. `celery[sqlalchemy]` : for using SQLAlchemy as a result backend (*supported*). `celery[pyro]` : for using the Pyro4 message transport (*experimental*). `celery[slmq]` : for using the SoftLayer Message Queue transport (*experimental*). `celery[consul]` : for using the Consul.io Key/Value store as a message transport or result backend (*experimental*). `celery[django]` : specifies the lowest version possible for Django support. You should probably not use this in your requirements, it\'s here for informational purposes only. `celery[gcs]` : for using the Google Cloud Storage as a result backend (*experimental*). `celery[gcpubsub]` : for using the Google Cloud Pub/Sub as a message transport (*experimental*).. ### Downloading and installing from source {#celery-installing-from-source} Download the latest version of Celery from PyPI: You can install it by doing the following,: ``` console $ tar xvfz celery-0.0.0.tar.gz $ cd celery-0.0.0 $ python setup.py build # python setup.py install ``` The last command must be executed as a privileged user if you aren\'t currently using a virtualenv. ### Using the development version {#celery-installing-from-git} #### With pip The Celery development version also requires the development versions of `kombu`{.interpreted-text role="pypi"}, `amqp`{.interpreted-text role="pypi"}, `billiard`{.interpreted-text role="pypi"}, and `vine`{.interpreted-text role="pypi"}. You can install the latest snapshot of these using the following pip commands: ``` console $ pip install https://github.com/celery/celery/zipball/main#egg=celery $ pip install https://github.com/celery/billiard/zipball/main#egg=billiard $ pip install https://github.com/celery/py-amqp/zipball/main#egg=amqp $ pip install https://github.com/celery/kombu/zipball/main#egg=kombu $ pip install https://github.com/celery/vine/zipball/main#egg=vine ``` #### With git Please see the `Contributing `{.interpreted-text role="ref"} section. --- # Next Steps The `first-steps`{.interpreted-text role="ref"} guide is intentionally minimal. In this guide I\'ll demonstrate what Celery offers in more detail, including how to add Celery support for your application and library. This document doesn\'t document all of Celery\'s features and best practices, so it\'s recommended that you also read the `User Guide `{.interpreted-text role="ref"} ::: {.contents local="" depth="1"} ::: ## Using Celery in your Application ### Our Project {#project-layout} Project layout: src/ proj/__init__.py /celery.py /tasks.py #### `proj/celery.py`{.interpreted-text role="file"} ::: {.literalinclude language="python"} ../../examples/next-steps/proj/celery.py ::: In this module you created our `@Celery`{.interpreted-text role="class"} instance (sometimes referred to as the *app*). To use Celery within your project you simply import this instance. - The `broker` argument specifies the URL of the broker to use. > See `celerytut-broker`{.interpreted-text role="ref"} for more > information. - The `backend` argument specifies the result backend to use. > It\'s used to keep track of task state and results. While results > are disabled by default I use the RPC result backend here because > I demonstrate how retrieving results work later. You may want to > use a different backend for your application. They all have > different strengths and weaknesses. If you don\'t need results, > it\'s better to disable them. Results can also be disabled for > individual tasks by setting the `@task(ignore_result=True)` > option. > > See `celerytut-keeping-results`{.interpreted-text role="ref"} for > more information. - The `include` argument is a list of modules to import when the worker starts. You need to add our tasks module here so that the worker is able to find our tasks. #### `proj/tasks.py`{.interpreted-text role="file"} ::: {.literalinclude language="python"} ../../examples/next-steps/proj/tasks.py ::: ### Starting the worker The `celery`{.interpreted-text role="program"} program can be used to start the worker (you need to run the worker in the directory above [proj]{.title-ref}, according to the example project layout the directory is [src]{.title-ref}): ``` console $ celery -A proj worker -l INFO ``` When the worker starts you should see a banner and some messages: --------------- celery@halcyon.local v4.0 (latentcall) --- ***** ----- -- ******* ---- [Configuration] - *** --- * --- . broker: amqp://guest@localhost:5672// - ** ---------- . app: __main__:0x1012d8590 - ** ---------- . concurrency: 8 (processes) - ** ---------- . events: OFF (enable -E to monitor this worker) - ** ---------- - *** --- * --- [Queues] -- ******* ---- . celery: exchange:celery(direct) binding:celery --- ***** ----- [2012-06-08 16:23:51,078: WARNING/MainProcess] celery@halcyon.local has started. \-- The *broker* is the URL you specified in the broker argument in our `celery` module. You can also specify a different broker on the command-line by using the `-b `{.interpreted-text role="option"} option. \-- *Concurrency* is the number of prefork worker process used to process your tasks concurrently. When all of these are busy doing work, new tasks will have to wait for one of the tasks to finish before it can be processed. The default concurrency number is the number of CPU\'s on that machine (including cores). You can specify a custom number using the `celery worker -c`{.interpreted-text role="option"} option. There\'s no recommended value, as the optimal number depends on a number of factors, but if your tasks are mostly I/O-bound then you can try to increase it. Experimentation has shown that adding more than twice the number of CPU\'s is rarely effective, and likely to degrade performance instead. Including the default prefork pool, Celery also supports using Eventlet, Gevent, and running in a single thread (see `concurrency`{.interpreted-text role="ref"}). \-- *Events* is an option that causes Celery to send monitoring messages (events) for actions occurring in the worker. These can be used by monitor programs like `celery events`, and Flower \-- the real-time Celery monitor, which you can read about in the `Monitoring and Management guide `{.interpreted-text role="ref"}. \-- *Queues* is the list of queues that the worker will consume tasks from. The worker can be told to consume from several queues at once, and this is used to route messages to specific workers as a means for Quality of Service, separation of concerns, and prioritization, all described in the `Routing Guide `{.interpreted-text role="ref"}. You can get a complete list of command-line arguments by passing in the `!--help`{.interpreted-text role="option"} flag: ``` console $ celery worker --help ``` These options are described in more detailed in the `Workers Guide `{.interpreted-text role="ref"}. #### Stopping the worker To stop the worker simply hit `Control-c`{.interpreted-text role="kbd"}. A list of signals supported by the worker is detailed in the `Workers Guide `{.interpreted-text role="ref"}. #### In the background In production you\'ll want to run the worker in the background, described in detail in the `daemonization tutorial `{.interpreted-text role="ref"}. The daemonization scripts uses the `celery multi`{.interpreted-text role="program"} command to start one or more workers in the background: ``` console $ celery multi start w1 -A proj -l INFO celery multi v4.0.0 (latentcall) > Starting nodes... > w1.halcyon.local: OK ``` You can restart it too: ``` console $ celery multi restart w1 -A proj -l INFO celery multi v4.0.0 (latentcall) > Stopping nodes... > w1.halcyon.local: TERM -> 64024 > Waiting for 1 node..... > w1.halcyon.local: OK > Restarting node w1.halcyon.local: OK celery multi v4.0.0 (latentcall) > Stopping nodes... > w1.halcyon.local: TERM -> 64052 ``` or stop it: ``` console $ celery multi stop w1 -A proj -l INFO ``` The `stop` command is asynchronous so it won\'t wait for the worker to shutdown. You\'ll probably want to use the `stopwait` command instead, which ensures that all currently executing tasks are completed before exiting: ``` console $ celery multi stopwait w1 -A proj -l INFO ``` ::: note ::: title Note ::: `celery multi`{.interpreted-text role="program"} doesn\'t store information about workers so you need to use the same command-line arguments when restarting. Only the same pidfile and logfile arguments must be used when stopping. ::: By default it\'ll create pid and log files in the current directory. To protect against multiple workers launching on top of each other you\'re encouraged to put these in a dedicated directory: ``` console $ mkdir -p /var/run/celery $ mkdir -p /var/log/celery $ celery multi start w1 -A proj -l INFO --pidfile=/var/run/celery/%n.pid \ --logfile=/var/log/celery/%n%I.log ``` With the multi command you can start multiple workers, and there\'s a powerful command-line syntax to specify arguments for different workers too, for example: ``` console $ celery multi start 10 -A proj -l INFO -Q:1-3 images,video -Q:4,5 data \ -Q default -L:4,5 debug ``` For more examples see the `~celery.bin.multi`{.interpreted-text role="mod"} module in the API reference. #### About the `--app `{.interpreted-text role="option"} argument {#app-argument} The `--app `{.interpreted-text role="option"} argument specifies the Celery app instance to use, in the form of `module.path:attribute` But it also supports a shortcut form. If only a package name is specified, it\'ll try to search for the app instance, in the following order: With `--app=proj `{.interpreted-text role="option"}: 1) an attribute named `proj.app`, or 2) an attribute named `proj.celery`, or 3) any attribute in the module `proj` where the value is a Celery application, or If none of these are found it\'ll try a submodule named `proj.celery`: 4) an attribute named `proj.celery.app`, or 5) an attribute named `proj.celery.celery`, or 6) Any attribute in the module `proj.celery` where the value is a Celery application. This scheme mimics the practices used in the documentation \-- that is, `proj:app` for a single contained module, and `proj.celery:app` for larger projects. ## Calling Tasks You can call a task using the `delay`{.interpreted-text role="meth"} method: ``` pycon >>> from proj.tasks import add >>> add.delay(2, 2) ``` This method is actually a star-argument shortcut to another method called `apply_async`{.interpreted-text role="meth"}: ``` pycon >>> add.apply_async((2, 2)) ``` The latter enables you to specify execution options like the time to run (countdown), the queue it should be sent to, and so on: ``` pycon >>> add.apply_async((2, 2), queue='lopri', countdown=10) ``` In the above example the task will be sent to a queue named `lopri` and the task will execute, at the earliest, 10 seconds after the message was sent. Applying the task directly will execute the task in the current process, so that no message is sent: ``` pycon >>> add(2, 2) 4 ``` These three methods - `delay`{.interpreted-text role="meth"}, `apply_async`{.interpreted-text role="meth"}, and applying (`__call__`), make up the Celery calling API, which is also used for signatures. A more detailed overview of the Calling API can be found in the `Calling User Guide `{.interpreted-text role="ref"}. Every task invocation will be given a unique identifier (an UUID) \-- this is the task id. The `delay` and `apply_async` methods return an `~@AsyncResult`{.interpreted-text role="class"} instance, which can be used to keep track of the tasks execution state. But for this you need to enable a `result backend `{.interpreted-text role="ref"} so that the state can be stored somewhere. Results are disabled by default because there is no result backend that suits every application; to choose one you need to consider the drawbacks of each individual backend. For many tasks keeping the return value isn\'t even very useful, so it\'s a sensible default to have. Also note that result backends aren\'t used for monitoring tasks and workers: for that Celery uses dedicated event messages (see `guide-monitoring`{.interpreted-text role="ref"}). If you have a result backend configured you can retrieve the return value of a task: ``` pycon >>> res = add.delay(2, 2) >>> res.get(timeout=1) 4 ``` You can find the task\'s id by looking at the `id`{.interpreted-text role="attr"} attribute: ``` pycon >>> res.id d6b3aea2-fb9b-4ebc-8da4-848818db9114 ``` You can also inspect the exception and traceback if the task raised an exception, in fact `result.get()` will propagate any errors by default: ``` pycon >>> res = add.delay(2, '2') >>> res.get(timeout=1) ``` ``` pytb Traceback (most recent call last): File "", line 1, in File "celery/result.py", line 221, in get return self.backend.wait_for_pending( File "celery/backends/asynchronous.py", line 195, in wait_for_pending return result.maybe_throw(callback=callback, propagate=propagate) File "celery/result.py", line 333, in maybe_throw self.throw(value, self._to_remote_traceback(tb)) File "celery/result.py", line 326, in throw self.on_ready.throw(*args, **kwargs) File "vine/promises.py", line 244, in throw reraise(type(exc), exc, tb) File "vine/five.py", line 195, in reraise raise value TypeError: unsupported operand type(s) for +: 'int' and 'str' ``` If you don\'t wish for the errors to propagate, you can disable that by passing `propagate`: ``` pycon >>> res.get(propagate=False) TypeError("unsupported operand type(s) for +: 'int' and 'str'") ``` In this case it\'ll return the exception instance raised instead \--so to check whether the task succeeded or failed, you\'ll have to use the corresponding methods on the result instance: ``` pycon >>> res.failed() True >>> res.successful() False ``` So how does it know if the task has failed or not? It can find out by looking at the tasks *state*: ``` pycon >>> res.state 'FAILURE' ``` A task can only be in a single state, but it can progress through several states. The stages of a typical task can be: PENDING -> STARTED -> SUCCESS The started state is a special state that\'s only recorded if the `task_track_started`{.interpreted-text role="setting"} setting is enabled, or if the `@task(track_started=True)` option is set for the task. The pending state is actually not a recorded state, but rather the default state for any task id that\'s unknown: this you can see from this example: ``` pycon >>> from proj.celery import app >>> res = app.AsyncResult('this-id-does-not-exist') >>> res.state 'PENDING' ``` If the task is retried the stages can become even more complex. To demonstrate, for a task that\'s retried two times the stages would be: ``` text PENDING -> STARTED -> RETRY -> STARTED -> RETRY -> STARTED -> SUCCESS ``` To read more about task states you should see the `task-states`{.interpreted-text role="ref"} section in the tasks user guide. Calling tasks is described in detail in the `Calling Guide `{.interpreted-text role="ref"}. ## *Canvas*: Designing Work-flows {#designing-workflows} You just learned how to call a task using the tasks `delay` method, and this is often all you need. But sometimes you may want to pass the signature of a task invocation to another process or as an argument to another function, for which Celery uses something called *signatures*. A signature wraps the arguments and execution options of a single task invocation in such a way that it can be passed to functions or even serialized and sent across the wire. You can create a signature for the `add` task using the arguments `(2, 2)`, and a countdown of 10 seconds like this: ``` pycon >>> add.signature((2, 2), countdown=10) tasks.add(2, 2) ``` There\'s also a shortcut using star arguments: ``` pycon >>> add.s(2, 2) tasks.add(2, 2) ``` ### And there\'s that calling API again... Signature instances also support the calling API, meaning they have `delay` and `apply_async` methods. But there\'s a difference in that the signature may already have an argument signature specified. The `add` task takes two arguments, so a signature specifying two arguments would make a complete signature: ``` pycon >>> s1 = add.s(2, 2) >>> res = s1.delay() >>> res.get() 4 ``` But, you can also make incomplete signatures to create what we call *partials*: ``` pycon # incomplete partial: add(?, 2) >>> s2 = add.s(2) ``` `s2` is now a partial signature that needs another argument to be complete, and this can be resolved when calling the signature: ``` pycon # resolves the partial: add(8, 2) >>> res = s2.delay(8) >>> res.get() 10 ``` Here you added the argument 8 that was prepended to the existing argument 2 forming a complete signature of `add(8, 2)`. Keyword arguments can also be added later; these are then merged with any existing keyword arguments, but with new arguments taking precedence: ``` pycon >>> s3 = add.s(2, 2, debug=True) >>> s3.delay(debug=False) # debug is now False. ``` As stated, signatures support the calling API: meaning that - `sig.apply_async(args=(), kwargs={}, **options)` > Calls the signature with optional partial arguments and partial > keyword arguments. Also supports partial execution options. - `sig.delay(*args, **kwargs)` Star argument version of `apply_async`. Any arguments will be prepended to the arguments in the signature, and keyword arguments is merged with any existing keys. So this all seems very useful, but what can you actually do with these? To get to that I must introduce the canvas primitives... ### The Primitives ::: topic **\\** ::: {.hlist columns="2"} - `group `{.interpreted-text role="ref"} - `chain `{.interpreted-text role="ref"} - `chord `{.interpreted-text role="ref"} - `map `{.interpreted-text role="ref"} - `starmap `{.interpreted-text role="ref"} - `chunks `{.interpreted-text role="ref"} ::: ::: These primitives are signature objects themselves, so they can be combined in any number of ways to compose complex work-flows. ::: note ::: title Note ::: These examples retrieve results, so to try them out you need to configure a result backend. The example project above already does that (see the backend argument to `~celery.Celery`{.interpreted-text role="class"}). ::: Let\'s look at some examples: #### Groups A `~celery.group`{.interpreted-text role="class"} calls a list of tasks in parallel, and it returns a special result instance that lets you inspect the results as a group, and retrieve the return values in order. ``` pycon >>> from celery import group >>> from proj.tasks import add >>> group(add.s(i, i) for i in range(10))().get() [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] ``` - Partial group ``` pycon >>> g = group(add.s(i) for i in range(10)) >>> g(10).get() [10, 11, 12, 13, 14, 15, 16, 17, 18, 19] ``` #### Chains Tasks can be linked together so that after one task returns the other is called: ``` pycon >>> from celery import chain >>> from proj.tasks import add, mul # (4 + 4) * 8 >>> chain(add.s(4, 4) | mul.s(8))().get() 64 ``` or a partial chain: ``` pycon >>> # (? + 4) * 8 >>> g = chain(add.s(4) | mul.s(8)) >>> g(4).get() 64 ``` Chains can also be written like this: ``` pycon >>> (add.s(4, 4) | mul.s(8))().get() 64 ``` #### Chords A chord is a group with a callback: ``` pycon >>> from celery import chord >>> from proj.tasks import add, xsum >>> chord((add.s(i, i) for i in range(10)), xsum.s())().get() 90 ``` A group chained to another task will be automatically converted to a chord: ``` pycon >>> (group(add.s(i, i) for i in range(10)) | xsum.s())().get() 90 ``` Since these primitives are all of the signature type they can be combined almost however you want, for example: ``` pycon >>> upload_document.s(file) | group(apply_filter.s() for filter in filters) ``` Be sure to read more about work-flows in the `Canvas `{.interpreted-text role="ref"} user guide. ## Routing Celery supports all of the routing facilities provided by AMQP, but it also supports simple routing where messages are sent to named queues. The `task_routes`{.interpreted-text role="setting"} setting enables you to route tasks by name and keep everything centralized in one location: ``` python app.conf.update( task_routes = { 'proj.tasks.add': {'queue': 'hipri'}, }, ) ``` You can also specify the queue at runtime with the `queue` argument to `apply_async`: ``` pycon >>> from proj.tasks import add >>> add.apply_async((2, 2), queue='hipri') ``` You can then make a worker consume from this queue by specifying the `celery worker -Q`{.interpreted-text role="option"} option: ``` console $ celery -A proj worker -Q hipri ``` You may specify multiple queues by using a comma-separated list. For example, you can make the worker consume from both the default queue and the `hipri` queue, where the default queue is named `celery` for historical reasons: ``` console $ celery -A proj worker -Q hipri,celery ``` The order of the queues doesn\'t matter as the worker will give equal weight to the queues. To learn more about routing, including taking use of the full power of AMQP routing, see the `Routing Guide `{.interpreted-text role="ref"}. ## Remote Control If you\'re using RabbitMQ (AMQP), Redis, or Qpid as the broker then you can control and inspect the worker at runtime. For example you can see what tasks the worker is currently working on: ``` console $ celery -A proj inspect active ``` This is implemented by using broadcast messaging, so all remote control commands are received by every worker in the cluster. You can also specify one or more workers to act on the request using the `--destination `{.interpreted-text role="option"} option. This is a comma-separated list of worker host names: ``` console $ celery -A proj inspect active --destination=celery@example.com ``` If a destination isn\'t provided then every worker will act and reply to the request. The `celery inspect`{.interpreted-text role="program"} command contains commands that don\'t change anything in the worker; it only returns information and statistics about what\'s going on inside the worker. For a list of inspect commands you can execute: ``` console $ celery -A proj inspect --help ``` Then there\'s the `celery control`{.interpreted-text role="program"} command, which contains commands that actually change things in the worker at runtime: ``` console $ celery -A proj control --help ``` For example you can force workers to enable event messages (used for monitoring tasks and workers): ``` console $ celery -A proj control enable_events ``` When events are enabled you can then start the event dumper to see what the workers are doing: ``` console $ celery -A proj events --dump ``` or you can start the curses interface: ``` console $ celery -A proj events ``` when you\'re finished monitoring you can disable events again: ``` console $ celery -A proj control disable_events ``` The `celery status`{.interpreted-text role="program"} command also uses remote control commands and shows a list of online workers in the cluster: ``` console $ celery -A proj status ``` You can read more about the `celery`{.interpreted-text role="program"} command and monitoring in the `Monitoring Guide `{.interpreted-text role="ref"}. ## Timezone All times and dates, internally and in messages use the UTC timezone. When the worker receives a message, for example with a countdown set it converts that UTC time to local time. If you wish to use a different timezone than the system timezone then you must configure that using the `timezone`{.interpreted-text role="setting"} setting: ``` python app.conf.timezone = 'Europe/London' ``` ## Optimization The default configuration isn\'t optimized for throughput. By default, it tries to walk the middle way between many short tasks and fewer long tasks, a compromise between throughput and fair scheduling. If you have strict fair scheduling requirements, or want to optimize for throughput then you should read the `Optimizing Guide `{.interpreted-text role="ref"}. ## What to do now? Now that you have read this document you should continue to the `User Guide `{.interpreted-text role="ref"}. There\'s also an `API reference `{.interpreted-text role="ref"} if you\'re so inclined. --- # Resources ::: {.contents local="" depth="2"} ::: ## Getting Help ::: warning ::: title Warning ::: Our [Google Groups account](https://groups.google.com/group/celery-users/) has been [compromised](https://github.com/celery/celery/discussions/9525). ::: ## Social Media Follow us on social media: - [X](https://x.com/celeryorg) - [LinkedIn](https://linkedin.com/company/celeryorg) These accounts will (mostly) mirror each other, but we encourage you to follow us on all platforms to ensure you don\'t miss any important updates. ## Bug tracker If you have any suggestions, bug reports, or annoyances please report them to our issue tracker at ## Contributing {#contributing-short} Development of [celery]{.title-ref} happens at GitHub: You\'re highly encouraged to participate in the development of [celery]{.title-ref}. If you don\'t like GitHub (for some reason) you\'re welcome to send regular patches. Be sure to also read the [Contributing to Celery]() section in the documentation. ::: {#Contributing to Celery} > ::: ## License This software is licensed under the [New BSD License]{.title-ref}. See the `LICENSE`{.interpreted-text role="file"} file in the top distribution directory for the full license text. --- # Glossary ::: {.glossary sorted=""} acknowledged : Workers acknowledge messages to signify that a message has been handled. Failing to acknowledge a message will cause the message to be redelivered. Exactly when a transaction is considered a failure varies by transport. In AMQP the transaction fails when the connection/channel is closed (or lost), but in Redis/SQS the transaction times out after a configurable amount of time (the `visibility_timeout`). ack : Short for `acknowledged`{.interpreted-text role="term"}. early acknowledgment : Task is `acknowledged`{.interpreted-text role="term"} just-in-time before being executed, meaning the task won\'t be redelivered to another worker if the machine loses power, or the worker instance is abruptly killed, mid-execution. Configured using `task_acks_late`{.interpreted-text role="setting"}. late acknowledgment : Task is `acknowledged`{.interpreted-text role="term"} after execution (both if successful, or if the task is raising an error), which means the task will be redelivered to another worker in the event of the machine losing power, or the worker instance being killed mid-execution. Configured using `task_acks_late`{.interpreted-text role="setting"}. early ack : Short for `early acknowledgment`{.interpreted-text role="term"} late ack : Short for `late acknowledgment`{.interpreted-text role="term"} ETA : \"Estimated Time of Arrival\", in Celery and Google Task Queue, etc., used as the term for a delayed message that should not be processed until the specified ETA time. See `calling-eta`{.interpreted-text role="ref"}. request : Task messages are converted to *requests* within the worker. The request information is also available as the task\'s `context`{.interpreted-text role="term"} (the `task.request` attribute). calling : Sends a task message so that the task function is `executed `{.interpreted-text role="term"} by a worker. kombu : Python messaging library used by Celery to send and receive messages. billiard : Fork of the Python multiprocessing library containing improvements required by Celery. executing : Workers *execute* task `requests `{.interpreted-text role="term"}. apply : Originally a synonym to `call `{.interpreted-text role="term"} but used to signify that a function is executed by the current process. context : The context of a task contains information like the id of the task, it\'s arguments and what queue it was delivered to. It can be accessed as the tasks `request` attribute. See `task-request-info`{.interpreted-text role="ref"} idempotent : Idempotence is a mathematical property that describes a function that can be called multiple times without changing the result. Practically it means that a function can be repeated many times without unintended effects, but not necessarily side-effect free in the pure sense (compare to `nullipotent`{.interpreted-text role="term"}). Further reading: nullipotent : describes a function that\'ll have the same effect, and give the same result, even if called zero or multiple times (side-effect free). A stronger version of `idempotent`{.interpreted-text role="term"}. reentrant : describes a function that can be interrupted in the middle of execution (e.g., by hardware interrupt or signal), and then safely called again later. Reentrancy isn\'t the same as `idempotence `{.interpreted-text role="term"} as the return value doesn\'t have to be the same given the same inputs, and a reentrant function may have side effects as long as it can be interrupted; An idempotent function is always reentrant, but the reverse may not be true. cipater : Celery release 3.1 named after song by Autechre () prefetch multiplier : The `prefetch count`{.interpreted-text role="term"} is configured by using the `worker_prefetch_multiplier`{.interpreted-text role="setting"} setting, which is multiplied by the number of pool slots (threads/processes/greenthreads). [prefetch count]{.title-ref} : Maximum number of unacknowledged messages a consumer can hold and if exceeded the transport shouldn\'t deliver any more messages to that consumer. See `optimizing-prefetch-limit`{.interpreted-text role="ref"}. pidbox : A process mailbox, used to implement remote control commands. ::: --- # Change history for Celery 1.0 {#changelog-1.0} ::: {.contents local=""} ::: ## 1.0.6 {#version-1.0.6} release-date : 2010-06-30 09:57 a.m. CEST release-by : Ask Solem - RabbitMQ 1.8.0 has extended their exchange equivalence tests to include [auto_delete]{.title-ref} and [durable]{.title-ref}. This broke the AMQP backend. If you\'ve already used the AMQP backend this means you have to delete the previous definitions: ``` console $ camqadm exchange.delete celeryresults ``` or: ``` console $ python manage.py camqadm exchange.delete celeryresults ``` ## 1.0.5 {#version-1.0.5} release-date : 2010-06-01 02:36 p.m. CEST release-by : Ask Solem ### Critical {#v105-critical} - `INT`{.interpreted-text role="sig"}/`Control-c`{.interpreted-text role="kbd"} killed the pool, abruptly terminating the currently executing tasks. > Fixed by making the pool worker processes ignore > `SIGINT`{.interpreted-text role="const"}. - Shouldn\'t close the consumers before the pool is terminated, just cancel the consumers. > See issue #122. - Now depends on `billiard`{.interpreted-text role="pypi"} \>= 0.3.1 - worker: Previously exceptions raised by worker components could stall start-up, now it correctly logs the exceptions and shuts down. - worker: Prefetch counts was set too late. QoS is now set as early as possible, so the worker: can\'t slurp in all the messages at start-up. ### Changes {#v105-changes} - `celery.contrib.abortable`{.interpreted-text role="mod"}: Abortable tasks. > Tasks that defines steps of execution, the task can then be > aborted after each step has completed. - `~celery.events.EventDispatcher`{.interpreted-text role="class"}: No longer creates AMQP channel if events are disabled - Added required RPM package names under [\[bdist_rpm\]]{.title-ref} section, to support building RPMs from the sources using `setup.py`{.interpreted-text role="file"}. - Running unit tests: `NOSE_VERBOSE`{.interpreted-text role="envvar"} environment var now enables verbose output from Nose. - `celery.execute.apply`{.interpreted-text role="func"}: Pass log file/log level arguments as task kwargs. > See issue #110. - celery.execute.apply: Should return exception, not `~billiard.einfo.ExceptionInfo`{.interpreted-text role="class"} on error. > See issue #111. - Added new entries to the `FAQs `{.interpreted-text role="ref"}: > - Should I use retry or acks_late? > - Can I call a task by name? ## 1.0.4 {#version-1.0.4} release-date : 2010-05-31 09:54 a.m. CEST release-by : Ask Solem - Changelog merged with 1.0.5 as the release was never announced. ## 1.0.3 {#version-1.0.3} release-date : 2010-05-15 03:00 p.m. CEST release-by : Ask Solem ### Important notes {#v103-important} - Messages are now acknowledged *just before* the task function is executed. > This is the behavior we\'ve wanted all along, but couldn\'t have > because of limitations in the multiprocessing module. The previous > behavior wasn\'t good, and the situation worsened with the release > of 1.0.1, so this change will definitely improve reliability, > performance and operations in general. > > For more information please see - Database result backend: result now explicitly sets [null=True]{.title-ref} as [django-picklefield]{.title-ref} version 0.1.5 changed the default behavior right under our noses :( > See: > > This means those who created their Celery tables (via `syncdb` or > `celeryinit`) with `django-picklefield`{.interpreted-text > role="pypi"}[ versions \>= 0.1.5 has to alter their tables to > allow the result field to be \`NULL]{.title-ref} manually. > > MySQL: > > ``` sql > ALTER TABLE celery_taskmeta MODIFY result TEXT NULL > ``` > > PostgreSQL: > > ``` sql > ALTER TABLE celery_taskmeta ALTER COLUMN result DROP NOT NULL > ``` - Removed [Task.rate_limit_queue_type]{.title-ref}, as it wasn\'t really useful and made it harder to refactor some parts. - Now depends on carrot \>= 0.10.4 - Now depends on billiard \>= 0.3.0 ### News {#v103-news} - AMQP backend: Added timeout support for [result.get()]{.title-ref} / [result.wait()]{.title-ref}. - New task option: [Task.acks_late]{.title-ref} (default: `CELERY_ACKS_LATE`{.interpreted-text role="setting"}) > Late ack means the task messages will be acknowledged **after** > the task has been executed, not *right before*, which is the > default behavior. > > ::: note > ::: title > Note > ::: > > This means the tasks may be executed twice if the worker crashes > in mid-execution. Not acceptable for most applications, but > desirable for others. > ::: - Added Crontab-like scheduling to periodic tasks. > Like a cronjob, you can specify units of time of when you\'d like > the task to execute. While not a full implementation of > `cron`{.interpreted-text role="command"}\'s features, it should > provide a fair degree of common scheduling needs. > > You can specify a minute (0-59), an hour (0-23), and/or a day of > the week (0-6 where 0 is Sunday, or by names: > `sun, mon, tue, wed, thu, fri, sat`). > > Examples: > > ``` python > from celery.schedules import crontab > from celery.decorators import periodic_task > > @periodic_task(run_every=crontab(hour=7, minute=30)) > def every_morning(): > print('Runs every morning at 7:30a.m') > > @periodic_task(run_every=crontab(hour=7, minute=30, day_of_week='mon')) > def every_monday_morning(): > print('Run every monday morning at 7:30a.m') > > @periodic_task(run_every=crontab(minutes=30)) > def every_hour(): > print('Runs every hour on the clock (e.g., 1:30, 2:30, 3:30 etc.).') > ``` > > ::: note > ::: title > Note > ::: > > This a late addition. While we have unit tests, due to the nature > of this feature we haven\'t been able to completely test this in > practice, so consider this experimental. > ::: - \`TaskPool.apply_async\`: Now supports the [accept_callback]{.title-ref} argument. - \`apply_async\`: Now raises `ValueError`{.interpreted-text role="exc"} if task args isn\'t a list, or kwargs isn\'t a tuple (Issue #95). - [Task.max_retries]{.title-ref} can now be [None]{.title-ref}, which means it will retry forever. - `celerybeat`: Now reuses the same connection when publishing large sets of tasks. - Modified the task locking example in the documentation to use [cache.add]{.title-ref} for atomic locking. - Added experimental support for a *started* status on tasks. > If [Task.track_started]{.title-ref} is enabled the task will > report its status as \"started\" when the task is executed by a > worker. > > The default value is [False]{.title-ref} as the normal behavior is > to not report that level of granularity. Tasks are either pending, > finished, or waiting to be retried. Having a \"started\" status > can be useful for when there are long running tasks and there\'s a > need to report which task is currently running. > > The global default can be overridden by the > `CELERY_TRACK_STARTED`{.interpreted-text role="setting"} setting. - User Guide: New section [Tips and Best Practices]{.title-ref}. > Contributions welcome! ### Remote control commands {#v103-remote-control} - Remote control commands can now send replies back to the caller. > Existing commands has been improved to send replies, and the > client interface in [celery.task.control]{.title-ref} has new > keyword arguments: [reply]{.title-ref}, [timeout]{.title-ref} and > [limit]{.title-ref}. Where reply means it will wait for replies, > timeout is the time in seconds to stop waiting for replies, and > limit is the maximum number of replies to get. > > By default, it will wait for as many replies as possible for one > second. > > - rate_limit(task_name, destination=all, reply=False, timeout=1, > limit=0) > > > Worker returns [{\'ok\': message}]{.title-ref} on success, > > or [{\'failure\': message}]{.title-ref} on failure. > > > > > \>\>\> from celery.task.control import rate_limit \>\>\> > > > rate_limit(\'tasks.add\', \'10/s\', reply=True) > > > \[{\'worker1\': {\'ok\': \'new rate limit set > > > successfully\'}}, {\'worker2\': {\'ok\': \'new rate limit > > > set successfully\'}}\] > > - ping(destination=all, reply=False, timeout=1, limit=0) > > > Worker returns the simple message [\"pong\"]{.title-ref}. > > > > > \>\>\> from celery.task.control import ping \>\>\> > > > ping(reply=True) \[{\'worker1\': \'pong\'}, {\'worker2\': > > > \'pong\'}, > > - revoke(destination=all, reply=False, timeout=1, limit=0) > > > Worker simply returns [True]{.title-ref}. > > > > > \>\>\> from celery.task.control import revoke \>\>\> > > > revoke(\'419e46eb-cf6a-4271-86a8-442b7124132c\', > > > reply=True) \[{\'worker1\': True}, {\'worker2\'; True}\] - You can now add your own remote control commands! > Remote control commands are functions registered in the command > registry. Registering a command is done using > `celery.worker.control.Panel.register`{.interpreted-text > role="meth"}: > > ``` python > from celery.task.control import Panel > > @Panel.register > def reset_broker_connection(state, **kwargs): > state.consumer.reset_connection() > return {'ok': 'connection re-established'} > ``` > > With this module imported in the worker, you can launch the > command using \`celery.task.control.broadcast\`: > > >>> from celery.task.control import broadcast > >>> broadcast('reset_broker_connection', reply=True) > [{'worker1': {'ok': 'connection re-established'}, > {'worker2': {'ok': 'connection re-established'}}] > > **TIP** You can choose the worker(s) to receive the command by > using the [destination]{.title-ref} argument: > > >>> broadcast('reset_broker_connection', destination=['worker1']) > [{'worker1': {'ok': 'connection re-established'}] - New remote control command: [dump_reserved]{.title-ref} > Dumps tasks reserved by the worker, waiting to be executed: > > >>> from celery.task.control import broadcast > >>> broadcast('dump_reserved', reply=True) > [{'myworker1': []}] - New remote control command: [dump_schedule]{.title-ref} > Dumps the workers currently registered ETA schedule. These are > tasks with an [eta]{.title-ref} (or [countdown]{.title-ref}) > argument waiting to be executed by the worker. > > > \>\>\> from celery.task.control import broadcast \>\>\> > > broadcast(\'dump_schedule\', reply=True) \[{\'w1\': \[\]}, > > {\'w3\': \[\]}, {\'w2\': \[\'0. 2010-05-12 11:06:00 pri0 > > \ > id:\'95b45760-4e73-4ce8-8eac-f100aa80273a\', args:\'(\ > freq_max:3600 freq_min:60 start:2184.0 stop:3276.0\>,)\', > > kwargs:\'{\'page\': 2}\'}\>\'\]}, {\'w4\': \[\'0. 2010-05-12 > > 11:00:00 pri0 \ > {name:\'opalfeeds.tasks.refresh_feed_slice\', > > id:\'c053480b-58fb-422f-ae68-8d30a464edfe\', args:\'(\ > freq_max:3600 freq_min:60 start:1092.0 stop:2184.0\>,)\', > > kwargs:\'{\'page\': 1}\'}\>\', \'1. 2010-05-12 11:12:00 pri0 > > \ > id:\'ab8bc59e-6cf8-44b8-88d0-f1af57789758\', args:\'(\ > freq_max:3600 freq_min:60 start:3276.0 stop:4365\>,)\', > > kwargs:\'{\'page\': 3}\'}\>\'\]}\] ### Fixes {#v103-fixes} - Mediator thread no longer blocks for more than 1 second. > With rate limits enabled and when there was a lot of remaining > time, the mediator thread could block shutdown (and potentially > block other jobs from coming in). - Remote rate limits wasn\'t properly applied (Issue #98). - Now handles exceptions with Unicode messages correctly in [TaskRequest.on_failure]{.title-ref}. - Database backend: \`TaskMeta.result\`: default value should be [None]{.title-ref} not empty string. ## 1.0.2 {#version-1.0.2} release-date : 2010-03-31 12:50 p.m. CET release-by : Ask Solem - Deprecated: `CELERY_BACKEND`{.interpreted-text role="setting"}, please use `CELERY_RESULT_BACKEND`{.interpreted-text role="setting"} instead. - We now use a custom logger in tasks. This logger supports task magic keyword arguments in formats. > The default format for tasks > (`CELERYD_TASK_LOG_FORMAT`{.interpreted-text role="setting"}) now > includes the id and the name of tasks so the origin of task log > messages can easily be traced. > > Example output:: > > : > > \[2010-03-25 13:11:20,317: INFO/PoolWorker-1\] > > : \[tasks.add(a6e1c5ad-60d9-42a0-8b24-9e39363125a4)\] Hello > from add > > To revert to the previous behavior you can set: > > CELERYD_TASK_LOG_FORMAT = """ > [%(asctime)s: %(levelname)s/%(processName)s] %(message)s > """.strip() - Unit tests: Don\'t disable the django test database tear down, instead fixed the underlying issue which was caused by modifications to the [DATABASE_NAME]{.title-ref} setting (Issue #82). - Django Loader: New config `CELERY_DB_REUSE_MAX`{.interpreted-text role="setting"} (max number of tasks to reuse the same database connection) > The default is to use a new connection for every task. We\'d very > much like to reuse the connection, but a safe number of reuses > isn\'t known, and we don\'t have any way to handle the errors that > might happen, which may even be database dependent. > > See: - worker: The worker components are now configurable: `CELERYD_POOL`{.interpreted-text role="setting"}, `CELERYD_CONSUMER`{.interpreted-text role="setting"}, `CELERYD_MEDIATOR`{.interpreted-text role="setting"}, and `CELERYD_ETA_SCHEDULER`{.interpreted-text role="setting"}. > The default configuration is as follows: > > ``` python > CELERYD_POOL = 'celery.concurrency.processes.TaskPool' > CELERYD_MEDIATOR = 'celery.worker.controllers.Mediator' > CELERYD_ETA_SCHEDULER = 'celery.worker.controllers.ScheduleController' > CELERYD_CONSUMER = 'celery.worker.consumer.Consumer' > ``` > > The `CELERYD_POOL`{.interpreted-text role="setting"} setting makes > it easy to swap out the multiprocessing pool with a threaded pool, > or how about a twisted/eventlet pool? > > Consider the competition for the first pool plug-in started! - Debian init-scripts: Use [-a]{.title-ref} not [&&]{.title-ref} (Issue #82). - Debian init-scripts: Now always preserves [\$CELERYD_OPTS]{.title-ref} from the [/etc/default/celeryd]{.title-ref} and [/etc/default/celerybeat]{.title-ref}. - celery.beat.Scheduler: Fixed a bug where the schedule wasn\'t properly flushed to disk if the schedule hadn\'t been properly initialized. - `celerybeat`: Now syncs the schedule to disk when receiving the `SIGTERM`{.interpreted-text role="sig"} and `SIGINT`{.interpreted-text role="sig"} signals. - Control commands: Make sure keywords arguments aren\'t in Unicode. - ETA scheduler: Was missing a logger object, so the scheduler crashed when trying to log that a task had been revoked. - `management.commands.camqadm`: Fixed typo [camqpadm]{.title-ref} -\> [camqadm]{.title-ref} (Issue #83). - PeriodicTask.delta_resolution: wasn\'t working for days and hours, now fixed by rounding to the nearest day/hour. - Fixed a potential infinite loop in [BaseAsyncResult.\_\_eq\_\_]{.title-ref}, although there\'s no evidence that it has ever been triggered. - worker: Now handles messages with encoding problems by acking them and emitting an error message. ## 1.0.1 {#version-1.0.1} release-date : 2010-02-24 07:05 p.m. CET release-by : Ask Solem - Tasks are now acknowledged early instead of late. > This is done because messages can only be acknowledged within the > same connection channel, so if the connection is lost we\'d\'ve to > re-fetch the message again to acknowledge it. > > This might or might not affect you, but mostly those running tasks > with a really long execution time are affected, as all tasks > that\'s made it all the way into the pool needs to be executed > before the worker can safely terminate (this is at most the number > of pool workers, multiplied by the > `CELERYD_PREFETCH_MULTIPLIER`{.interpreted-text role="setting"} > setting). > > We multiply the prefetch count by default to increase the > performance at times with bursts of tasks with a short execution > time. If this doesn\'t apply to your use case, you should be able > to set the prefetch multiplier to zero, without sacrificing > performance. > > ::: note > ::: title > Note > ::: > > A patch to `multiprocessing`{.interpreted-text role="mod"} is > currently being worked on, this patch would enable us to use a > better solution, and is scheduled for inclusion in the > [2.0.0]{.title-ref} release. > ::: - The worker now shutdowns cleanly when receiving the `SIGTERM`{.interpreted-text role="sig"} signal. - The worker now does a cold shutdown if the `SIGINT`{.interpreted-text role="sig"} signal is received (`Control-c`{.interpreted-text role="kbd"}), this means it tries to terminate as soon as possible. - Caching of results now moved to the base backend classes, so no need to implement this functionality in the base classes. - Caches are now also limited in size, so their memory usage doesn\'t grow out of control. > You can set the maximum number of results the cache can hold using > the `CELERY_MAX_CACHED_RESULTS`{.interpreted-text role="setting"} > setting (the default is five thousand results). In addition, you > can re-fetch already retrieved results using > [backend.reload_task_result]{.title-ref} + > [backend.reload_taskset_result]{.title-ref} (that\'s for those who > want to send results incrementally). - The worker now works on Windows again. > ::: warning > ::: title > Warning > ::: > > If you\'re using Celery with Django, you can\'t use > [project.settings]{.title-ref} as the settings module name, but > the following should work: > > ``` console > $ python manage.py celeryd --settings=settings > ``` > ::: - Execution: [.messaging.TaskPublisher.send_task]{.title-ref} now incorporates all the functionality apply_async previously did. > Like converting countdowns to ETA, so > `celery.execute.apply_async`{.interpreted-text role="func"} is now > simply a convenient front-end to > `celery.messaging.TaskPublisher.send_task`{.interpreted-text > role="meth"}, using the task classes default options. > > Also `celery.execute.send_task`{.interpreted-text role="func"} has > been introduced, which can apply tasks using just the task name > (useful if the client doesn\'t have the destination task in its > task registry). > > Example: > > > \>\>\> from celery.execute import send_task \>\>\> result = > > send_task(\'celery.ping\', args=\[\], kwargs={}) \>\>\> > > result.get() \'pong\' - \`camqadm\`: This is a new utility for command-line access to the AMQP API. > Excellent for deleting queues/bindings/exchanges, experimentation > and testing: > > ``` console > $ camqadm > 1> help > ``` > > Gives an interactive shell, type [help]{.title-ref} for a list of > commands. > > When using Django, use the management command instead: > > ``` console > $ python manage.py camqadm > 1> help > ``` - Redis result backend: To conform to recent Redis API changes, the following settings has been deprecated: > - [REDIS_TIMEOUT]{.title-ref} > - [REDIS_CONNECT_RETRY]{.title-ref} > > These will emit a [DeprecationWarning]{.title-ref} if used. > > A [REDIS_PASSWORD]{.title-ref} setting has been added, so you can > use the new simple authentication mechanism in Redis. - The redis result backend no longer calls [SAVE]{.title-ref} when disconnecting, as this is apparently better handled by Redis itself. - If [settings.DEBUG]{.title-ref} is on, the worker now warns about the possible memory leak it can result in. - The ETA scheduler now sleeps at most two seconds between iterations. - The ETA scheduler now deletes any revoked tasks it might encounter. > As revokes aren\'t yet persistent, this is done to make sure the > task is revoked even though, for example, it\'s currently being > hold because its ETA is a week into the future. - The [task_id]{.title-ref} argument is now respected even if the task is executed eagerly (either using apply, or `CELERY_ALWAYS_EAGER`{.interpreted-text role="setting"}). - The internal queues are now cleared if the connection is reset. - New magic keyword argument: [delivery_info]{.title-ref}. > Used by retry() to resend the task to its original destination > using the same exchange/routing_key. - Events: Fields wasn\'t passed by [.send()]{.title-ref} (fixes the UUID key errors in celerymon) - Added [\--schedule]{.title-ref}/[-s]{.title-ref} option to the worker, so it is possible to specify a custom schedule filename when using an embedded `celerybeat` server (the [-B]{.title-ref}/[\--beat]{.title-ref}) option. - Better Python 2.4 compatibility. The test suite now passes. - task decorators: Now preserve docstring as [cls.\_\_doc\_\_]{.title-ref}, (was previously copied to [cls.run.\_\_doc\_\_]{.title-ref}) - The [testproj]{.title-ref} directory has been renamed to [tests]{.title-ref} and we\'re now using [nose]{.title-ref} + [django-nose]{.title-ref} for test discovery, and [unittest2]{.title-ref} for test cases. - New pip requirements files available in `requirements`{.interpreted-text role="file"}. - TaskPublisher: Declarations are now done once (per process). - Added [Task.delivery_mode]{.title-ref} and the `CELERY_DEFAULT_DELIVERY_MODE`{.interpreted-text role="setting"} setting. > These can be used to mark messages non-persistent (i.e., so > they\'re lost if the broker is restarted). - Now have our own [ImproperlyConfigured]{.title-ref} exception, instead of using the Django one. - Improvements to the Debian init-scripts: Shows an error if the program is not executable. Does not modify [CELERYD]{.title-ref} when using django with virtualenv. ## 1.0.0 {#version-1.0.0} release-date : 2010-02-10 04:00 p.m. CET release-by : Ask Solem ### Backward incompatible changes {#v100-incompatible} - Celery doesn\'t support detaching anymore, so you have to use the tools available on your platform, or something like `supervisor`{.interpreted-text role="pypi"} to make `celeryd`/`celerybeat`/`celerymon` into background processes. > We\'ve had too many problems with the worker daemonizing itself, > so it was decided it has to be removed. Example start-up scripts > has been added to the [extra/]{.title-ref} directory: > > - Debian, Ubuntu, (`start-stop-daemon`{.interpreted-text > role="command"}) > > > [extra/debian/init.d/celeryd]{.title-ref} > > [extra/debian/init.d/celerybeat]{.title-ref} > > - macOS `launchd`{.interpreted-text role="command"} > > > [extra/mac/org.celeryq.celeryd.plist]{.title-ref} > > [extra/mac/org.celeryq.celerybeat.plist]{.title-ref} > > [extra/mac/org.celeryq.celerymon.plist]{.title-ref} > > - Supervisor () > > > [extra/supervisord/supervisord.conf]{.title-ref} > > In addition to [\--detach]{.title-ref}, the following program > arguments has been removed: [\--uid]{.title-ref}, > [\--gid]{.title-ref}, [\--workdir]{.title-ref}, > [\--chroot]{.title-ref}, [\--pidfile]{.title-ref}, > [\--umask]{.title-ref}. All good daemonization tools should > support equivalent functionality, so don\'t worry. > > Also the following configuration keys has been removed: > [CELERYD_PID_FILE]{.title-ref}, [CELERYBEAT_PID_FILE]{.title-ref}, > [CELERYMON_PID_FILE]{.title-ref}. - Default worker loglevel is now [WARN]{.title-ref}, to enable the previous log level start the worker with [\--loglevel=INFO]{.title-ref}. - Tasks are automatically registered. > This means you no longer have to register your tasks manually. You > don\'t have to change your old code right away, as it doesn\'t > matter if a task is registered twice. > > If you don\'t want your task to be automatically registered you > can set the [abstract]{.title-ref} attribute > > ``` python > class MyTask(Task): > abstract = True > ``` > > By using [abstract]{.title-ref} only tasks subclassing this task > will be automatically registered (this works like the Django ORM). > > If you don\'t want subclasses to be registered either, you can set > the [autoregister]{.title-ref} attribute to [False]{.title-ref}. > > Incidentally, this change also fixes the problems with automatic > name assignment and relative imports. So you also don\'t have to > specify a task name anymore if you use relative imports. - You can no longer use regular functions as tasks. > This change was added because it makes the internals a lot more > clean and simple. However, you can now turn functions into tasks > by using the [\@task]{.title-ref} decorator: > > ``` python > from celery.decorators import task > > @task() > def add(x, y): > return x + y > ``` > > ::: seealso > `guide-tasks`{.interpreted-text role="ref"} for more information > about the task decorators. > ::: - The periodic task system has been rewritten to a centralized solution. > This means the worker no longer schedules periodic tasks by > default, but a new daemon has been introduced: > [celerybeat]{.title-ref}. > > To launch the periodic task scheduler you have to run > `celerybeat`: > > ``` console > $ celerybeat > ``` > > Make sure this is running on one server only, if you run it twice, > all periodic tasks will also be executed twice. > > If you only have one worker server you can embed it into the > worker like this: > > ``` console > $ celeryd --beat # Embed celerybeat in celeryd. > ``` - The supervisor has been removed. > This means the [-S]{.title-ref} and [\--supervised]{.title-ref} > options to [celeryd]{.title-ref} is no longer supported. Please > use something like instead. - [TaskSet.join]{.title-ref} has been removed, use [TaskSetResult.join]{.title-ref} instead. - The task status [\"DONE\"]{.title-ref} has been renamed to [\"SUCCESS\"]{.title-ref}. - [AsyncResult.is_done]{.title-ref} has been removed, use [AsyncResult.successful]{.title-ref} instead. - The worker no longer stores errors if [Task.ignore_result]{.title-ref} is set, to revert to the previous behavior set `CELERY_STORE_ERRORS_EVEN_IF_IGNORED`{.interpreted-text role="setting"} to [True]{.title-ref}. - The statistics functionality has been removed in favor of events, so the [-S]{.title-ref} and \--statistics\` switches has been removed. - The module [celery.task.strategy]{.title-ref} has been removed. - [celery.discovery]{.title-ref} has been removed, and it\'s `autodiscover` function is now in [celery.loaders.djangoapp]{.title-ref}. Reason: Internal API. - The `CELERY_LOADER`{.interpreted-text role="envvar"} environment variable now needs loader class name in addition to module name, > For example, where you previously had: > [\"celery.loaders.default\"]{.title-ref}, you now need > [\"celery.loaders.default.Loader\"]{.title-ref}, using the > previous syntax will result in a [DeprecationWarning]{.title-ref}. - Detecting the loader is now lazy, and so isn\'t done when importing [celery.loaders]{.title-ref}. > To make this happen [celery.loaders.settings]{.title-ref} has been > renamed to [load_settings]{.title-ref} and is now a function > returning the settings object. > [celery.loaders.current_loader]{.title-ref} is now also a > function, returning the current loader. > > So: > > loader = current_loader > > needs to be changed to: > > loader = current_loader() ### Deprecations {#v100-deprecations} - The following configuration variables has been renamed and will be deprecated in v2.0: > - `CELERYD_DAEMON_LOG_FORMAT` -\> `CELERYD_LOG_FORMAT` > - `CELERYD_DAEMON_LOG_LEVEL` -\> `CELERYD_LOG_LEVEL` > - `CELERY_AMQP_CONNECTION_TIMEOUT` -\> > `CELERY_BROKER_CONNECTION_TIMEOUT` > - `CELERY_AMQP_CONNECTION_RETRY` -\> > `CELERY_BROKER_CONNECTION_RETRY` > - `CELERY_AMQP_CONNECTION_MAX_RETRIES` -\> > `CELERY_BROKER_CONNECTION_MAX_RETRIES` > - `SEND_CELERY_TASK_ERROR_EMAILS` -\> > `CELERY_SEND_TASK_ERROR_EMAILS` - The public API names in celery.conf has also changed to a consistent naming scheme. - We now support consuming from an arbitrary number of queues. > To do this we had to rename the configuration syntax. If you use > any of the custom AMQP routing options > (queue/exchange/routing_key, etc.), you should read the new FAQ > entry: `faq-task-routing`{.interpreted-text role="ref"}. > > The previous syntax is deprecated and scheduled for removal in > v2.0. - [TaskSet.run]{.title-ref} has been renamed to [TaskSet.apply_async]{.title-ref}. > [TaskSet.run]{.title-ref} has now been deprecated, and is > scheduled for removal in v2.0. ### News - Rate limiting support (per task type, or globally). - New periodic task system. - Automatic registration. - New cool task decorator syntax. - worker: now sends events if enabled with the [-E]{.title-ref} argument. > Excellent for monitoring tools, one is already in the making > (). > > Current events include: `worker-heartbeat`{.interpreted-text > role="event"}, task-\[received/succeeded/failed/retried\], > `worker-online`{.interpreted-text role="event"}, > `worker-offline`{.interpreted-text role="event"}. - You can now delete (revoke) tasks that\'s already been applied. - You can now set the hostname the worker identifies as using the [\--hostname]{.title-ref} argument. - Cache backend now respects the `CELERY_TASK_RESULT_EXPIRES`{.interpreted-text role="setting"} setting. - Message format has been standardized and now uses ISO-8601 format for dates instead of datetime. - worker now responds to the `SIGHUP`{.interpreted-text role="sig"} signal by restarting itself. - Periodic tasks are now scheduled on the clock. > That is, [timedelta(hours=1)]{.title-ref} means every hour at :00 > minutes, not every hour from the server starts. To revert to the > previous behavior you can set [PeriodicTask.relative = > True]{.title-ref}. - Now supports passing execute options to a TaskSets list of args. > Example: > > ``` pycon > >>> ts = TaskSet(add, [([2, 2], {}, {'countdown': 1}), > ... ([4, 4], {}, {'countdown': 2}), > ... ([8, 8], {}, {'countdown': 3})]) > >>> ts.run() > ``` - Got a 3x performance gain by setting the prefetch count to four times the concurrency, (from an average task round-trip of 0.1s to 0.03s!). > A new setting has been added: > `CELERYD_PREFETCH_MULTIPLIER`{.interpreted-text role="setting"}, > which is set to [4]{.title-ref} by default. - Improved support for webhook tasks. > [celery.task.rest]{.title-ref} is now deprecated, replaced with > the new and shiny [celery.task.http]{.title-ref}. With more > reflective names, sensible interface, and it\'s possible to > override the methods used to perform HTTP requests. - The results of task sets are now cached by storing it in the result backend. ### Changes {#v100-changes} - Now depends on `carrot`{.interpreted-text role="pypi"} \>= 0.8.1 - New dependencies: `billiard`{.interpreted-text role="pypi"}, `python-dateutil`{.interpreted-text role="pypi"}, `django-picklefield`{.interpreted-text role="pypi"}. - No longer depends on python-daemon - The [uuid]{.title-ref} distribution is added as a dependency when running Python 2.4. - Now remembers the previously detected loader by keeping it in the `CELERY_LOADER`{.interpreted-text role="envvar"} environment variable. > This may help on windows where fork emulation is used. - ETA no longer sends datetime objects, but uses ISO 8601 date format in a string for better compatibility with other platforms. - No longer sends error mails for retried tasks. - Task can now override the backend used to store results. - Refactored the ExecuteWrapper, [apply]{.title-ref} and `CELERY_ALWAYS_EAGER`{.interpreted-text role="setting"} now also executes the task callbacks and signals. - Now using a proper scheduler for the tasks with an ETA. > This means waiting ETA tasks are sorted by time, so we don\'t have > to poll the whole list all the time. - Now also imports modules listed in `CELERY_IMPORTS`{.interpreted-text role="setting"} when running with django (as documented). - Log level for stdout/stderr changed from INFO to ERROR - ImportErrors are now properly propagated when auto-discovering tasks. - You can now use [celery.messaging.establish_connection]{.title-ref} to establish a connection to the broker. - When running as a separate service the periodic task scheduler does some smart moves to not poll too regularly. > If you need faster poll times you can lower the value of > `CELERYBEAT_MAX_LOOP_INTERVAL`{.interpreted-text role="setting"}. - You can now change periodic task intervals at runtime, by making [run_every]{.title-ref} a property, or subclassing [PeriodicTask.is_due]{.title-ref}. - The worker now supports control commands enabled through the use of a broadcast queue, you can remotely revoke tasks or set the rate limit for a task type. See `celery.task.control`{.interpreted-text role="mod"}. - The services now sets informative process names (as shown in [ps]{.title-ref} listings) if the `setproctitle`{.interpreted-text role="pypi"} module is installed. - `~@NotRegistered`{.interpreted-text role="exc"} now inherits from `KeyError`{.interpreted-text role="exc"}, and [TaskRegistry.\_\_getitem\_\_]{.title-ref}+\`pop\` raises [NotRegistered]{.title-ref} instead - You can set the loader via the `CELERY_LOADER`{.interpreted-text role="envvar"} environment variable. - You can now set `CELERY_IGNORE_RESULT`{.interpreted-text role="setting"} to ignore task results by default (if enabled, tasks doesn\'t save results or errors to the backend used). - The worker now correctly handles malformed messages by throwing away and acknowledging the message, instead of crashing. ### Bugs {#v100-bugs} - Fixed a race condition that could happen while storing task results in the database. ### Documentation {#v100-documentation} - Reference now split into two sections; API reference and internal module reference. ## 0.8.4 {#version-0.8.4} release-date : 2010-02-05 01:52 p.m. CEST release-by : Ask Solem - Now emits a warning if the \--detach argument is used. \--detach shouldn\'t be used anymore, as it has several not easily fixed bugs related to it. Instead, use something like start-stop-daemon, `supervisor`{.interpreted-text role="pypi"} or `launchd`{.interpreted-text role="command"} (macOS). - Make sure logger class is process aware, even if running Python \>= 2.6. - Error emails are not sent anymore when the task is retried. ## 0.8.3 {#version-0.8.3} release-date : 2009-12-22 09:43 a.m. CEST release-by : Ask Solem - Fixed a possible race condition that could happen when storing/querying task results using the database backend. - Now has console script entry points in the `setup.py`{.interpreted-text role="file"} file, so tools like `zc.buildout`{.interpreted-text role="pypi"} will correctly install the programs `celeryd` and `celeryinit`. ## 0.8.2 {#version-0.8.2} release-date : 2009-11-20 03:40 p.m. CEST release-by : Ask Solem - QOS Prefetch count wasn\'t applied properly, as it was set for every message received (which apparently behaves like, \"receive one more\"), instead of only set when our wanted value changed. ## 0.8.1 {#version-0.8.1} release-date : 2009-11-16 05:21 p.m. CEST release-by : Ask Solem ### Very important note {#v081-very-important} This release (with carrot 0.8.0) enables AMQP QoS (quality of service), which means the workers will only receive as many messages as it can handle at a time. As with any release, you should test this version upgrade on your development servers before rolling it out to production! ### Important changes {#v081-important} - If you\'re using Python \< 2.6 and you use the multiprocessing backport, then multiprocessing version 2.6.2.1 is required. - All [AMQP]()\* settings has been renamed to [BROKER]()\*, and in addition AMQP_SERVER has been renamed to BROKER_HOST, so before where you had: AMQP_SERVER = 'localhost' AMQP_PORT = 5678 AMQP_USER = 'myuser' AMQP_PASSWORD = 'mypassword' AMQP_VHOST = 'celery' You need to change that to: BROKER_HOST = 'localhost' BROKER_PORT = 5678 BROKER_USER = 'myuser' BROKER_PASSWORD = 'mypassword' BROKER_VHOST = 'celery' - Custom carrot backends now need to include the backend class name, so before where you had: CARROT_BACKEND = 'mycustom.backend.module' you need to change it to: CARROT_BACKEND = 'mycustom.backend.module.Backend' where [Backend]{.title-ref} is the class name. This is probably [\"Backend\"]{.title-ref}, as that was the previously implied name. - New version requirement for carrot: 0.8.0 ### Changes {#v081-changes} - Incorporated the multiprocessing backport patch that fixes the [processName]{.title-ref} error. - Ignore the result of PeriodicTask\'s by default. - Added a Redis result store backend - Allow `/etc/default/celeryd`{.interpreted-text role="file"} to define additional options for the `celeryd` init-script. - MongoDB periodic tasks issue when using different time than UTC fixed. - Windows specific: Negate test for available `os.fork` (thanks `miracle2k`{.interpreted-text role="github_user"}). - Now tried to handle broken PID files. - Added a Django test runner to contrib that sets [CELERY_ALWAYS_EAGER = True]{.title-ref} for testing with the database backend. - Added a `CELERY_CACHE_BACKEND`{.interpreted-text role="setting"} setting for using something other than the Django-global cache backend. - Use custom implementation of `functools.partial` for Python 2.4 support (Probably still problems with running on 2.4, but it will eventually be supported) - Prepare exception to pickle when saving `RETRY`{.interpreted-text role="state"} status for all backends. - SQLite no concurrency limit should only be effective if the database backend is used. ## 0.8.0 {#version-0.8.0} release-date : 2009-09-22 03:06 p.m. CEST release-by : Ask Solem ### Backward incompatible changes {#v080-incompatible} - Add traceback to result value on failure. > ::: note > ::: title > Note > ::: > > If you use the database backend you have to re-create the database > table [celery_taskmeta]{.title-ref}. > > Contact the `mailing-list`{.interpreted-text role="ref"} or > `irc-channel`{.interpreted-text role="ref"} channel for help doing > this. > ::: - Database tables are now only created if the database backend is used, so if you change back to the database backend at some point, be sure to initialize tables (django: [syncdb]{.title-ref}, python: [celeryinit]{.title-ref}). ::: note ::: title Note ::: This is only applies if using Django version 1.1 or higher. ::: - Now depends on [carrot]{.title-ref} version 0.6.0. - Now depends on python-daemon 1.4.8 ### Important changes {#v080-important} - Celery can now be used in pure Python (outside of a Django project). > This means Celery is no longer Django specific. > > For more information see the FAQ entry > `faq-is-celery-for-django-only`{.interpreted-text role="ref"}. - Celery now supports task retries. > See `task-retry`{.interpreted-text role="ref"} for more > information. - We now have an AMQP result store backend. > It uses messages to publish task return value and status. And > it\'s incredibly fast! > > See issue #6 for more info! - AMQP QoS (prefetch count) implemented: > This to not receive more messages than we can handle. - Now redirects stdout/stderr to the workers log file when detached - Now uses [inspect.getargspec]{.title-ref} to only pass default arguments : the task supports. - Add Task.on_success, .on_retry, .on_failure handlers : See `celery.task.base.Task.on_success`{.interpreted-text role="meth"}, : `celery.task.base.Task.on_retry`{.interpreted-text role="meth"}, `celery.task.base.Task.on_failure`{.interpreted-text role="meth"}, - \`celery.utils.gen_unique_id\`: Workaround for : - You can now customize what happens at worker start, at process init, etc., : by creating your own loaders (see `celery.loaders.default`{.interpreted-text role="mod"}, `celery.loaders.djangoapp`{.interpreted-text role="mod"}, `celery.loaders`{.interpreted-text role="mod"}). - Support for multiple AMQP exchanges and queues. > This feature misses documentation and tests, so anyone interested > is encouraged to improve this situation. - The worker now survives a restart of the AMQP server! Automatically re-establish AMQP broker connection if it\'s lost. New settings: > - > > AMQP_CONNECTION_RETRY > > : Set to [True]{.title-ref} to enable connection retries. > > - > > AMQP_CONNECTION_MAX_RETRIES. > > : Maximum number of restarts before we give up. Default: > [100]{.title-ref}. ### News {#v080-news} - Fix an incompatibility between python-daemon and multiprocessing, : which resulted in the [\[Errno 10\] No child processes]{.title-ref} problem when detaching. - Fixed a possible DjangoUnicodeDecodeError being raised when saving pickled : data to Django\`s Memcached cache backend. - Better Windows compatibility. - New version of the pickled field (taken from : ) - New signals introduced: [task_sent]{.title-ref}, [task_prerun]{.title-ref} and : [task_postrun]{.title-ref}, see `celery.signals`{.interpreted-text role="mod"} for more information. - [TaskSetResult.join]{.title-ref} caused [TypeError]{.title-ref} when [timeout=None]{.title-ref}. : Thanks Jerzy Kozera. Closes #31 - [views.apply]{.title-ref} should return [HttpResponse]{.title-ref} instance. : Thanks to Jerzy Kozera. Closes #32 - \`PeriodicTask\`: Save conversion of [run_every]{.title-ref} from [int]{.title-ref} : to [timedelta]{.title-ref} to the class attribute instead of on the instance. - Exceptions has been moved to [celery.exceptions]{.title-ref}, but are still : available in the previous module. - Try to rollback transaction and retry saving result if an error happens : while setting task status with the database backend. - jail() refactored into `celery.execute.ExecuteWrapper`{.interpreted-text role="class"}. - [views.apply]{.title-ref} now correctly sets mime-type to \"application/json\" - [views.task_status]{.title-ref} now returns exception if state is `RETRY`{.interpreted-text role="state"} - [views.task_status]{.title-ref} now returns traceback if state is `FAILURE`{.interpreted-text role="state"} : or `RETRY`{.interpreted-text role="state"} - Documented default task arguments. - Add a sensible \_\_repr\_\_ to ExceptionInfo for easier debugging - Fix documentation typo [.. import map]{.title-ref} -\> [.. import dmap]{.title-ref}. : Thanks to `mikedizon`{.interpreted-text role="github_user"}. ## 0.6.0 {#version-0.6.0} release-date : 2009-08-07 06:54 a.m. CET release-by : Ask Solem ### Important changes {#v060-important} - Fixed a bug where tasks raising unpickleable exceptions crashed pool : workers. So if you\'ve had pool workers mysteriously disappearing, or problems with the worker stopping working, this has been fixed in this version. - Fixed a race condition with periodic tasks. - The task pool is now supervised, so if a pool worker crashes, : goes away or stops responding, it is automatically replaced with a new one. - Task.name is now automatically generated out of class module+name, for example [\"djangotwitter.tasks.UpdateStatusesTask\"]{.title-ref}. Very convenient. No idea why we didn\'t do this before. Some documentation is updated to not manually specify a task name. ### News {#v060-news} - Tested with Django 1.1 - New Tutorial: Creating a click counter using Carrot and Celery - Database entries for periodic tasks are now created at the workers : start-up instead of for each check (which has been a forgotten TODO/XXX in the code for a long time) - New settings variable: `CELERY_TASK_RESULT_EXPIRES`{.interpreted-text role="setting"} : Time (in seconds, or a [datetime.timedelta]{.title-ref} object) for when after stored task results are deleted. For the moment this only works for the database backend. - The worker now emits a debug log message for which periodic tasks : has been launched. - The periodic task table is now locked for reading while getting : periodic task status (MySQL only so far, seeking patches for other engines) - A lot more debugging information is now available by turning on the : [DEBUG]{.title-ref} log level ([\--loglevel=DEBUG]{.title-ref}). - Functions/methods with a timeout argument now works correctly. - New: \`celery.strategy.even_time_distribution\`: : With an iterator yielding task args, kwargs tuples, evenly distribute the processing of its tasks throughout the time window available. - Log message [Unknown task ignored\...]{.title-ref} now has log level [ERROR]{.title-ref} - Log message when task is received is now emitted for all tasks, even if : the task has an ETA (estimated time of arrival). Also the log message now includes the ETA for the task (if any). - Acknowledgment now happens in the pool callback. Can\'t do ack in the job : target, as it\'s not pickleable (can\'t share AMQP connection, etc.). - Added note about .delay hanging in README - Tests now passing in Django 1.1 - Fixed discovery to make sure app is in INSTALLED_APPS - Previously overridden pool behavior (process reap, wait until pool worker : available, etc.) is now handled by [multiprocessing.Pool]{.title-ref} itself. - Convert statistics data to Unicode for use as kwargs. Thanks Lucy! ## 0.4.1 {#version-0.4.1} release-date : 2009-07-02 01:42 p.m. CET release-by : Ask Solem - Fixed a bug with parsing the message options ([mandatory]{.title-ref}, [routing_key]{.title-ref}, [priority]{.title-ref}, [immediate]{.title-ref}) ## 0.4.0 {#version-0.4.0} release-date : 2009-07-01 07:29 p.m. CET release-by : Ask Solem - Adds eager execution. [celery.execute.apply]{.title-ref}\|[Task.apply]{.title-ref} executes the function blocking until the task is done, for API compatibility it returns a [celery.result.EagerResult]{.title-ref} instance. You can configure Celery to always run tasks locally by setting the `CELERY_ALWAYS_EAGER`{.interpreted-text role="setting"} setting to [True]{.title-ref}. - Now depends on [anyjson]{.title-ref}. - 99% coverage using Python [coverage]{.title-ref} 3.0. ## 0.3.20 {#version-0.3.20} release-date : 2009-06-25 08:42 p.m. CET release-by : Ask Solem - New arguments to [apply_async]{.title-ref} (the advanced version of [delay_task]{.title-ref}), [countdown]{.title-ref} and [eta]{.title-ref}; > \>\>\> \# Run 10 seconds into the future. \>\>\> res = > apply_async(MyTask, countdown=10); > > \>\>\> \# Run 1 day from now \>\>\> res = apply_async(MyTask, \... > eta=datetime.now() + timedelta(days=1)) - Now unlinks stale PID files - Lots of more tests. - Now compatible with carrot \>= 0.5.0. - **IMPORTANT** The [subtask_ids]{.title-ref} attribute on the [TaskSetResult]{.title-ref} instance has been removed. To get this information instead use: > \>\>\> subtask_ids = \[subtask.id for subtask in ts_res.subtasks\] - [Taskset.run()]{.title-ref} now respects extra message options from the task class. - Task: Add attribute \`ignore_result\`: Don\'t store the status and return value. This means you can\'t use the [celery.result.AsyncResult]{.title-ref} to check if the task is done, or get its return value. Only use if you need the performance and is able live without these features. Any exceptions raised will store the return value/status as usual. - Task: Add attribute [disable_error_emails]{.title-ref} to disable sending error emails for that task. - Should now work on Windows (although running in the background won\'t work, so using the [\--detach]{.title-ref} argument results in an exception being raised). - Added support for statistics for profiling and monitoring. To start sending statistics start the worker with the [\--statistics option. Then after a while you can dump the results by running \`python manage.py celerystats]{.title-ref}. See [celery.monitoring]{.title-ref} for more information. - The Celery daemon can now be supervised (i.e., it is automatically restarted if it crashes). To use this start the worker with the \--supervised\` option (or alternatively [-S]{.title-ref}). - views.apply: View calling a task. > Example: > > ``` text > http://e.com/celery/apply/task_name/arg1/arg2//?kwarg1=a&kwarg2=b > ``` > > ::: warning > ::: title > Warning > ::: > > Use with caution! Don\'t expose this URL to the public without > first ensuring that your code is safe! > ::: - Refactored [celery.task]{.title-ref}. It\'s now split into three modules: > - `celery.task` > > > Contains [apply_async]{.title-ref}, > > [delay_task]{.title-ref}, [discard_all]{.title-ref}, and > > task shortcuts, plus imports objects from > > [celery.task.base]{.title-ref} and > > [celery.task.builtins]{.title-ref} > > - `celery.task.base` > > > Contains task base classes: [Task]{.title-ref}, > > [PeriodicTask]{.title-ref}, [TaskSet]{.title-ref}, > > [AsynchronousMapTask]{.title-ref}, > > [ExecuteRemoteTask]{.title-ref}. > > - `celery.task.builtins` > > > Built-in tasks: [PingTask]{.title-ref}, > > [DeleteExpiredTaskMetaTask]{.title-ref}. ## 0.3.7 {#version-0.3.7} release-date : 2008-06-16 11:41 p.m. CET release-by : Ask Solem - **IMPORTANT** Now uses AMQP\`s [basic.consume]{.title-ref} instead of [basic.get]{.title-ref}. This means we\'re no longer polling the broker for new messages. - **IMPORTANT** Default concurrency limit is now set to the number of CPUs available on the system. - **IMPORTANT** \`tasks.register\`: Renamed [task_name]{.title-ref} argument to [name]{.title-ref}, so: >>> tasks.register(func, task_name='mytask') has to be replaced with: >>> tasks.register(func, name='mytask') - The daemon now correctly runs if the pidfile is stale. - Now compatible with carrot 0.4.5 - Default AMQP connection timeout is now 4 seconds. - [AsyncResult.read()]{.title-ref} was always returning [True]{.title-ref}. - Only use README as long_description if the file exists so easy_install doesn\'t break. - \`celery.view\`: JSON responses now properly set its mime-type. - [apply_async]{.title-ref} now has a [connection]{.title-ref} keyword argument so you can re-use the same AMQP connection if you want to execute more than one task. - Handle failures in task_status view such that it won\'t throw 500s. - Fixed typo [AMQP_SERVER]{.title-ref} in documentation to [AMQP_HOST]{.title-ref}. - Worker exception emails sent to administrators now works properly. - No longer depends on [django]{.title-ref}, so installing [celery]{.title-ref} won\'t affect the preferred Django version installed. - Now works with PostgreSQL (`psycopg2`{.interpreted-text role="pypi"}) again by registering the [PickledObject]{.title-ref} field. - Worker: Added [\--detach]{.title-ref} option as an alias to [\--daemon]{.title-ref}, and it\'s the term used in the documentation from now on. - Make sure the pool and periodic task worker thread is terminated properly at exit (so `Control-c`{.interpreted-text role="kbd"} works again). - Now depends on [python-daemon]{.title-ref}. - Removed dependency to [simplejson]{.title-ref} - Cache Backend: Re-establishes connection for every task process if the Django cache backend is `python-memcached`{.interpreted-text role="pypi"}/`libmemcached`{.interpreted-text role="pypi"}. - Tyrant Backend: Now re-establishes the connection for every task executed. ## 0.3.3 {#version-0.3.3} release-date : 2009-06-08 01:07 p.m. CET release-by : Ask Solem - The [PeriodicWorkController]{.title-ref} now sleeps for 1 second between checking for periodic tasks to execute. ## 0.3.2 {#version-0.3.2} release-date : 2009-06-08 01:07 p.m. CET release-by : Ask Solem - worker: Added option \`\--discard\`: Discard (delete!) all waiting messages in the queue. - Worker: The [\--wakeup-after]{.title-ref} option wasn\'t handled as a float. ## 0.3.1 {#version-0.3.1} release-date : 2009-06-08 01:07 p.m. CET release-by : Ask Solem - The [PeriodicTask]{.title-ref} worker is now running in its own thread instead of blocking the [TaskController]{.title-ref} loop. - Default [QUEUE_WAKEUP_AFTER]{.title-ref} has been lowered to [0.1]{.title-ref} (was [0.3]{.title-ref}) ## 0.3.0 {#version-0.3.0} release-date : 2009-06-08 12:41 p.m. CET release-by : Ask Solem ::: warning ::: title Warning ::: This is a development version, for the stable release, please see versions 0.2.x. ::: **VERY IMPORTANT:** Pickle is now the encoder used for serializing task arguments, so be sure to flush your task queue before you upgrade. - **IMPORTANT** TaskSet.run() now returns a `celery.result.TaskSetResult` instance, which lets you inspect the status and return values of a taskset as it was a single entity. - **IMPORTANT** Celery now depends on carrot \>= 0.4.1. - The Celery daemon now sends task errors to the registered admin emails. To turn off this feature, set [SEND_CELERY_TASK_ERROR_EMAILS]{.title-ref} to [False]{.title-ref} in your [settings.py]{.title-ref}. Thanks to Grégoire Cachet. - You can now run the Celery daemon by using \`manage.py\`: ``` console $ python manage.py celeryd ``` Thanks to Grégoire Cachet. - Added support for message priorities, topic exchanges, custom routing keys for tasks. This means we\'ve introduced [celery.task.apply_async]{.title-ref}, a new way of executing tasks. You can use [celery.task.delay]{.title-ref} and [celery.Task.delay]{.title-ref} like usual, but if you want greater control over the message sent, you want [celery.task.apply_async]{.title-ref} and [celery.Task.apply_async]{.title-ref}. This also means the AMQP configuration has changed. Some settings has been renamed, while others are new: > - `CELERY_AMQP_EXCHANGE` > - `CELERY_AMQP_PUBLISHER_ROUTING_KEY` > - `CELERY_AMQP_CONSUMER_ROUTING_KEY` > - `CELERY_AMQP_CONSUMER_QUEUE` > - `CELERY_AMQP_EXCHANGE_TYPE` See the entry `faq-task-routing`{.interpreted-text role="ref"} in the `FAQ `{.interpreted-text role="ref"} for more information. - Task errors are now logged using log level [ERROR]{.title-ref} instead of [INFO]{.title-ref}, and stack-traces are dumped. Thanks to Grégoire Cachet. - Make every new worker process re-establish it\'s Django DB connection, this solving the \"MySQL connection died?\" exceptions. Thanks to Vitaly Babiy and Jirka Vejrazka. - **IMPORTANT** Now using pickle to encode task arguments. This means you now can pass complex Python objects to tasks as arguments. - Removed dependency to [yadayada]{.title-ref}. - Added a FAQ, see [docs/faq.rst]{.title-ref}. - Now converts any Unicode keys in task [kwargs]{.title-ref} to regular strings. Thanks Vitaly Babiy. - Renamed the [TaskDaemon]{.title-ref} to [WorkController]{.title-ref}. - [celery.datastructures.TaskProcessQueue]{.title-ref} is now renamed to [celery.pool.TaskPool]{.title-ref}. - The pool algorithm has been refactored for greater performance and stability. ## 0.2.0 {#version-0.2.0} release-date : 2009-05-20 05:14 p.m. CET release-by : Ask Solem - Final release of 0.2.0 - Compatible with carrot version 0.4.0. - Fixes some syntax errors related to fetching results from the database backend. ## 0.2.0-pre3 {#version-0.2.0-pre3} release-date : 2009-05-20 05:14 p.m. CET release-by : Ask Solem - *Internal release*. Improved handling of unpickleable exceptions, [get_result]{.title-ref} now tries to recreate something looking like the original exception. ## 0.2.0-pre2 {#version-0.2.0-pre2} release-date : 2009-05-20 01:56 p.m. CET release-by : Ask Solem - Now handles unpickleable exceptions (like the dynamically generated subclasses of [django.core.exception.MultipleObjectsReturned]{.title-ref}). ## 0.2.0-pre1 {#version-0.2.0-pre1} release-date : 2009-05-20 12:33 p.m. CET release-by : Ask Solem - It\'s getting quite stable, with a lot of new features, so bump version to 0.2. This is a pre-release. - [celery.task.mark_as_read()]{.title-ref} and [celery.task.mark_as_failure()]{.title-ref} has been removed. Use [celery.backends.default_backend.mark_as_read()]{.title-ref}, and [celery.backends.default_backend.mark_as_failure()]{.title-ref} instead. ## 0.1.15 {#version-0.1.15} release-date : 2009-05-19 04:13 p.m. CET release-by : Ask Solem - The Celery daemon was leaking AMQP connections, this should be fixed, if you have any problems with too many files open (like [emfile]{.title-ref} errors in [rabbit.log]{.title-ref}, please contact us! ## 0.1.14 {#version-0.1.14} release-date : 2009-05-19 01:08 p.m. CET release-by : Ask Solem - Fixed a syntax error in the [TaskSet]{.title-ref} class (no such variable [TimeOutError]{.title-ref}). ## 0.1.13 {#version-0.1.13} release-date : 2009-05-19 12:36 p.m. CET release-by : Ask Solem - Forgot to add [yadayada]{.title-ref} to install requirements. - Now deletes all expired task results, not just those marked as done. - Able to load the Tokyo Tyrant backend class without django configuration, can specify tyrant settings directly in the class constructor. - Improved API documentation - Now using the Sphinx documentation system, you can build the html documentation by doing: > ``` console > $ cd docs > $ make html > ``` and the result will be in [docs/\_build/html]{.title-ref}. ## 0.1.12 {#version-0.1.12} release-date : 2009-05-18 04:38 p.m. CET release-by : Ask Solem - [delay_task()]{.title-ref} etc. now returns [celery.task.AsyncResult]{.title-ref} object, which lets you check the result and any failure that might\'ve happened. It kind of works like the [multiprocessing.AsyncResult]{.title-ref} class returned by [multiprocessing.Pool.map_async]{.title-ref}. - Added `dmap()` and `dmap_async()`. This works like the [multiprocessing.Pool]{.title-ref} versions except they\'re tasks distributed to the Celery server. Example: > ``` pycon > >>> from celery.task import dmap > >>> import operator > >>> dmap(operator.add, [[2, 2], [4, 4], [8, 8]]) > >>> [4, 8, 16] > > >>> from celery.task import dmap_async > >>> import operator > >>> result = dmap_async(operator.add, [[2, 2], [4, 4], [8, 8]]) > >>> result.ready() > False > >>> time.sleep(1) > >>> result.ready() > True > >>> result.result > [4, 8, 16] > ``` - Refactored the task meta-data cache and database backends, and added a new backend for Tokyo Tyrant. You can set the backend in your django settings file. > Example: > > ``` python > CELERY_RESULT_BACKEND = 'database'; # Uses the database > CELERY_RESULT_BACKEND = 'cache'; # Uses the django cache framework > CELERY_RESULT_BACKEND = 'tyrant'; # Uses Tokyo Tyrant > TT_HOST = 'localhost'; # Hostname for the Tokyo Tyrant server. > TT_PORT = 6657; # Port of the Tokyo Tyrant server. > ``` ## 0.1.11 {#version-0.1.11} release-date : 2009-05-12 02:08 p.m. CET release-by : Ask Solem - The logging system was leaking file descriptors, resulting in servers stopping with the EMFILES (too many open files) error (fixed). ## 0.1.10 {#version-0.1.10} release-date : 2009-05-11 12:46 p.m. CET release-by : Ask Solem - Tasks now supports both positional arguments and keyword arguments. - Requires carrot 0.3.8. - The daemon now tries to reconnect if the connection is lost. ## 0.1.8 {#version-0.1.8} release-date : 2009-05-07 12:27 p.m. CET release-by : Ask Solem - Better test coverage - More documentation - The worker doesn\'t emit [Queue is empty]{.title-ref} message if [settings.CELERYD_EMPTY_MSG_EMIT_EVERY]{.title-ref} is 0. ## 0.1.7 {#version-0.1.7} release-date : 2009-04-30 01:50 p.m. CET release-by : Ask Solem - Added some unit tests - Can now use the database for task meta-data (like if the task has been executed or not). Set [settings.CELERY_TASK_META]{.title-ref} - Can now run [python setup.py test]{.title-ref} to run the unit tests from within the [tests]{.title-ref} project. - Can set the AMQP exchange/routing key/queue using [settings.CELERY_AMQP_EXCHANGE]{.title-ref}, [settings.CELERY_AMQP_ROUTING_KEY]{.title-ref}, and [settings.CELERY_AMQP_CONSUMER_QUEUE]{.title-ref}. ## 0.1.6 {#version-0.1.6} release-date : 2009-04-28 02:13 p.m. CET release-by : Ask Solem - Introducing [TaskSet]{.title-ref}. A set of subtasks is executed and you can find out how many, or if all them, are done (excellent for progress bars and such) - Now catches all exceptions when running [Task.\_\_call\_\_]{.title-ref}, so the daemon doesn\'t die. This doesn\'t happen for pure functions yet, only [Task]{.title-ref} classes. - [autodiscover()]{.title-ref} now works with zipped eggs. - Worker: Now adds current working directory to [sys.path]{.title-ref} for convenience. - The [run_every]{.title-ref} attribute of [PeriodicTask]{.title-ref} classes can now be a [datetime.timedelta()]{.title-ref} object. - Worker: You can now set the [DJANGO_PROJECT_DIR]{.title-ref} variable for the worker and it will add that to [sys.path]{.title-ref} for easy launching. - Can now check if a task has been executed or not via HTTP. - You can do this by including the Celery [urls.py]{.title-ref} into your project, > \>\>\> url(r\'\^celery/\$\', include(\'celery.urls\')) then visiting the following URL: ``` text http://mysite/celery/$task_id/done/ ``` this will return a JSON dictionary, for example: ``` json {"task": {"id": "TASK_ID", "executed": true}} ``` - [delay_task]{.title-ref} now returns string id, not [uuid.UUID]{.title-ref} instance. - Now has [PeriodicTasks]{.title-ref}, to have [cron]{.title-ref} like functionality. - Project changed name from [crunchy]{.title-ref} to [celery]{.title-ref}. The details of the name change request is in [docs/name_change_request.txt]{.title-ref}. ## 0.1.0 {#version-0.1.0} release-date : 2009-04-24 11:28 a.m. CET release-by : Ask Solem - Initial release Sphinx started sucking by removing images from \_static, so we need to add them here into actual content to ensure they are included :-( ![image](../images/celery-banner.png) ![image](../images/celery-banner-small.png) --- # Change history for Celery 2.0 {#changelog-2.0} ::: {.contents local=""} ::: ## 2.0.3 {#version-2.0.3} release-date : 2010-08-27 12:00 p.m. CEST release-by : Ask Solem ### Fixes {#v203-fixes} - Worker: Properly handle connection errors happening while closing consumers. - Worker: Events are now buffered if the connection is down, then sent when the connection is re-established. - No longer depends on the `mailer`{.interpreted-text role="pypi"} package. > This package had a name space collision with > [django-mailer]{.title-ref}, so its functionality was replaced. - Redis result backend: Documentation typos: Redis doesn\'t have database names, but database numbers. The default database is now 0. - `~celery.task.control.inspect`{.interpreted-text role="class"}: [registered_tasks]{.title-ref} was requesting an invalid command because of a typo. > See issue #170. - `CELERY_ROUTES`{.interpreted-text role="setting"}: Values defined in the route should now have precedence over values defined in `CELERY_QUEUES`{.interpreted-text role="setting"} when merging the two. > With the follow settings: > > ``` python > CELERY_QUEUES = {'cpubound': {'exchange': 'cpubound', > 'routing_key': 'cpubound'}} > > CELERY_ROUTES = {'tasks.add': {'queue': 'cpubound', > 'routing_key': 'tasks.add', > 'serializer': 'json'}} > ``` > > The final routing options for [tasks.add]{.title-ref} will become: > > ``` python > {'exchange': 'cpubound', > 'routing_key': 'tasks.add', > 'serializer': 'json'} > ``` > > This wasn\'t the case before: the values in > `CELERY_QUEUES`{.interpreted-text role="setting"} would take > precedence. - Worker crashed if the value of `CELERY_TASK_ERROR_WHITELIST`{.interpreted-text role="setting"} was not an iterable - `~celery.execute.apply`{.interpreted-text role="func"}: Make sure [kwargs\[\'task_id\'\]]{.title-ref} is always set. - \`AsyncResult.traceback\`: Now returns `None`{.interpreted-text role="const"}, instead of raising `KeyError`{.interpreted-text role="exc"} if traceback is missing. - `~celery.task.control.inspect`{.interpreted-text role="class"}: Replies didn\'t work correctly if no destination was specified. - Can now store result/meta-data for custom states. - Worker: A warning is now emitted if the sending of task error emails fails. - `celeryev`: Curses monitor no longer crashes if the terminal window is resized. > See issue #160. - Worker: On macOS it isn\'t possible to run [os.exec\*]{.title-ref} in a process that\'s threaded. > This breaks the SIGHUP restart handler, and is now disabled on > macOS, emitting a warning instead. > > See issue #152. - `celery.execute.trace`{.interpreted-text role="mod"}: Properly handle [raise(str)]{.title-ref}, which is still allowed in Python 2.4. > See issue #175. - Using urllib2 in a periodic task on macOS crashed because of the proxy auto detection used in macOS. > This is now fixed by using a workaround. See issue #143. - Debian init-scripts: Commands shouldn\'t run in a sub shell > See issue #163. - Debian init-scripts: Use the absolute path of `celeryd` program to allow stat > See issue #162. ### Documentation {#v203-documentation} - getting-started/broker-installation: Fixed typo > [set_permissions \"\"]{.title-ref} -\> [set_permissions > \".\*\"]{.title-ref}. - Tasks User Guide: Added section on database transactions. > See issue #169. - Routing User Guide: Fixed typo [\"feed\": -\> {\"queue\": \"feeds\"}]{.title-ref}. > See issue #169. - Documented the default values for the `CELERYD_CONCURRENCY`{.interpreted-text role="setting"} and `CELERYD_PREFETCH_MULTIPLIER`{.interpreted-text role="setting"} settings. - Tasks User Guide: Fixed typos in the subtask example - celery.signals: Documented worker_process_init. - Daemonization cookbook: Need to export DJANGO_SETTINGS_MODULE in [/etc/default/celeryd]{.title-ref}. - Added some more FAQs from stack overflow - Daemonization cookbook: Fixed typo [CELERYD_LOGFILE/CELERYD_PIDFILE]{.title-ref} > to [CELERYD_LOG_FILE]{.title-ref} / [CELERYD_PID_FILE]{.title-ref} > > Also added troubleshooting section for the init-scripts. ## 2.0.2 {#version-2.0.2} release-date : 2010-07-22 11:31 a.m. CEST release-by : Ask Solem - Routes: When using the dict route syntax, the exchange for a task could disappear making the task unroutable. > See issue #158. - Test suite now passing on Python 2.4 - No longer have to type [PYTHONPATH=.]{.title-ref} to use `celeryconfig` in the current directory. > This is accomplished by the default loader ensuring that the > current directory is in [sys.path]{.title-ref} when loading the > config module. [sys.path]{.title-ref} is reset to its original > state after loading. > > Adding the current working directory to [sys.path]{.title-ref} > without the user knowing may be a security issue, as this means > someone can drop a Python module in the users directory that > executes arbitrary commands. This was the original reason not to > do this, but if done *only when loading the config module*, this > means that the behavior will only apply to the modules imported in > the config module, which I think is a good compromise (certainly > better than just explicitly setting [PYTHONPATH=.]{.title-ref} > anyway) - Experimental Cassandra backend added. - Worker: SIGHUP handler accidentally propagated to worker pool processes. > In combination with > `7a7c44e39344789f11b5346e9cc8340f5fe4846c`{.interpreted-text > role="sha"} this would make each child process start a new worker > instance when the terminal window was closed :/ - Worker: Don\'t install SIGHUP handler if running from a terminal. > This fixes the problem where the worker is launched in the > background when closing the terminal. - Worker: Now joins threads at shutdown. > See issue #152. - Test tear down: Don\'t use [atexit]{.title-ref} but nose\'s [teardown()]{.title-ref} functionality instead. > See issue #154. - Debian worker init-script: Stop now works correctly. - Task logger: [warn]{.title-ref} method added (synonym for [warning]{.title-ref}) - Can now define a white list of errors to send error emails for. > Example: > > ``` python > CELERY_TASK_ERROR_WHITELIST = ('myapp.MalformedInputError',) > ``` > > See issue #153. - Worker: Now handles overflow exceptions in [time.mktime]{.title-ref} while parsing the ETA field. - LoggerWrapper: Try to detect loggers logging back to stderr/stdout making an infinite loop. - Added `celery.task.control.inspect`{.interpreted-text role="class"}: Inspects a running worker. > Examples: > > ``` pycon > # Inspect a single worker > >>> i = inspect('myworker.example.com') > > # Inspect several workers > >>> i = inspect(['myworker.example.com', 'myworker2.example.com']) > > # Inspect all workers consuming on this vhost. > >>> i = inspect() > > ### Methods > > # Get currently executing tasks > >>> i.active() > > # Get currently reserved tasks > >>> i.reserved() > > # Get the current ETA schedule > >>> i.scheduled() > > # Worker statistics and info > >>> i.stats() > > # List of currently revoked tasks > >>> i.revoked() > > # List of registered tasks > >>> i.registered_tasks() > ``` - Remote control commands [dump_active]{.title-ref}/[dump_reserved]{.title-ref}/[dump_schedule]{.title-ref} now replies with detailed task requests. > Containing the original arguments and fields of the task > requested. > > In addition the remote control command [set_loglevel]{.title-ref} > has been added, this only changes the log level for the main > process. - Worker control command execution now catches errors and returns their string representation in the reply. - Functional test suite added > `celery.tests.functional.case`{.interpreted-text role="mod"} > contains utilities to start and stop an embedded worker process, > for use in functional testing. ## 2.0.1 {#version-2.0.1} release-date : 2010-07-09 03:02 p.m. CEST release-by : Ask Solem - multiprocessing.pool: Now handles encoding errors, so that pickling errors doesn\'t crash the worker processes. - The remote control command replies wasn\'t working with RabbitMQ 1.8.0\'s stricter equivalence checks. > If you\'ve already hit this problem you may have to delete the > declaration: > > ``` console > $ camqadm exchange.delete celerycrq > ``` > > or: > > ``` console > $ python manage.py camqadm exchange.delete celerycrq > ``` - A bug sneaked in the ETA scheduler that made it only able to execute one task per second(!) > The scheduler sleeps between iterations so it doesn\'t consume too > much CPU. It keeps a list of the scheduled items sorted by time, > at each iteration it sleeps for the remaining time of the item > with the nearest deadline. If there are no ETA tasks it will sleep > for a minimum amount of time, one second by default. > > A bug sneaked in here, making it sleep for one second for every > task that was scheduled. This has been fixed, so now it should > move tasks like hot knife through butter. > > In addition a new setting has been added to control the minimum > sleep interval; > `CELERYD_ETA_SCHEDULER_PRECISION`{.interpreted-text > role="setting"}. A good value for this would be a float between 0 > and 1, depending on the needed precision. A value of 0.8 means > that when the ETA of a task is met, it will take at most 0.8 > seconds for the task to be moved to the ready queue. - Pool: Supervisor didn\'t release the semaphore. > This would lead to a deadlock if all workers terminated > prematurely. - Added Python version trove classifiers: 2.4, 2.5, 2.6 and 2.7 - Tests now passing on Python 2.7. - Task.\_\_reduce\_\_: Tasks created using the task decorator can now be pickled. - `setup.py`{.interpreted-text role="file"}: `nose`{.interpreted-text role="pypi"} added to [tests_require]{.title-ref}. - Pickle should now work with SQLAlchemy 0.5.x - New homepage design by Jan Henrik Helmers: - New Sphinx theme by Armin Ronacher: - Fixed \"pending_xref\" errors shown in the HTML rendering of the documentation. Apparently this was caused by new changes in Sphinx 1.0b2. - Router classes in `CELERY_ROUTES`{.interpreted-text role="setting"} are now imported lazily. > Importing a router class in a module that also loads the Celery > environment would cause a circular dependency. This is solved by > importing it when needed after the environment is set up. - `CELERY_ROUTES`{.interpreted-text role="setting"} was broken if set to a single dict. > This example in the docs should now work again: > > ``` python > CELERY_ROUTES = {'feed.tasks.import_feed': 'feeds'} > ``` - [CREATE_MISSING_QUEUES]{.title-ref} wasn\'t honored by apply_async. - New remote control command: [stats]{.title-ref} > Dumps information about the worker, like pool process ids, and > total number of tasks executed by type. > > Example reply: > > ``` python > [{'worker.local': > 'total': {'tasks.sleeptask': 6}, > 'pool': {'timeouts': [None, None], > 'processes': [60376, 60377], > 'max-concurrency': 2, > 'max-tasks-per-child': None, > 'put-guarded-by-semaphore': True}}] > ``` - New remote control command: [dump_active]{.title-ref} > Gives a list of tasks currently being executed by the worker. By > default arguments are passed through repr in case there are > arguments that\'s not JSON encodable. If you know the arguments > are JSON safe, you can pass the argument [safe=True]{.title-ref}. > > Example reply: > > ``` pycon > >>> broadcast('dump_active', arguments={'safe': False}, reply=True) > [{'worker.local': [ > {'args': '(1,)', > 'time_start': 1278580542.6300001, > 'name': 'tasks.sleeptask', > 'delivery_info': { > 'consumer_tag': '30', > 'routing_key': 'celery', > 'exchange': 'celery'}, > 'hostname': 'casper.local', > 'acknowledged': True, > 'kwargs': '{}', > 'id': '802e93e9-e470-47ed-b913-06de8510aca2', > } > ]}] > ``` - Added experimental support for persistent revokes. > Use the [-S\|\--statedb]{.title-ref} argument to the worker to > enable it: > > ``` console > $ celeryd --statedb=/var/run/celeryd > ``` > > This will use the file: [/var/run/celeryd.db]{.title-ref}, as the > [shelve]{.title-ref} module automatically adds the > [.db]{.title-ref} suffix. ## 2.0.0 {#version-2.0.0} release-date : 2010-07-02 02:30 p.m. CEST release-by : Ask Solem ### Foreword Celery 2.0 contains backward incompatible changes, the most important being that the Django dependency has been removed so Celery no longer supports Django out of the box, but instead as an add-on package called `django-celery`{.interpreted-text role="pypi"}. We\'re very sorry for breaking backwards compatibility, but there\'s also many new and exciting features to make up for the time you lose upgrading, so be sure to read the `News `{.interpreted-text role="ref"} section. Quite a lot of potential users have been upset about the Django dependency, so maybe this is a chance to get wider adoption by the Python community as well. Big thanks to all contributors, testers and users! ### Upgrading for Django-users {#v200-django-upgrade} Django integration has been moved to a separate package: `django-celery`{.interpreted-text role="pypi"}. - To upgrade you need to install the `django-celery`{.interpreted-text role="pypi"} module and change: ``` python INSTALLED_APPS = 'celery' ``` to: ``` python INSTALLED_APPS = 'djcelery' ``` - If you use [mod_wsgi]{.title-ref} you need to add the following line to your [.wsgi]{.title-ref} file: > ``` python > import os > os.environ['CELERY_LOADER'] = 'django' > ``` - The following modules has been moved to `django-celery`{.interpreted-text role="pypi"}: > **Module name** **Replace with** > ---------------------------------------- ------------------------------------------ > [celery.models]{.title-ref} [djcelery.models]{.title-ref} > [celery.managers]{.title-ref} [djcelery.managers]{.title-ref} > [celery.views]{.title-ref} [djcelery.views]{.title-ref} > [celery.urls]{.title-ref} [djcelery.urls]{.title-ref} > [celery.management]{.title-ref} [djcelery.management]{.title-ref} > [celery.loaders.djangoapp]{.title-ref} [djcelery.loaders]{.title-ref} > [celery.backends.database]{.title-ref} [djcelery.backends.database]{.title-ref} > [celery.backends.cache]{.title-ref} [djcelery.backends.cache]{.title-ref} Importing `djcelery`{.interpreted-text role="mod"} will automatically setup Celery to use Django loader. loader. It does this by setting the `CELERY_LOADER`{.interpreted-text role="envvar"} environment variable to [\"django\"]{.title-ref} (it won\'t change it if a loader is already set). When the Django loader is used, the \"database\" and \"cache\" result backend aliases will point to the `djcelery`{.interpreted-text role="mod"} backends instead of the built-in backends, and configuration will be read from the Django settings. ### Upgrading for others {#v200-upgrade} #### Database result backend {#v200-upgrade-database} The database result backend is now using [SQLAlchemy](http://www.sqlalchemy.org) instead of the Django ORM, see [Supported Databases](http://www.sqlalchemy.org/docs/core/engines.html#supported-databases) for a table of supported databases. The [DATABASE\_\*]{.title-ref} settings has been replaced by a single setting: `CELERY_RESULT_DBURI`{.interpreted-text role="setting"}. The value here should be an [SQLAlchemy Connection String](http://www.sqlalchemy.org/docs/core/engines.html#database-urls), some examples include: ``` python # sqlite (filename) CELERY_RESULT_DBURI = 'sqlite:///celerydb.sqlite' # mysql CELERY_RESULT_DBURI = 'mysql://scott:tiger@localhost/foo' # postgresql CELERY_RESULT_DBURI = 'postgresql://scott:tiger@localhost/mydatabase' # oracle CELERY_RESULT_DBURI = 'oracle://scott:tiger@127.0.0.1:1521/sidname' ``` See [SQLAlchemy Connection Strings](http://www.sqlalchemy.org/docs/core/engines.html#database-urls) for more information about connection strings. To specify additional SQLAlchemy database engine options you can use the `CELERY_RESULT_ENGINE_OPTIONS`{.interpreted-text role="setting"} setting: > ``` python > # echo enables verbose logging from SQLAlchemy. > CELERY_RESULT_ENGINE_OPTIONS = {'echo': True} > ``` #### Cache result backend {#v200-upgrade-cache} The cache result backend is no longer using the Django cache framework, but it supports mostly the same configuration syntax: > ``` python > CELERY_CACHE_BACKEND = 'memcached://A.example.com:11211;B.example.com' > ``` To use the cache backend you must either have the `pylibmc`{.interpreted-text role="pypi"} or `python-memcached`{.interpreted-text role="pypi"} library installed, of which the former is regarded as the best choice. The support backend types are [memcached://]{.title-ref} and [memory://]{.title-ref}, we haven\'t felt the need to support any of the other backends provided by Django. ### Backward incompatible changes {#v200-incompatible} - Default (python) loader now prints warning on missing [celeryconfig.py]{.title-ref} instead of raising `ImportError`{.interpreted-text role="exc"}. > The worker raises `~@ImproperlyConfigured`{.interpreted-text > role="exc"} if the configuration isn\'t set up. This makes it > possible to use [\--help]{.title-ref} etc., without having a > working configuration. > > Also this makes it possible to use the client side of Celery > without being configured: > > ``` pycon > >>> from carrot.connection import BrokerConnection > >>> conn = BrokerConnection('localhost', 'guest', 'guest', '/') > >>> from celery.execute import send_task > >>> r = send_task('celery.ping', args=(), kwargs={}, connection=conn) > >>> from celery.backends.amqp import AMQPBackend > >>> r.backend = AMQPBackend(connection=conn) > >>> r.get() > 'pong' > ``` - The following deprecated settings has been removed (as scheduled by the `deprecation-timeline`{.interpreted-text role="ref"}): > **Setting name** **Replace with** > ------------------------------------------------- -------------------------------------------- > [CELERY_AMQP_CONSUMER_QUEUES]{.title-ref} [CELERY_QUEUES]{.title-ref} > [CELERY_AMQP_EXCHANGE]{.title-ref} [CELERY_DEFAULT_EXCHANGE]{.title-ref} > [CELERY_AMQP_EXCHANGE_TYPE]{.title-ref} [CELERY_DEFAULT_EXCHANGE_TYPE]{.title-ref} > [CELERY_AMQP_CONSUMER_ROUTING_KEY]{.title-ref} [CELERY_QUEUES]{.title-ref} > [CELERY_AMQP_PUBLISHER_ROUTING_KEY]{.title-ref} [CELERY_DEFAULT_ROUTING_KEY]{.title-ref} - The [celery.task.rest]{.title-ref} module has been removed, use [celery.task.http]{.title-ref} instead (as scheduled by the `deprecation-timeline`{.interpreted-text role="ref"}). - It\'s no longer allowed to skip the class name in loader names. (as scheduled by the `deprecation-timeline`{.interpreted-text role="ref"}): > Assuming the implicit [Loader]{.title-ref} class name is no longer > supported, for example, if you use: > > ``` python > CELERY_LOADER = 'myapp.loaders' > ``` > > You need to include the loader class name, like this: > > ``` python > CELERY_LOADER = 'myapp.loaders.Loader' > ``` - `CELERY_TASK_RESULT_EXPIRES`{.interpreted-text role="setting"} now defaults to 1 day. > Previous default setting was to expire in 5 days. - AMQP backend: Don\'t use different values for [auto_delete]{.title-ref}. > This bug became visible with RabbitMQ 1.8.0, which no longer > allows conflicting declarations for the auto_delete and durable > settings. > > If you\'ve already used Celery with this backend chances are you > have to delete the previous declaration: > > ``` console > $ camqadm exchange.delete celeryresults > ``` - Now uses pickle instead of cPickle on Python versions \<= 2.5 > cPickle is broken in Python \<= 2.5. > > It unsafely and incorrectly uses relative instead of absolute > imports, so for example: > > ``` python > exceptions.KeyError > ``` > > becomes: > > ``` python > celery.exceptions.KeyError > ``` > > Your best choice is to upgrade to Python 2.6, as while the pure > pickle version has worse performance, it is the only safe option > for older Python versions. ### News {#v200-news} - **celeryev**: Curses Celery Monitor and Event Viewer. > This is a simple monitor allowing you to see what tasks are > executing in real-time and investigate tracebacks and results of > ready tasks. It also enables you to set new rate limits and revoke > tasks. > > Screenshot: > > ![](../images/celeryevshotsm.jpg) > > If you run [celeryev]{.title-ref} with the [-d]{.title-ref} switch > it will act as an event dumper, simply dumping the events it > receives to standard out: > > ``` console > $ celeryev -d > -> celeryev: starting capture... > casper.local [2010-06-04 10:42:07.020000] heartbeat > casper.local [2010-06-04 10:42:14.750000] task received: > tasks.add(61a68756-27f4-4879-b816-3cf815672b0e) args=[2, 2] kwargs={} > eta=2010-06-04T10:42:16.669290, retries=0 > casper.local [2010-06-04 10:42:17.230000] task started > tasks.add(61a68756-27f4-4879-b816-3cf815672b0e) args=[2, 2] kwargs={} > casper.local [2010-06-04 10:42:17.960000] task succeeded: > tasks.add(61a68756-27f4-4879-b816-3cf815672b0e) > args=[2, 2] kwargs={} result=4, runtime=0.782663106918 > > The fields here are, in order: *sender hostname*, *timestamp*, *event type* and > *additional event fields*. > ``` - AMQP result backend: Now supports [.ready()]{.title-ref}, [.successful()]{.title-ref}, [.result]{.title-ref}, [.status]{.title-ref}, and even responds to changes in task state - New user guides: > - `guide-workers`{.interpreted-text role="ref"} > - `guide-canvas`{.interpreted-text role="ref"} > - `guide-routing`{.interpreted-text role="ref"} - Worker: Standard out/error is now being redirected to the log file. - `billiard`{.interpreted-text role="pypi"} has been moved back to the Celery repository. > **Module name** **celery equivalent** > ----------------------------------------- ------------------------------------------------- > [billiard.pool]{.title-ref} [celery.concurrency.processes.pool]{.title-ref} > [billiard.serialization]{.title-ref} [celery.serialization]{.title-ref} > [billiard.utils.functional]{.title-ref} [celery.utils.functional]{.title-ref} > > The `billiard`{.interpreted-text role="pypi"} distribution may be > maintained, depending on interest. - now depends on `carrot`{.interpreted-text role="pypi"} \>= 0.10.5 - now depends on `pyparsing`{.interpreted-text role="pypi"} - Worker: Added [\--purge]{.title-ref} as an alias to [\--discard]{.title-ref}. - Worker: `Control-c`{.interpreted-text role="kbd"} (SIGINT) once does warm shutdown, hitting `Control-c`{.interpreted-text role="kbd"} twice forces termination. - Added support for using complex Crontab-expressions in periodic tasks. For example, you can now use: > ``` pycon > >>> crontab(minute='*/15') > ``` > > or even: > > ``` pycon > >>> crontab(minute='*/30', hour='8-17,1-2', day_of_week='thu-fri') > ``` See `guide-beat`{.interpreted-text role="ref"}. - Worker: Now waits for available pool processes before applying new tasks to the pool. > This means it doesn\'t have to wait for dozens of tasks to finish > at shutdown because it has applied prefetched tasks without having > any pool processes available to immediately accept them. > > See issue #122. - New built-in way to do task callbacks using `~celery.subtask`{.interpreted-text role="class"}. See `guide-canvas`{.interpreted-text role="ref"} for more information. - TaskSets can now contain several types of tasks. `~celery.task.sets.TaskSet`{.interpreted-text role="class"} has been refactored to use a new syntax, please see `guide-canvas`{.interpreted-text role="ref"} for more information. The previous syntax is still supported, but will be deprecated in version 1.4. - TaskSet failed() result was incorrect. > See issue #132. - Now creates different loggers per task class. > See issue #129. - Missing queue definitions are now created automatically. > You can disable this using the > `CELERY_CREATE_MISSING_QUEUES`{.interpreted-text role="setting"} > setting. > > The missing queues are created with the following options: > > ``` python > CELERY_QUEUES[name] = {'exchange': name, > 'exchange_type': 'direct', > 'routing_key': 'name} > ``` > > This feature is added for easily setting up routing using the > [-Q]{.title-ref} option to the worker: > > ``` console > $ celeryd -Q video, image > ``` > > See the new routing section of the User Guide for more > information: `guide-routing`{.interpreted-text role="ref"}. - New Task option: [Task.queue]{.title-ref} > If set, message options will be taken from the corresponding entry > in `CELERY_QUEUES`{.interpreted-text role="setting"}. > [exchange]{.title-ref}, [exchange_type]{.title-ref} and > [routing_key]{.title-ref} will be ignored - Added support for task soft and hard time limits. > New settings added: > > - `CELERYD_TASK_TIME_LIMIT`{.interpreted-text role="setting"} > > > Hard time limit. The worker processing the task will be > > killed and replaced with a new one when this is exceeded. > > - `CELERYD_TASK_SOFT_TIME_LIMIT`{.interpreted-text > role="setting"} > > > Soft time limit. The > > `~@SoftTimeLimitExceeded`{.interpreted-text role="exc"} > > exception will be raised when this is exceeded. The task can > > catch this to, for example, clean up before the hard time > > limit comes. > > New command-line arguments to `celeryd` added: > [\--time-limit]{.title-ref} and [\--soft-time-limit]{.title-ref}. > > What\'s left? > > This won\'t work on platforms not supporting signals (and > specifically the [SIGUSR1]{.title-ref} signal) yet. So an > alternative the ability to disable the feature all together on > nonconforming platforms must be implemented. > > Also when the hard time limit is exceeded, the task result should > be a [TimeLimitExceeded]{.title-ref} exception. - Test suite is now passing without a running broker, using the carrot in-memory backend. - Log output is now available in colors. > **Log level** **Color** > ------------------------ ----------- > [DEBUG]{.title-ref} Blue > [WARNING]{.title-ref} Yellow > [CRITICAL]{.title-ref} Magenta > [ERROR]{.title-ref} Red > > This is only enabled when the log output is a tty. You can > explicitly enable/disable this feature using the > `CELERYD_LOG_COLOR`{.interpreted-text role="setting"} setting. - Added support for task router classes (like the django multi-db routers) > - New setting: `CELERY_ROUTES`{.interpreted-text role="setting"} > > This is a single, or a list of routers to traverse when sending > tasks. Dictionaries in this list converts to a > `celery.routes.MapRoute`{.interpreted-text role="class"} instance. > > Examples: > > > \>\>\> CELERY_ROUTES = {\'celery.ping\': \'default\', > > > > : \'mytasks.add\': \'cpu-bound\', \'video.encode\': { > > \'queue\': \'video\', \'exchange\': \'media\' > > \'routing_key\': \'media.video.encode\'}} > > > > \>\>\> CELERY_ROUTES = (\'myapp.tasks.Router\', > > > > : {\'celery.ping\': \'default\'}) > > Where [myapp.tasks.Router]{.title-ref} could be: > > ``` python > class Router(object): > > def route_for_task(self, task, args=None, kwargs=None): > if task == 'celery.ping': > return 'default' > ``` > > route_for_task may return a string or a dict. A string then means > it\'s a queue name in `CELERY_QUEUES`{.interpreted-text > role="setting"}, a dict means it\'s a custom route. > > When sending tasks, the routers are consulted in order. The first > router that doesn\'t return [None]{.title-ref} is the route to > use. The message options is then merged with the found route > settings, where the routers settings have priority. > > Example if `~celery.execute.apply_async`{.interpreted-text > role="func"} has these arguments: > > ``` pycon > >>> Task.apply_async(immediate=False, exchange='video', > ... routing_key='video.compress') > ``` > > and a router returns: > > ``` python > {'immediate': True, > 'exchange': 'urgent'} > ``` > > the final message options will be: > > ``` pycon > >>> task.apply_async( > ... immediate=True, > ... exchange='urgent', > ... routing_key='video.compress', > ... ) > ``` > > (and any default message options defined in the > `~celery.task.base.Task`{.interpreted-text role="class"} class) - New Task handler called after the task returns: `~celery.task.base.Task.after_return`{.interpreted-text role="meth"}. - `~billiard.einfo.ExceptionInfo`{.interpreted-text role="class"} now passed to : `~celery.task.base.Task.on_retry`{.interpreted-text role="meth"}/ `~celery.task.base.Task.on_failure`{.interpreted-text role="meth"} as `einfo` keyword argument. - Worker: Added `CELERYD_MAX_TASKS_PER_CHILD`{.interpreted-text role="setting"} / `celery worker --maxtasksperchild`. > Defines the maximum number of tasks a pool worker can process > before the process is terminated and replaced by a new one. - Revoked tasks now marked with state `REVOKED`{.interpreted-text role="state"}, and [result.get()]{.title-ref} will now raise `~@TaskRevokedError`{.interpreted-text role="exc"}. - `celery.task.control.ping`{.interpreted-text role="func"} now works as expected. - [apply(throw=True)]{.title-ref} / `CELERY_EAGER_PROPAGATES_EXCEPTIONS`{.interpreted-text role="setting"}: Makes eager execution re-raise task errors. - New signal: `~celery.signals.worker_process_init`{.interpreted-text role="signal"}: Sent inside the pool worker process at init. - Worker: `celery worker -Q`{.interpreted-text role="option"} option: Ability to specify list of queues to use, disabling other configured queues. > For example, if `CELERY_QUEUES`{.interpreted-text role="setting"} > defines four queues: [image]{.title-ref}, [video]{.title-ref}, > [data]{.title-ref} and [default]{.title-ref}, the following > command would make the worker only consume from the > [image]{.title-ref} and [video]{.title-ref} queues: > > ``` console > $ celeryd -Q image,video > ``` - Worker: New return value for the [revoke]{.title-ref} control command: > Now returns: > > ``` python > {'ok': 'task $id revoked'} > ``` > > instead of `True`{.interpreted-text role="const"}. - Worker: Can now enable/disable events using remote control > Example usage: > > > \>\>\> from celery.task.control import broadcast \>\>\> > > broadcast(\'enable_events\') \>\>\> > > broadcast(\'disable_events\') - Removed top-level tests directory. Test config now in celery.tests.config > This means running the unit tests doesn\'t require any special > setup. [celery/tests/\_\_init\_\_]{.title-ref} now configures the > `CELERY_CONFIG_MODULE`{.interpreted-text role="envvar"} and > `CELERY_LOADER`{.interpreted-text role="envvar"} environment > variables, so when [nosetests]{.title-ref} imports that, the unit > test environment is all set up. > > Before you run the tests you need to install the test > requirements: > > ``` console > $ pip install -r requirements/test.txt > ``` > > Running all tests: > > ``` console > $ nosetests > ``` > > Specifying the tests to run: > > ``` console > $ nosetests celery.tests.test_task > ``` > > Producing HTML coverage: > > ``` console > $ nosetests --with-coverage3 > ``` > > The coverage output is then located in > [celery/tests/cover/index.html]{.title-ref}. - Worker: New option \`\--version\`: Dump version info and exit. - `celeryd-multi `{.interpreted-text role="mod"}: Tool for shell scripts to start multiple workers. > Some examples: > > - Advanced example with 10 workers: > > > - Three of the workers processes the images and video > > queue > > - Two of the workers processes the data queue with > > loglevel DEBUG > > - the rest processes the default\' queue. > > > > ``` console > > $ celeryd-multi start 10 -l INFO -Q:1-3 images,video -Q:4,5:data -Q default -L:4,5 DEBUG > > ``` > > - Get commands to start 10 workers, with 3 processes each > > > ``` console > > $ celeryd-multi start 3 -c 3 > > celeryd -n celeryd1.myhost -c 3 > > celeryd -n celeryd2.myhost -c 3 > > celeryd -n celeryd3.myhost -c 3 > > ``` > > - Start 3 named workers > > > ``` console > > $ celeryd-multi start image video data -c 3 > > celeryd -n image.myhost -c 3 > > celeryd -n video.myhost -c 3 > > celeryd -n data.myhost -c 3 > > ``` > > - Specify custom hostname > > > ``` console > > $ celeryd-multi start 2 -n worker.example.com -c 3 > > celeryd -n celeryd1.worker.example.com -c 3 > > celeryd -n celeryd2.worker.example.com -c 3 > > ``` > > > > Additional options are added to each `celeryd`, but you can > > also modify the options for ranges of or single workers > > - 3 workers: Two with 3 processes, and one with 10 processes. > > > ``` console > > $ celeryd-multi start 3 -c 3 -c:1 10 > > celeryd -n celeryd1.myhost -c 10 > > celeryd -n celeryd2.myhost -c 3 > > celeryd -n celeryd3.myhost -c 3 > > ``` > > - Can also specify options for named workers > > > ``` console > > $ celeryd-multi start image video data -c 3 -c:image 10 > > celeryd -n image.myhost -c 10 > > celeryd -n video.myhost -c 3 > > celeryd -n data.myhost -c 3 > > ``` > > - Ranges and lists of workers in options is also allowed: > (`-c:1-3` can also be written as `-c:1,2,3`) > > > ``` console > > $ celeryd-multi start 5 -c 3 -c:1-3 10 > > celeryd-multi -n celeryd1.myhost -c 10 > > celeryd-multi -n celeryd2.myhost -c 10 > > celeryd-multi -n celeryd3.myhost -c 10 > > celeryd-multi -n celeryd4.myhost -c 3 > > celeryd-multi -n celeryd5.myhost -c 3 > > ``` > > - Lists also work with named workers: > > > ``` console > > $ celeryd-multi start foo bar baz xuzzy -c 3 -c:foo,bar,baz 10 > > celeryd-multi -n foo.myhost -c 10 > > celeryd-multi -n bar.myhost -c 10 > > celeryd-multi -n baz.myhost -c 10 > > celeryd-multi -n xuzzy.myhost -c 3 > > ``` - The worker now calls the result backends [process_cleanup]{.title-ref} method *after* task execution instead of before. - AMQP result backend now supports Pika. --- # Change history for Celery 2.1 {#changelog-2.1} ::: {.contents local=""} ::: ## 2.1.4 {#version-2.1.4} release-date : 2010-12-03 12:00 p.m. CEST release-by : Ask Solem ### Fixes {#v214-fixes} - Execution options to [apply_async]{.title-ref} now takes precedence over options returned by active routers. This was a regression introduced recently (Issue #244). - curses monitor: Long arguments are now truncated so curses doesn\'t crash with out of bounds errors (Issue #235). - multi: Channel errors occurring while handling control commands no longer crash the worker but are instead logged with severity error. - SQLAlchemy database backend: Fixed a race condition occurring when the client wrote the pending state. Just like the Django database backend, it does no longer save the pending state (Issue #261 + Issue #262). - Error email body now uses [repr(exception)]{.title-ref} instead of [str(exception)]{.title-ref}, as the latter could result in Unicode decode errors (Issue #245). - Error email timeout value is now configurable by using the `EMAIL_TIMEOUT`{.interpreted-text role="setting"} setting. - \`celeryev\`: Now works on Windows (but the curses monitor won\'t work without having curses). - Unit test output no longer emits non-standard characters. - worker: The broadcast consumer is now closed if the connection is reset. - worker: Now properly handles errors occurring while trying to acknowledge the message. - [TaskRequest.on_failure]{.title-ref} now encodes traceback using the current file-system : encoding (Issue #286). - [EagerResult]{.title-ref} can now be pickled (Issue #288). ### Documentation {#v214-documentation} - Adding `contributing`{.interpreted-text role="ref"}. - Added `guide-optimizing`{.interpreted-text role="ref"}. - Added `faq-security`{.interpreted-text role="ref"} section to the FAQ. ## 2.1.3 {#version-2.1.3} release-date : 2010-11-09 05:00 p.m. CEST release-by : Ask Solem ::: {#v213-fixes} - Fixed deadlocks in [timer2]{.title-ref} which could lead to [djcelerymon]{.title-ref}/[celeryev -c]{.title-ref} hanging. - \`EventReceiver\`: now sends heartbeat request to find workers. > This means `celeryev`{.interpreted-text role="program"} and > friends finds workers immediately at start-up. - `celeryev` curses monitor: Set screen_delay to 10ms, so the screen refreshes more often. - Fixed pickling errors when pickling `AsyncResult`{.interpreted-text role="class"} on older Python versions. - worker: prefetch count was decremented by ETA tasks even if there were no active prefetch limits. ::: ## 2.1.2 {#version-2.1.2} release-data : TBA ### Fixes {#v212-fixes} - worker: Now sends the `task-retried`{.interpreted-text role="event"} event for retried tasks. - worker: Now honors ignore result for `~@WorkerLostError`{.interpreted-text role="exc"} and timeout errors. - `celerybeat`: Fixed `UnboundLocalError`{.interpreted-text role="exc"} in `celerybeat` logging when using logging setup signals. - worker: All log messages now includes [exc_info]{.title-ref}. ## 2.1.1 {#version-2.1.1} release-date : 2010-10-14 02:00 p.m. CEST release-by : Ask Solem ### Fixes {#v211-fixes} - Now working on Windows again. > Removed dependency on the `pwd`{.interpreted-text > role="mod"}/`grp`{.interpreted-text role="mod"} modules. - snapshots: Fixed race condition leading to loss of events. - worker: Reject tasks with an ETA that cannot be converted to a time stamp. > See issue #209 - concurrency.processes.pool: The semaphore was released twice for each task (both at ACK and result ready). > This has been fixed, and it is now released only once per task. - docs/configuration: Fixed typo [CELERYD_TASK_SOFT_TIME_LIMIT]{.title-ref} -\> `CELERYD_TASK_SOFT_TIME_LIMIT`{.interpreted-text role="setting"}. > See issue #214 - control command \`dump_scheduled\`: was using old .info attribute - multi: Fixed [set changed size during iteration]{.title-ref} bug : occurring in the restart command. - worker: Accidentally tried to use additional command-line arguments. > This would lead to an error like: > > > [got multiple values for keyword argument > > \'concurrency\']{.title-ref}. > > > > Additional command-line arguments are now ignored, and doesn\'t > > produce this error. However \-- we do reserve the right to use > > positional arguments in the future, so please don\'t depend on > > this behavior. - `celerybeat`: Now respects routers and task execution options again. - `celerybeat`: Now reuses the publisher instead of the connection. - Cache result backend: Using `float`{.interpreted-text role="class"} as the expires argument to [cache.set]{.title-ref} is deprecated by the Memcached libraries, so we now automatically cast to `int`{.interpreted-text role="class"}. - unit tests: No longer emits logging and warnings in test output. ### News {#v211-news} - Now depends on carrot version 0.10.7. - Added `CELERY_REDIRECT_STDOUTS`{.interpreted-text role="setting"}, and `CELERYD_REDIRECT_STDOUTS_LEVEL`{.interpreted-text role="setting"} settings. > `CELERY_REDIRECT_STDOUTS`{.interpreted-text role="setting"} is > used by the worker and beat. All output to [stdout]{.title-ref} > and [stderr]{.title-ref} will be redirected to the current logger > if enabled. > > `CELERY_REDIRECT_STDOUTS_LEVEL`{.interpreted-text role="setting"} > decides the log level used and is `WARNING`{.interpreted-text > role="const"} by default. - Added `CELERYBEAT_SCHEDULER`{.interpreted-text role="setting"} setting. > This setting is used to define the default for the -S option to > `celerybeat`{.interpreted-text role="program"}. > > Example: > > ``` python > CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler' > ``` - Added Task.expires: Used to set default expiry time for tasks. - New remote control commands: [add_consumer]{.title-ref} and [cancel_consumer]{.title-ref}. > ::: {.method module=""} > add_consumer(queue, exchange, exchange_type, routing_key, > \*\*options) > > Tells the worker to declare and consume from the specified > declaration. > ::: > > ::: {.method module=""} > cancel_consumer(queue_name) > > Tells the worker to stop consuming from queue (by queue name). > ::: > > Commands also added to `celeryctl`{.interpreted-text > role="program"} and > `~celery.task.control.inspect`{.interpreted-text role="class"}. > > Example using `celeryctl` to start consuming from queue \"queue\", > in exchange \"exchange\", of type \"direct\" using binding key > \"key\": > > ``` console > $ celeryctl inspect add_consumer queue exchange direct key > $ celeryctl inspect cancel_consumer queue > ``` > > See `monitoring-control`{.interpreted-text role="ref"} for more > information about the `celeryctl`{.interpreted-text > role="program"} program. > > Another example using > `~celery.task.control.inspect`{.interpreted-text role="class"}: > > ``` pycon > >>> from celery.task.control import inspect > >>> inspect.add_consumer(queue='queue', exchange='exchange', > ... exchange_type='direct', > ... routing_key='key', > ... durable=False, > ... auto_delete=True) > > >>> inspect.cancel_consumer('queue') > ``` - `celerybeat`: Now logs the traceback if a message can\'t be sent. - `celerybeat`: Now enables a default socket timeout of 30 seconds. - `README`/introduction/homepage: Added link to [Flask-Celery](https://github.com/ask/flask-celery). ## 2.1.0 {#version-2.1.0} release-date : 2010-10-08 12:00 p.m. CEST release-by : Ask Solem ### Important Notes {#v210-important} - Celery is now following the versioning semantics defined by [semver](http://semver.org). > This means we\'re no longer allowed to use odd/even versioning > semantics By our previous versioning scheme this stable release > should\'ve been version 2.2. - Now depends on Carrot 0.10.7. - No longer depends on SQLAlchemy, this needs to be installed separately if the database result backend is used. - `django-celery`{.interpreted-text role="pypi"} now comes with a monitor for the Django Admin interface. This can also be used if you\'re not a Django user. (Update: Django-Admin monitor has been replaced with Flower, see the Monitoring guide). - If you get an error after upgrading saying: [AttributeError: \'module\' object has no attribute \'system\']{.title-ref}, > Then this is because the [celery.platform]{.title-ref} module has > been renamed to [celery.platforms]{.title-ref} to not collide with > the built-in `platform`{.interpreted-text role="mod"} module. > > You have to remove the old `platform.py`{.interpreted-text > role="file"} (and maybe `platform.pyc`{.interpreted-text > role="file"}) file from your previous Celery installation. > > To do this use `python`{.interpreted-text role="program"} to find > the location of this module: > > ``` console > $ python > >>> import celery.platform > >>> celery.platform > > ``` > > Here the compiled module is in > `/opt/devel/celery/celery/`{.interpreted-text role="file"}, to > remove the offending files do: > > ``` console > $ rm -f /opt/devel/celery/celery/platform.py* > ``` ### News {#v210-news} - Added support for expiration of AMQP results (requires RabbitMQ 2.1.0) > The new configuration option > `CELERY_AMQP_TASK_RESULT_EXPIRES`{.interpreted-text > role="setting"} sets the expiry time in seconds (can be int or > float): > > ``` python > CELERY_AMQP_TASK_RESULT_EXPIRES = 30 * 60 # 30 minutes. > CELERY_AMQP_TASK_RESULT_EXPIRES = 0.80 # 800 ms. > ``` - `celeryev`: Event Snapshots > If enabled, the worker sends messages about what the worker is > doing. These messages are called \"events\". The events are used > by real-time monitors to show what the cluster is doing, but > they\'re not very useful for monitoring over a longer period of > time. Snapshots lets you take \"pictures\" of the clusters state > at regular intervals. This can then be stored in a database to > generate statistics with, or even monitoring over longer time > periods. > > `django-celery`{.interpreted-text role="pypi"} now comes with a > Celery monitor for the Django Admin interface. To use this you > need to run the `django-celery`{.interpreted-text role="pypi"} > snapshot camera, which stores snapshots to the database at > configurable intervals. > > To use the Django admin monitor you need to do the following: > > 1. Create the new database tables: > > > ``` console > > $ python manage.py syncdb > > ``` > > 2. Start the `django-celery`{.interpreted-text role="pypi"} > snapshot camera: > > > ``` console > > $ python manage.py celerycam > > ``` > > 3. Open up the django admin to monitor your cluster. > > The admin interface shows tasks, worker nodes, and even lets you > perform some actions, like revoking and rate limiting tasks, and > shutting down worker nodes. > > There\'s also a Debian init.d script for > `~celery.bin.events`{.interpreted-text role="mod"} available, see > `daemonizing`{.interpreted-text role="ref"} for more information. > > New command-line arguments to `celeryev`: > > > - `celery events --camera`{.interpreted-text role="option"}: > > Snapshot camera class to use. > > - `celery events --logfile`{.interpreted-text role="option"}: > > Log file > > - `celery events --loglevel`{.interpreted-text role="option"}: > > Log level > > - `celery events --maxrate`{.interpreted-text role="option"}: > > Shutter rate limit. > > - `celery events --freq`{.interpreted-text role="option"}: > > Shutter frequency > > The `--camera `{.interpreted-text > role="option"} argument is the name of a class used to take > snapshots with. It must support the interface defined by > `celery.events.snapshot.Polaroid`{.interpreted-text role="class"}. > > Shutter frequency controls how often the camera thread wakes up, > while the rate limit controls how often it will actually take a > snapshot. The rate limit can be an integer (snapshots/s), or a > rate limit string which has the same syntax as the task rate limit > strings ([\"200/m\"]{.title-ref}, [\"10/s\"]{.title-ref}, > [\"1/h\",]{.title-ref} etc). > > For the Django camera case, this rate limit can be used to control > how often the snapshots are written to the database, and the > frequency used to control how often the thread wakes up to check > if there\'s anything new. > > The rate limit is off by default, which means it will take a > snapshot for every > `--frequency `{.interpreted-text > role="option"} seconds. - `~celery.task.control.broadcast`{.interpreted-text role="func"}: Added callback argument, this can be used to process replies immediately as they arrive. - `celeryctl`: New command line utility to manage and inspect worker nodes, apply tasks and inspect the results of tasks. > ::: seealso > The `monitoring-control`{.interpreted-text role="ref"} section in > the `guide`{.interpreted-text role="ref"}. > ::: > > Some examples: > > ``` console > $ celeryctl apply tasks.add -a '[2, 2]' --countdown=10 > > $ celeryctl inspect active > $ celeryctl inspect registered_tasks > $ celeryctl inspect scheduled > $ celeryctl inspect --help > $ celeryctl apply --help > ``` - Added the ability to set an expiry date and time for tasks. > Example: > > >>> # Task expires after one minute from now. > >>> task.apply_async(args, kwargs, expires=60) > >>> # Also supports datetime > >>> task.apply_async(args, kwargs, > ... expires=datetime.now() + timedelta(days=1) > > When a worker receives a task that\'s been expired it will be > marked as revoked (`~@TaskRevokedError`{.interpreted-text > role="exc"}). - Changed the way logging is configured. > We now configure the root logger instead of only configuring our > custom logger. In addition we don\'t hijack the multiprocessing > logger anymore, but instead use a custom logger name for different > applications: > > **Application** **Logger Name** > ----------------- ----------------- > `celeryd` `"celery"` > `celerybeat` `"celery.beat"` > `celeryev` `"celery.ev"` > > This means that the [loglevel]{.title-ref} and > [logfile]{.title-ref} arguments will affect all registered loggers > (even those from third-party libraries). Unless you configure the > loggers manually as shown below, that is. > > *Users can choose to configure logging by subscribing to the > :signal:\`\~celery.signals.setup_logging\` signal:* > > ``` python > from logging.config import fileConfig > from celery import signals > > @signals.setup_logging.connect > def setup_logging(**kwargs): > fileConfig('logging.conf') > ``` > > If there are no receivers for this signal, the logging subsystem > will be configured using the > `--loglevel `{.interpreted-text > role="option"}/ > `--logfile `{.interpreted-text > role="option"} arguments, this will be used for *all defined > loggers*. > > Remember that the worker also redirects stdout and stderr to the > Celery logger, if manually configure logging you also need to > redirect the standard outs manually: > > ``` python > from logging.config import fileConfig > from celery import log > > def setup_logging(**kwargs): > import logging > fileConfig('logging.conf') > stdouts = logging.getLogger('mystdoutslogger') > log.redirect_stdouts_to_logger(stdouts, loglevel=logging.WARNING) > ``` - worker Added command line option `--include `{.interpreted-text role="option"}: > A comma separated list of (task) modules to be imported. > > Example: > > ``` console > $ celeryd -I app1.tasks,app2.tasks > ``` - worker: now emits a warning if running as the root user (euid is 0). - `celery.messaging.establish_connection`{.interpreted-text role="func"}: Ability to override defaults used using keyword argument \"defaults\". - worker: Now uses [multiprocessing.freeze_support()]{.title-ref} so that it should work with **py2exe**, **PyInstaller**, **cx_Freeze**, etc. - worker: Now includes more meta-data for the `STARTED`{.interpreted-text role="state"} state: PID and host name of the worker that started the task. > See issue #181 - subtask: Merge additional keyword arguments to [subtask()]{.title-ref} into task keyword arguments. > For example: > > ``` pycon > >>> s = subtask((1, 2), {'foo': 'bar'}, baz=1) > >>> s.args > (1, 2) > >>> s.kwargs > {'foo': 'bar', 'baz': 1} > ``` > > See issue #182. - worker: Now emits a warning if there\'s already a worker node using the same name running on the same virtual host. - AMQP result backend: Sending of results are now retried if the connection is down. - AMQP result backend: \`result.get()[: Wait for next state if state isn\'t in :data:]{.title-ref}\~celery.states.READY_STATES\`. - TaskSetResult now supports subscription. > >>> res = TaskSet(tasks).apply_async() > >>> res[0].get() - Added [Task.send_error_emails]{.title-ref} + [Task.error_whitelist]{.title-ref}, so these can be configured per task instead of just by the global setting. - Added [Task.store_errors_even_if_ignored]{.title-ref}, so it can be changed per Task, not just by the global setting. - The Crontab scheduler no longer wakes up every second, but implements [remaining_estimate]{.title-ref} (*Optimization*). - worker: Store `FAILURE`{.interpreted-text role="state"} result if the : `~@WorkerLostError`{.interpreted-text role="exc"} exception occurs (worker process disappeared). - worker: Store `FAILURE`{.interpreted-text role="state"} result if one of the [\*TimeLimitExceeded]{.title-ref} exceptions occurs. - Refactored the periodic task responsible for cleaning up results. > - > > The backend cleanup task is now only added to the schedule if > > : `CELERY_TASK_RESULT_EXPIRES`{.interpreted-text > role="setting"} is set. > > - If the schedule already contains a periodic task named > \"celery.backend_cleanup\" it won\'t change it, so the > behavior of the backend cleanup task can be easily changed. > > - The task is now run every day at 4:00 AM, rather than every > day since the first time it was run (using Crontab schedule > instead of [run_every]{.title-ref}) > > - > > Renamed [celery.task.builtins.DeleteExpiredTaskMetaTask]{.title-ref} > > : -\> > `celery.task.builtins.backend_cleanup`{.interpreted-text > role="class"} > > - The task itself has been renamed from > \"celery.delete_expired_task_meta\" to > \"celery.backend_cleanup\" > > See issue #134. - Implemented [AsyncResult.forget]{.title-ref} for SQLAlchemy/Memcached/Redis/Tokyo Tyrant backends (forget and remove task result). > See issue #184. - `TaskSetResult.join `{.interpreted-text role="meth"}: Added \'propagate=True\' argument. When set to `False`{.interpreted-text role="const"} exceptions occurring in subtasks will not be re-raised. - Added [Task.update_state(task_id, state, meta)]{.title-ref} as a shortcut to [task.backend.store_result(task_id, meta, state)]{.title-ref}. > The backend interface is \"private\" and the terminology outdated, > so better to move this to > `~celery.task.base.Task`{.interpreted-text role="class"} so it can > be used. - timer2: Set [self.running=False]{.title-ref} in `~celery.utils.timer2.Timer.stop`{.interpreted-text role="meth"} so it won\'t try to join again on subsequent calls to [stop()]{.title-ref}. - Log colors are now disabled by default on Windows. - [celery.platform]{.title-ref} renamed to `celery.platforms`{.interpreted-text role="mod"}, so it doesn\'t collide with the built-in `platform`{.interpreted-text role="mod"} module. - Exceptions occurring in Mediator+Pool callbacks are now caught and logged instead of taking down the worker. - Redis result backend: Now supports result expiration using the Redis [EXPIRE]{.title-ref} command. - unit tests: Don\'t leave threads running at tear down. - worker: Task results shown in logs are now truncated to 46 chars. - [Task.\_\_name\_\_]{.title-ref} is now an alias to [self.\_\_class\_\_.\_\_name\_\_]{.title-ref}. : This way tasks introspects more like regular functions. - \`Task.retry\`: Now raises `TypeError`{.interpreted-text role="exc"} if kwargs argument is empty. > See issue #164. - `timedelta_seconds`: Use `timedelta.total_seconds` if running on Python 2.7 - `~kombu.utils.limits.TokenBucket`{.interpreted-text role="class"}: Generic Token Bucket algorithm - `celery.events.state`{.interpreted-text role="mod"}: Recording of cluster state can now be paused and resumed, including support for buffering. > ::: method > State.freeze(buffer=True) > > Pauses recording of the stream. > > If [buffer]{.title-ref} is true, events received while being > frozen will be buffered, and may be replayed later. > ::: > > ::: method > State.thaw(replay=True) > > Resumes recording of the stream. > > If [replay]{.title-ref} is true, then the recorded buffer will be > applied. > ::: > > ::: method > State.freeze_while(fun) > > With a function to apply, freezes the stream before, and replays > the buffer after the function returns. > ::: - `EventReceiver.capture `{.interpreted-text role="meth"} Now supports a timeout keyword argument. - worker: The mediator thread is now disabled if `CELERY_RATE_LIMITS`{.interpreted-text role="setting"} is enabled, and tasks are directly sent to the pool without going through the ready queue (*Optimization*). ### Fixes {#v210-fixes} - Pool: Process timed out by [TimeoutHandler]{.title-ref} must be joined by the Supervisor, so don\'t remove it from the internal process list. > See issue #192. - [TaskPublisher.delay_task]{.title-ref} now supports exchange argument, so exchange can be overridden when sending tasks in bulk using the same publisher > See issue #187. - the worker no longer marks tasks as revoked if `CELERY_IGNORE_RESULT`{.interpreted-text role="setting"} is enabled. > See issue #207. - AMQP Result backend: Fixed bug with [result.get()]{.title-ref} if `CELERY_TRACK_STARTED`{.interpreted-text role="setting"} enabled. > [result.get()]{.title-ref} would stop consuming after receiving > the `STARTED`{.interpreted-text role="state"} state. - Fixed bug where new processes created by the pool supervisor becomes stuck while reading from the task Queue. > See - Fixed timing issue when declaring the remote control command reply queue > This issue could result in replies being lost, but have now been > fixed. - Backward compatible [LoggerAdapter]{.title-ref} implementation: Now works for Python 2.4. > Also added support for several new methods: [fatal]{.title-ref}, > [makeRecord]{.title-ref}, [\_log]{.title-ref}, [log]{.title-ref}, > [isEnabledFor]{.title-ref}, [addHandler]{.title-ref}, > [removeHandler]{.title-ref}. ### Experimental {#v210-experimental} - multi: Added daemonization support. > multi can now be used to start, stop and restart worker nodes: > > ``` console > $ celeryd-multi start jerry elaine george kramer > ``` > > This also creates PID files and log files > (`celeryd@jerry.pid`{.interpreted-text role="file"}, \..., > `celeryd@jerry.log`{.interpreted-text role="file"}. To specify a > location for these files use the [\--pidfile]{.title-ref} and > [\--logfile]{.title-ref} arguments with the [%n]{.title-ref} > format: > > ``` console > $ celeryd-multi start jerry elaine george kramer \ > --logfile=/var/log/celeryd@%n.log \ > --pidfile=/var/run/celeryd@%n.pid > ``` > > Stopping: > > ``` console > $ celeryd-multi stop jerry elaine george kramer > ``` > > Restarting. The nodes will be restarted one by one as the old ones > are shutdown: > > ``` console > $ celeryd-multi restart jerry elaine george kramer > ``` > > Killing the nodes (**WARNING**: Will discard currently executing > tasks): > > ``` console > $ celeryd-multi kill jerry elaine george kramer > ``` > > See [celeryd-multi help]{.title-ref} for help. - multi: [start]{.title-ref} command renamed to [show]{.title-ref}. > [celeryd-multi start]{.title-ref} will now actually start and > detach worker nodes. To just generate the commands you have to use > [celeryd-multi show]{.title-ref}. - worker: Added [\--pidfile]{.title-ref} argument. > The worker will write its pid when it starts. The worker will not > be started if this file exists and the pid contained is still > alive. - Added generic init.d script using [celeryd-multi]{.title-ref} > ### Documentation {#v210-documentation} - Added User guide section: Monitoring - Added user guide section: Periodic Tasks > Moved from [getting-started/periodic-tasks]{.title-ref} and > updated. - tutorials/external moved to new section: \"community\". - References has been added to all sections in the documentation. > This makes it easier to link between documents. --- # Change history for Celery 2.2 {#changelog-2.2} ::: {.contents local=""} ::: ## 2.2.8 {#version-2.2.8} release-date : 2011-11-25 04:00 p.m. GMT release-by : Ask Solem ### Security Fixes {#v228-security-fixes} - \[Security: [CELERYSA-0001](https://github.com/celery/celery/tree/master/docs/sec/CELERYSA-0001.txt)\] Daemons would set effective id\'s rather than real id\'s when the `!--uid`{.interpreted-text role="option"}/ `!--gid`{.interpreted-text role="option"} arguments to `celery multi`{.interpreted-text role="program"}, `celeryd_detach`{.interpreted-text role="program"}, `celery beat`{.interpreted-text role="program"} and `celery events`{.interpreted-text role="program"} were used. This means privileges weren\'t properly dropped, and that it would be possible to regain supervisor privileges later. ## 2.2.7 {#version-2.2.7} release-date : 2011-06-13 04:00 p.m. BST release-by : Ask Solem - New signals: `after_setup_logger`{.interpreted-text role="signal"} and `after_setup_task_logger`{.interpreted-text role="signal"} > These signals can be used to augment logging configuration after > Celery has set up logging. - Redis result backend now works with Redis 2.4.4. - multi: The `!--gid`{.interpreted-text role="option"} option now works correctly. - worker: Retry wrongfully used the repr of the traceback instead of the string representation. - App.config_from_object: Now loads module, not attribute of module. - Fixed issue where logging of objects would give \"\\" ## 2.2.6 {#version-2.2.6} release-date : 2011-04-15 04:00 p.m. CEST release-by : Ask Solem ### Important Notes {#v226-important} - Now depends on `Kombu`{.interpreted-text role="pypi"} 1.1.2. - Dependency lists now explicitly specifies that we don\'t want `python-dateutil`{.interpreted-text role="pypi"} 2.x, as this version only supports Python 3. > If you have installed dateutil 2.0 by accident you should > downgrade to the 1.5.0 version: > > ``` console > $ pip install -U python-dateutil==1.5.0 > ``` > > or by `easy_install`: > > ``` console > $ easy_install -U python-dateutil==1.5.0 > ``` ### Fixes {#v226-fixes} - The new `WatchedFileHandler` broke Python 2.5 support (Issue #367). - Task: Don\'t use `app.main` if the task name is set explicitly. - Sending emails didn\'t work on Python 2.5, due to a bug in the version detection code (Issue #378). - Beat: Adds method `ScheduleEntry._default_now` > This method can be overridden to change the default value of > `last_run_at`. - An error occurring in process cleanup could mask task errors. We no longer propagate errors happening at process cleanup, but log them instead. This way they won\'t interfere with publishing the task result (Issue #365). - Defining tasks didn\'t work properly when using the Django `shell_plus` utility (Issue #366). - `AsyncResult.get` didn\'t accept the `interval` and `propagate` : arguments. - worker: Fixed a bug where the worker wouldn\'t shutdown if a : `socket.error`{.interpreted-text role="exc"} was raised. ## 2.2.5 {#version-2.2.5} release-date : 2011-03-28 06:00 p.m. CEST release-by : Ask Solem ### Important Notes {#v225-important} - Now depends on Kombu 1.0.7 ### News {#v225-news} - Our documentation is now hosted by Read The Docs (), and all links have been changed to point to the new URL. - Logging: Now supports log rotation using external tools like [logrotate.d](http://www.ducea.com/2006/06/06/rotating-linux-log-files-part-2-logrotate/) (Issue #321) > This is accomplished by using the `WatchedFileHandler`, which > re-opens the file if it\'s renamed or deleted. - `otherqueues` tutorial now documents how to configure Redis/Database result : backends. - gevent: Now supports ETA tasks. > But gevent still needs `CELERY_DISABLE_RATE_LIMITS=True` to work. - TaskSet User Guide: now contains TaskSet callback recipes. - Eventlet: New signals: > - `eventlet_pool_started` > - `eventlet_pool_preshutdown` > - `eventlet_pool_postshutdown` > - `eventlet_pool_apply` > > See `celery.signals`{.interpreted-text role="mod"} for more > information. - New `BROKER_TRANSPORT_OPTIONS`{.interpreted-text role="setting"} setting can be used to pass additional arguments to a particular broker transport. - worker: `worker_pid` is now part of the request info as returned by broadcast commands. - TaskSet.apply/Taskset.apply_async now accepts an optional `taskset_id` argument. - The taskset_id (if any) is now available in the Task request context. - SQLAlchemy result backend: taskset_id and taskset_id columns now have a unique constraint (tables need to recreated for this to take affect). - Task user guide: Added section about choosing a result backend. - Removed unused attribute `AsyncResult.uuid`. ### Fixes {#v225-fixes} - multiprocessing.Pool: Fixes race condition when marking job with `WorkerLostError` (Issue #268). > The process may have published a result before it was terminated, > but we have no reliable way to detect that this is the case. > > So we have to wait for 10 seconds before marking the result with > WorkerLostError. This gives the result handler a chance to > retrieve the result. - multiprocessing.Pool: Shutdown could hang if rate limits disabled. > There was a race condition when the MainThread was waiting for the > pool semaphore to be released. The ResultHandler now terminates > after 5 seconds if there are unacked jobs, but no worker processes > left to start them (it needs to timeout because there could still > be an ack+result that we haven\'t consumed from the result queue. > It is unlikely we\'ll receive any after 5 seconds with no worker > processes). - `celerybeat`: Now creates pidfile even if the `--detach` option isn\'t set. - eventlet/gevent: The broadcast command consumer is now running in a separate green-thread. > This ensures broadcast commands will take priority even if there > are many active tasks. - Internal module `celery.worker.controllers` renamed to `celery.worker.mediator`. - worker: Threads now terminates the program by calling `os._exit`, as it is the only way to ensure exit in the case of syntax errors, or other unrecoverable errors. - Fixed typo in `maybe_timedelta` (Issue #352). - worker: Broadcast commands now logs with loglevel debug instead of warning. - AMQP Result Backend: Now resets cached channel if the connection is lost. - Polling results with the AMQP result backend wasn\'t working properly. - Rate limits: No longer sleeps if there are no tasks, but rather waits for the task received condition (Performance improvement). - ConfigurationView: `iter(dict)` should return keys, not items (Issue #362). - `celerybeat`: PersistentScheduler now automatically removes a corrupted schedule file (Issue #346). - Programs that doesn\'t support positional command-line arguments now provides a user friendly error message. - Programs no longer tries to load the configuration file when showing `--version` (Issue #347). - Autoscaler: The \"all processes busy\" log message is now severity debug instead of error. - worker: If the message body can\'t be decoded, it\'s now passed through `safe_str` when logging. > This to ensure we don\'t get additional decoding errors when > trying to log the failure. - `app.config_from_object`/`app.config_from_envvar` now works for all loaders. - Now emits a user-friendly error message if the result backend name is unknown (Issue #349). - `celery.contrib.batches`: Now sets loglevel and logfile in the task request so `task.get_logger` works with batch tasks (Issue #357). - worker: An exception was raised if using the amqp transport and the prefetch count value exceeded 65535 (Issue #359). > The prefetch count is incremented for every received task with an > ETA/countdown defined. The prefetch count is a short, so can only > support a maximum value of 65535. If the value exceeds the maximum > value we now disable the prefetch count, it\'s re-enabled as soon > as the value is below the limit again. - `cursesmon`: Fixed unbound local error (Issue #303). - eventlet/gevent is now imported on demand so autodoc can import the modules without having eventlet/gevent installed. - worker: Ack callback now properly handles `AttributeError`. - `Task.after_return` is now always called *after* the result has been written. - Cassandra Result Backend: Should now work with the latest `pycassa` version. - multiprocessing.Pool: No longer cares if the `putlock` semaphore is released too many times (this can happen if one or more worker processes are killed). - SQLAlchemy Result Backend: Now returns accidentally removed `date_done` again (Issue #325). - Task.request context is now always initialized to ensure calling the task function directly works even if it actively uses the request context. - Exception occurring when iterating over the result from `TaskSet.apply` fixed. - eventlet: Now properly schedules tasks with an ETA in the past. ## 2.2.4 {#version-2.2.4} release-date : 2011-02-19 00:00 AM CET release-by : Ask Solem ### Fixes {#v224-fixes} - worker: 2.2.3 broke error logging, resulting in tracebacks not being logged. - AMQP result backend: Polling task states didn\'t work properly if there were more than one result message in the queue. - `TaskSet.apply_async()` and `TaskSet.apply()` now supports an optional `taskset_id` keyword argument (Issue #331). - The current taskset id (if any) is now available in the task context as `request.taskset` (Issue #329). - SQLAlchemy result backend: [date_done]{.title-ref} was no longer part of the results as it had been accidentally removed. It\'s now available again (Issue #325). - SQLAlchemy result backend: Added unique constraint on [Task.id]{.title-ref} and [TaskSet.taskset_id]{.title-ref}. Tables needs to be recreated for this to take effect. - Fixed exception raised when iterating on the result of `TaskSet.apply()`. - Tasks user guide: Added section on choosing a result backend. ## 2.2.3 {#version-2.2.3} release-date : 2011-02-12 04:00 p.m. CET release-by : Ask Solem ### Fixes {#v223-fixes} - Now depends on `Kombu`{.interpreted-text role="pypi"} 1.0.3 - Task.retry now supports a `max_retries` argument, used to change the default value. - [multiprocessing.cpu_count]{.title-ref} may raise `NotImplementedError`{.interpreted-text role="exc"} on platforms where this isn\'t supported (Issue #320). - Coloring of log messages broke if the logged object wasn\'t a string. - Fixed several typos in the init-script documentation. - A regression caused [Task.exchange]{.title-ref} and [Task.routing_key]{.title-ref} to no longer have any effect. This is now fixed. - Routing user guide: Fixes typo, routers in `CELERY_ROUTES`{.interpreted-text role="setting"} must be instances, not classes. - `celeryev`{.interpreted-text role="program"} didn\'t create pidfile even though the `--pidfile `{.interpreted-text role="option"} argument was set. - Task logger format was no longer used (Issue #317). > The id and name of the task is now part of the log message again. - A safe version of `repr()` is now used in strategic places to ensure objects with a broken `__repr__` doesn\'t crash the worker, or otherwise make errors hard to understand (Issue #298). - Remote control command `active_queues`{.interpreted-text role="control"}: didn\'t account for queues added at runtime. > In addition the dictionary replied by this command now has a > different structure: the exchange key is now a dictionary > containing the exchange declaration in full. - The `celery worker -Q`{.interpreted-text role="option"} option removed unused queue declarations, so routing of tasks could fail. > Queues are no longer removed, but rather > [app.amqp.queues.consume_from()]{.title-ref} is used as the list > of queues to consume from. > > This ensures all queues are available for routing purposes. - `celeryctl`: Now supports the [inspect active_queues]{.title-ref} command. ## 2.2.2 {#version-2.2.2} release-date : 2011-02-03 04:00 p.m. CET release-by : Ask Solem ### Fixes {#v222-fixes} - `celerybeat` couldn\'t read the schedule properly, so entries in `CELERYBEAT_SCHEDULE`{.interpreted-text role="setting"} wouldn\'t be scheduled. - Task error log message now includes [exc_info]{.title-ref} again. - The [eta]{.title-ref} argument can now be used with [task.retry]{.title-ref}. > Previously it was overwritten by the countdown argument. - `celery multi`/`celeryd_detach`: Now logs errors occurring when executing the [celery worker]{.title-ref} command. - daemonizing tutorial: Fixed typo `--time-limit 300` -\> `--time-limit=300` - Colors in logging broke non-string objects in log messages. - `setup_task_logger` no longer makes assumptions about magic task kwargs. ## 2.2.1 {#version-2.2.1} release-date : 2011-02-02 04:00 p.m. CET release-by : Ask Solem ### Fixes {#v221-fixes} - Eventlet pool was leaking memory (Issue #308). - Deprecated function `celery.execute.delay_task` was accidentally removed, now available again. - `BasePool.on_terminate` stub didn\'t exist - `celeryd_detach`: Adds readable error messages if user/group name doesn\'t exist. - Smarter handling of unicode decode errors when logging errors. ## 2.2.0 {#version-2.2.0} release-date : 2011-02-01 10:00 AM CET release-by : Ask Solem ### Important Notes {#v220-important} - Carrot has been replaced with `Kombu`{.interpreted-text role="pypi"} > Kombu is the next generation messaging library for Python, fixing > several flaws present in Carrot that was hard to fix without > breaking backwards compatibility. > > Also it adds: > > - First-class support for virtual transports; Redis, Django ORM, > SQLAlchemy, Beanstalk, MongoDB, CouchDB and in-memory. > - Consistent error handling with introspection, > - The ability to ensure that an operation is performed by > gracefully handling connection and channel errors, > - Message compression (`zlib`{.interpreted-text role="mod"}, > `bz2`{.interpreted-text role="mod"}, or custom compression > schemes). > > This means that [ghettoq]{.title-ref} is no longer needed as the > functionality it provided is already available in Celery by > default. The virtual transports are also more feature complete > with support for exchanges (direct and topic). The Redis transport > even supports fanout exchanges so it\'s able to perform worker > remote control commands. - Magic keyword arguments pending deprecation. > The magic keyword arguments were responsible for many problems and > quirks: notably issues with tasks and decorators, and name > collisions in keyword arguments for the unaware. > > It wasn\'t easy to find a way to deprecate the magic keyword > arguments, but we think this is a solution that makes sense and it > won\'t have any adverse effects for existing code. > > The path to a magic keyword argument free world is: > > > - the [celery.decorators]{.title-ref} module is deprecated and > > the decorators can now be found in > > [celery.task]{.title-ref}. > > - The decorators in [celery.task]{.title-ref} disables keyword > > arguments by default > > - All examples in the documentation have been changed to use > > [celery.task]{.title-ref}. > > > > This means that the following will have magic keyword arguments > > enabled (old style): > > > > > ``` python > > > from celery.decorators import task > > > > > > @task() > > > def add(x, y, **kwargs): > > > print('In task %s' % kwargs['task_id']) > > > return x + y > > > ``` > > > > And this won\'t use magic keyword arguments (new style): > > > > > ``` python > > > from celery.task import task > > > > > > @task() > > > def add(x, y): > > > print('In task %s' % add.request.id) > > > return x + y > > > ``` > > In addition, tasks can choose not to accept magic keyword > arguments by setting the [task.accept_magic_kwargs]{.title-ref} > attribute. > > ::: admonition > Deprecation > > Using the decorators in `celery.decorators`{.interpreted-text > role="mod"} emits a `PendingDeprecationWarning`{.interpreted-text > role="class"} with a helpful message urging you to change your > code, in version 2.4 this will be replaced with a > `DeprecationWarning`{.interpreted-text role="class"}, and in > version 4.0 the `celery.decorators`{.interpreted-text role="mod"} > module will be removed and no longer exist. > > Similarly, the [task.accept_magic_kwargs]{.title-ref} attribute > will no longer have any effect starting from version 4.0. > ::: - The magic keyword arguments are now available as [task.request]{.title-ref} > This is called *the context*. Using thread-local storage the > context contains state that\'s related to the current request. > > It\'s mutable and you can add custom attributes that\'ll only be > seen by the current task request. > > The following context attributes are always available: > > **Magic Keyword Argument** **Replace with** > ------------------------------------------- ------------------------------------------ > [kwargs\[\'task_id\'\]]{.title-ref} [self.request.id]{.title-ref} > [kwargs\[\'delivery_info\'\]]{.title-ref} [self.request.delivery_info]{.title-ref} > [kwargs\[\'task_retries\'\]]{.title-ref} [self.request.retries]{.title-ref} > [kwargs\[\'logfile\'\]]{.title-ref} [self.request.logfile]{.title-ref} > [kwargs\[\'loglevel\'\]]{.title-ref} [self.request.loglevel]{.title-ref} > [kwargs\[\'task_is_eager\'\]]{.title-ref} [self.request.is_eager]{.title-ref} > **NEW** [self.request.args]{.title-ref} > **NEW** [self.request.kwargs]{.title-ref} > > In addition, the following methods now automatically uses the > current context, so you don\'t have to pass [kwargs]{.title-ref} > manually anymore: > > > - [task.retry]{.title-ref} > > - [task.get_logger]{.title-ref} > > - [task.update_state]{.title-ref} - [Eventlet](http://eventlet.net) support. > This is great news for I/O-bound tasks! > > To change pool implementations you use the > `celery worker --pool`{.interpreted-text role="option"} argument, > or globally using the `CELERYD_POOL`{.interpreted-text > role="setting"} setting. This can be the full name of a class, or > one of the following aliases: [processes]{.title-ref}, > [eventlet]{.title-ref}, [gevent]{.title-ref}. > > For more information please see the > `concurrency-eventlet`{.interpreted-text role="ref"} section in > the User Guide. > > ::: admonition > Why not gevent? > > For our first alternative concurrency implementation we\'ve > focused on [Eventlet](http://eventlet.net), but there\'s also an > experimental [gevent](http://gevent.org) pool available. This is > missing some features, notably the ability to schedule ETA tasks. > > Hopefully the [gevent](http://gevent.org) support will be feature > complete by version 2.3, but this depends on user demand (and > contributions). > ::: - Python 2.4 support deprecated! > We\'re happy\^H\^H\^H\^H\^Hsad to announce that this is the last > version to support Python 2.4. > > You\'re urged to make some noise if you\'re currently stuck with > Python 2.4. Complain to your package maintainers, sysadmins and > bosses: tell them it\'s time to move on! > > Apart from wanting to take advantage of `with`{.interpreted-text > role="keyword"} statements, coroutines, conditional expressions > and enhanced `try`{.interpreted-text role="keyword"} blocks, the > code base now contains so many 2.4 related hacks and workarounds > it\'s no longer just a compromise, but a sacrifice. > > If it really isn\'t your choice, and you don\'t have the option to > upgrade to a newer version of Python, you can just continue to use > Celery 2.2. Important fixes can be back ported for as long as > there\'s interest. - worker: Now supports Autoscaling of child worker processes. > The `--autoscale `{.interpreted-text > role="option"} option can be used to configure the minimum and > maximum number of child worker processes: > > ``` text > --autoscale=AUTOSCALE > Enable autoscaling by providing > max_concurrency,min_concurrency. Example: > --autoscale=10,3 (always keep 3 processes, but grow to > 10 if necessary). > ``` - Remote Debugging of Tasks > `celery.contrib.rdb` is an extended version of > `pdb`{.interpreted-text role="mod"} that enables remote debugging > of processes that doesn\'t have terminal access. > > Example usage: > > ``` text > from celery.contrib import rdb > from celery.task import task > > @task() > def add(x, y): > result = x + y > # set breakpoint > rdb.set_trace() > return result > > :func:`~celery.contrib.rdb.set_trace` sets a breakpoint at the current > location and creates a socket you can telnet into to remotely debug > your task. > > The debugger may be started by multiple processes at the same time, > so rather than using a fixed port the debugger will search for an > available port, starting from the base port (6900 by default). > The base port can be changed using the environment variable > :envvar:`CELERY_RDB_PORT`. > > By default the debugger will only be available from the local host, > to enable access from the outside you have to set the environment > variable :envvar:`CELERY_RDB_HOST`. > > When the worker encounters your breakpoint it will log the following > information:: > > [INFO/MainProcess] Received task: > tasks.add[d7261c71-4962-47e5-b342-2448bedd20e8] > [WARNING/PoolWorker-1] Remote Debugger:6900: > Please telnet 127.0.0.1 6900. Type `exit` in session to continue. > [2011-01-18 14:25:44,119: WARNING/PoolWorker-1] Remote Debugger:6900: > Waiting for client... > > If you telnet the port specified you'll be presented > with a ``pdb`` shell: > > .. code-block:: console > > $ telnet localhost 6900 > Connected to localhost. > Escape character is '^]'. > > /opt/devel/demoapp/tasks.py(128)add() > -> return result > (Pdb) > > Enter ``help`` to get a list of available commands, > It may be a good idea to read the `Python Debugger Manual`_ if > you have never used `pdb` before. > ``` - Events are now transient and is using a topic exchange (instead of direct). > The [CELERYD_EVENT_EXCHANGE]{.title-ref}, > [CELERYD_EVENT_ROUTING_KEY]{.title-ref}, > [CELERYD_EVENT_EXCHANGE_TYPE]{.title-ref} settings are no longer > in use. > > This means events won\'t be stored until there\'s a consumer, and > the events will be gone as soon as the consumer stops. Also it > means there can be multiple monitors running at the same time. > > The routing key of an event is the type of event (e.g., > [worker.started]{.title-ref}, [worker.heartbeat]{.title-ref}, > [task.succeeded]{.title-ref}, etc. This means a consumer can > filter on specific types, to only be alerted of the events it > cares about. > > Each consumer will create a unique queue, meaning it\'s in effect > a broadcast exchange. > > This opens up a lot of possibilities, for example the workers > could listen for worker events to know what workers are in the > neighborhood, and even restart workers when they go down (or use > this information to optimize tasks/autoscaling). > > ::: note > ::: title > Note > ::: > > The event exchange has been renamed from `"celeryevent"` to > `"celeryev"` so it doesn\'t collide with older versions. > > If you\'d like to remove the old exchange you can do so by > executing the following command: > > ``` console > $ camqadm exchange.delete celeryevent > ``` > ::: - The worker now starts without configuration, and configuration can be specified directly on the command-line. Configuration options must appear after the last argument, separated by two dashes: ``` console $ celery worker -l info -I tasks -- broker.host=localhost broker.vhost=/app ``` - Configuration is now an alias to the original configuration, so changes to the original will reflect Celery at runtime. - [celery.conf]{.title-ref} has been deprecated, and modifying [celery.conf.ALWAYS_EAGER]{.title-ref} will no longer have any effect. > The default configuration is now available in the > `celery.app.defaults`{.interpreted-text role="mod"} module. The > available configuration options and their types can now be > introspected. - Remote control commands are now provided by [kombu.pidbox]{.title-ref}, the generic process mailbox. - Internal module [celery.worker.listener]{.title-ref} has been renamed to [celery.worker.consumer]{.title-ref}, and [.CarrotListener]{.title-ref} is now [.Consumer]{.title-ref}. - Previously deprecated modules [celery.models]{.title-ref} and [celery.management.commands]{.title-ref} have now been removed as per the deprecation time-line. - \[Security: Low severity\] Removed [celery.task.RemoteExecuteTask]{.title-ref} and : accompanying functions: [dmap]{.title-ref}, [dmap_async]{.title-ref}, and [execute_remote]{.title-ref}. Executing arbitrary code using pickle is a potential security issue if someone gains unrestricted access to the message broker. If you really need this functionality, then you\'d\'ve to add this to your own project. - \[Security: Low severity\] The [stats]{.title-ref} command no longer transmits the broker password. > One would\'ve needed an authenticated broker connection to receive > this password in the first place, but sniffing the password at the > wire level would\'ve been possible if using unencrypted > communication. ### News {#v220-news} - The internal module [celery.task.builtins]{.title-ref} has been removed. - The module [celery.task.schedules]{.title-ref} is deprecated, and [celery.schedules]{.title-ref} should be used instead. > For example if you have: > > from celery.task.schedules import crontab > > You should replace that with: > > from celery.schedules import crontab > > The module needs to be renamed because it must be possible to > import schedules without importing the [celery.task]{.title-ref} > module. - The following functions have been deprecated and is scheduled for removal in version 2.3: > - [celery.execute.apply_async]{.title-ref} > > > Use [task.apply_async()]{.title-ref} instead. > > - [celery.execute.apply]{.title-ref} > > > Use [task.apply()]{.title-ref} instead. > > - [celery.execute.delay_task]{.title-ref} > > > Use [registry.tasks\[name\].delay()]{.title-ref} instead. - Importing [TaskSet]{.title-ref} from [celery.task.base]{.title-ref} is now deprecated. > You should use: > > >>> from celery.task import TaskSet > > instead. - New remote control commands: > - [active_queues]{.title-ref} > > > Returns the queue declarations a worker is currently > > consuming from. - Added the ability to retry publishing the task message in the event of connection loss or failure. > This is disabled by default but can be enabled using the > `CELERY_TASK_PUBLISH_RETRY`{.interpreted-text role="setting"} > setting, and tweaked by the > `CELERY_TASK_PUBLISH_RETRY_POLICY`{.interpreted-text > role="setting"} setting. > > In addition [retry]{.title-ref}, and [retry_policy]{.title-ref} > keyword arguments have been added to > [Task.apply_async]{.title-ref}. > > ::: note > ::: title > Note > ::: > > Using the [retry]{.title-ref} argument to > [apply_async]{.title-ref} requires you to handle the > publisher/connection manually. > ::: - Periodic Task classes ([\@periodic_task]{.title-ref}/[PeriodicTask]{.title-ref}) will *not* be deprecated as previously indicated in the source code. > But you\'re encouraged to use the more flexible > `CELERYBEAT_SCHEDULE`{.interpreted-text role="setting"} setting. - Built-in daemonization support of the worker using [celery multi]{.title-ref} is no longer experimental and is considered production quality. > See `daemon-generic`{.interpreted-text role="ref"} if you want to > use the new generic init scripts. - Added support for message compression using the `CELERY_MESSAGE_COMPRESSION`{.interpreted-text role="setting"} setting, or the [compression]{.title-ref} argument to [apply_async]{.title-ref}. This can also be set using routers. - worker: Now logs stack-trace of all threads when receiving the : [SIGUSR1]{.title-ref} signal (doesn\'t work on CPython 2.4, Windows or Jython). > Inspired by - Can now remotely terminate/kill the worker process currently processing a task. > The [revoke]{.title-ref} remote control command now supports a > [terminate]{.title-ref} argument Default signal is > [TERM]{.title-ref}, but can be specified using the > [signal]{.title-ref} argument. Signal can be the uppercase name of > any signal defined in the `signal`{.interpreted-text role="mod"} > module in the Python Standard Library. > > Terminating a task also revokes it. > > Example: > > >>> from celery.task.control import revoke > > >>> revoke(task_id, terminate=True) > >>> revoke(task_id, terminate=True, signal='KILL') > >>> revoke(task_id, terminate=True, signal='SIGKILL') - \`TaskSetResult.join_native\`: Backend-optimized version of [join()]{.title-ref}. > If available, this version uses the backends ability to retrieve > multiple results at once, unlike [join()]{.title-ref} which > fetches the results one by one. > > So far only supported by the AMQP result backend. Support for > Memcached and Redis may be added later. - Improved implementations of [TaskSetResult.join]{.title-ref} and [AsyncResult.wait]{.title-ref}. > An [interval]{.title-ref} keyword argument have been added to both > so the polling interval can be specified (default interval is 0.5 > seconds). > > > A [propagate]{.title-ref} keyword argument have been added to > > [result.wait()]{.title-ref}, errors will be returned instead of > > raised if this is set to False. > > > > ::: warning > > ::: title > > Warning > > ::: > > > > You should decrease the polling interval when using the database > > result backend, as frequent polling can result in high database > > load. > > ::: - The PID of the child worker process accepting a task is now sent as a field with the `task-started`{.interpreted-text role="event"} event. - The following fields have been added to all events in the worker class: > - \`sw_ident\`: Name of worker software (e.g., `"py-celery"`). > - \`sw_ver\`: Software version (e.g., 2.2.0). > - \`sw_sys\`: Operating System (e.g., Linux, Windows, Darwin). - For better accuracy the start time reported by the multiprocessing worker process is used when calculating task duration. > Previously the time reported by the accept callback was used. - \`celerybeat\`: New built-in daemonization support using the [\--detach]{.title-ref} : option. - \`celeryev\`: New built-in daemonization support using the [\--detach]{.title-ref} : option. - \`TaskSet.apply_async\`: Now supports custom publishers by using the [publisher]{.title-ref} argument. - Added `CELERY_SEND_TASK_SENT_EVENT`{.interpreted-text role="setting"} setting. > If enabled an event will be sent with every task, so monitors can > track tasks before the workers receive them. - \`celerybeat\`: Now reuses the broker connection when calling : scheduled tasks. - The configuration module and loader to use can now be specified on the command-line. > For example: > > ``` console > $ celery worker --config=celeryconfig.py --loader=myloader.Loader > ``` - Added signals: [beat_init]{.title-ref} and [beat_embedded_init]{.title-ref} > - `celery.signals.beat_init`{.interpreted-text role="signal"} > > > Dispatched when `celerybeat`{.interpreted-text > > role="program"} starts (either standalone or embedded). > > Sender is the `celery.beat.Service`{.interpreted-text > > role="class"} instance. > > - `celery.signals.beat_embedded_init`{.interpreted-text > role="signal"} > > > Dispatched in addition to the `beat_init`{.interpreted-text > > role="signal"} signal when `celerybeat`{.interpreted-text > > role="program"} is started as an embedded process. Sender is > > the `celery.beat.Service`{.interpreted-text role="class"} > > instance. - Redis result backend: Removed deprecated settings [REDIS_TIMEOUT]{.title-ref} and [REDIS_CONNECT_RETRY]{.title-ref}. - CentOS init-script for `celery worker`{.interpreted-text role="program"} now available in [extra/centos]{.title-ref}. - Now depends on `pyparsing`{.interpreted-text role="pypi"} version 1.5.0 or higher. > There have been reported issues using Celery with > `pyparsing`{.interpreted-text role="pypi"} 1.4.x, so please > upgrade to the latest version. - Lots of new unit tests written, now with a total coverage of 95%. ### Fixes {#v220-fixes} - [celeryev]{.title-ref} Curses Monitor: Improved resize handling and UI layout (Issue #274 + Issue #276) - AMQP Backend: Exceptions occurring while sending task results are now propagated instead of silenced. > the worker will then show the full traceback of these errors in > the log. - AMQP Backend: No longer deletes the result queue after successful poll, as this should be handled by the `CELERY_AMQP_TASK_RESULT_EXPIRES`{.interpreted-text role="setting"} setting instead. - AMQP Backend: Now ensures queues are declared before polling results. - Windows: worker: Show error if running with [-B]{.title-ref} option. > Running `celerybeat` embedded is known not to work on Windows, so > users are encouraged to run `celerybeat` as a separate service > instead. - Windows: Utilities no longer output ANSI color codes on Windows - `camqadm`: Now properly handles `Control-c`{.interpreted-text role="kbd"} by simply exiting instead of showing confusing traceback. - Windows: All tests are now passing on Windows. - Remove bin/ directory, and [scripts]{.title-ref} section from `setup.py`{.interpreted-text role="file"}. > This means we now rely completely on setuptools entry-points. ### Experimental {#v220-experimental} - Jython: worker now runs on Jython using the threaded pool. > All tests pass, but there may still be bugs lurking around the > corners. - PyPy: worker now runs on PyPy. > It runs without any pool, so to get parallel execution you must > start multiple instances (e.g., using `multi`{.interpreted-text > role="program"}). > > Sadly an initial benchmark seems to show a 30% performance > decrease on `pypy-1.4.1` + JIT. We would like to find out why this > is, so stay tuned. - `PublisherPool`{.interpreted-text role="class"}: Experimental pool of task publishers and connections to be used with the [retry]{.title-ref} argument to [apply_async]{.title-ref}. The example code below will re-use connections and channels, and retry sending of the task message if the connection is lost. ``` python from celery import current_app # Global pool pool = current_app().amqp.PublisherPool(limit=10) def my_view(request): with pool.acquire() as publisher: add.apply_async((2, 2), publisher=publisher, retry=True) ``` --- # Change history for Celery 2.3 {#changelog-2.3} ::: {.contents local=""} ::: ## 2.3.4 {#version-2.3.4} release-date : 2011-11-25 04:00 p.m. GMT release-by : Ask Solem ### Security Fixes {#v234-security-fixes} - \[Security: [CELERYSA-0001](https://github.com/celery/celery/tree/master/docs/sec/CELERYSA-0001.txt)\] Daemons would set effective id\'s rather than real id\'s when the `!--uid`{.interpreted-text role="option"}/ `!--gid`{.interpreted-text role="option"} arguments to `celery multi`{.interpreted-text role="program"}, `celeryd_detach`{.interpreted-text role="program"}, `celery beat`{.interpreted-text role="program"} and `celery events`{.interpreted-text role="program"} were used. This means privileges weren\'t properly dropped, and that it would be possible to regain supervisor privileges later. ### Fixes - Backported fix for #455 from 2.4 to 2.3. - StateDB wasn\'t saved at shutdown. - Fixes worker sometimes hanging when hard time limit exceeded. ## 2.3.3 {#version-2.3.3} release-date : 2011-16-09 05:00 p.m. BST release-by : Mher Movsisyan - Monkey patching `sys.stdout`{.interpreted-text role="attr"} could result in the worker crashing if the replacing object didn\'t define `isatty`{.interpreted-text role="meth"} (Issue #477). - `CELERYD` option in `/etc/default/celeryd`{.interpreted-text role="file"} shouldn\'t be used with generic init-scripts. ## 2.3.2 {#version-2.3.2} release-date : 2011-10-07 05:00 p.m. BST release-by : Ask Solem ### News {#v232-news} - Improved Contributing guide. > If you\'d like to contribute to Celery you should read the > `Contributing Gudie `{.interpreted-text role="ref"}. > > We\'re looking for contributors at all skill levels, so don\'t > hesitate! - Now depends on Kombu 1.3.1 - `Task.request` now contains the current worker host name (Issue #460). > Available as `task.request.hostname`. - It\'s now easier for app subclasses to extend how they\'re pickled. : (see `celery.app.AppPickler`{.interpreted-text role="class"}). ### Fixes {#v232-fixes} - [purge/discard_all]{.title-ref} wasn\'t working correctly (Issue #455). - The coloring of log messages didn\'t handle non-ASCII data well (Issue #427). - \[Windows\] the multiprocessing pool tried to import `os.kill` even though this isn\'t available there (Issue #450). - Fixes case where the worker could become unresponsive because of tasks exceeding the hard time limit. - The `task-sent`{.interpreted-text role="event"} event was missing from the event reference. - `ResultSet.iterate` now returns results as they finish (Issue #459). > This wasn\'t the case previously, even though the documentation > states this was the expected behavior. - Retries will no longer be performed when tasks are called directly (using `__call__`). > Instead the exception passed to `retry` will be re-raised. - Eventlet no longer crashes if autoscale is enabled. > growing and shrinking eventlet pools is still not supported. - `py24` target removed from `tox.ini`{.interpreted-text role="file"}. ## 2.3.1 {#version-2.3.1} release-date : 2011-08-07 08:00 p.m. BST release-by : Ask Solem ### Fixes - The `CELERY_AMQP_TASK_RESULT_EXPIRES`{.interpreted-text role="setting"} setting didn\'t work, resulting in an AMQP related error about not being able to serialize floats while trying to publish task states (Issue #446). ## 2.3.0 {#version-2.3.0} release-date : 2011-08-05 12:00 p.m. BST tested : CPython: 2.5, 2.6, 2.7; PyPy: 1.5; Jython: 2.5.2 release-by : Ask Solem ### Important Notes {#v230-important} - Now requires Kombu 1.2.1 - Results are now disabled by default. > The AMQP backend wasn\'t a good default because often the users > were not consuming the results, resulting in thousands of queues. > > While the queues can be configured to expire if left unused, it > wasn\'t possible to enable this by default because this was only > available in recent RabbitMQ versions (2.1.1+) > > With this change enabling a result backend will be a conscious > choice, which will hopefully lead the user to read the > documentation and be aware of any common pitfalls with the > particular backend. > > The default backend is now a dummy backend > (`celery.backends.base.DisabledBackend`{.interpreted-text > role="class"}). Saving state is simply an no-op, and > AsyncResult.wait(), .result, .state, etc. will raise a > `NotImplementedError`{.interpreted-text role="exc"} telling the > user to configure the result backend. > > For help choosing a backend please see > `task-result-backends`{.interpreted-text role="ref"}. > > If you depend on the previous default which was the AMQP backend, > then you have to set this explicitly before upgrading: > > CELERY_RESULT_BACKEND = 'amqp' > > ::: note > ::: title > Note > ::: > > For `django-celery`{.interpreted-text role="pypi"} users the > default backend is still `database`, and results are not disabled > by default. > ::: - The Debian init-scripts have been deprecated in favor of the generic-init.d init-scripts. > In addition generic init-scripts for `celerybeat` and `celeryev` > has been added. ### News {#v230-news} - Automatic connection pool support. > The pool is used by everything that requires a broker connection, > for example calling tasks, sending broadcast commands, retrieving > results with the AMQP result backend, and so on. > > The pool is disabled by default, but you can enable it by > configuring the `BROKER_POOL_LIMIT`{.interpreted-text > role="setting"} setting: > > BROKER_POOL_LIMIT = 10 > > A limit of 10 means a maximum of 10 simultaneous connections can > co-exist. Only a single connection will ever be used in a > single-thread environment, but in a concurrent environment > (threads, greenlets, etc., but not processes) when the limit has > been exceeded, any try to acquire a connection will block the > thread and wait for a connection to be released. This is something > to take into consideration when choosing a limit. > > A limit of `None`{.interpreted-text role="const"} or 0 means no > limit, and connections will be established and closed every time. - Introducing Chords (taskset callbacks). > A chord is a task that only executes after all of the tasks in a > taskset has finished executing. It\'s a fancy term for \"taskset > callbacks\" adopted from > [Cω](http://research.microsoft.com/en-us/um/cambridge/projects/comega/)). > > It works with all result backends, but the best implementation is > currently provided by the Redis result backend. > > Here\'s an example chord: > > >>> chord(add.subtask((i, i)) > ... for i in xrange(100))(tsum.subtask()).get() > 9900 > > Please read the > `Chords section in the user guide `{.interpreted-text > role="ref"}, if you want to know more. - Time limits can now be set for individual tasks. > To set the soft and hard time limits for a task use the > `time_limit` and `soft_time_limit` attributes: > > ``` python > import time > > @task(time_limit=60, soft_time_limit=30) > def sleeptask(seconds): > time.sleep(seconds) > ``` > > If the attributes are not set, then the workers default time > limits will be used. > > New in this version you can also change the time limits for a task > at runtime using the `time_limit`{.interpreted-text role="func"} > remote control command: > > >>> from celery.task import control > >>> control.time_limit('tasks.sleeptask', > ... soft=60, hard=120, reply=True) > [{'worker1.example.com': {'ok': 'time limits set successfully'}}] > > Only tasks that starts executing after the time limit change will > be affected. > > ::: note > ::: title > Note > ::: > > Soft time limits will still not work on Windows or other platforms > that don\'t have the `SIGUSR1` signal. > ::: - Redis backend configuration directive names changed to include the : `CELERY_` prefix. > **Old setting name** **Replace with** > ------------------------------ ------------------------------------- > [REDIS_HOST]{.title-ref} [CELERY_REDIS_HOST]{.title-ref} > [REDIS_PORT]{.title-ref} [CELERY_REDIS_PORT]{.title-ref} > [REDIS_DB]{.title-ref} [CELERY_REDIS_DB]{.title-ref} > [REDIS_PASSWORD]{.title-ref} [CELERY_REDIS_PASSWORD]{.title-ref} > > The old names are still supported but pending deprecation. - PyPy: The default pool implementation used is now multiprocessing if running on PyPy 1.5. - multi: now supports \"pass through\" options. > Pass through options makes it easier to use Celery without a > configuration file, or just add last-minute options on the command > line. > > Example use: > > ``` console > $ celery multi start 4 -c 2 -- broker.host=amqp.example.com \ > broker.vhost=/ \ > celery.disable_rate_limits=yes > ``` - `celerybeat`: Now retries establishing the connection (Issue #419). - `celeryctl`: New `list bindings` command. > Lists the current or all available bindings, depending on the > broker transport used. - Heartbeat is now sent every 30 seconds (previously every 2 minutes). - `ResultSet.join_native()` and `iter_native()` is now supported by the Redis and Cache result backends. > This is an optimized version of `join()` using the underlying > backends ability to fetch multiple results at once. - Can now use SSL when sending error e-mails by enabling the `EMAIL_USE_SSL`{.interpreted-text role="setting"} setting. - `events.default_dispatcher()`: Context manager to easily obtain an event dispatcher instance using the connection pool. - Import errors in the configuration module won\'t be silenced anymore. - ResultSet.iterate: Now supports the `timeout`, `propagate` and `interval` arguments. - `with_default_connection` -\> `with default_connection` - TaskPool.apply_async: Keyword arguments `callbacks` and `errbacks` has been renamed to `callback` and `errback` and take a single scalar value instead of a list. - No longer propagates errors occurring during process cleanup (Issue #365) - Added `TaskSetResult.delete()`, which will delete a previously saved taskset result. - `celerybeat` now syncs every 3 minutes instead of only at shutdown (Issue #382). - Monitors now properly handles unknown events, so user-defined events are displayed. - Terminating a task on Windows now also terminates all of the tasks child processes (Issue #384). - worker: `-I|--include` option now always searches the current directory to import the specified modules. - Cassandra backend: Now expires results by using TTLs. - Functional test suite in `funtests` is now actually working properly, and passing tests. ### Fixes {#v230-fixes} - `celeryev` was trying to create the pidfile twice. - celery.contrib.batches: Fixed problem where tasks failed silently (Issue #393). - Fixed an issue where logging objects would give \"\ This replaces the old \"Other queues\" tutorial, and adds > documentation for MongoDB, Beanstalk and CouchDB. ## 2.4.4 {#version-2.4.4} release-date : 2011-11-25 04:00 p.m. GMT release-by : Ask Solem ### Security Fixes {#v244-security-fixes} - \[Security: [CELERYSA-0001](https://github.com/celery/celery/tree/master/docs/sec/CELERYSA-0001.txt)\] Daemons would set effective id\'s rather than real id\'s when the `!--uid`{.interpreted-text role="option"}/ `!--gid`{.interpreted-text role="option"} arguments to `celery multi`{.interpreted-text role="program"}, `celeryd_detach`{.interpreted-text role="program"}, `celery beat`{.interpreted-text role="program"} and `celery events`{.interpreted-text role="program"} were used. This means privileges weren\'t properly dropped, and that it would be possible to regain supervisor privileges later. ### Fixes {#v244-fixes} - Processes pool: Fixed rare deadlock at shutdown (Issue #523). > Fix contributed by Ionel Maries Christian. - Webhook tasks issued the wrong HTTP POST headers (Issue #515). > The *Content-Type* header has been changed from `application/json` > ⇒ `application/x-www-form-urlencoded`, and adds a proper > *Content-Length* header. > > Fix contributed by Mitar. - Daemonization tutorial: Adds a configuration example using Django and virtualenv together (Issue #505). > Contributed by Juan Ignacio Catalano. - generic init-scripts now automatically creates log and pid file directories (Issue #545). > Contributed by Chris Streeter. ## 2.4.3 {#version-2.4.3} release-date : 2011-11-22 06:00 p.m. GMT release-by : Ask Solem - Fixes module import typo in [celeryctl]{.title-ref} (Issue #538). > Fix contributed by Chris Streeter. ## 2.4.2 {#version-2.4.2} release-date : 2011-11-14 12:00 p.m. GMT release-by : Ask Solem - Program module no longer uses relative imports so that it\'s possible to do `python -m celery.bin.name`. ## 2.4.1 {#version-2.4.1} release-date : 2011-11-07 06:00 p.m. GMT release-by : Ask Solem - `celeryctl inspect` commands was missing output. - processes pool: Decrease polling interval for less idle CPU usage. - processes pool: MaybeEncodingError wasn\'t wrapped in ExceptionInfo (Issue #524). - worker: would silence errors occurring after task consumer started. - logging: Fixed a bug where unicode in stdout redirected log messages couldn\'t be written (Issue #522). ## 2.4.0 {#version-2.4.0} release-date : 2011-11-04 04:00 p.m. GMT release-by : Ask Solem ### Important Notes {#v240-important} - Now supports Python 3. - Fixed deadlock in worker process handling (Issue #496). > A deadlock could occur after spawning new child processes because > the logging library\'s mutex wasn\'t properly reset after fork. > > The symptoms of this bug affecting would be that the worker simply > stops processing tasks, as none of the workers child processes are > functioning. There was a greater chance of this bug occurring with > `maxtasksperchild` or a time-limit enabled. > > This is a workaround for > . > > Be aware that while this fixes the logging library lock, there > could still be other locks initialized in the parent process, > introduced by custom code. > > Fix contributed by Harm Verhagen. - AMQP Result backend: Now expires results by default. > The default expiration value is now taken from the > `CELERY_TASK_RESULT_EXPIRES`{.interpreted-text role="setting"} > setting. > > The old `CELERY_AMQP_TASK_RESULT_EXPIRES`{.interpreted-text > role="setting"} setting has been deprecated and will be removed in > version 4.0. > > Note that this means that the result backend requires RabbitMQ > 2.1.0 or higher, and that you have to disable expiration if > you\'re running with an older version. You can do so by disabling > the `CELERY_TASK_RESULT_EXPIRES`{.interpreted-text role="setting"} > setting: > > CELERY_TASK_RESULT_EXPIRES = None - Eventlet: Fixed problem with shutdown (Issue #457). - Broker transports can be now be specified using URLs > The broker can now be specified as a URL instead. This URL must > have the format: > > ``` text > transport://user:password@hostname:port/virtual_host > ``` > > for example the default broker is written as: > > ``` text > amqp://guest:guest@localhost:5672// > ``` > > The scheme is required, so that the host is identified as a URL > and not just a host name. User, password, port and virtual_host > are optional and defaults to the particular transports default > value. > > ::: note > ::: title > Note > ::: > > Note that the path component (virtual_host) always starts with a > forward-slash. This is necessary to distinguish between the > virtual host `''` (empty) and `'/'`, which are both acceptable > virtual host names. > > A virtual host of `'/'` becomes: > > ``` text > amqp://guest:guest@localhost:5672// > ``` > > and a virtual host of `''` (empty) becomes: > > ``` text > amqp://guest:guest@localhost:5672/ > ``` > > So the leading slash in the path component is **always required**. > ::: > > In addition the `BROKER_URL`{.interpreted-text role="setting"} > setting has been added as an alias to `BROKER_HOST`. Any broker > setting specified in both the URL and in the configuration will be > ignored, if a setting isn\'t provided in the URL then the value > from the configuration will be used as default. > > Also, programs now support the > `--broker `{.interpreted-text role="option"} > option to specify a broker URL on the command-line: > > ``` console > $ celery worker -b redis://localhost > > $ celery inspect -b amqp://guest:guest@localhost//e > ``` > > The environment variable `CELERY_BROKER_URL`{.interpreted-text > role="envvar"} can also be used to easily override the default > broker used. - The deprecated `celery.loaders.setup_loader`{.interpreted-text role="func"} function has been removed. - The `CELERY_TASK_ERROR_WHITELIST`{.interpreted-text role="setting"} setting has been replaced by a more flexible approach (Issue #447). > The error mail sending logic is now available as `Task.ErrorMail`, > with the implementation (for reference) in > `celery.utils.mail`{.interpreted-text role="mod"}. > > The error mail class can be sub-classed to gain complete control > of when error messages are sent, thus removing the need for a > separate white-list setting. > > The `CELERY_TASK_ERROR_WHITELIST`{.interpreted-text > role="setting"} setting has been deprecated, and will be removed > completely in version 4.0. - Additional Deprecations > The following functions has been deprecated and is scheduled for > removal in version 4.0: > > **Old function** **Alternative** > --------------------------------------------- ----------------------------------------- > [celery.loaders.current_loader]{.title-ref} [celery.current_app.loader]{.title-ref} > [celery.loaders.load_settings]{.title-ref} [celery.current_app.conf]{.title-ref} > [celery.execute.apply]{.title-ref} [Task.apply]{.title-ref} > [celery.execute.apply_async]{.title-ref} [Task.apply_async]{.title-ref} > [celery.execute.delay_task]{.title-ref} [celery.execute.send_task]{.title-ref} > > The following settings has been deprecated and is scheduled for > removal in version 4.0: > > **Old setting** **Alternative** > ------------------------------------ ----------------------------- > [CELERYD_LOG_LEVEL]{.title-ref} `celery worker --loglevel=` > [CELERYD_LOG_FILE]{.title-ref} `celery worker --logfile=` > [CELERYBEAT_LOG_LEVEL]{.title-ref} `celery beat --loglevel=` > [CELERYBEAT_LOG_FILE]{.title-ref} `celery beat --logfile=` > [CELERYMON_LOG_LEVEL]{.title-ref} `celerymon --loglevel=` > [CELERYMON_LOG_FILE]{.title-ref} `celerymon --logfile=` ### News {#v240-news} - No longer depends on `pyparsing`{.interpreted-text role="pypi"}. - Now depends on Kombu 1.4.3. - CELERY_IMPORTS can now be a scalar value (Issue #485). > It\'s too easy to forget to add the comma after the sole element > of a tuple, and this is something that often affects newcomers. > > The docs should probably use a list in examples, as using a tuple > for this doesn\'t even make sense. Nonetheless, there are many > tutorials out there using a tuple, and this change should be a > help to new users. > > Suggested by `jsaxon-cars`{.interpreted-text role="github_user"}. - Fixed a memory leak when using the thread pool (Issue #486). > Contributed by Kornelijus Survila. - The `statedb` wasn\'t saved at exit. > This has now been fixed and it should again remember previously > revoked tasks when a `--statedb` is enabled. - Adds `EMAIL_USE_TLS`{.interpreted-text role="setting"} to enable secure SMTP connections (Issue #418). > Contributed by Stefan Kjartansson. - Now handles missing fields in task messages as documented in the message format documentation. > - Missing required field throws > `~@InvalidTaskError`{.interpreted-text role="exc"} > - Missing args/kwargs is assumed empty. > > Contributed by Chris Chamberlin. - Fixed race condition in `celery.events.state`{.interpreted-text role="mod"} (`celerymon`/`celeryev`) where task info would be removed while iterating over it (Issue #501). - The Cache, Cassandra, MongoDB, Redis and Tyrant backends now respects the `CELERY_RESULT_SERIALIZER`{.interpreted-text role="setting"} setting (Issue #435). > This means that only the database (Django/SQLAlchemy) backends > currently doesn\'t support using custom serializers. > > Contributed by Steeve Morin - Logging calls no longer manually formats messages, but delegates that to the logging system, so tools like Sentry can easier work with the messages (Issue #445). > Contributed by Chris Adams. - `multi` now supports a `stop_verify` command to wait for processes to shutdown. - Cache backend didn\'t work if the cache key was unicode (Issue #504). > Fix contributed by Neil Chintomby. - New setting `CELERY_RESULT_DB_SHORT_LIVED_SESSIONS`{.interpreted-text role="setting"} added, which if enabled will disable the caching of SQLAlchemy sessions (Issue #449). > Contributed by Leo Dirac. - All result backends now implements `__reduce__` so that they can be pickled (Issue #441). > Fix contributed by Remy Noel - multi didn\'t work on Windows (Issue #472). - New-style `CELERY_REDIS_*` settings now takes precedence over the old `REDIS_*` configuration keys (Issue #508). > Fix contributed by Joshua Ginsberg - Generic beat init-script no longer sets [bash -e]{.title-ref} (Issue #510). > Fix contributed by Roger Hu. - Documented that Chords don\'t work well with `redis-server`{.interpreted-text role="command"} versions before 2.2. > Contributed by Dan McGee. - The `CELERYBEAT_MAX_LOOP_INTERVAL`{.interpreted-text role="setting"} setting wasn\'t respected. - `inspect.registered_tasks` renamed to `inspect.registered` for naming consistency. > The previous name is still available as an alias. > > Contributed by Mher Movsisyan - Worker logged the string representation of args and kwargs without safe guards (Issue #480). - RHEL init-script: Changed worker start-up priority. > The default start / stop priorities for MySQL on RHEL are: > > ``` console > # chkconfig: - 64 36 > ``` > > Therefore, if Celery is using a database as a broker / message > store, it should be started after the database is up and running, > otherwise errors will ensue. This commit changes the priority in > the init-script to: > > ``` console > # chkconfig: - 85 15 > ``` > > which are the default recommended settings for 3-rd party > applications and assure that Celery will be started after the > database service & shut down before it terminates. > > Contributed by Yury V. Zaytsev. - KeyValueStoreBackend.get_many didn\'t respect the `timeout` argument (Issue #512). - beat/events\'s `--workdir` option didn\'t `chdir(2)`{.interpreted-text role="manpage"} before after configuration was attempted (Issue #506). - After deprecating 2.4 support we can now name modules correctly, since we can take use of absolute imports. > Therefore the following internal modules have been renamed: > > > `celery.concurrency.evlet` -\> `celery.concurrency.eventlet` > > `celery.concurrency.evg` -\> `celery.concurrency.gevent` - `AUTHORS`{.interpreted-text role="file"} file is now sorted alphabetically. > Also, as you may have noticed the contributors of new > features/fixes are now mentioned in the Changelog. --- # Change history for Celery 2.5 {#changelog-2.5} This document contains change notes for bugfix releases in the 2.5.x series, please see `whatsnew-2.5`{.interpreted-text role="ref"} for an overview of what\'s new in Celery 2.5. If you\'re looking for versions prior to 2.5 you should visit our `history`{.interpreted-text role="ref"} of releases. ::: {.contents local=""} ::: ## 2.5.5 {#version-2.5.5} release-date : 2012-06-06 04:00 p.m. BST release-by : Ask Solem This is a dummy release performed for the following goals: - Protect against force upgrading to Kombu 2.2.0 - Version parity with `django-celery`{.interpreted-text role="pypi"} ## 2.5.3 {#version-2.5.3} release-date : 2012-04-16 07:00 p.m. BST release-by : Ask Solem - A bug causes messages to be sent with UTC time-stamps even though `CELERY_ENABLE_UTC`{.interpreted-text role="setting"} wasn\'t enabled (Issue #636). - `celerybeat`: No longer crashes if an entry\'s args is set to None (Issue #657). - Auto-reload didn\'t work if a module\'s `__file__` attribute was set to the modules `.pyc` file. (Issue #647). - Fixes early 2.5 compatibility where `__package__` doesn\'t exist (Issue #638). ## 2.5.2 {#version-2.5.2} release-date : 2012-04-13 04:30 p.m. GMT release-by : Ask Solem ### News {#v252-news} - Now depends on Kombu 2.1.5. - Django documentation has been moved to the main Celery docs. > See `django`{.interpreted-text role="ref"}. - New `celeryd_init`{.interpreted-text role="signal"} signal can be used to configure workers by hostname. - Signal.connect can now be used as a decorator. > Example: > > ``` python > from celery.signals import task_sent > > @task_sent.connect > def on_task_sent(**kwargs): > print('sent task: %r' % (kwargs,)) > ``` - Invalid task messages are now rejected instead of acked. > This means that they will be moved to the dead-letter queue > introduced in the latest RabbitMQ version (but must be enabled > manually, consult the RabbitMQ documentation). - Internal logging calls has been cleaned up to work better with tools like Sentry. > Contributed by David Cramer. - New method `subtask.clone()` can be used to clone an existing subtask with augmented arguments/options. > Example: > > ``` pycon > >>> s = add.subtask((5,)) > >>> new = s.clone(args=(10,), countdown=5}) > >>> new.args > (10, 5) > > >>> new.options > {'countdown': 5} > ``` - Chord callbacks are now triggered in eager mode. ### Fixes {#v252-fixes} - Programs now verifies that the pidfile is actually written correctly (Issue #641). > Hopefully this will crash the worker immediately if the system is > out of space to store the complete pidfile. > > In addition, we now verify that existing pidfiles contain a new > line so that a partially written pidfile is detected as broken, as > before doing: > > ``` console > $ echo -n "1" > celeryd.pid > ``` > > would cause the worker to think that an existing instance was > already running (init has pid 1 after all). - Fixed 2.5 compatibility issue with use of print_exception. > Fix contributed by Martin Melin. - Fixed 2.5 compatibility issue with imports. > Fix contributed by Iurii Kriachko. - All programs now fix up `__package__` when called as main. > This fixes compatibility with Python 2.5. > > Fix contributed by Martin Melin. - \[celery control\|inspect\] can now be configured on the command-line. > Like with the worker it is now possible to configure Celery > settings on the command-line for celery control\|inspect > > ``` console > $ celery inspect -- broker.pool_limit=30 > ``` - Version dependency for `python-dateutil`{.interpreted-text role="pypi"} fixed to be strict. > Fix contributed by Thomas Meson. - `Task.__call__` is now optimized away in the task tracer rather than when the task class is created. > This fixes a bug where a custom \_\_call\_\_ may mysteriously > disappear. - Auto-reload\'s `inotify` support has been improved. > Contributed by Mher Movsisyan. - The Django broker documentation has been improved. - Removed confusing warning at top of routing user guide. ## 2.5.1 {#version-2.5.1} release-date : 2012-03-01 01:00 p.m. GMT release-by : Ask Solem ### Fixes {#v251-fixes} - Eventlet/Gevent: A small typo caused the worker to hang when eventlet/gevent was used, this was because the environment wasn\'t monkey patched early enough. - Eventlet/Gevent: Another small typo caused the mediator to be started with eventlet/gevent, which would make the worker sometimes hang at shutdown. - `multiprocessing`{.interpreted-text role="mod"}: Fixed an error occurring if the pool was stopped before it was properly started. - Proxy objects now redirects `__doc__` and `__name__` so `help(obj)` works. - Internal timer (timer2) now logs exceptions instead of swallowing them (Issue #626). - celery shell: can now be started with `--eventlet `{.interpreted-text role="option"} or `--gevent `{.interpreted-text role="option"} options to apply their monkey patches. ## 2.5.0 {#version-2.5.0} release-date : 2012-02-24 04:00 p.m. GMT release-by : Ask Solem See `whatsnew-2.5`{.interpreted-text role="ref"}. Since the changelog has gained considerable size, we decided to do things differently this time: by having separate \"what\'s new\" documents for major version changes. Bugfix releases will still be found in the changelog. --- # Change history for Celery 3.0 {#changelog-3.0} ::: {.contents local=""} ::: If you\'re looking for versions prior to 3.0.x you should go to `history`{.interpreted-text role="ref"}. ## 3.0.24 {#version-3.0.24} release-date : 2013-10-11 04:40 p.m. BST release-by : Ask Solem - Now depends on `Kombu 2.5.15 `{.interpreted-text role="ref"}. - Now depends on `billiard`{.interpreted-text role="pypi"} version 2.7.3.34. - AMQP Result backend: No longer caches queue declarations. > The queues created by the AMQP result backend are always unique, > so caching the declarations caused a slow memory leak. - Worker: Fixed crash when hostname contained Unicode characters. > Contributed by Daodao. - The worker would no longer start if the [-P solo]{.title-ref} pool was selected (Issue #1548). - Redis/Cache result backends wouldn\'t complete chords if any of the tasks were retried (Issue #1401). - Task decorator is no longer lazy if app is finalized. - AsyncResult: Fixed bug with `copy(AsyncResult)` when no `current_app` available. - ResultSet: Now properly propagates app when passed string id\'s. - Loader now ignores `CELERY_CONFIG_MODULE`{.interpreted-text role="envvar"} if value is empty string. - Fixed race condition in Proxy object where it tried to delete an attribute twice, resulting in `AttributeError`{.interpreted-text role="exc"}. - Task methods now works with the `CELERY_ALWAYS_EAGER`{.interpreted-text role="setting"} setting (Issue #1478). - `~kombu.common.Broadcast`{.interpreted-text role="class"} queues were accidentally declared when publishing tasks (Issue #1540). - New `C_FAKEFORK`{.interpreted-text role="envvar"} environment variable can be used to debug the init-scripts. > Setting this will skip the daemonization step so that errors > printed to stderr after standard outs are closed can be seen: > > ``` console > $ C_FAKEFORK /etc/init.d/celeryd start > ``` > > This works with the [celery multi]{.title-ref} command in general. - `get_pickleable_etype` didn\'t always return a value (Issue #1556). - Fixed bug where `app.GroupResult.restore` would fall back to the default app. - Fixed rare bug where built-in tasks would use the current_app. - `~celery.platforms.maybe_fileno`{.interpreted-text role="func"} now handles `ValueError`{.interpreted-text role="exc"}. ## 3.0.23 {#version-3.0.23} release-date : 2013-09-02 01:00 p.m. BST release-by : Ask Solem - Now depends on `Kombu 2.5.14 `{.interpreted-text role="ref"}. - `send_task` didn\'t honor `link` and `link_error` arguments. > This had the side effect of chains not calling unregistered tasks, > silently discarding them. > > Fix contributed by Taylor Nelson. - `celery.state`{.interpreted-text role="mod"}: Optimized precedence lookup. > Contributed by Matt Robenolt. - POSIX: Daemonization didn\'t redirect `sys.stdin` to `/dev/null`. > Fix contributed by Alexander Smirnov. - Canvas: group bug caused fallback to default app when `.apply_async` used (Issue #1516) - Canvas: generator arguments wasn\'t always pickleable. ## 3.0.22 {#version-3.0.22} release-date : 2013-08-16 04:30 p.m. BST release-by : Ask Solem - Now depends on `Kombu 2.5.13 `{.interpreted-text role="ref"}. - Now depends on `billiard`{.interpreted-text role="pypi"} 2.7.3.32 - Fixed bug with monthly and yearly Crontabs (Issue #1465). > Fix contributed by Guillaume Gauvrit. - Fixed memory leak caused by time limits (Issue #1129, Issue #1427) - Worker will now sleep if being restarted more than 5 times in one second to avoid spamming with `worker-online` events. - Includes documentation fixes > Contributed by: Ken Fromm, Andreas Savvides, Alex Kiriukha, > Michael Fladischer. ## 3.0.21 {#version-3.0.21} release-date : 2013-07-05 04:30 p.m. BST release-by : Ask Solem - Now depends on `billiard`{.interpreted-text role="pypi"} 2.7.3.31. > This version fixed a bug when running without the billiard C > extension. - 3.0.20 broke eventlet/gevent support (worker not starting). - Fixed memory leak problem when MongoDB result backend was used with the gevent pool. > Fix contributed by Ross Lawley. ## 3.0.20 {#version-3.0.20} release-date : 2013-06-28 04:00 p.m. BST release-by : Ask Solem - Contains workaround for deadlock problems. > A better solution will be part of Celery 3.1. - Now depends on `Kombu 2.5.12 `{.interpreted-text role="ref"}. - Now depends on `billiard`{.interpreted-text role="pypi"} 2.7.3.30. - `--loader `{.interpreted-text role="option"} argument no longer supported importing loaders from the current directory. - \[Worker\] Fixed memory leak when restarting after connection lost (Issue #1325). - \[Worker\] Fixed UnicodeDecodeError at start-up (Issue #1373). > Fix contributed by Jessica Tallon. - \[Worker\] Now properly rewrites unpickleable exceptions again. - Fixed possible race condition when evicting items from the revoked task set. - \[generic-init.d\] Fixed compatibility with Ubuntu\'s minimal Dash shell (Issue #1387). > Fix contributed by `monkut`{.interpreted-text role="github_user"}. - `Task.apply`/`ALWAYS_EAGER` now also executes callbacks and errbacks (Issue #1336). - \[Worker\] The `worker-shutdown`{.interpreted-text role="signal"} signal was no longer being dispatched (Issue #1339)j - \[Python 3\] Fixed problem with threading.Event. > Fix contributed by Xavier Ordoquy. - \[Python 3\] Now handles `io.UnsupportedOperation` that may be raised by `file.fileno()` in Python 3. - \[Python 3\] Fixed problem with `qualname`. - \[events.State\] Now ignores unknown event-groups. - \[MongoDB backend\] No longer uses deprecated `safe` parameter. > Fix contributed by `rfkrocktk`{.interpreted-text > role="github_user"}. - The eventlet pool now imports on Windows. - \[Canvas\] Fixed regression where immutable chord members may receive arguments (Issue #1340). > Fix contributed by Peter Brook. - \[Canvas\] chain now accepts generator argument again (Issue #1319). - `celery.migrate` command now consumes from all queues if no queues specified. > Fix contributed by John Watson. ## 3.0.19 {#version-3.0.19} release-date : 2013-04-17 04:30:00 p.m. BST release-by : Ask Solem - Now depends on `billiard`{.interpreted-text role="pypi"} 2.7.3.28 - A Python 3 related fix managed to disable the deadlock fix announced in 3.0.18. > Tests have been added to make sure this doesn\'t happen again. - Task retry policy: Default max_retries is now 3. > This ensures clients won\'t be hanging while the broker is down. > > ::: note > ::: title > Note > ::: > > You can set a longer retry for the worker by using the > `celeryd_after_setup`{.interpreted-text role="signal"} signal: > > ``` python > from celery.signals import celeryd_after_setup > > @celeryd_after_setup.connect > def configure_worker(instance, conf, **kwargs): > conf.CELERY_TASK_PUBLISH_RETRY_POLICY = { > 'max_retries': 100, > 'interval_start': 0, > 'interval_max': 1, > 'interval_step': 0.2, > } > ``` > ::: - Worker: Will now properly display message body in error messages even if the body is a buffer instance. - 3.0.18 broke the MongoDB result backend (Issue #1303). ## 3.0.18 {#version-3.0.18} release-date : 2013-04-12 05:00:00 p.m. BST release-by : Ask Solem - Now depends on `kombu`{.interpreted-text role="pypi"} 2.5.10. > See the `kombu changelog `{.interpreted-text > role="ref"}. - Now depends on `billiard`{.interpreted-text role="pypi"} 2.7.3.27. - Can now specify a white-list of accepted serializers using the new `CELERY_ACCEPT_CONTENT`{.interpreted-text role="setting"} setting. > This means that you can force the worker to discard messages > serialized with pickle and other untrusted serializers. For > example to only allow JSON serialized messages use: > > CELERY_ACCEPT_CONTENT = ['json'] > > you can also specify MIME types in the white-list: > > CELERY_ACCEPT_CONTENT = ['application/json'] - Fixed deadlock in multiprocessing\'s pool caused by the semaphore not being released when terminated by signal. - Processes Pool: It\'s now possible to debug pool processes using GDB. - `celery report` now censors possibly secret settings, like passwords and secret tokens. > You should still check the output before pasting anything on the > internet. - Connection URLs now ignore multiple \'+\' tokens. - Worker/`statedb`: Now uses pickle protocol 2 (Python 2.5+) - Fixed Python 3 compatibility issues. - Worker: A warning is now given if a worker is started with the same node name as an existing worker. - Worker: Fixed a deadlock that could occur while revoking tasks (Issue #1297). - Worker: The `HUP`{.interpreted-text role="sig"} handler now closes all open file descriptors before restarting to ensure file descriptors doesn\'t leak (Issue #1270). - Worker: Optimized storing/loading the revoked tasks list (Issue #1289). > After this change the `celery worker --statedb`{.interpreted-text > role="option"} file will take up more disk space, but loading from > and storing the revoked tasks will be considerably faster (what > before took 5 minutes will now take less than a second). - Celery will now suggest alternatives if there\'s a typo in the broker transport name (e.g., `ampq` -\> `amqp`). - Worker: The auto-reloader would cause a crash if a monitored file was unlinked. > Fix contributed by Agris Ameriks. - Fixed AsyncResult pickling error. > Fix contributed by Thomas Minor. - Fixed handling of Unicode in logging output when using log colors (Issue #427). - `~celery.app.utils.ConfigurationView`{.interpreted-text role="class"} is now a `MutableMapping`. > Contributed by Aaron Harnly. - Fixed memory leak in LRU cache implementation. > Fix contributed by Romuald Brunet. - `celery.contrib.rdb`: Now works when sockets are in non-blocking mode. > Fix contributed by Theo Spears. - The [inspect reserved]{.title-ref} remote control command included active (started) tasks with the reserved tasks (Issue #1030). - The `task_failure`{.interpreted-text role="signal"} signal received a modified traceback object meant for pickling purposes, this has been fixed so that it now receives the real traceback instead. - The `@task` decorator silently ignored positional arguments, it now raises the expected `TypeError`{.interpreted-text role="exc"} instead (Issue #1125). - The worker will now properly handle messages with invalid ETA/expires fields (Issue #1232). - The `pool_restart` remote control command now reports an error if the `CELERYD_POOL_RESTARTS`{.interpreted-text role="setting"} setting isn\'t set. - `@add_defaults`{.interpreted-text role="meth"}\` can now be used with non-dict objects. - Fixed compatibility problems in the Proxy class (Issue #1087). > The class attributes `__module__`, `__name__` and `__doc__` are > now meaningful string objects. > > Thanks to Marius Gedminas. - MongoDB Backend: The `MONGODB_BACKEND_SETTINGS`{.interpreted-text role="setting"} setting now accepts a `option` key that lets you forward arbitrary kwargs to the underlying `pymongo.Connection` object (Issue #1015). - Beat: The daily backend cleanup task is no longer enabled for result backends that support automatic result expiration (Issue #1031). - Canvas list operations now takes application instance from the first task in the list, instead of depending on the `current_app` (Issue #1249). - Worker: Message decoding error log message now includes traceback information. - Worker: The start-up banner now includes system platform. - `celery inspect|status|control` now gives an error if used with a SQL based broker transport. ## 3.0.17 {#version-3.0.17} release-date : 2013-03-22 04:00:00 p.m. UTC release-by : Ask Solem - Now depends on kombu 2.5.8 - Now depends on billiard 2.7.3.23 - RabbitMQ/Redis: thread-less and lock-free rate-limit implementation. > This means that rate limits pose minimal overhead when used with > RabbitMQ/Redis or future transports using the event-loop, and that > the rate-limit implementation is now thread-less and lock-free. > > The thread-based transports will still use the old implementation > for now, but the plan is to use the timer also for other broker > transports in Celery 3.1. - Rate limits now works with eventlet/gevent if using RabbitMQ/Redis as the broker. - A regression caused `task.retry` to ignore additional keyword arguments. > Extra keyword arguments are now used as execution options again. > Fix contributed by Simon Engledew. - Windows: Fixed problem with the worker trying to pickle the Django settings module at worker start-up. - generic-init.d: No longer double quotes `$CELERYD_CHDIR` (Issue #1235). - generic-init.d: Removes bash-specific syntax. > Fix contributed by Pär Wieslander. - Cassandra Result Backend: Now handles the `~pycassa.AllServersUnavailable`{.interpreted-text role="exc"} error (Issue #1010). > Fix contributed by Jared Biel. - Result: Now properly forwards apps to GroupResults when deserializing (Issue #1249). > Fix contributed by Charles-Axel Dein. - `GroupResult.revoke` now supports the `terminate` and `signal` keyword arguments. - Worker: Multiprocessing pool workers now import task modules/configuration before setting up the logging system so that logging signals can be connected before they\'re dispatched. - chord: The `AsyncResult` instance returned now has its `parent` attribute set to the header `GroupResult`. > This is consistent with how `chain` works. ## 3.0.16 {#version-3.0.16} release-date : 2013-03-07 04:00:00 p.m. UTC release-by : Ask Solem - Happy International Women\'s Day! > We have a long way to go, so this is a chance for you to get > involved in one of the organizations working for making our > communities more diverse. > > > - PyLadies --- > > - Girls Who Code --- > > - Women Who Code --- > > - Now depends on `kombu`{.interpreted-text role="pypi"} version 2.5.7 - Now depends on `billiard`{.interpreted-text role="pypi"} version 2.7.3.22 - AMQP heartbeats are now disabled by default. > Some users experiences issues with heartbeats enabled, and it\'s > not strictly necessary to use them. > > If you\'re experiencing problems detecting connection failures, > you can re-enable heartbeats by configuring the > `BROKER_HEARTBEAT`{.interpreted-text role="setting"} setting. - Worker: Now propagates connection errors occurring in multiprocessing callbacks, so that the connection can be reset (Issue #1226). - Worker: Now propagates connection errors occurring in timer callbacks, so that the connection can be reset. - The modules in `CELERY_IMPORTS`{.interpreted-text role="setting"} and `CELERY_INCLUDE`{.interpreted-text role="setting"} are now imported in the original order (Issue #1161). > The modules in `CELERY_IMPORTS`{.interpreted-text role="setting"} > will be imported first, then continued by > `CELERY_INCLUDE`{.interpreted-text role="setting"}. > > Thanks to Joey Wilhelm. - New bash completion for `celery` available in the git repository: > > > You can source this file or put it in `bash_completion.d` to get > auto-completion for the `celery` command-line utility. - The node name of a worker can now include unicode characters (Issue #1186). - The repr of a `crontab` object now displays correctly (Issue #972). - `events.State` no longer modifies the original event dictionary. - No longer uses `Logger.warn` deprecated in Python 3. - Cache Backend: Now works with chords again (Issue #1094). - Chord unlock now handles errors occurring while calling the callback. - Generic worker init.d script: Status check is now performed by querying the pid of the instance instead of sending messages. > Contributed by Milen Pavlov. - Improved init-scripts for CentOS. > - Updated to support Celery 3.x conventions. > - Now uses CentOS built-in `status` and `killproc` > - Support for multi-node / multi-pid worker services. > - Standard color-coded CentOS service-init output. > - A test suite. > > Contributed by Milen Pavlov. - `ResultSet.join` now always works with empty result set (Issue #1219). - A `group` consisting of a single task is now supported (Issue #1219). - Now supports the `pycallgraph` program (Issue #1051). - Fixed Jython compatibility problems. - Django tutorial: Now mentions that the example app must be added to `INSTALLED_APPS` (Issue #1192). ## 3.0.15 {#version-3.0.15} release-date : 2013-02-11 04:30:00 p.m. UTC release-by : Ask Solem - Now depends on billiard 2.7.3.21 which fixed a syntax error crash. - Fixed bug with `CELERY_SEND_TASK_SENT_EVENT`{.interpreted-text role="setting"}. ## 3.0.14 {#version-3.0.14} release-date : 2013-02-08 05:00:00 p.m. UTC release-by : Ask Solem - Now depends on Kombu 2.5.6 - Now depends on billiard 2.7.3.20 - `execv` is now disabled by default. > It was causing too many problems for users, you can still enable > it using the [CELERYD_FORCE_EXECV]{.title-ref} setting. > > execv was only enabled when transports other than AMQP/Redis was > used, and it\'s there to prevent deadlocks caused by mutexes not > being released before the process forks. Unfortunately it also > changes the environment introducing many corner case bugs that\'re > hard to fix without adding horrible hacks. Deadlock issues are > reported far less often than the bugs that execv are causing, so > we now disable it by default. > > Work is in motion to create non-blocking versions of these > transports so that execv isn\'t necessary (which is the situation > with the amqp and redis broker transports) - Chord exception behavior defined (Issue #1172). > From Celery 3.1 the chord callback will change state to FAILURE > when a task part of a chord raises an exception. > > It was never documented what happens in this case, and the actual > behavior was very unsatisfactory, indeed it will just forward the > exception value to the chord callback. > > For backward compatibility reasons we don\'t change to the new > behavior in a bugfix release, even if the current behavior was > never documented. Instead you can enable the > `CELERY_CHORD_PROPAGATES`{.interpreted-text role="setting"} > setting to get the new behavior that\'ll be default from Celery > 3.1. > > See more at `chord-errors`{.interpreted-text role="ref"}. - worker: Fixes bug with ignored and retried tasks. > The `on_chord_part_return` and `Task.after_return` callbacks, nor > the `task_postrun` signal should be called when the task was > retried/ignored. > > Fix contributed by Vlad. - `GroupResult.join_native` now respects the `propagate` argument. - `subtask.id` added as an alias to `subtask['options'].id` > ``` pycon > >>> s = add.s(2, 2) > >>> s.id = 'my-id' > >>> s['options'] > {'task_id': 'my-id'} > > >>> s.id > 'my-id' > ``` - worker: Fixed error [Could not start worker processes]{.title-ref} occurring when restarting after connection failure (Issue #1118). - Adds new signal `task-retried`{.interpreted-text role="signal"} (Issue #1169). - [celery events \--dumper]{.title-ref} now handles connection loss. - Will now retry sending the task-sent event in case of connection failure. - amqp backend: Now uses `Message.requeue` instead of republishing the message after poll. - New `BROKER_HEARTBEAT_CHECKRATE`{.interpreted-text role="setting"} setting introduced to modify the rate at which broker connection heartbeats are monitored. > The default value was also changed from 3.0 to 2.0. - `celery.events.state.State`{.interpreted-text role="class"} is now pickleable. > Fix contributed by Mher Movsisyan. - `celery.utils.functional.LRUCache`{.interpreted-text role="class"} is now pickleable. > Fix contributed by Mher Movsisyan. - The stats broadcast command now includes the workers pid. > Contributed by Mher Movsisyan. - New `conf` remote control command to get a workers current configuration. > Contributed by Mher Movsisyan. - Adds the ability to modify the chord unlock task\'s countdown argument (Issue #1146). > Contributed by Jun Sakai - beat: The scheduler now uses the [now()]{.title-ref}\` method of the schedule, so that schedules can provide a custom way to get the current date and time. > Contributed by Raphaël Slinckx - Fixed pickling of configuration modules on Windows or when execv is used (Issue #1126). - Multiprocessing logger is now configured with loglevel `ERROR` by default. > Since 3.0 the multiprocessing loggers were disabled by default > (only configured when the `MP_LOG`{.interpreted-text > role="envvar"} environment variable was set). ## 3.0.13 {#version-3.0.13} release-date : 2013-01-07 04:00:00 p.m. UTC release-by : Ask Solem - Now depends on Kombu 2.5 > - `amqp`{.interpreted-text role="pypi"} has replaced > `amqplib`{.interpreted-text role="pypi"} as the default > transport, gaining support for AMQP 0.9, and the RabbitMQ > extensions, including Consumer Cancel Notifications and > heartbeats. > - support for multiple connection URLs for failover. > - Read more in the > `Kombu 2.5 changelog `{.interpreted-text > role="ref"}. - Now depends on billiard 2.7.3.19 - Fixed a deadlock issue that could occur when the producer pool inherited the connection pool instance of the parent process. - The `--loader `{.interpreted-text role="option"} option now works again (Issue #1066). - `celery`{.interpreted-text role="program"} umbrella command: All sub-commands now supports the `--workdir `{.interpreted-text role="option"} option (Issue #1063). - Groups included in chains now give GroupResults (Issue #1057) > Previously it would incorrectly add a regular result instead of a > group result, but now this works: > > ``` pycon > >>> # [4 + 4, 4 + 8, 16 + 8] > >>> res = (add.s(2, 2) | group(add.s(4), add.s(8), add.s(16)))() > >>> res > 4346501c-cb99-4ad8-8577-12256c7a22b1, > b12ead10-a622-4d44-86e9-3193a778f345, > 26c7a420-11f3-4b33-8fac-66cd3b62abfd]> > ``` - Chains can now chain other chains and use partial arguments (Issue #1057). > Example: > > ``` pycon > >>> c1 = (add.s(2) | add.s(4)) > >>> c2 = (add.s(8) | add.s(16)) > > >>> c3 = (c1 | c2) > > >>> # 8 + 2 + 4 + 8 + 16 > >>> assert c3(8).get() == 38 > ``` - Subtasks can now be used with unregistered tasks. > You can specify subtasks even if you just have the name: > > >>> s = subtask(task_name, args=(), kwargs=()) > >>> s.delay() - The `celery shell`{.interpreted-text role="program"} command now always adds the current directory to the module path. - The worker will now properly handle the `pytz.AmbiguousTimeError`{.interpreted-text role="exc"} exception raised when an ETA/countdown is prepared while being in DST transition (Issue #1061). - force_execv: Now makes sure that task symbols in the original task modules will always use the correct app instance (Issue #1072). - AMQP Backend: Now republishes result messages that have been polled (using `result.ready()` and friends, `result.get()` won\'t do this in this version). - Crontab schedule values can now \"wrap around\" > This means that values like `11-1` translates to `[11, 12, 1]`. > > Contributed by Loren Abrams. - `multi stopwait` command now shows the pid of processes. > Contributed by Loren Abrams. - Handling of ETA/countdown fixed when the `CELERY_ENABLE_UTC`{.interpreted-text role="setting"} : setting is disabled (Issue #1065). - A number of unneeded properties were included in messages, caused by accidentally passing `Queue.as_dict` as message properties. - Rate limit values can now be float > This also extends the string format so that values like `"0.5/s"` > works. > > Contributed by Christoph Krybus - Fixed a typo in the broadcast routing documentation (Issue #1026). - Rewrote confusing section about idempotence in the task user guide. - Fixed typo in the daemonization tutorial (Issue #1055). - Fixed several typos in the documentation. > Contributed by Marius Gedminas. - Batches: Now works when using the eventlet pool. > Fix contributed by Thomas Grainger. - Batches: Added example sending results to `celery.contrib.batches`. > Contributed by Thomas Grainger. - MongoDB backend: Connection `max_pool_size` can now be set in `CELERY_MONGODB_BACKEND_SETTINGS`{.interpreted-text role="setting"}. > Contributed by Craig Younkins. - Fixed problem when using earlier versions of `pytz`{.interpreted-text role="pypi"}. > Fix contributed by Vlad. - Docs updated to include the default value for the `CELERY_TASK_RESULT_EXPIRES`{.interpreted-text role="setting"} setting. - Improvements to the `django-celery`{.interpreted-text role="pypi"} tutorial. > Contributed by Locker537. - The `add_consumer` control command didn\'t properly persist the addition of new queues so that they survived connection failure (Issue #1079). ## 3.0.12 release-date : 2012-11-06 02:00 p.m. UTC release-by : Ask Solem - Now depends on kombu 2.4.8 > - \[Redis\] New and improved fair queue cycle algorithm (Kevin > McCarthy). > > - \[Redis\] Now uses a Redis-based mutex when restoring > messages. > > - > > \[Redis\] Number of messages that can be restored in one interval is no > > : longer limited (but can be set using the > `unacked_restore_limit` > `transport option `{.interpreted-text > role="setting"}). > > - Heartbeat value can be specified in broker URLs (Mher > Movsisyan). > > - Fixed problem with msgpack on Python 3 (Jasper Bryant-Greene). - Now depends on billiard 2.7.3.18 - Celery can now be used with static analysis tools like PyDev/PyCharm/pylint etc. - Development documentation has moved to Read The Docs. > The new URL is: - New `CELERY_QUEUE_HA_POLICY`{.interpreted-text role="setting"} setting used to set the default HA policy for queues when using RabbitMQ. - New method `Task.subtask_from_request` returns a subtask using the current request. - Results get_many method didn\'t respect timeout argument. > Fix contributed by Remigiusz Modrzejewski - generic_init.d scripts now support setting `CELERY_CREATE_DIRS`{.interpreted-text role="envvar"} to always create log and pid directories (Issue #1045). > This can be set in your `/etc/default/celeryd`{.interpreted-text > role="file"}. - Fixed strange kombu import problem on Python 3.2 (Issue #1034). - Worker: ETA scheduler now uses millisecond precision (Issue #1040). - The `--config `{.interpreted-text role="option"} argument to programs is now supported by all loaders. - The `CASSANDRA_OPTIONS`{.interpreted-text role="setting"} setting has now been documented. > Contributed by Jared Biel. - Task methods (`celery.contrib.methods`{.interpreted-text role="mod"}) cannot be used with the old task base class, the task decorator in that module now inherits from the new. - An optimization was too eager and caused some logging messages to never emit. - `celery.contrib.batches` now works again. - Fixed missing white-space in `bdist_rpm` requirements (Issue #1046). - Event state\'s `tasks_by_name` applied limit before filtering by name. > Fix contributed by Alexander A. Sosnovskiy. ## 3.0.11 {#version-3.0.11} release-date : 2012-09-26 04:00 p.m. UTC release-by : Ask Solem - \[security:low\] generic-init.d scripts changed permissions of /var/log & /var/run > In the daemonization tutorial the recommended directories were as > follows: > > ``` bash > CELERYD_LOG_FILE="/var/log/celery/%n.log" > CELERYD_PID_FILE="/var/run/celery/%n.pid" > ``` > > But in the scripts themselves the default files were > `/var/log/celery%n.log` and `/var/run/celery%n.pid`, so if the > user didn\'t change the location by configuration, the directories > `/var/log` and `/var/run` would be created - and worse have their > permissions and owners changed. > > This change means that: > > > - Default pid file is `/var/run/celery/%n.pid` > > - Default log file is `/var/log/celery/%n.log` > > - The directories are only created and have their permissions > > changed if *no custom locations are set*. > > Users can force paths to be created by calling the `create-paths` > sub-command: > > ``` console > $ sudo /etc/init.d/celeryd create-paths > ``` > > ::: admonition > Upgrading Celery won\'t update init-scripts > > To update the init-scripts you have to re-download the files from > source control and update them manually. You can find the > init-scripts for version 3.0.x at: > > > > ::: - Now depends on billiard 2.7.3.17 - Fixes request stack protection when app is initialized more than once (Issue #1003). - ETA tasks now properly works when system timezone isn\'t same as the configured timezone (Issue #1004). - Terminating a task now works if the task has been sent to the pool but not yet acknowledged by a pool process (Issue #1007). > Fix contributed by Alexey Zatelepin - Terminating a task now properly updates the state of the task to revoked, and sends a `task-revoked` event. - Generic worker init-script now waits for workers to shutdown by default. - Multi: No longer parses \--app option (Issue #1008). - Multi: `stop_verify` command renamed to `stopwait`. - Daemonization: Now delays trying to create pidfile/logfile until after the working directory has been changed into. - `celery worker`{.interpreted-text role="program"} and `celery beat`{.interpreted-text role="program"} commands now respects the `--no-color `{.interpreted-text role="option"} option (Issue #999). - Fixed typos in eventlet examples (Issue #1000) > Fix contributed by Bryan Bishop. Congratulations on opening bug > #1000! - Tasks that raise `~celery.exceptions.Ignore`{.interpreted-text role="exc"} are now acknowledged. - Beat: Now shows the name of the entry in `sending due task` logs. ## 3.0.10 {#version-3.0.10} release-date : 2012-09-20 05:30 p.m. BST release-by : Ask Solem - Now depends on kombu 2.4.7 - Now depends on billiard 2.7.3.14 > - Fixes crash at start-up when using Django and pre-1.4 projects > (`setup_environ`). > > - Hard time limits now sends the KILL signal shortly after TERM, > to terminate processes that have signal handlers blocked by C > extensions. > > - Billiard now installs even if the C extension cannot be built. > > > It\'s still recommended to build the C extension if you\'re > > using a transport other than RabbitMQ/Redis (or use forced > > execv for some other reason). > > - Pool now sets a `current_process().index` attribute that can > be used to create as many log files as there are processes in > the pool. - Canvas: chord/group/chain no longer modifies the state when called > Previously calling a chord/group/chain would modify the ids of > subtasks so that: > > ``` pycon > >>> c = chord([add.s(2, 2), add.s(4, 4)], xsum.s()) > >>> c() > >>> c() <-- call again > ``` > > at the second time the ids for the tasks would be the same as in > the previous invocation. This is now fixed, so that calling a > subtask won\'t mutate any options. - Canvas: Chaining a chord to another task now works (Issue #965). - Worker: Fixed a bug where the request stack could be corrupted if relative imports are used. > Problem usually manifested itself as an exception while trying to > send a failed task result (`NoneType does not have id attribute`). > > Fix contributed by Sam Cooke. - Tasks can now raise `~celery.exceptions.Ignore`{.interpreted-text role="exc"} to skip updating states or events after return. > Example: > > ``` python > from celery.exceptions import Ignore > > @task > def custom_revokes(): > if redis.sismember('tasks.revoked', custom_revokes.request.id): > raise Ignore() > ``` - The worker now makes sure the request/task stacks aren\'t modified by the initial `Task.__call__`. > This would previously be a problem if a custom task class defined > `__call__` and also called `super()`. - Because of problems the fast local optimization has been disabled, and can only be enabled by setting the `USE_FAST_LOCALS`{.interpreted-text role="envvar"} attribute. - Worker: Now sets a default socket timeout of 5 seconds at shutdown so that broken socket reads don\'t hinder proper shutdown (Issue #975). - More fixes related to late eventlet/gevent patching. - Documentation for settings out of sync with reality: > - `CELERY_TASK_PUBLISH_RETRY`{.interpreted-text role="setting"} > > > Documented as disabled by default, but it was enabled by > > default since 2.5 as stated by the 2.5 changelog. > > - `CELERY_TASK_PUBLISH_RETRY_POLICY`{.interpreted-text > role="setting"} > > > The default max_retries had been set to 100, but documented > > as being 3, and the interval_max was set to 1 but documented > > as 0.2. The default setting are now set to 3 and 0.2 as it > > was originally documented. > > Fix contributed by Matt Long. - Worker: Log messages when connection established and lost have been improved. - The repr of a Crontab schedule value of \'0\' should be \'\*\' (Issue #972). - Revoked tasks are now removed from reserved/active state in the worker (Issue #969) > Fix contributed by Alexey Zatelepin. - gevent: Now supports hard time limits using `gevent.Timeout`. - Documentation: Links to init-scripts now point to the 3.0 branch instead of the development branch (master). - Documentation: Fixed typo in signals user guide (Issue #986). > `instance.app.queues` -\> `instance.app.amqp.queues`. - Eventlet/gevent: The worker didn\'t properly set the custom app for new greenlets. - Eventlet/gevent: Fixed a bug where the worker could not recover from connection loss (Issue #959). > Also, because of a suspected bug in gevent the > `BROKER_CONNECTION_TIMEOUT`{.interpreted-text role="setting"} > setting has been disabled when using gevent ## 3.0.9 release-date : 2012-08-31 06:00 p.m. BST release-by : Ask Solem - Important note for users of Django and the database scheduler! > Recently a timezone issue has been fixed for periodic tasks, but > erroneous timezones could have already been stored in the > database, so for the fix to work you need to reset the > `last_run_at` fields. > > You can do this by executing the following command: > > ``` console > $ python manage.py shell > >>> from djcelery.models import PeriodicTask > >>> PeriodicTask.objects.update(last_run_at=None) > ``` > > You also have to do this if you change the timezone or > `CELERY_ENABLE_UTC`{.interpreted-text role="setting"} setting. - Note about the `CELERY_ENABLE_UTC`{.interpreted-text role="setting"} setting. > If you previously disabled this just to force periodic tasks to > work with your timezone, then you\'re now *encouraged to re-enable > it*. - Now depends on Kombu 2.4.5 which fixes PyPy + Jython installation. - Fixed bug with timezones when `CELERY_ENABLE_UTC`{.interpreted-text role="setting"} is disabled (Issue #952). - Fixed a typo in the `celerybeat` upgrade mechanism (Issue #951). - Make sure the [exc_info]{.title-ref} argument to logging is resolved (Issue #899). - Fixed problem with Python 3.2 and thread join timeout overflow (Issue #796). - A test case was occasionally broken for Python 2.5. - Unit test suite now passes for PyPy 1.9. - App instances now supports the `with`{.interpreted-text role="keyword"} statement. > This calls the new `@close`{.interpreted-text role="meth"} method > at exit, which cleans up after the app like closing pool > connections. > > Note that this is only necessary when dynamically creating apps, > for example \"temporary\" apps. - Support for piping a subtask to a chain. > For example: > > ``` python > pipe = sometask.s() | othertask.s() > new_pipe = mytask.s() | pipe > ``` > > Contributed by Steve Morin. - Fixed problem with group results on non-pickle serializers. > Fix contributed by Steeve Morin. ## 3.0.8 {#version-3.0.8} release-date : 2012-08-29 05:00 p.m. BST release-by : Ask Solem - Now depends on Kombu 2.4.4 - Fixed problem with `amqplib`{.interpreted-text role="pypi"} and receiving larger message payloads (Issue #922). > The problem would manifest itself as either the worker hanging, or > occasionally a `Framing error` exception appearing. > > Users of the new `pyamqp://` transport must upgrade to > `amqp`{.interpreted-text role="pypi"} 0.9.3. - Beat: Fixed another timezone bug with interval and Crontab schedules (Issue #943). - Beat: The schedule file is now automatically cleared if the timezone is changed. > The schedule is also cleared when you upgrade to 3.0.8 from an > earlier version, this to register the initial timezone info. - Events: The `worker-heartbeat`{.interpreted-text role="event"} event now include processed and active count fields. > Contributed by Mher Movsisyan. - Fixed error with error email and new task classes (Issue #931). - `BaseTask.__call__` is no longer optimized away if it has been monkey patched. - Fixed shutdown issue when using gevent (Issue #911 & Issue #936). > Fix contributed by Thomas Meson. ## 3.0.7 {#version-3.0.7} release-date : 2012-08-24 05:00 p.m. BST release-by : Ask Solem - Fixes several problems with periodic tasks and timezones (Issue #937). - Now depends on kombu 2.4.2 > - Redis: Fixes a race condition crash > - Fixes an infinite loop that could happen when retrying > establishing the broker connection. - Daemons now redirect standard file descriptors to `/dev/null`{.interpreted-text role="file"} > Though by default the standard outs are also redirected to the > logger instead, but you can disable this by changing the > `CELERY_REDIRECT_STDOUTS`{.interpreted-text role="setting"} > setting. - Fixes possible problems when eventlet/gevent is patched too late. - `LoggingProxy` no longer defines `fileno()` (Issue #928). - Results are now ignored for the chord unlock task. > Fix contributed by Steeve Morin. - Cassandra backend now works if result expiry is disabled. > Fix contributed by Steeve Morin. - The traceback object is now passed to signal handlers instead of the string representation. > Fix contributed by Adam DePue. - Celery command: Extensions are now sorted by name. - A regression caused the `task-failed`{.interpreted-text role="event"} event to be sent with the exception object instead of its string representation. - The worker daemon would try to create the pid file before daemonizing to catch errors, but this file wasn\'t immediately released (Issue #923). - Fixes Jython compatibility. - `billiard.forking_enable` was called by all pools not just the processes pool, which would result in a useless warning if the billiard C extensions weren\'t installed. ## 3.0.6 {#version-3.0.6} release-date : 2012-08-17 11:00 p.mp.m. Ask Solem - Now depends on kombu 2.4.0 - Now depends on billiard 2.7.3.12 - Redis: Celery now tries to restore messages whenever there are no messages in the queue. - Crontab schedules now properly respects `CELERY_TIMEZONE`{.interpreted-text role="setting"} setting. > It\'s important to note that Crontab schedules uses UTC time by > default unless this setting is set. > > Issue #904 and `django-celery`{.interpreted-text role="pypi"} > #150. - `billiard.enable_forking` is now only set by the processes pool. - The transport is now properly shown by `celery report`{.interpreted-text role="program"} (Issue #913). - The [\--app]{.title-ref} argument now works if the last part is a module name (Issue #921). - Fixed problem with unpickleable exceptions (billiard #12). - Adds `task_name` attribute to `EagerResult` which is always `None`{.interpreted-text role="const"} (Issue #907). - Old Task class in `celery.task`{.interpreted-text role="mod"} no longer accepts magic kwargs by default (Issue #918). > A regression long ago disabled magic kwargs for these, and since > no one has complained about it we don\'t have any incentive to fix > it now. - The `inspect reserved` control command didn\'t work properly. - Should now play better with tools for static analysis by explicitly specifying dynamically created attributes in the `celery`{.interpreted-text role="mod"} and `celery.task`{.interpreted-text role="mod"} modules. - Terminating a task now results in `~celery.exceptions.RevokedTaskError`{.interpreted-text role="exc"} instead of a `WorkerLostError`. - `AsyncResult.revoke` now accepts `terminate` and `signal` arguments. - The `task-revoked`{.interpreted-text role="event"} event now includes new fields: `terminated`, `signum`, and `expired`. - The argument to `~celery.exceptions.TaskRevokedError`{.interpreted-text role="class"} is now one of the reasons `revoked`, `expired` or `terminated`. - Old Task class does no longer use `classmethod`{.interpreted-text role="class"} for `push_request` and `pop_request` (Issue #912). - `GroupResult` now supports the `children` attribute (Issue #916). - `AsyncResult.collect` now respects the `intermediate` argument (Issue #917). - Fixes example task in documentation (Issue #902). - Eventlet fixed so that the environment is patched as soon as possible. - eventlet: Now warns if Celery related modules that depends on threads are imported before eventlet is patched. - Improved event and camera examples in the monitoring guide. - Disables celery command setuptools entry-points if the command can\'t be loaded. - Fixed broken `dump_request` example in the tasks guide. ## 3.0.5 {#version-3.0.5} release-date : 2012-08-01 04:00 p.m. BST release-by : Ask Solem - Now depends on kombu 2.3.1 + billiard 2.7.3.11 - Fixed a bug with the -B option (`cannot pickle thread.lock objects`) (Issue #894 + Issue #892, + `django-celery`{.interpreted-text role="pypi"} #154). - The `restart_pool`{.interpreted-text role="control"} control command now requires the `CELERYD_POOL_RESTARTS`{.interpreted-text role="setting"} setting to be enabled > This change was necessary as the multiprocessing event that the > restart command depends on is responsible for creating many > semaphores/file descriptors, resulting in problems in some > environments. - `chain.apply` now passes args to the first task (Issue #889). - Documented previously secret options to the `django-celery`{.interpreted-text role="pypi"} monitor in the monitoring user guide (Issue #396). - Old changelog are now organized in separate documents for each series, see `history`{.interpreted-text role="ref"}. ## 3.0.4 {#version-3.0.4} release-date : 2012-07-26 07:00 p.m. BST release-by : Ask Solem - Now depends on Kombu 2.3 - New experimental standalone Celery monitor: Flower > See `monitoring-flower`{.interpreted-text role="ref"} to read more > about it! > > Contributed by Mher Movsisyan. - Now supports AMQP heartbeats if using the new `pyamqp://` transport. > - The `amqp`{.interpreted-text role="pypi"} transport requires > the `amqp`{.interpreted-text role="pypi"} library to be > installed: > > > ``` console > > $ pip install amqp > > ``` > > - Then you need to set the transport URL prefix to `pyamqp://`. > > - The default heartbeat value is 10 seconds, but this can be > changed using the `BROKER_HEARTBEAT`{.interpreted-text > role="setting"} setting: > > BROKER_HEARTBEAT = 5.0 > > - If the broker heartbeat is set to 10 seconds, the heartbeats > will be monitored every 5 seconds (double the heartbeat rate). > > See the > `Kombu 2.3 changelog `{.interpreted-text > role="ref"} for more information. - Now supports RabbitMQ Consumer Cancel Notifications, using the `pyamqp://` transport. > This is essential when running RabbitMQ in a cluster. > > See the > `Kombu 2.3 changelog `{.interpreted-text > role="ref"} for more information. - Delivery info is no longer passed directly through. > It was discovered that the SQS transport adds objects that can\'t > be pickled to the delivery info mapping, so we had to go back to > using the white-list again. > > Fixing this bug also means that the SQS transport is now working > again. - The semaphore wasn\'t properly released when a task was revoked (Issue #877). > This could lead to tasks being swallowed and not released until a > worker restart. > > Thanks to Hynek Schlawack for debugging the issue. - Retrying a task now also forwards any linked tasks. > This means that if a task is part of a chain (or linked in some > other way) and that even if the task is retried, then the next > task in the chain will be executed when the retry succeeds. - Chords: Now supports setting the interval and other keyword arguments to the chord unlock task. > - The interval can now be set as part of the chord subtasks > kwargs: > > chord(header)(body, interval=10.0) > > - In addition the chord unlock task now honors the > Task.default_retry_delay option, used when none is specified, > which also means that the default interval can also be changed > using annotations: > > > ``` python > > CELERY_ANNOTATIONS = { > > 'celery.chord_unlock': { > > 'default_retry_delay': 10.0, > > } > > } > > ``` - New `@add_defaults`{.interpreted-text role="meth"} method can add new default configuration dictionaries to the applications configuration. > For example: > > config = {'FOO': 10} > > app.add_defaults(config) > > is the same as `app.conf.update(config)` except that data won\'t > be copied, and that it won\'t be pickled when the worker spawns > child processes. > > In addition the method accepts a callable: > > def initialize_config(): > # insert heavy stuff that can't be done at import time here. > > app.add_defaults(initialize_config) > > which means the same as the above except that it won\'t happen > until the Celery configuration is actually used. > > As an example, Celery can lazily use the configuration of a Flask > app: > > flask_app = Flask() > app = Celery() > app.add_defaults(lambda: flask_app.config) - Revoked tasks weren\'t marked as revoked in the result backend (Issue #871). > Fix contributed by Hynek Schlawack. - Event-loop now properly handles the case when the `epoll`{.interpreted-text role="manpage"} poller object has been closed (Issue #882). - Fixed syntax error in `funtests/test_leak.py` > Fix contributed by Catalin Iacob. - group/chunks: Now accepts empty task list (Issue #873). - New method names: > - `Celery.default_connection()` ➠ > `~@connection_or_acquire`{.interpreted-text role="meth"}. > - `Celery.default_producer()` ➠ > `~@producer_or_acquire`{.interpreted-text role="meth"}. > > The old names still work for backward compatibility. ## 3.0.3 {#version-3.0.3} release-date : 2012-07-20 09:17 p.m. BST release-by : Ask Solem - `amqplib`{.interpreted-text role="pypi"} passes the channel object as part of the delivery_info and it\'s not pickleable, so we now remove it. ## 3.0.2 {#version-3.0.2} release-date : 2012-07-20 04:00 p.m. BST release-by : Ask Solem - A bug caused the following task options to not take defaults from the : configuration (Issue #867 + Issue #858) > The following settings were affected: > > - `CELERY_IGNORE_RESULT`{.interpreted-text role="setting"} > - `CELERYD_SEND_TASK_ERROR_EMAILS`{.interpreted-text > role="setting"} > - `CELERY_TRACK_STARTED`{.interpreted-text role="setting"} > - `CElERY_STORE_ERRORS_EVEN_IF_IGNORED`{.interpreted-text > role="setting"} > > Fix contributed by John Watson. - Task Request: `delivery_info` is now passed through as-is (Issue #807). - The ETA argument now supports datetime\'s with a timezone set (Issue #855). - The worker\'s banner displayed the autoscale settings in the wrong order (Issue #859). - Extension commands are now loaded after concurrency is set up so that they don\'t interfere with things like eventlet patching. - Fixed bug in the threaded pool (Issue #863) - The task failure handler mixed up the fields in `sys.exc_info`{.interpreted-text role="func"}. > Fix contributed by Rinat Shigapov. - Fixed typos and wording in the docs. > Fix contributed by Paul McMillan - New setting: `CELERY_WORKER_DIRECT`{.interpreted-text role="setting"} > If enabled each worker will consume from their own dedicated queue > which can be used to route tasks to specific workers. - Fixed several edge case bugs in the add consumer remote control command. - `~celery.contrib.migrate`{.interpreted-text role="mod"}: Can now filter and move tasks to specific workers if `CELERY_WORKER_DIRECT`{.interpreted-text role="setting"} is enabled. > Among other improvements, the following functions have been added: > > > - `move_direct(filterfun, **opts)` > > - `move_direct_by_id(task_id, worker_hostname, **opts)` > > - `move_direct_by_idmap({task_id: worker_hostname, ...}, **opts)` > > - `move_direct_by_taskmap({task_name: worker_hostname, ...}, **opts)` - `~celery.Celery.default_connection`{.interpreted-text role="meth"} now accepts a pool argument that if set to false causes a new connection to be created instead of acquiring one from the pool. - New signal: `celeryd_after_setup`{.interpreted-text role="signal"}. - Default loader now keeps lowercase attributes from the configuration module. ## 3.0.1 {#version-3.0.1} release-date : 2012-07-10 06:00 p.m. BST release-by : Ask Solem - Now depends on kombu 2.2.5 - inspect now supports limit argument: myapp.control.inspect(limit=1).ping() - Beat: now works with timezone aware datetime\'s. - Task classes inheriting `from celery import Task` mistakenly enabled `accept_magic_kwargs`. - Fixed bug in `inspect scheduled` (Issue #829). - Beat: Now resets the schedule to upgrade to UTC. - The `celery worker`{.interpreted-text role="program"} command now works with eventlet/gevent. > Previously it wouldn\'t patch the environment early enough. - The `celery`{.interpreted-text role="program"} command now supports extension commands using setuptools entry-points. > Libraries can add additional commands to the > `celery`{.interpreted-text role="program"} command by adding an > entry-point like: > > setup( > entry_points=[ > 'celery.commands': [ > 'foo = my.module:Command', > ], > ], > ...) > > The command must then support the interface of > `celery.bin.base.Command`{.interpreted-text role="class"}. - contrib.migrate: New utilities to move tasks from one queue to another. > - `~celery.contrib.migrate.move_tasks`{.interpreted-text > role="func"} > - `~celery.contrib.migrate.move_task_by_id`{.interpreted-text > role="func"} - The `task-sent`{.interpreted-text role="event"} event now contains `exchange` and `routing_key` fields. - Fixes bug with installing on Python 3. > Fix contributed by Jed Smith. ## 3.0.0 (Chiastic Slide) {#version-3.0.0} release-date : 2012-07-07 01:30 p.m. BST release-by : Ask Solem See `whatsnew-3.0`{.interpreted-text role="ref"}. --- # Change history {#changelog-3.1} This document contains change notes for bugfix releases in the 3.1.x series (Cipater), please see `whatsnew-3.1`{.interpreted-text role="ref"} for an overview of what\'s new in Celery 3.1. ## 3.1.26 {#version-3.1.26} release-date : 2018-23-03 16:00 PM IST release-by : Omer Katz - Fixed a crash caused by tasks cycling between Celery 3 and Celery 4 workers. ## 3.1.25 {#version-3.1.25} release-date : 2016-10-10 12:00 PM PDT release-by : Ask Solem - **Requirements** > - Now depends on > `Kombu 3.0.37 `{.interpreted-text > role="ref"} - Fixed problem with chords in group introduced in 3.1.24 (Issue #3504). ## 3.1.24 {#version-3.1.24} release-date : 2016-09-30 04:21 PM PDT release-by : Ask Solem - **Requirements** > - Now depends on > `Kombu 3.0.36 `{.interpreted-text > role="ref"}. - Now supports Task protocol 2 from the future 4.0 release. > Workers running 3.1.24 are now able to process messages sent using > the [new task message > protocol](https://docs.celeryq.dev/en/master/internals/protocol.html#version-2) > to be introduced in Celery 4.0. > > Users upgrading to Celery 4.0 when this is released are encouraged > to upgrade to this version as an intermediate step, as this means > workers not yet upgraded will be able to process messages from > clients/workers running 4.0. - `Task.send_events` can now be set to disable sending of events for that task only. > Example when defining the task: > > ``` python > @app.task(send_events=False) > def add(x, y): > return x + y > ``` - **Utils**: Fixed compatibility with recent `psutil`{.interpreted-text role="pypi"} versions (Issue #3262). - **Canvas**: Chord now forwards partial arguments to its subtasks. > Fix contributed by Tayfun Sen. - **App**: Arguments to app such as `backend`, `broker`, etc are now pickled and sent to the child processes on Windows. > Fix contributed by Jeremy Zafran. - **Deployment**: Generic init scripts now supports being symlinked in runlevel directories (Issue #3208). - **Deployment**: Updated CentOS scripts to work with CentOS 7. > Contributed by Joe Sanford. - **Events**: The curses monitor no longer crashes when the result of a task is empty. > Fix contributed by Dongweiming. - **Worker**: `repr(worker)` would crash when called early in the startup process (Issue #2514). - **Tasks**: GroupResult now defines \_\_bool\_\_ and \_\_nonzero\_\_. > This is to fix an issue where a ResultSet or GroupResult with an > empty result list are not properly tupled with the as_tuple() > method when it is a parent result. This is due to the as_tuple() > method performing a logical and operation on the ResultSet. > > Fix contributed by Colin McIntosh. - **Worker**: Fixed wrong values in autoscale related logging message. > Fix contributed by `@raducc`. - Documentation improvements by > - Alexandru Chirila > - Michael Aquilina > - Mikko Ekström > - Mitchel Humpherys > - Thomas A. Neil > - Tiago Moreira Vieira > - Yuriy Syrovetskiy > - `@dessant` ## 3.1.23 {#version-3.1.23} release-date : 2016-03-09 06:00 P.M PST release-by : Ask Solem - **Programs**: Last release broke support for the `--hostnmame` argument to `celery multi`{.interpreted-text role="program"} and `celery worker --detach`{.interpreted-text role="program"} (Issue #3103). - **Results**: MongoDB result backend could crash the worker at startup if not configured using an URL. ## 3.1.22 {#version-3.1.22} release-date : 2016-03-07 01:30 P.M PST release-by : Ask Solem - **Programs**: The worker would crash immediately on startup on `backend.as_uri()` when using some result backends (Issue #3094). - **Programs**: `celery multi`{.interpreted-text role="program"}/`celery worker --detach`{.interpreted-text role="program"} would create an extraneous logfile including literal formats (e.g. `%I`) in the filename (Issue #3096). ## 3.1.21 {#version-3.1.21} release-date : 2016-03-04 11:16 a.m. PST release-by : Ask Solem - **Requirements** > - Now depends on > `Kombu 3.0.34 `{.interpreted-text > role="ref"}. > - Now depends on `billiard`{.interpreted-text role="mod"} > 3.3.0.23. - **Prefork pool**: Fixes 100% CPU loop on Linux `epoll`{.interpreted-text role="manpage"} (Issue #1845). > Also potential fix for: Issue #2142, Issue #2606 - **Prefork pool**: Fixes memory leak related to processes exiting (Issue #2927). - **Worker**: Fixes crash at start-up when trying to censor passwords in MongoDB and Cache result backend URLs (Issue #3079, Issue #3045, Issue #3049, Issue #3068, Issue #3073). > Fix contributed by Maxime Verger. - **Task**: An exception is now raised if countdown/expires is less than -2147483648 (Issue #3078). - **Programs**: `celery shell --ipython`{.interpreted-text role="program"} now compatible with newer `IPython`{.interpreted-text role="pypi"} versions. - **Programs**: The DuplicateNodeName warning emitted by inspect/control now includes a list of the node names returned. > Contributed by Sebastian Kalinowski. - **Utils**: The `.discard(item)` method of `~celery.utils.collections.LimitedSet`{.interpreted-text role="class"} didn\'t actually remove the item (Issue #3087). > Fix contributed by Dave Smith. - **Worker**: Node name formatting now emits less confusing error message for unmatched format keys (Issue #3016). - **Results**: RPC/AMQP backends: Fixed deserialization of JSON exceptions (Issue #2518). > Fix contributed by Allard Hoeve. - **Prefork pool**: The [process inqueue damaged]{.title-ref} error message now includes the original exception raised. - **Documentation**: Includes improvements by: > - Jeff Widman. ## 3.1.20 {#version-3.1.20} release-date : 2016-01-22 06:50 p.m. UTC release-by : Ask Solem - **Requirements** > - Now depends on > `Kombu 3.0.33 `{.interpreted-text > role="ref"}. > > - Now depends on `billiard`{.interpreted-text role="mod"} > 3.3.0.22. > > > Includes binary wheels for Microsoft Windows x86 and x86_64! - **Task**: Error emails now uses `utf-8` character set by default (Issue #2737). - **Task**: Retry now forwards original message headers (Issue #3017). - **Worker**: Bootsteps can now hook into `on_node_join`/`leave`/`lost`. > See `extending-consumer-attributes`{.interpreted-text role="ref"} > for an example. - **Events**: Fixed handling of DST timezones (Issue #2983). - **Results**: Redis backend stopped respecting certain settings. > Contributed by Jeremy Llewellyn. - **Results**: Database backend now properly supports JSON exceptions (Issue #2441). - **Results**: Redis `new_join` didn\'t properly call task errbacks on chord error (Issue #2796). - **Results**: Restores Redis compatibility with Python `redis`{.interpreted-text role="pypi"} \< 2.10.0 (Issue #2903). - **Results**: Fixed rare issue with chord error handling (Issue #2409). - **Tasks**: Using queue-name values in `CELERY_ROUTES`{.interpreted-text role="setting"} now works again (Issue #2987). - **General**: Result backend password now sanitized in report output (Issue #2812, Issue #2004). - **Configuration**: Now gives helpful error message when the result backend configuration points to a module, and not a class (Issue #2945). - **Results**: Exceptions sent by JSON serialized workers are now properly handled by pickle configured workers. - **Programs**: `celery control autoscale` now works (Issue #2950). - **Programs**: `celery beat --detached` now runs after fork callbacks. - **General**: Fix for LRU cache implementation on Python 3.5 (Issue #2897). > Contributed by Dennis Brakhane. > > Python 3.5\'s `OrderedDict` doesn\'t allow mutation while it is > being iterated over. This breaks \"update\" if it is called with a > dict larger than the maximum size. > > This commit changes the code to a version that doesn\'t iterate > over the dict, and should also be a little bit faster. - **Init-scripts**: The beat init-script now properly reports service as down when no pid file can be found. > Eric Zarowny - **Beat**: Added cleaning of corrupted scheduler files for some storage backend errors (Issue #2985). > Fix contributed by Aleksandr Kuznetsov. - **Beat**: Now syncs the schedule even if the schedule is empty. > Fix contributed by Colin McIntosh. - **Supervisord**: Set higher process priority in the `supervisord`{.interpreted-text role="pypi"} : example. Contributed by George Tantiras. - **Documentation**: Includes improvements by: > `Bryson`{.interpreted-text role="github_user"} Caleb Mingle > Christopher Martin Dieter Adriaenssens Jason Veatch Jeremy Cline > Juan Rossi Kevin Harvey Kevin McCarthy Kirill Pavlov Marco Buttu > `Mayflower`{.interpreted-text role="github_user"} Mher Movsisyan > Michael Floering `michael-k`{.interpreted-text role="github_user"} > Nathaniel Varona Rudy Attias Ryan Luckie Steven Parker > `squfrans`{.interpreted-text role="github_user"} Tadej Janež > TakesxiSximada Tom S ## 3.1.19 {#version-3.1.19} release-date : 2015-10-26 01:00 p.m. UTC release-by : Ask Solem - **Requirements** > - Now depends on > `Kombu 3.0.29 `{.interpreted-text > role="ref"}. > - Now depends on `billiard`{.interpreted-text role="mod"} > 3.3.0.21. - **Results**: Fixed MongoDB result backend URL parsing problem (Issue celery/kombu#375). - **Worker**: Task request now properly sets `priority` in delivery_info. > Fix contributed by Gerald Manipon. - **Beat**: PyPy shelve may raise `KeyError` when setting keys (Issue #2862). - **Programs**: `celery beat --deatched`{.interpreted-text role="program"} now working on PyPy. > Fix contributed by Krzysztof Bujniewicz. - **Results**: Redis result backend now ensures all pipelines are cleaned up. > Contributed by Justin Patrin. - **Results**: Redis result backend now allows for timeout to be set in the query portion of the result backend URL. > For example `CELERY_RESULT_BACKEND = 'redis://?timeout=10'` > > Contributed by Justin Patrin. - **Results**: `result.get` now properly handles failures where the exception value is set to `None`{.interpreted-text role="const"} (Issue #2560). - **Prefork pool**: Fixed attribute error `proc.dead`. - **Worker**: Fixed worker hanging when gossip/heartbeat disabled (Issue #1847). > Fix contributed by Aaron Webber and Bryan Helmig. - **Results**: MongoDB result backend now supports pymongo 3.x (Issue #2744). > Fix contributed by Sukrit Khera. - **Results**: RPC/AMQP backends didn\'t deserialize exceptions properly (Issue #2691). > Fix contributed by Sukrit Khera. - **Programs**: Fixed problem with `celery amqp`{.interpreted-text role="program"}\'s `basic_publish` (Issue #2013). - **Worker**: Embedded beat now properly sets app for thread/process (Issue #2594). - **Documentation**: Many improvements and typos fixed. > Contributions by: > > > Carlos Garcia-Dubus > > D. Yu `jerry`{.interpreted-text role="github_user"} Jocelyn > > Delalande Josh Kupershmidt Juan Rossi > > `kanemra`{.interpreted-text role="github_user"} Paul Pearce > > Pavel Savchenko Sean Wang Seungha Kim Zhaorong Ma ## 3.1.18 {#version-3.1.18} release-date : 2015-04-22 05:30 p.m. UTC release-by : Ask Solem - **Requirements** > - Now depends on > `Kombu 3.0.25 `{.interpreted-text > role="ref"}. > - Now depends on `billiard`{.interpreted-text role="mod"} > 3.3.0.20. - **Django**: Now supports Django 1.8 (Issue #2536). > Fix contributed by Bence Tamas and Mickaël Penhard. - **Results**: MongoDB result backend now compatible with pymongo 3.0. > Fix contributed by Fatih Sucu. - **Tasks**: Fixed bug only happening when a task has multiple callbacks (Issue #2515). > Fix contributed by NotSqrt. - **Commands**: Preload options now support `--arg value` syntax. > Fix contributed by John Anderson. - **Compat**: A typo caused `celery.log.setup_logging_subsystem` to be undefined. > Fix contributed by Gunnlaugur Thor Briem. - **init-scripts**: The beat generic init-script now uses `/bin/sh`{.interpreted-text role="file"} instead of `bash`{.interpreted-text role="command"} (Issue #2496). > Fix contributed by Jelle Verstraaten. - **Django**: Fixed a `TypeError`{.interpreted-text role="exc"} sometimes occurring in logging when validating models. > Fix contributed by Alexander. - **Commands**: Worker now supports new `--executable `{.interpreted-text role="option"} argument that can be used with `celery worker --detach`{.interpreted-text role="option"}. > Contributed by Bert Vanderbauwhede. - **Canvas**: Fixed crash in chord unlock fallback task (Issue #2404). - **Worker**: Fixed rare crash occurring with `--autoscale `{.interpreted-text role="option"} enabled (Issue #2411). - **Django**: Properly recycle worker Django database connections when the Django `CONN_MAX_AGE` setting is enabled (Issue #2453). > Fix contributed by Luke Burden. ## 3.1.17 {#version-3.1.17} release-date : 2014-11-19 03:30 p.m. UTC release-by : Ask Solem ::: admonition Don\'t enable the [CELERYD_FORCE_EXECV]{.title-ref} setting! Please review your configuration and disable this option if you\'re using the RabbitMQ or Redis transport. Keeping this option enabled after 3.1 means the async based prefork pool will be disabled, which can easily cause instability. ::: - **Requirements** > - Now depends on > `Kombu 3.0.24 `{.interpreted-text > role="ref"}. > > > Includes the new Qpid transport coming in Celery 3.2, > > backported to support those who may still require Python 2.6 > > compatibility. > > - Now depends on `billiard`{.interpreted-text role="mod"} > 3.3.0.19. > > - `celery[librabbitmq]` now depends on librabbitmq 1.6.1. - **Task**: The timing of ETA/countdown tasks were off after the example `LocalTimezone` implementation in the Python documentation no longer works in Python 3.4. (Issue #2306). - **Task**: Raising `~celery.exceptions.Ignore`{.interpreted-text role="exc"} no longer sends `task-failed` event (Issue #2365). - **Redis result backend**: Fixed unbound local errors. > Fix contributed by Thomas French. - **Task**: Callbacks wasn\'t called properly if `link` was a list of signatures (Issue #2350). - **Canvas**: chain and group now handles json serialized signatures (Issue #2076). - **Results**: `.join_native()` would accidentally treat the `STARTED` state as being ready (Issue #2326). > This could lead to the chord callback being called with invalid > arguments when using chords with the > `CELERY_TRACK_STARTED`{.interpreted-text role="setting"} setting > enabled. - **Canvas**: The `chord_size` attribute is now set for all canvas primitives, making sure more combinations will work with the `new_join` optimization for Redis (Issue #2339). - **Task**: Fixed problem with app not being properly propagated to `trace_task` in all cases. > Fix contributed by `kristaps`{.interpreted-text > role="github_user"}. - **Worker**: Expires from task message now associated with a timezone. > Fix contributed by Albert Wang. - **Cassandra result backend**: Fixed problems when using detailed mode. > When using the Cassandra backend in detailed mode, a regression > caused errors when attempting to retrieve results. > > Fix contributed by Gino Ledesma. - **Mongodb Result backend**: Pickling the backend instance will now include the original URL (Issue #2347). > Fix contributed by Sukrit Khera. - **Task**: Exception info wasn\'t properly set for tasks raising `~celery.exceptions.Reject`{.interpreted-text role="exc"} (Issue #2043). - **Worker**: Duplicates are now removed when loading the set of revoked tasks from the worker state database (Issue #2336). - **celery.contrib.rdb**: Fixed problems with `rdb.set_trace` calling stop from the wrong frame. > Fix contributed by `llllllllll`{.interpreted-text > role="github_user"}. - **Canvas**: `chain` and `chord` can now be immutable. - **Canvas**: `chord.apply_async` will now keep partial args set in `self.args` (Issue #2299). - **Results**: Small refactoring so that results are decoded the same way in all result backends. - **Logging**: The `processName` format was introduced in Python 2.6.2 so for compatibility this format is now excluded when using earlier versions (Issue #1644). ## 3.1.16 {#version-3.1.16} release-date : 2014-10-03 06:00 p.m. UTC release-by : Ask Solem - **Worker**: 3.1.15 broke `-Ofair `{.interpreted-text role="option"} behavior (Issue #2286). > This regression could result in all tasks executing in a single > child process if `-Ofair` was enabled. - **Canvas**: `celery.signature` now properly forwards app argument in all cases. - **Task**: `.retry()` didn\'t raise the exception correctly when called without a current exception. > Fix contributed by Andrea Rabbaglietti. - **Worker**: The `enable_events` remote control command disabled worker-related events by mistake (Issue #2272). > Fix contributed by Konstantinos Koukopoulos. - **Django**: Adds support for Django 1.7 class names in INSTALLED_APPS when using `app.autodiscover_tasks()` (Issue #2248). - **Sphinx**: `celery.contrib.sphinx` now uses `getfullargspec` on Python 3 (Issue #2302). - **Redis/Cache Backends**: Chords will now run at most once if one or more tasks in the chord are executed multiple times for some reason. ## 3.1.15 {#version-3.1.15} release-date : 2014-09-14 11:00 p.m. UTC release-by : Ask Solem - **Django**: Now makes sure `django.setup()` is called before importing any task modules (Django 1.7 compatibility, Issue #2227) - **Results**: `result.get()` was misbehaving by calling `backend.get_task_meta` in a `finally`{.interpreted-text role="keyword"} call leading to AMQP result backend queues not being properly cleaned up (Issue #2245). ## 3.1.14 {#version-3.1.14} release-date : 2014-09-08 03:00 p.m. UTC release-by : Ask Solem - **Requirements** > - Now depends on > `Kombu 3.0.22 `{.interpreted-text > role="ref"}. - **Init-scripts**: The generic worker init-scripts `status` command now gets an accurate pidfile list (Issue #1942). - **Init-scripts**: The generic beat script now implements the `status` : command. > Contributed by John Whitlock. - **Commands**: Multi now writes informational output to stdout instead of stderr. - **Worker**: Now ignores not implemented error for `pool.restart` (Issue #2153). - **Task**: Retry no longer raises retry exception when executed in eager mode (Issue #2164). - **AMQP Result backend**: Now ensured `on_interval` is called at least every second for blocking calls to properly propagate parent errors. - **Django**: Compatibility with Django 1.7 on Windows (Issue #2126). - **Programs**: `!--umask`{.interpreted-text role="option"} argument can now be specified in both octal (if starting with 0) or decimal. ## 3.1.13 {#version-3.1.13} ### Security Fixes - \[Security: [CELERYSA-0002](https://github.com/celery/celery/tree/master/docs/sec/CELERYSA-0002.txt)\] Insecure default umask. > The built-in utility used to daemonize the Celery worker service > sets an insecure umask by default (umask 0). > > This means that any files or directories created by the worker > will end up having world-writable permissions. > > Special thanks to Red Hat for originally discovering and reporting > the issue! > > This version will no longer set a default umask by default, so if > unset the umask of the parent process will be used. ### News - **Requirements** > - Now depends on > `Kombu 3.0.21 `{.interpreted-text > role="ref"}. > - Now depends on `billiard`{.interpreted-text role="mod"} > 3.3.0.18. - **App**: `backend` argument now also sets the `CELERY_RESULT_BACKEND`{.interpreted-text role="setting"} setting. - **Task**: `signature_from_request` now propagates `reply_to` so that the RPC backend works with retried tasks (Issue #2113). - **Task**: `retry` will no longer attempt to re-queue the task if sending the retry message fails. > Unrelated exceptions being raised could cause a message loop, so > it was better to remove this behavior. - **Beat**: Accounts for standard 1ms drift by always waking up 0.010s earlier. > This will adjust the latency so that the periodic tasks won\'t > move 1ms after every invocation. - Documentation fixes > Contributed by Yuval Greenfield, Lucas Wiman, > `nicholsonjf`{.interpreted-text role="github_user"}. - **Worker**: Removed an outdated assert statement that could lead to errors being masked (Issue #2086). ## 3.1.12 {#version-3.1.12} release-date : 2014-06-09 10:12 p.m. UTC release-by : Ask Solem - **Requirements** > Now depends on > `Kombu 3.0.19 `{.interpreted-text > role="ref"}. - **App**: Connections weren\'t being closed after fork due to an error in the after fork handler (Issue #2055). > This could manifest itself by causing framing errors when using > RabbitMQ. (`Unexpected frame`). - **Django**: `django.setup()` was being called too late when using Django 1.7 (Issue #1802). - **Django**: Fixed problems with event timezones when using Django (`Substantial drift`). > Celery didn\'t take into account that Django modifies the > `time.timeone` attributes and friends. - **Canvas**: `Signature.link` now works when the link option is a scalar value (Issue #2019). - **Prefork pool**: Fixed race conditions for when file descriptors are removed from the event loop. > Fix contributed by Roger Hu. - **Prefork pool**: Improved solution for dividing tasks between child processes. > This change should improve performance when there are many child > processes, and also decrease the chance that two subsequent tasks > are written to the same child process. - **Worker**: Now ignores unknown event types, instead of crashing. > Fix contributed by Illes Solt. - **Programs**: `celery worker --detach`{.interpreted-text role="program"} no longer closes open file descriptors when `C_FAKEFORK`{.interpreted-text role="envvar"} is used so that the workers output can be seen. - **Programs**: The default working directory for `celery worker --detach`{.interpreted-text role="program"} is now the current working directory, not `/`. - **Canvas**: `signature(s, app=app)` didn\'t upgrade serialized signatures to their original class (`subtask_type`) when the `app` keyword argument was used. - **Control**: The `duplicate nodename` warning emitted by control commands now shows the duplicate node name. - **Tasks**: Can now call `ResultSet.get()` on a result set without members. > Fix contributed by Alexey Kotlyarov. - **App**: Fixed strange traceback mangling issue for `app.connection_or_acquire`. - **Programs**: The `celery multi stopwait`{.interpreted-text role="program"} command is now documented in usage. - **Other**: Fixed cleanup problem with `PromiseProxy` when an error is raised while trying to evaluate the promise. - **Other**: The utility used to censor configuration values now handles non-string keys. > Fix contributed by Luke Pomfrey. - **Other**: The `inspect conf` command didn\'t handle non-string keys well. > Fix contributed by Jay Farrimond. - **Programs**: Fixed argument handling problem in `celery worker --detach`{.interpreted-text role="program"}. > Fix contributed by Dmitry Malinovsky. - **Programs**: `celery worker --detach`{.interpreted-text role="program"} didn\'t forward working directory option (Issue #2003). - **Programs**: `celery inspect registered`{.interpreted-text role="program"} no longer includes the list of built-in tasks. - **Worker**: The `requires` attribute for boot steps weren\'t being handled correctly (Issue #2002). - **Eventlet**: The eventlet pool now supports the `pool_grow` and `pool_shrink` remote control commands. > Contributed by Mher Movsisyan. - **Eventlet**: The eventlet pool now implements statistics for `` `celery inspect stats ``{.interpreted-text role="program"}\`. > Contributed by Mher Movsisyan. - **Documentation**: Clarified `Task.rate_limit` behavior. > Contributed by Jonas Haag. - **Documentation**: `AbortableTask` examples now updated to use the new API (Issue #1993). - **Documentation**: The security documentation examples used an out of date import. > Fix contributed by Ian Dees. - **Init-scripts**: The CentOS init-scripts didn\'t quote `CELERY_CHDIR`{.interpreted-text role="envvar"}. > Fix contributed by `ffeast`{.interpreted-text role="github_user"}. ## 3.1.11 {#version-3.1.11} release-date : 2014-04-16 11:00 p.m. UTC release-by : Ask Solem - **Now compatible with RabbitMQ 3.3.0** > You need to run Celery 3.1.11 or later when using RabbitMQ 3.3, > and if you use the `librabbitmq` module you also have to upgrade > to librabbitmq 1.5.0: > > ``` bash > $ pip install -U librabbitmq > ``` - **Requirements**: > - Now depends on > `Kombu 3.0.15 `{.interpreted-text > role="ref"}. > - Now depends on [billiard > 3.3.0.17](https://github.com/celery/billiard/blob/master/CHANGES.txt). > - Bundle `celery[librabbitmq]` now depends on > `librabbitmq`{.interpreted-text role="mod"} 1.5.0. - **Tasks**: The `CELERY_DEFAULT_DELIVERY_MODE`{.interpreted-text role="setting"} setting was being ignored (Issue #1953). - **Worker**: New `celery worker --heartbeat-interval`{.interpreted-text role="option"} can be used to change the time (in seconds) between sending event heartbeats. > Contributed by Matthew Duggan and Craig Northway. - **App**: Fixed memory leaks occurring when creating lots of temporary app instances (Issue #1949). - **MongoDB**: SSL configuration with non-MongoDB transport breaks MongoDB results backend (Issue #1973). > Fix contributed by Brian Bouterse. - **Logging**: The color formatter accidentally modified `record.msg` (Issue #1939). - **Results**: Fixed problem with task trails being stored multiple times, causing `result.collect()` to hang (Issue #1936, Issue #1943). - **Results**: `ResultSet` now implements a `.backend` attribute for compatibility with `AsyncResult`. - **Results**: `.forget()` now also clears the local cache. - **Results**: Fixed problem with multiple calls to `result._set_cache` (Issue #1940). - **Results**: `join_native` populated result cache even if disabled. - **Results**: The YAML result serializer should now be able to handle storing exceptions. - **Worker**: No longer sends task error emails for expected errors (in `@task(throws=(..., )))`. - **Canvas**: Fixed problem with exception deserialization when using the JSON serializer (Issue #1987). - **Eventlet**: Fixes crash when `celery.contrib.batches` attempted to cancel a non-existing timer (Issue #1984). - Can now import `celery.version_info_t`, and `celery.five` (Issue #1968). ## 3.1.10 {#version-3.1.10} release-date : 2014-03-22 09:40 p.m. UTC release-by : Ask Solem - **Requirements**: > - Now depends on > `Kombu 3.0.14 `{.interpreted-text > role="ref"}. - **Results**: > Reliability improvements to the SQLAlchemy database backend. > Previously the connection from the MainProcess was improperly > shared with the workers. (Issue #1786) - **Redis:** Important note about events (Issue #1882). > There\'s a new transport option for Redis that enables monitors to > filter out unwanted events. Enabling this option in the workers > will increase performance considerably: > > ``` python > BROKER_TRANSPORT_OPTIONS = {'fanout_patterns': True} > ``` > > Enabling this option means that your workers won\'t be able to see > workers with the option disabled (or is running an older version > of Celery), so if you do enable it then make sure you do so on all > nodes. > > See `redis-caveats`{.interpreted-text role="ref"}. > > This will be the default in Celery 3.2. - **Results**: The `@AsyncResult`{.interpreted-text role="class"} object now keeps a local cache of the final state of the task. > This means that the global result cache can finally be disabled, > and you can do so by setting > `CELERY_MAX_CACHED_RESULTS`{.interpreted-text role="setting"} to > `-1`{.interpreted-text role="const"}. The lifetime of the cache > will then be bound to the lifetime of the result object, which > will be the default behavior in Celery 3.2. - **Events**: The \"Substantial drift\" warning message is now logged once per node name only (Issue #1802). - **Worker**: Ability to use one log file per child process when using the prefork pool. > This can be enabled by using the new `%i` and `%I` format > specifiers for the log file name. See > `worker-files-process-index`{.interpreted-text role="ref"}. - **Redis**: New experimental chord join implementation. > This is an optimization for chords when using the Redis result > backend, where the join operation is now considerably faster and > using less resources than the previous strategy. > > The new option can be set in the result backend URL: > > ``` python > CELERY_RESULT_BACKEND = 'redis://localhost?new_join=1' > ``` > > This must be enabled manually as it\'s incompatible with workers > and clients not using it, so be sure to enable the option in all > clients and workers if you decide to use it. - **Multi**: With `-opt:index` (e.g., `-c:1`) the index now always refers to the position of a node in the argument list. > This means that referring to a number will work when specifying a > list of node names and not just for a number range: > > ``` bash > celery multi start A B C D -c:1 4 -c:2-4 8 > ``` > > In this example `1` refers to node A (as it\'s the first node in > the list). - **Signals**: The sender argument to `Signal.connect` can now be a proxy object, which means that it can be used with the task decorator (Issue #1873). - **Task**: A regression caused the `queue` argument to `Task.retry` to be ignored (Issue #1892). - **App**: Fixed error message for `~@Celery.config_from_envvar`{.interpreted-text role="meth"}. > Fix contributed by Dmitry Malinovsky. - **Canvas**: Chords can now contain a group of other chords (Issue #1921). - **Canvas**: Chords can now be combined when using the amqp result backend (a chord where the callback is also a chord). - **Canvas**: Calling `result.get()` for a chain task will now complete even if one of the tasks in the chain is `ignore_result=True` (Issue #1905). - **Canvas**: Worker now also logs chord errors. - **Canvas**: A chord task raising an exception will now result in any errbacks (`link_error`) to the chord callback to also be called. - **Results**: Reliability improvements to the SQLAlchemy database backend (Issue #1786). > Previously the connection from the `MainProcess` was improperly > inherited by child processes. > > Fix contributed by Ionel Cristian Mărieș. - **Task**: Task callbacks and errbacks are now called using the group primitive. - **Task**: `Task.apply` now properly sets `request.headers` (Issue #1874). - **Worker**: Fixed `UnicodeEncodeError`{.interpreted-text role="exc"} occurring when worker is started by `supervisor`{.interpreted-text role="pypi"}. > Fix contributed by Codeb Fan. - **Beat**: No longer attempts to upgrade a newly created database file (Issue #1923). - **Beat**: New setting `` `CELERYBEAT_SYNC_EVERY ``{.interpreted-text role="setting"}\` can be be used to control file sync by specifying the number of tasks to send between each sync. > Contributed by Chris Clark. - **Commands**: `celery inspect memdump`{.interpreted-text role="program"} no longer crashes if the `psutil`{.interpreted-text role="mod"} module isn\'t installed (Issue #1914). - **Worker**: Remote control commands now always accepts json serialized messages (Issue #1870). - **Worker**: Gossip will now drop any task related events it receives by mistake (Issue #1882). ## 3.1.9 {#version-3.1.9} release-date : 2014-02-10 06:43 p.m. UTC release-by : Ask Solem - **Requirements**: > - Now depends on > `Kombu 3.0.12 `{.interpreted-text > role="ref"}. - **Prefork pool**: Better handling of exiting child processes. > Fix contributed by Ionel Cristian Mărieș. - **Prefork pool**: Now makes sure all file descriptors are removed from the hub when a process is cleaned up. > Fix contributed by Ionel Cristian Mărieș. - **New Sphinx extension**: for autodoc documentation of tasks: `celery.contrib.spinx`{.interpreted-text role="mod"} (Issue #1833). - **Django**: Now works with Django 1.7a1. - **Task**: Task.backend is now a property that forwards to `app.backend` if no custom backend has been specified for the task (Issue #1821). - **Generic init-scripts**: Fixed bug in stop command. > Fix contributed by Rinat Shigapov. - **Generic init-scripts**: Fixed compatibility with GNU `stat`{.interpreted-text role="manpage"}. > Fix contributed by Paul Kilgo. - **Generic init-scripts**: Fixed compatibility with the minimal `dash`{.interpreted-text role="program"} shell (Issue #1815). - **Commands**: The `celery amqp basic.publish`{.interpreted-text role="program"} command wasn\'t working properly. > Fix contributed by Andrey Voronov. - **Commands**: Did no longer emit an error message if the pidfile exists and the process is still alive (Issue #1855). - **Commands**: Better error message for missing arguments to preload options (Issue #1860). - **Commands**: `celery -h`{.interpreted-text role="program"} didn\'t work because of a bug in the argument parser (Issue #1849). - **Worker**: Improved error message for message decoding errors. - **Time**: Now properly parses the [Z]{.title-ref} timezone specifier in ISO 8601 date strings. > Fix contributed by Martin Davidsson. - **Worker**: Now uses the *negotiated* heartbeat value to calculate how often to run the heartbeat checks. - **Beat**: Fixed problem with beat hanging after the first schedule iteration (Issue #1822). > Fix contributed by Roger Hu. - **Signals**: The header argument to `before_task_publish`{.interpreted-text role="signal"} is now always a dictionary instance so that signal handlers can add headers. - **Worker**: A list of message headers is now included in message related errors. ## 3.1.8 {#version-3.1.8} release-date : 2014-01-17 10:45 p.m. UTC release-by : Ask Solem - **Requirements**: > - Now depends on > `Kombu 3.0.10 `{.interpreted-text > role="ref"}. > - Now depends on [billiard > 3.3.0.14](https://github.com/celery/billiard/blob/master/CHANGES.txt). - **Worker**: The event loop wasn\'t properly reinitialized at consumer restart which would force the worker to continue with a closed `epoll` instance on Linux, resulting in a crash. - **Events:** Fixed issue with both heartbeats and task events that could result in the data not being kept in sorted order. > As a result this would force the worker to log \"heartbeat > missed\" events even though the remote node was sending heartbeats > in a timely manner. - **Results:** The pickle serializer no longer converts group results to tuples, and will keep the original type (*Issue #1750*). - **Results:** `ResultSet.iterate` is now pending deprecation. > The method will be deprecated in version 3.2 and removed in > version 3.3. > > Use `result.get(callback=)` (or `result.iter_native()` where > available) instead. - **Worker**\|eventlet/gevent: A regression caused `Control-c`{.interpreted-text role="kbd"} to be ineffective for shutdown. - **Redis result backend:** Now using a pipeline to store state changes for improved performance. > Contributed by Pepijn de Vos. - **Redis result backend:** Will now retry storing the result if disconnected. - **Worker**\|gossip: Fixed attribute error occurring when another node leaves. > Fix contributed by Brodie Rao. - **Generic init-scripts:** Now runs a check at start-up to verify that any configuration scripts are owned by root and that they aren\'t world/group writable. > The init-script configuration is a shell script executed by root, > so this is a preventive measure to ensure that users don\'t leave > this file vulnerable to changes by unprivileged users. > > ::: note > ::: title > Note > ::: > > Note that upgrading Celery won\'t update the init-scripts, instead > you need to manually copy the improved versions from the source > distribution: > > ::: - **Commands**: The `celery purge`{.interpreted-text role="program"} command now warns that the operation will delete all tasks and prompts the user for confirmation. > A new `-f `{.interpreted-text role="option"} was > added that can be used to disable interactive mode. - **Task**: `.retry()` didn\'t raise the value provided in the `exc` argument when called outside of an error context (*Issue #1755*). - **Commands:** The `celery multi`{.interpreted-text role="program"} command didn\'t forward command line configuration to the target workers. > The change means that multi will forward the special `--` argument > and configuration content at the end of the arguments line to the > specified workers. > > Example using command-line configuration to set a broker heartbeat > from `celery multi`{.interpreted-text role="program"}: > > ``` bash > $ celery multi start 1 -c3 -- broker.heartbeat=30 > ``` > > Fix contributed by Antoine Legrand. - **Canvas:** `chain.apply_async()` now properly forwards execution options. > Fix contributed by Konstantin Podshumok. - **Redis result backend:** Now takes `connection_pool` argument that can be used to change the connection pool class/constructor. - **Worker:** Now truncates very long arguments and keyword arguments logged by the pool at debug severity. - **Worker:** The worker now closes all open files on `SIGHUP`{.interpreted-text role="sig"} (regression) (*Issue #1768*). > Fix contributed by Brodie Rao - **Worker:** Will no longer accept remote control commands while the worker start-up phase is incomplete (*Issue #1741*). - **Commands:** The output of the event dump utility (`celery events -d`{.interpreted-text role="program"}) can now be piped into other commands. - **Documentation:** The RabbitMQ installation instructions for macOS was updated to use modern Homebrew practices. > Contributed by Jon Chen. - **Commands:** The `celery inspect conf`{.interpreted-text role="program"} utility now works. - **Commands:** The `--no-color `{.interpreted-text role="option"} argument was not respected by all commands (*Issue #1799*). - **App:** Fixed rare bug with `autodiscover_tasks()` (*Issue #1797*). - **Distribution:** The sphinx docs will now always add the parent directory to path so that the current Celery source code is used as a basis for API documentation (*Issue #1782*). - **Documentation:** `supervisor`{.interpreted-text role="pypi"} examples contained an extraneous \'-\' in a `--logfile `{.interpreted-text role="option"} argument example. > Fix contributed by Mohammad Almeer. ## 3.1.7 {#version-3.1.7} release-date : 2013-12-17 06:00 p.m. UTC release-by : Ask Solem ### Important Notes {#v317-important} ### Init-script security improvements Where the generic init-scripts (for `celeryd`, and `celerybeat`) before delegated the responsibility of dropping privileges to the target application, it will now use `su` instead, so that the Python program isn\'t trusted with superuser privileges. This isn\'t in reaction to any known exploit, but it will limit the possibility of a privilege escalation bug being abused in the future. You have to upgrade the init-scripts manually from this directory: #### AMQP result backend The 3.1 release accidentally left the amqp backend configured to be non-persistent by default. Upgrading from 3.0 would give a \"not equivalent\" error when attempting to set or retrieve results for a task. That\'s unless you manually set the persistence setting: CELERY_RESULT_PERSISTENT = True This version restores the previous value so if you already forced the upgrade by removing the existing exchange you must either keep the configuration by setting `CELERY_RESULT_PERSISTENT = False` or delete the `celeryresults` exchange again. #### Synchronous subtasks Tasks waiting for the result of a subtask will now emit a `RuntimeWarning`{.interpreted-text role="exc"} warning when using the prefork pool, and in 3.2 this will result in an exception being raised. It\'s not legal for tasks to block by waiting for subtasks as this is likely to lead to resource starvation and eventually deadlock when using the prefork pool (see also `task-synchronous-subtasks`{.interpreted-text role="ref"}). If you really know what you\'re doing you can avoid the warning (and the future exception being raised) by moving the operation in a white-list block: ``` python from celery.result import allow_join_result @app.task def misbehaving(): result = other_task.delay() with allow_join_result(): result.get() ``` Note also that if you wait for the result of a subtask in any form when using the prefork pool you must also disable the pool prefetching behavior with the worker `-Ofair option `{.interpreted-text role="ref"}. ### Fixes {#v317-fixes} - Now depends on `Kombu 3.0.8 `{.interpreted-text role="ref"}. - Now depends on `billiard`{.interpreted-text role="mod"} 3.3.0.13 - Events: Fixed compatibility with non-standard json libraries that sends float as `decimal.Decimal`{.interpreted-text role="class"} (Issue #1731) - Events: State worker objects now always defines attributes: `active`, `processed`, `loadavg`, `sw_ident`, `sw_ver` and `sw_sys`. - Worker: Now keeps count of the total number of tasks processed, not just by type (`all_active_count`). - Init-scripts: Fixed problem with reading configuration file when the init-script is symlinked to a runlevel (e.g., `S02celeryd`). (Issue #1740). > This also removed a rarely used feature where you can symlink the > script to provide alternative configurations. You instead copy the > script and give it a new name, but perhaps a better solution is to > provide arguments to `CELERYD_OPTS` to separate them: > > ``` bash > CELERYD_NODES="X1 X2 Y1 Y2" > CELERYD_OPTS="-A:X1 x -A:X2 x -A:Y1 y -A:Y2 y" > ``` - Fallback chord unlock task is now always called after the chord header (Issue #1700). > This means that the unlock task won\'t be started if there\'s an > error sending the header. - Celery command: Fixed problem with arguments for some control commands. > Fix contributed by Konstantin Podshumok. - Fixed bug in `utcoffset` where the offset when in DST would be completely wrong (Issue #1743). - Worker: Errors occurring while attempting to serialize the result of a task will now cause the task to be marked with failure and a `kombu.exceptions.EncodingError`{.interpreted-text role="class"} error. > Fix contributed by Ionel Cristian Mărieș. - Worker with `-B `{.interpreted-text role="option"} argument didn\'t properly shut down the beat instance. - Worker: The `%n` and `%h` formats are now also supported by the `--logfile `{.interpreted-text role="option"}, `--pidfile `{.interpreted-text role="option"} and `--statedb `{.interpreted-text role="option"} arguments. > Example: > > ``` bash > $ celery -A proj worker -n foo@%h --logfile=%n.log --statedb=%n.db > ``` - Redis/Cache result backends: Will now timeout if keys evicted while trying to join a chord. - The fallback unlock chord task now raises `Retry`{.interpreted-text role="exc"} so that the retry even is properly logged by the worker. - Multi: Will no longer apply Eventlet/gevent monkey patches (Issue #1717). - Redis result backend: Now supports UNIX sockets. > Like the Redis broker transport the result backend now also > supports using `redis+socket:///tmp/redis.sock` URLs. > > Contributed by Alcides Viamontes Esquivel. - Events: Events sent by clients was mistaken for worker related events (Issue #1714). > For `events.State` the tasks now have a `Task.client` attribute > that\'s set when a `task-sent` event is being received. > > Also, a clients logical clock isn\'t in sync with the cluster so > they live in a \"time bubble.\" So for this reason monitors will > no longer attempt to merge with the clock of an event sent by a > client, instead it will fake the value by using the current clock > with a skew of -1. - Prefork pool: The method used to find terminated processes was flawed in that it didn\'t also take into account missing `popen` objects. - Canvas: `group` and `chord` now works with anon signatures as long as the group/chord object is associated with an app instance (Issue #1744). > You can pass the app by using `group(..., app=app)`. ## 3.1.6 {#version-3.1.6} release-date : 2013-12-02 06:00 p.m. UTC release-by : Ask Solem - Now depends on `billiard`{.interpreted-text role="mod"} 3.3.0.10. - Now depends on `Kombu 3.0.7 `{.interpreted-text role="ref"}. - Fixed problem where Mingle caused the worker to hang at start-up (Issue #1686). - Beat: Would attempt to drop privileges twice (Issue #1708). - Windows: Fixed error with `geteuid` not being available (Issue #1676). - Tasks can now provide a list of expected error classes (Issue #1682). > The list should only include errors that the task is expected to > raise during normal operation: > > @task(throws=(KeyError, HttpNotFound)) > > What happens when an exceptions is raised depends on the type of > error: > > - Expected errors (included in `Task.throws`) > > > Will be logged using severity `INFO`, and traceback is > > excluded. > > - Unexpected errors > > > Will be logged using severity `ERROR`, with traceback > > included. - Cache result backend now compatible with Python 3 (Issue #1697). - CentOS init-script: Now compatible with SysV style init symlinks. > Fix contributed by Jonathan Jordan. - Events: Fixed problem when task name isn\'t defined (Issue #1710). > Fix contributed by Mher Movsisyan. - Task: Fixed unbound local errors (Issue #1684). > Fix contributed by Markus Ullmann. - Canvas: Now unrolls groups with only one task (optimization) (Issue #1656). - Task: Fixed problem with ETA and timezones. > Fix contributed by Alexander Koval. - Django: Worker now performs model validation (Issue #1681). - Task decorator now emits less confusing errors when used with incorrect arguments (Issue #1692). - Task: New method `Task.send_event` can be used to send custom events to Flower and other monitors. - Fixed a compatibility issue with non-abstract task classes - Events from clients now uses new node name format (`gen@`). - Fixed rare bug with Callable not being defined at interpreter shutdown (Issue #1678). > Fix contributed by Nick Johnson. - Fixed Python 2.6 compatibility (Issue #1679). ## 3.1.5 {#version-3.1.5} release-date : 2013-11-21 06:20 p.m. UTC release-by : Ask Solem - Now depends on `Kombu 3.0.6 `{.interpreted-text role="ref"}. - Now depends on `billiard`{.interpreted-text role="mod"} 3.3.0.8 - App: `config_from_object` is now lazy (Issue #1665). - App: `autodiscover_tasks` is now lazy. > Django users should now wrap access to the settings object in a > lambda: > > app.autodiscover_tasks(lambda: settings.INSTALLED_APPS) > > this ensures that the settings object isn\'t prepared prematurely. - Fixed regression for `--app `{.interpreted-text role="option"} argument experienced by some users (Issue #1653). - Worker: Now respects the `--uid `{.interpreted-text role="option"} and `--gid `{.interpreted-text role="option"} arguments even if `--detach `{.interpreted-text role="option"} isn\'t enabled. - Beat: Now respects the `--uid `{.interpreted-text role="option"} and `--gid `{.interpreted-text role="option"} arguments even if `--detach `{.interpreted-text role="option"} isn\'t enabled. - Python 3: Fixed unorderable error occurring with the worker `-B `{.interpreted-text role="option"} argument enabled. - `celery.VERSION` is now a named tuple. - `maybe_signature(list)` is now applied recursively (Issue #1645). - `celery shell` command: Fixed `IPython.frontend` deprecation warning. - The default app no longer includes the built-in fix-ups. > This fixes a bug where `celery multi` would attempt to load the > Django settings module before entering the target working > directory. - The Django daemonization tutorial was changed. > Users no longer have to explicitly export `DJANGO_SETTINGS_MODULE` > in `/etc/default/celeryd`{.interpreted-text role="file"} when the > new project layout is used. - Redis result backend: expiry value can now be 0 (Issue #1661). - Censoring settings now accounts for non-string keys (Issue #1663). - App: New `autofinalize` option. > Apps are automatically finalized when the task registry is > accessed. You can now disable this behavior so that an exception > is raised instead. > > Example: > > ``` python > app = Celery(autofinalize=False) > > # raises RuntimeError > tasks = app.tasks > > @app.task > def add(x, y): > return x + y > > # raises RuntimeError > add.delay(2, 2) > > app.finalize() > # no longer raises: > tasks = app.tasks > add.delay(2, 2) > ``` - The worker didn\'t send monitoring events during shutdown. - Worker: Mingle and gossip is now automatically disabled when used with an unsupported transport (Issue #1664). - `celery` command: Preload options now supports the rare `--opt value` format (Issue #1668). - `celery` command: Accidentally removed options appearing before the sub-command, these are now moved to the end instead. - Worker now properly responds to `inspect stats` commands even if received before start-up is complete (Issue #1659). - `task_postrun`{.interpreted-text role="signal"} is now sent within a `finally`{.interpreted-text role="keyword"} block, to make sure the signal is always sent. - Beat: Fixed syntax error in string formatting. > Contributed by `nadad`{.interpreted-text role="github_user"}. - Fixed typos in the documentation. > Fixes contributed by Loic Bistuer, `sunfinite`{.interpreted-text > role="github_user"}. - Nested chains now works properly when constructed using the `chain` type instead of the `|` operator (Issue #1656). ## 3.1.4 {#version-3.1.4} release-date : 2013-11-15 11:40 p.m. UTC release-by : Ask Solem - Now depends on `Kombu 3.0.5 `{.interpreted-text role="ref"}. - Now depends on `billiard`{.interpreted-text role="mod"} 3.3.0.7 - Worker accidentally set a default socket timeout of 5 seconds. - Django: Fix-up now sets the default app so that threads will use the same app instance (e.g., for `manage.py runserver`{.interpreted-text role="command"}). - Worker: Fixed Unicode error crash at start-up experienced by some users. - Calling `.apply_async` on an empty chain now works again (Issue #1650). - The `celery multi show` command now generates the same arguments as the start command does. - The `--app `{.interpreted-text role="option"} argument could end up using a module object instead of an app instance (with a resulting crash). - Fixed a syntax error problem in the beat init-script. > Fix contributed by Vsevolod. - Tests now passing on PyPy 2.1 and 2.2. ## 3.1.3 {#version-3.1.3} release-date : 2013-11-13 00:55 a.m. UTC release-by : Ask Solem - Fixed compatibility problem with Python 2.7.0 - 2.7.5 (Issue #1637) > `unpack_from` started supporting `memoryview` arguments in Python > 2.7.6. - Worker: `-B `{.interpreted-text role="option"} argument accidentally closed files used for logging. - Task decorated tasks now keep their docstring (Issue #1636) ## 3.1.2 {#version-3.1.2} release-date : 2013-11-12 08:00 p.m. UTC release-by : Ask Solem - Now depends on `billiard`{.interpreted-text role="mod"} 3.3.0.6 - No longer needs the billiard C extension to be installed. - The worker silently ignored task errors. - Django: Fixed `ImproperlyConfigured` error raised when no database backend specified. > Fix contributed by `j0hnsmith`{.interpreted-text > role="github_user"}. - Prefork pool: Now using `_multiprocessing.read` with `memoryview` if available. - `close_open_fds` now uses `os.closerange` if available. - `get_fdmax` now takes value from `sysconfig` if possible. ## 3.1.1 {#version-3.1.1} release-date : 2013-11-11 06:30 p.m. UTC release-by : Ask Solem - Now depends on `billiard`{.interpreted-text role="mod"} 3.3.0.4. - Python 3: Fixed compatibility issues. - Windows: Accidentally showed warning that the billiard C extension wasn\'t installed (Issue #1630). - Django: Tutorial updated with a solution that sets a default `DJANGO_SETTINGS_MODULE`{.interpreted-text role="envvar"} so that it doesn\'t have to be typed in with the `celery`{.interpreted-text role="program"} command. > Also fixed typos in the tutorial, and added the settings required > to use the Django database backend. > > Thanks to Chris Ward, `orarbel`{.interpreted-text > role="github_user"}. - Django: Fixed a problem when using the Django settings in Django 1.6. - Django: Fix-up shouldn\'t be applied if the django loader is active. - Worker: Fixed attribute error for `human_write_stats` when using the compatibility prefork pool implementation. - Worker: Fixed compatibility with billiard without C extension. - Inspect.conf: Now supports a `with_defaults` argument. - Group.restore: The backend argument wasn\'t respected. ## 3.1.0 {#version-3.1.0} release-date : 2013-11-09 11:00 p.m. UTC release-by : Ask Solem See `whatsnew-3.1`{.interpreted-text role="ref"}. --- # Change history {#changelog-4.0} This document contains change notes for bugfix releases in the 4.0.x series (latentcall), please see `whatsnew-4.0`{.interpreted-text role="ref"} for an overview of what\'s new in Celery 4.0. ## 4.0.2 {#version-4.0.2} release-date : 2016-12-15 03:40 PM PST release-by : Ask Solem - **Requirements** > - Now depends on > `Kombu 4.0.2 `{.interpreted-text > role="ref"}. - **Tasks**: Fixed problem with JSON serialization of [group]{.title-ref} (`keys must be string` error, Issue #3688). - **Worker**: Fixed JSON serialization issue when using `inspect active` and friends (Issue #3667). - **App**: Fixed saferef errors when using signals (Issue #3670). - **Prefork**: Fixed bug with pack requiring bytes argument on Python 2.7.5 and earlier (Issue #3674). - **Tasks**: Saferepr did not handle unicode in bytestrings on Python 2 (Issue #3676). - **Testing**: Added new `celery_worker_paremeters` fixture. > Contributed by **Michael Howitz**. - **Tasks**: Added new `app` argument to `GroupResult.restore` (Issue #3669). > This makes the restore method behave the same way as the > `GroupResult` constructor. > > Contributed by **Andreas Pelme**. - **Tasks**: Fixed type checking crash when task takes `*args` on Python 3 (Issue #3678). - Documentation and examples improvements by: > - **BLAGA Razvan-Paul** > - **Michael Howitz** > - `paradox41`{.interpreted-text role="github_user"} ## 4.0.1 {#version-4.0.1} release-date : 2016-12-08 05:22 PM PST release-by : Ask Solem - \[Security: [CELERYSA-0003](https://github.com/celery/celery/tree/master/docs/sec/CELERYSA-0003.txt)\] Insecure default configuration > The default `accept_content`{.interpreted-text role="setting"} > setting was set to allow deserialization of pickled messages in > Celery 4.0.0. > > The insecure default has been fixed in 4.0.1, and you can also > configure the 4.0.0 version to explicitly only allow json > serialized messages: > > ``` python > app.conf.accept_content = ['json'] > ``` - **Tasks**: Added new method to register class-based tasks (Issue #3615). > To register a class based task you should now call > `app.register_task`: > > ``` python > from celery import Celery, Task > > app = Celery() > > class CustomTask(Task): > > def run(self): > return 'hello' > > app.register_task(CustomTask()) > ``` - **Tasks**: Argument checking now supports keyword-only arguments on Python3 (Issue #3658). > Contributed by `sww`{.interpreted-text role="github_user"}. - **Tasks**: The `task-sent` event was not being sent even if configured to do so (Issue #3646). - **Worker**: Fixed AMQP heartbeat support for eventlet/gevent pools (Issue #3649). - **App**: `app.conf.humanize()` would not work if configuration not finalized (Issue #3652). - **Utils**: `saferepr` attempted to show iterables as lists and mappings as dicts. - **Utils**: `saferepr` did not handle unicode-errors when attempting to format `bytes` on Python 3 (Issue #3610). - **Utils**: `saferepr` should now properly represent byte strings with non-ascii characters (Issue #3600). - **Results**: Fixed bug in elasticsearch where \_index method missed the body argument (Issue #3606). > Fix contributed by **何翔宇** (Sean Ho). - **Canvas**: Fixed `ValueError`{.interpreted-text role="exc"} in chord with single task header (Issue #3608). > Fix contributed by **Viktor Holmqvist**. - **Task**: Ensure class-based task has name prior to registration (Issue #3616). > Fix contributed by **Rick Wargo**. - **Beat**: Fixed problem with strings in shelve (Issue #3644). > Fix contributed by **Alli**. - **Worker**: Fixed `KeyError`{.interpreted-text role="exc"} in `inspect stats` when `-O` argument set to something other than `fast` or `fair` (Issue #3621). - **Task**: Retried tasks were no longer sent to the original queue (Issue #3622). - **Worker**: Python 3: Fixed None/int type comparison in `apps/worker.py`{.interpreted-text role="file"} (Issue #3631). - **Results**: Redis has a new `redis_socket_connect_timeout`{.interpreted-text role="setting"} setting. - **Results**: Redis result backend passed the `socket_connect_timeout` argument to UNIX socket based connections by mistake, causing a crash. - **Worker**: Fixed missing logo in worker splash screen when running on Python 3.x (Issue #3627). > Fix contributed by **Brian Luan**. - **Deps**: Fixed `celery[redis]` bundle installation (Issue #3643). > Fix contributed by **Rémi Marenco**. - **Deps**: Bundle `celery[sqs]` now also requires `pycurl`{.interpreted-text role="pypi"} (Issue #3619). - **Worker**: Hard time limits were no longer being respected (Issue #3618). - **Worker**: Soft time limit log showed `Trues` instead of the number of seconds. - **App**: `registry_cls` argument no longer had any effect (Issue #3613). - **Worker**: Event producer now uses `connection_for_Write` (Issue #3525). - **Results**: Redis/memcache backends now uses `result_expires`{.interpreted-text role="setting"} to expire chord counter (Issue #3573). > Contributed by **Tayfun Sen**. - **Django**: Fixed command for upgrading settings with Django (Issue #3563). > Fix contributed by **François Voron**. - **Testing**: Added a `celery_parameters` test fixture to be able to use customized `Celery` init parameters. (#3626) > Contributed by **Steffen Allner**. - Documentation improvements contributed by > - `csfeathers`{.interpreted-text role="github_user"} > - **Moussa Taifi** > - **Yuhannaa** > - **Laurent Peuch** > - **Christian** > - **Bruno Alla** > - **Steven Johns** > - `tnir`{.interpreted-text role="github_user"} > - **GDR!** ## 4.0.0 {#version-4.0.0} release-date : 2016-11-04 02:00 P.M PDT release-by : Ask Solem See `whatsnew-4.0`{.interpreted-text role="ref"} (in `docs/whatsnew-4.0.rst`{.interpreted-text role="file"}). ## 4.0.0rc7 {#version-4.0.0rc7} release-date : 2016-11-02 01:30 P.M PDT ### Important notes - Database result backend related setting names changed from `sqlalchemy_*` -\> `database_*`. > The `sqlalchemy_` named settings won\'t work at all in this > version so you need to rename them. This is a last minute change, > and as they were not supported in 3.1 we will not be providing > aliases. - `chain(A, B, C)` now works the same way as `A | B | C`. > This means calling `chain()` might not actually return a chain, it > can return a group or any other type depending on how the workflow > can be optimized. --- # Change history {#changelog-4.1} This document contains change notes for bugfix releases in the 4.1.x series, please see `whatsnew-4.2`{.interpreted-text role="ref"} for an overview of what\'s new in Celery 4.2. ## 4.1.1 {#version-4.1.1} release-date : 2018-05-21 12:48 PM PST release-by : Omer Katz ::: important ::: title Important ::: Please upgrade as soon as possible or pin Kombu to 4.1.0. ::: - **Breaking Change**: The module [async]{.title-ref} in Kombu changed to [asynchronous]{.title-ref}. Contributed by **Omer Katz & Asif Saifuddin Auvi** ## 4.1.0 {#version-4.1.0} release-date : 2017-07-25 00:00 PM PST release-by : Omer Katz - **Configuration**: CELERY_SEND_EVENTS instead of CELERYD_SEND_EVENTS for 3.1.x compatibility (#3997) > Contributed by **abhinav nilaratna**. - **App**: Restore behavior so Broadcast queues work. (#3934) > Contributed by **Patrick Cloke**. - **Sphinx**: Make appstr use standard format (#4134) (#4139) > Contributed by **Preston Moore**. - **App**: Make id, name always accessible from logging.Formatter via extra (#3994) > Contributed by **Yoichi NAKAYAMA**. - **Worker**: Add worker_shutting_down signal (#3998) > Contributed by **Daniel Huang**. - **PyPy**: Support PyPy version 5.8.0 (#4128) > Contributed by **Omer Katz**. - **Results**: Elasticsearch: Fix serializing keys (#3924) > Contributed by `staticfox`{.interpreted-text role="github_user"}. - **Canvas**: Deserialize all tasks in a chain (#4015) > Contributed by `fcoelho`{.interpreted-text role="github_user"}. - **Systemd**: Recover loglevel for ExecStart in systemd config (#4023) > Contributed by **Yoichi NAKAYAMA**. - **Sphinx**: Use the Sphinx add_directive_to_domain API. (#4037) > Contributed by **Patrick Cloke**. - **App**: Pass properties to before_task_publish signal (#4035) > Contributed by **Javier Domingo Cansino**. - **Results**: Add SSL option for Redis backends (#3831) > Contributed by **Chris Kuehl**. - **Beat**: celery.schedule.crontab: fix reduce (#3826) (#3827) > Contributed by **Taylor C. Richberger**. - **State**: Fix celery issues when using flower REST API > Contributed by **Thierry RAMORASOAVINA**. - **Results**: Elasticsearch: Fix serializing document id. > Contributed by **Acey9**. - **Beat**: Make shallow copy of schedules dictionary > Contributed by **Brian May**. - **Beat**: Populate heap when periodic tasks are changed > Contributed by **Wojciech Żywno**. - **Task**: Allow class methods to define tasks (#3952) > Contributed by **georgepsarakis**. - **Platforms**: Always return boolean value when checking if signal is supported (#3962). > Contributed by **Jian Yu**. - **Canvas**: Avoid duplicating chains in chords (#3779) > Contributed by **Ryan Hiebert**. - **Canvas**: Lookup task only if list has items (#3847) > Contributed by **Marc Gibbons**. - **Results**: Allow unicode message for exception raised in task (#3903) > Contributed by **George Psarakis**. - **Python3**: Support for Python 3.6 (#3904, #3903, #3736) > Contributed by **Jon Dufresne**, **George Psarakis**, **Asif Saifuddin > Auvi**, **Omer Katz**. - **App**: Fix retried tasks with expirations (#3790) > Contributed by **Brendan MacDonell**. - - Fixes items format route in docs (#3875) > Contributed by **Slam**. - **Utils**: Fix maybe_make_aware (#3850) > Contributed by **Taylor C. Richberger**. - **Task**: Fix task ETA issues when timezone is defined in configuration (#3867) > Contributed by **George Psarakis**. - **Concurrency**: Consumer does not shutdown properly when embedded in gevent application (#3746) > Contributed by **Arcadiy Ivanov**. - **Canvas**: Fix #3725: Task replaced with group does not complete (#3731) > Contributed by **Morgan Doocy**. - **Task**: Correct order in chains with replaced tasks (#3730) > Contributed by **Morgan Doocy**. - **Result**: Enable synchronous execution of sub-tasks (#3696) > Contributed by **shalev67**. - **Task**: Fix request context for blocking task apply (added hostname) (#3716) > Contributed by **Marat Sharafutdinov**. - **Utils**: Fix task argument handling (#3678) (#3693) > Contributed by **Roman Sichny**. - **Beat**: Provide a transparent method to update the Scheduler heap (#3721) > Contributed by **Alejandro Pernin**. - **Beat**: Specify default value for pidfile option of celery beat. (#3722) > Contributed by **Arnaud Rocher**. - **Results**: Elasticsearch: Stop generating a new field every time when a new result is being put (#3708) > Contributed by **Mike Chen**. - **Requirements** > - Now depends on > `Kombu 4.1.0 `{.interpreted-text > role="ref"}. - **Results**: Elasticsearch now reuses fields when new results are added. > Contributed by **Mike Chen**. - **Results**: Fixed MongoDB integration when using binary encodings (Issue #3575). > Contributed by **Andrew de Quincey**. - **Worker**: Making missing `*args` and `**kwargs` in Task protocol 1 return empty value in protocol 2 (Issue #3687). > Contributed by **Roman Sichny**. - **App**: Fixed `TypeError`{.interpreted-text role="exc"} in AMQP when using deprecated signal (Issue #3707). > Contributed by `michael-k`{.interpreted-text role="github_user"}. - **Beat**: Added a transparent method to update the scheduler heap. > Contributed by **Alejandro Pernin**. - **Task**: Fixed handling of tasks with keyword arguments on Python 3 (Issue #3657). > Contributed by **Roman Sichny**. - **Task**: Fixed request context for blocking task apply by adding missing hostname attribute. > Contributed by **Marat Sharafutdinov**. - **Task**: Added option to run subtasks synchronously with `disable_sync_subtasks` argument. > Contributed by `shalev67`{.interpreted-text role="github_user"}. - **App**: Fixed chaining of replaced tasks (Issue #3726). > Contributed by **Morgan Doocy**. - **Canvas**: Fixed bug where replaced tasks with groups were not completing (Issue #3725). > Contributed by **Morgan Doocy**. - **Worker**: Fixed problem where consumer does not shutdown properly when embedded in a gevent application (Issue #3745). > Contributed by **Arcadiy Ivanov**. - **Results**: Added support for using AWS DynamoDB as a result backend (#3736). > Contributed by **George Psarakis**. - **Testing**: Added caching on pip installs. > Contributed by `orf`{.interpreted-text role="github_user"}. - **Worker**: Prevent consuming queue before ready on startup (Issue #3620). > Contributed by **Alan Hamlett**. - **App**: Fixed task ETA issues when timezone is defined in configuration (Issue #3753). > Contributed by **George Psarakis**. - **Utils**: `maybe_make_aware` should not modify datetime when it is already timezone-aware (Issue #3849). > Contributed by **Taylor C. Richberger**. - **App**: Fixed retrying tasks with expirations (Issue #3734). > Contributed by **Brendan MacDonell**. - **Results**: Allow unicode message for exceptions raised in task (Issue #3858). > Contributed by `staticfox`{.interpreted-text role="github_user"}. - **Canvas**: Fixed `IndexError`{.interpreted-text role="exc"} raised when chord has an empty header. > Contributed by **Marc Gibbons**. - **Canvas**: Avoid duplicating chains in chords (Issue #3771). > Contributed by **Ryan Hiebert** and **George Psarakis**. - **Utils**: Allow class methods to define tasks (Issue #3863). > Contributed by **George Psarakis**. - **Beat**: Populate heap when periodic tasks are changed. > Contributed by `wzywno`{.interpreted-text role="github_user"} and > **Brian May**. - **Results**: Added support for Elasticsearch backend options settings. > Contributed by `Acey9`{.interpreted-text role="github_user"}. - **Events**: Ensure `Task.as_dict()` works when not all information about task is available. > Contributed by `tramora`{.interpreted-text role="github_user"}. - **Schedules**: Fixed pickled crontab schedules to restore properly (Issue #3826). > Contributed by **Taylor C. Richberger**. - **Results**: Added SSL option for redis backends (Issue #3830). > Contributed by **Chris Kuehl**. - Documentation and examples improvements by: > - **Bruno Alla** > - **Jamie Alessio** > - **Vivek Anand** > - **Peter Bittner** > - **Kalle Bronsen** > - **Jon Dufresne** > - **James Michael DuPont** > - **Sergey Fursov** > - **Samuel Dion-Girardeau** > - **Daniel Hahler** > - **Mike Helmick** > - **Marc Hörsken** > - **Christopher Hoskin** > - **Daniel Huang** > - **Primož Kerin** > - **Michal Kuffa** > - **Simon Legner** > - **Anthony Lukach** > - **Ed Morley** > - **Jay McGrath** > - **Rico Moorman** > - **Viraj Navkal** > - **Ross Patterson** > - **Dmytro Petruk** > - **Luke Plant** > - **Eric Poelke** > - **Salvatore Rinchiera** > - **Arnaud Rocher** > - **Kirill Romanov** > - **Simon Schmidt** > - **Tamer Sherif** > - **YuLun Shih** > - **Ask Solem** > - **Tom \'Biwaa\' Riat** > - **Arthur Vigil** > - **Joey Wilhelm** > - **Jian Yu** > - **YuLun Shih** > - **Arthur Vigil** > - **Joey Wilhelm** > - `baixuexue123`{.interpreted-text role="github_user"} > - `bronsen`{.interpreted-text role="github_user"} > - `michael-k`{.interpreted-text role="github_user"} > - `orf`{.interpreted-text role="github_user"} > - `3lnc`{.interpreted-text role="github_user"} --- # Change history {#changelog-4.2} This document contains change notes for bugfix releases in the 4.2.x series, please see `whatsnew-4.2`{.interpreted-text role="ref"} for an overview of what\'s new in Celery 4.2. ## 4.2.1 release-date : 2018-07-18 11:00 AM IST release-by : Omer Katz - **Result Backend**: Fix deserialization of exceptions that are present in the producer codebase but not in the consumer codebase. Contributed by **John Arnold** - **Message Protocol Compatibility**: Fix error caused by an invalid (None) timelimit value in the message headers when migrating messages from 3.x to 4.x. Contributed by **Robert Kopaczewski** - **Result Backend**: Fix serialization of exception arguments when exception arguments are not JSON serializable by default. Contributed by **Tom Booth** - **Worker**: Fixed multiple issues with rate limited tasks Maintain scheduling order. Fix possible scheduling of a `celery.worker.request.Request`{.interpreted-text role="class"} with the wrong `kombu.utils.limits.TokenBucket`{.interpreted-text role="class"} which could cause tasks\' rate limit to behave incorrectly. Fix possible duplicated execution of tasks that were rate limited or if ETA/Countdown was provided for them. Contributed by `ideascf`{.interpreted-text role="github_user"} - **Worker**: Defensively handle invalid timelimit header values in requests. Contributed by **Omer Katz** Documentation fixes: > - **Matt Wiens** > - **Seunghun Lee** > - **Lewis M. Kabui** > - **Prathamesh Salunkhe** ## 4.2.0 release-date : 2018-06-10 21:30 PM IST release-by : Omer Katz - **Task**: Add `ignore_result` as task execution option (#4709, #3834) > Contributed by **Andrii Kostenko** and **George Psarakis**. - **Redis Result Backend**: Do not create PubSub subscriptions when results are ignored (#4709, #3834) > Contributed by **Andrii Kostenko** and **George Psarakis**. - **Redis Result Backend**: Result consumer always unsubscribes when task state is ready (#4666) > Contributed by **George Psarakis**. - **Development/Testing**: Add docker-compose and base Dockerfile for development (#4482) > Contributed by **Chris Mitchell**. - **Documentation/Sphinx**: Teach autodoc to document tasks if undoc-members is not set (#4588) > Contributed by **Leo Singer**. - **Documentation/Sphinx**: Put back undoc-members option in sphinx test (#4586) > Contributed by **Leo Singer**. - **Documentation/Sphinx**: Sphinx autodoc picks up tasks automatically only if [undoc-members]{.title-ref} is set (#4584) > Contributed by **Leo Singer**. - **Task**: Fix shadow_name issue when using previous version Task class (#4572) > Contributed by `pachewise`{.interpreted-text role="github_user"}. - **Task**: Add support for bound tasks as [link_error]{.title-ref} parameter (Fixes #3723) (#4545) > Contributed by `brabiega`{.interpreted-text role="github_user"}. - **Deployment**: Add a command line option for setting the Result Backend URL (#4549) > Contributed by `y0ngdi`{.interpreted-text role="github_user"}. - **CI**: Enable pip cache in appveyor build (#4546) > Contributed by **Thijs Triemstra**. - **Concurrency/Asynpool**: Fix errno property name shadowing. > Contributed by **Omer Katz**. - **DynamoDB Backend**: Configurable endpoint URL (#4532) > Contributed by **Bohdan Rybak**. - **Timezones**: Correctly detect UTC timezone and timezone from settings (Fixes #4517) (#4519) > Contributed by `last-partizan`{.interpreted-text > role="github_user"}. - **Control**: Cleanup the mailbox\'s producer pool after forking (#4472) > Contributed by **Nick Eaket**. - **Documentation**: Start Celery and Celery Beat on Azure WebJob (#4484) > Contributed by **PauloPeres**. - **Celery Beat**: Schedule due tasks on startup, after Beat restart has occurred (#4493) > Contributed by **Igor Kasianov**. - **Worker**: Use absolute time when task is accepted by worker pool (#3684) > Contributed by **Régis Behmo**. - **Canvas**: Propagate arguments to chains inside groups (#4481) > Contributed by **Chris Mitchell**. - **Canvas**: Fix [Task.replace]{.title-ref} behavior in nested chords (fixes #4368) (#4369) > Contributed by **Denis Shirokov** & **Alex Hill**. - **Installation**: Pass python_requires argument to setuptools (#4479) > Contributed by **Jon Dufresne**. - **Message Protocol Compatibility**: Handle \"hybrid\" messages that have moved between Celery versions (#4358) (Issue #4356) > Contributed by **Russell Keith-Magee**. - **Canvas**: request on_timeout now ignores soft time limit exception (fixes #4412) (#4473) > Contributed by **Alex Garel**. - **Redis Result Backend**: Integration test to verify PubSub unsubscriptions (#4468) > Contributed by **George Psarakis**. - **Message Protocol Properties**: Allow the shadow keyword argument and the shadow_name method to set shadow properly (#4381) > Contributed by `hclihn`{.interpreted-text role="github_user"}. - **Canvas**: Run chord_unlock on same queue as chord body (#4448) (Issue #4337) > Contributed by **Alex Hill**. - **Canvas**: Support chords with empty header group (#4443) > Contributed by **Alex Hill**. - **Timezones**: make astimezone call in localize more safe (#4324) > Contributed by **Matt Davis**. - **Canvas**: Fix length-1 and nested chords (#4437) (Issues #4393, #4055, #3885, #3597, #3574, #3323, #4301) > Contributed by **Alex Hill**. - **CI**: Run [Openstack Bandit](https://pypi.org/project/bandit/1.0.1/) in Travis CI in order to detect security issues. > Contributed by **Omer Katz**. - **CI**: Run [isort](https://github.com/timothycrosley/isort) in Travis CI in order to lint Python **import** statements. > Contributed by **Omer Katz**. - **Canvas**: Resolve TypeError on [.get]{.title-ref} from nested groups (#4432) (Issue #4274) > Contributed by **Misha Wolfson**. - **CouchDB Backend**: Correct CouchDB key string type for Python 2/3 compatibility (#4166) > Contributed by `fmind`{.interpreted-text role="github_user"} && > **Omer Katz**. - **Group Result**: Fix current_app fallback in GroupResult.restore() (#4431) > Contributed by **Alex Hill**. - **Consul Backend**: Correct key string type for Python 2/3 compatibility (#4416) > Contributed by **Wido den Hollander**. - **Group Result**: Correctly restore an empty GroupResult (#2202) (#4427) > Contributed by **Alex Hill** & **Omer Katz**. - **Result**: Disable synchronous waiting for sub-tasks on eager mode(#4322) > Contributed by **Denis Podlesniy**. - **Celery Beat**: Detect timezone or Daylight Saving Time changes (#1604) (#4403) > Contributed by **Vincent Barbaresi**. - **Canvas**: Fix append to an empty chain. Fixes #4047. (#4402) > Contributed by **Omer Katz**. - **Task**: Allow shadow to override task name in trace and logging messages. (#4379) > Contributed by `hclihn`{.interpreted-text role="github_user"}. - **Documentation/Sphinx**: Fix getfullargspec Python 2.x compatibility in contrib/sphinx.py (#4399) > Contributed by **Javier Martin Montull**. - **Documentation**: Updated installation instructions for SQS broker (#4382) > Contributed by **Sergio Fernandez**. - **Celery Beat**: Better equality comparison for ScheduleEntry instances (#4312) > Contributed by `mariia-zelenova`{.interpreted-text > role="github_user"}. - **Task**: Adding \'shadow\' property to as_task_v2 (#4350) > Contributed by **Marcelo Da Cruz Pinto**. - Try to import directly, do not use deprecated imp method (#4216) > Contributed by **Tobias Kunze**. - **Task**: Enable [kwargsrepr]{.title-ref} and [argsrepr]{.title-ref} override for modifying task argument representation (#4260) > Contributed by **James M. Allen**. - **Result Backend**: Add Redis Sentinel backend (#4144) > Contributed by **Geoffrey Bauduin**. - Use unique time values for Collections/LimitedSet (#3879 and #3891) (#3892) > Contributed by `lead2gold`{.interpreted-text role="github_user"}. - **CI**: Report coverage for all result backends. > Contributed by **Omer Katz**. - **Django**: Use Django DB max age connection setting (fixes #4116) (#4292) > Contributed by **Marco Schweighauser**. - **Canvas**: Properly take into account chain tasks link_error (#4240) > Contributed by `agladkov`{.interpreted-text role="github_user"}. - **Canvas**: Allow to create group with single task (fixes issue #4255) (#4280) > Contributed by `agladkov`{.interpreted-text role="github_user"}. - **Canvas**: Copy dictionary parameter in chord.from_dict before modifying (fixes issue #4223) (#4278) > Contributed by `agladkov`{.interpreted-text role="github_user"}. - **Results Backend**: Add Cassandra options (#4224) > Contributed by **Scott Cooper**. - **Worker**: Apply rate limiting for tasks with ETA (#4251) > Contributed by `arpanshah29`{.interpreted-text > role="github_user"}. - **Celery Beat**: support scheduler entries without a schedule (#4235) > Contributed by **Markus Kaiserswerth**. - **SQS Broker**: Updated SQS requirements file with correct boto3 version (#4231) > Contributed by **Alejandro Varas**. - Remove unused code from \_create_app contextmanager (#4204) > Contributed by **Ryan P Kilby**. - **Group Result**: Modify GroupResult.as_tuple() to include parent (fixes #4106) (#4205) > Contributed by `pachewise`{.interpreted-text role="github_user"}. - **Beat**: Set default scheduler class in beat command. (#4189) > Contributed by `Kxrr`{.interpreted-text role="github_user"}. - **Worker**: Retry signal receiver after raised exception (#4192) > Contributed by **David Davis**. - **Task**: Allow custom Request class for tasks (#3977) > Contributed by **Manuel Vázquez Acosta**. - **Django**: Django fixup should close all cache backends (#4187) > Contributed by **Raphaël Riel**. - **Deployment**: Adds stopasgroup to the supervisor scripts (#4200) > Contributed by `martialp`{.interpreted-text role="github_user"}. - Using Exception.args to serialize/deserialize exceptions (#4085) > Contributed by **Alexander Ovechkin**. - **Timezones**: Correct calculation of application current time with timezone (#4173) > Contributed by **George Psarakis**. - **Remote Debugger**: Set the SO_REUSEADDR option on the socket (#3969) > Contributed by **Theodore Dubois**. - **Django**: Celery ignores exceptions raised during [django.setup()]{.title-ref} (#4146) > Contributed by **Kevin Gu**. - Use heartbeat setting from application configuration for Broker connection (#4148) > Contributed by `mperice`{.interpreted-text role="github_user"}. - **Celery Beat**: Fixed exception caused by next_transit receiving an unexpected argument. (#4103) > Contributed by **DDevine**. - **Task** Introduce exponential backoff with Task auto-retry (#4101) > Contributed by **David Baumgold**. - **AsyncResult**: Remove weak-references to bound methods in AsyncResult promises. (#4131) > Contributed by **Vinod Chandru**. - **Development/Testing**: Allow eager application of canvas structures (#4576) > Contributed by **Nicholas Pilon**. - **Command Line**: Flush stderr before exiting with error code 1. > Contributed by **Antonin Delpeuch**. - **Task**: Escapes single quotes in kwargsrepr strings. > Contributed by **Kareem Zidane** - **AsyncResult**: Restore ability to join over ResultSet after fixing celery/#3818. > Contributed by **Derek Harland** - **Redis Results Backend**: Unsubscribe on message success. Previously Celery would leak channels, filling the memory of the Redis instance. Contributed by **George Psarakis** - **Task**: Only convert eta to isoformat when it is not already a string. Contributed by **Omer Katz** - **Redis Results Backend**: The result_backend setting now supports URIs Contributed by **James Remeika** - **Canvas** Keyword arguments are passed to tasks in chain as expected. Contributed by `tothegump`{.interpreted-text role="github_user"} - **Django** Fix a regression causing Celery to crash when using Django. Contributed by **Jonas Haag** - **Canvas** Chain with one task now runs as expected. Contributed by `tothegump`{.interpreted-text role="github_user"} - **Kombu** Celery 4.2 now requires Kombu 4.2 or better. Contributed by **Omer Katz & Asif Saifuddin Auvi** - [GreenletExit]{.title-ref} is not in [\_\_all\_\_]{.title-ref} in greenlet.py which can not be imported by Python 3.6. The import was adjusted to work on Python 3.6 as well. Contributed by **Hsiaoming Yang** - Fixed a regression that occurred during the development of Celery 4.2 which caused [celery report]{.title-ref} to crash when Django is installed. Contributed by **Josue Balandrano Coronel** - Matched the behavior of [GroupResult.as_tuple()]{.title-ref} to that of [AsyncResult.as_tuple()]{.title-ref}. The group\'s parent is now serialized correctly. Contributed by **Josue Balandrano Coronel** - Use Redis coercion mechanism for converting URI query parameters. Contributed by **Justin Patrin** - Fixed the representation of [GroupResult]{.title-ref}. The dependency graph is now presented correctly. Contributed by **Josue Balandrano Coronel** Documentation, CI, Installation and Tests fixes: > - **Sammie S. Taunton** > - **Dan Wilson** > - `pachewise`{.interpreted-text role="github_user"} > - **Sergi Almacellas Abellana** > - **Omer Katz** > - **Alex Zaitsev** > - **Leo Singer** > - **Rachel Johnson** > - **Jon Dufresne** > - **Samuel Dion-Girardeau** > - **Ryan Guest** > - **Huang Huang** > - **Geoffrey Bauduin** > - **Andrew Wong** > - **Mads Jensen** > - **Jackie Leng** > - **Harry Moreno** > - `michael-k`{.interpreted-text role="github_user"} > - **Nicolas Mota** > - **Armenak Baburyan** > - **Patrick Zhang** > - `anentropic`{.interpreted-text role="github_user"} > - `jairojair`{.interpreted-text role="github_user"} > - **Ben Welsh** > - **Michael Peake** > - **Fengyuan Chen** > - `arpanshah29`{.interpreted-text role="github_user"} > - **Xavier Hardy** > - **Shitikanth** > - **Igor Kasianov** > - **John Arnold** > - `dmollerm`{.interpreted-text role="github_user"} > - **Robert Knight** > - **Asif Saifuddin Auvi** > - **Eduardo Ramírez** > - **Kamil Breguła** > - **Juan Gutierrez** --- # Change history {#changelog-4.3} This document contains change notes for bugfix releases in the 4.3.x series, please see `whatsnew-4.3`{.interpreted-text role="ref"} for an overview of what\'s new in Celery 4.3. ## 4.3.1 release-date : 2020-09-10 1:00 P.M UTC+3:00 release-by : Omer Katz - Limit vine version to be below 5.0.0. Contributed by **Omer Katz** ## 4.3.0 release-date : 2019-03-31 7:00 P.M UTC+3:00 release-by : Omer Katz - Added support for broadcasting using a regular expression pattern or a glob pattern to multiple Pidboxes. This allows you to inspect or ping multiple workers at once. Contributed by **Dmitry Malinovsky** & **Jason Held** - Added support for PEP 420 namespace packages. This allows you to load tasks from namespace packages. Contributed by **Colin Watson** - Added `acks_on_failure_or_timeout`{.interpreted-text role="setting"} as a setting instead of a task only option. This was missing from the original PR but now added for completeness. Contributed by **Omer Katz** - Added the `task_received`{.interpreted-text role="signal"} signal. Contributed by **Omer Katz** - Fixed a crash of our CLI that occurred for everyone using Python \< 3.6. The crash was introduced in [acd6025](https://github.com/celery/celery/commit/acd6025b7dc4db112a31020686fc8b15e1722c67) by using the `ModuleNotFoundError`{.interpreted-text role="class"} exception which was introduced in Python 3.6. Contributed by **Omer Katz** - Fixed a crash that occurred when using the Redis result backend while the `result_expires`{.interpreted-text role="setting"} is set to None. Contributed by **Toni Ruža** & **Omer Katz** - Added support the [DNS seedlist connection format](https://docs.mongodb.com/manual/reference/connection-string/#dns-seedlist-connection-format) for the MongoDB result backend. This requires the [dnspython]{.title-ref} package which will be installed by default when installing the dependencies for the MongoDB result backend. Contributed by **George Psarakis** - Bump the minimum eventlet version to 0.24.1. Contributed by **George Psarakis** - Replace the [msgpack-python]{.title-ref} package with [msgpack]{.title-ref}. We\'re no longer using the deprecated package. See our `important notes `{.interpreted-text role="ref"} for this release for further details on how to upgrade. Contributed by **Daniel Hahler** - Allow scheduling error handlers which are not registered tasks in the current worker. These kind of error handlers are now possible: ``` python from celery import Signature Signature( 'bar', args=['foo'], link_error=Signature('msg.err', queue='msg') ).apply_async() ``` - Additional fixes and enhancements to the SSL support of the Redis broker and result backend. Contributed by **Jeremy Cohen** Code Cleanups, Test Coverage & CI Improvements by: > - **Omer Katz** > - **Florian Chardin** Documentation Fixes by: > - **Omer Katz** > - **Samuel Huang** > - **Amir Hossein Saeid Mehr** > - **Dmytro Litvinov** ## 4.3.0 RC2 release-date : 2019-03-03 9:30 P.M UTC+2:00 release-by : Omer Katz - **Filesystem Backend**: Added meaningful error messages for filesystem backend. Contributed by **Lars Rinn** - **New Result Backend**: Added the ArangoDB backend. Contributed by **Dilip Vamsi Moturi** - **Django**: Prepend current working directory instead of appending so that the project directory will have precedence over system modules as expected. Contributed by **Antonin Delpeuch** - Bump minimum py-redis version to 3.2.0. Due to multiple bugs in earlier versions of py-redis that were causing issues for Celery, we were forced to bump the minimum required version to 3.2.0. Contributed by **Omer Katz** - **Dependencies**: Bump minimum required version of Kombu to 4.4 Contributed by **Omer Katz** ## 4.3.0 RC1 release-date : 2019-02-20 5:00 PM IST release-by : Omer Katz - **Canvas**: `celery.chain.apply`{.interpreted-text role="meth"} does not ignore keyword arguments anymore when applying the chain. Contributed by **Korijn van Golen** - **Result Set**: Don\'t attempt to cache results in a `celery.result.ResultSet`{.interpreted-text role="class"}. During a join, the results cache was populated using `celery.result.ResultSet.get`{.interpreted-text role="meth"}, if one of the results contains an exception, joining unexpectedly failed. The results cache is now removed. Contributed by **Derek Harland** - **Application**: `celery.Celery.autodiscover_tasks`{.interpreted-text role="meth"} now attempts to import the package itself when the [related_name]{.title-ref} keyword argument is [None]{.title-ref}. Contributed by **Alex Ioannidis** - **Windows Support**: On Windows 10, stale PID files prevented celery beat to run. We now remove them when a `SystemExit`{.interpreted-text role="class"} is raised. Contributed by **:github_user:\`na387\`** - **Task**: Added the new `task_acks_on_failure_or_timeout`{.interpreted-text role="setting"} setting. Acknowledging SQS messages on failure or timing out makes it impossible to use dead letter queues. We introduce the new option acks_on_failure_or_timeout, to ensure we can totally fallback on native SQS message lifecycle, using redeliveries for retries (in case of slow processing or failure) and transitions to dead letter queue after defined number of times. Contributed by **Mario Kostelac** - **RabbitMQ Broker**: Adjust HA headers to work on RabbitMQ 3.x. This change also means we\'re ending official support for RabbitMQ 2.x. Contributed by **Asif Saif Uddin** - **Command Line**: Improve `celery update`{.interpreted-text role="program"} error handling. Contributed by **Federico Bond** - **Canvas**: Support chords with `task_always_eager`{.interpreted-text role="setting"} set to [True]{.title-ref}. Contributed by **Axel Haustant** - **Result Backend**: Optionally store task properties in result backend. Setting the `result_extended`{.interpreted-text role="setting"} configuration option to [True]{.title-ref} enables storing additional task properties in the result backend. Contributed by **John Arnold** - **Couchbase Result Backend**: Allow the Couchbase result backend to automatically detect the serialization format. Contributed by **Douglas Rohde** - **New Result Backend**: Added the Azure Block Blob Storage result backend. The backend is implemented on top of the azure-storage library which uses Azure Blob Storage for a scalable low-cost PaaS backend. The backend was load tested via a simple nginx/gunicorn/sanic app hosted on a DS4 virtual machine (4 vCores, 16 GB RAM) and was able to handle 600+ concurrent users at \~170 RPS. The commit also contains a live end-to-end test to facilitate verification of the backend functionality. The test is activated by setting the [AZUREBLOCKBLOB_URL]{.title-ref} environment variable to [azureblockblob://{ConnectionString}]{.title-ref} where the value for [ConnectionString]{.title-ref} can be found in the [Access Keys]{.title-ref} pane of a Storage Account resources in the Azure Portal. Contributed by **Clemens Wolff** - **Task**: `celery.app.task.update_state`{.interpreted-text role="meth"} now accepts keyword arguments. This allows passing extra fields to the result backend. These fields are unused by default but custom result backends can use them to determine how to store results. Contributed by **Christopher Dignam** - Gracefully handle consumer `kombu.exceptions.DecodeError`{.interpreted-text role="class"}. When using the v2 protocol the worker no longer crashes when the consumer encounters an error while decoding a message. Contributed by **Steven Sklar** - **Deployment**: Fix init.d service stop. Contributed by **Marcus McHale** - **Django**: Drop support for Django \< 1.11. Contributed by **Asif Saif Uddin** - **Django**: Remove old djcelery loader. Contributed by **Asif Saif Uddin** - **Result Backend**: `celery.worker.request.Request`{.interpreted-text role="class"} now passes `celery.app.task.Context`{.interpreted-text role="class"} to the backend\'s store_result functions. Since the class currently passes [self]{.title-ref} to these functions, revoking a task resulted in corrupted task result data when django-celery-results was used. Contributed by **Kiyohiro Yamaguchi** - **Worker**: Retry if the heartbeat connection dies. Previously, we keep trying to write to the broken connection. This results in a memory leak because the event dispatcher will keep appending the message to the outbound buffer. Contributed by **Raf Geens** - **Celery Beat**: Handle microseconds when scheduling. Contributed by **K Davis** - **Asynpool**: Fixed deadlock when closing socket. Upon attempting to close a socket, `celery.concurrency.asynpool.AsynPool`{.interpreted-text role="class"} only removed the queue writer from the hub but did not remove the reader. This led to a deadlock on the file descriptor and eventually the worker stopped accepting new tasks. We now close both the reader and the writer file descriptors in a single loop iteration which prevents the deadlock. Contributed by **Joshua Engelman** - **Celery Beat**: Correctly consider timezone when calculating timestamp. Contributed by **:github_user:\`yywing\`** - **Celery Beat**: `celery.beat.Scheduler.schedules_equal`{.interpreted-text role="meth"} can now handle either arguments being a [None]{.title-ref} value. Contributed by **:github_user:\` ratson\`** - **Documentation/Sphinx**: Fixed Sphinx support for shared_task decorated functions. Contributed by **Jon Banafato** - **New Result Backend**: Added the CosmosDB result backend. This change adds a new results backend. The backend is implemented on top of the pydocumentdb library which uses Azure CosmosDB for a scalable, globally replicated, high-performance, low-latency and high-throughput PaaS backend. Contributed by **Clemens Wolff** - **Application**: Added configuration options to allow separate multiple apps to run on a single RabbitMQ vhost. The newly added `event_exchange`{.interpreted-text role="setting"} and `control_exchange`{.interpreted-text role="setting"} configuration options allow users to use separate Pidbox exchange and a separate events exchange. This allow different Celery applications to run separately on the same vhost. Contributed by **Artem Vasilyev** - **Result Backend**: Forget parent result metadata when forgetting a result. Contributed by **:github_user:\`tothegump\`** - **Task** Store task arguments inside `celery.exceptions.MaxRetriesExceededError`{.interpreted-text role="class"}. Contributed by **Anthony Ruhier** - **Result Backend**: Added the `result_accept_content`{.interpreted-text role="setting"} setting. This feature allows to configure different accepted content for the result backend. A special serializer ([auth]{.title-ref}) is used for signed messaging, however the result_serializer remains in json, because we don\'t want encrypted content in our result backend. To accept unsigned content from the result backend, we introduced this new configuration option to specify the accepted content from the backend. - **Canvas**: Fixed error callback processing for class based tasks. Contributed by **Victor Mireyev** - **New Result Backend**: Added the S3 result backend. Contributed by **Florian Chardin** - **Task**: Added support for Cythonized Celery tasks. Contributed by **Andrey Skabelin** - **Riak Result Backend**: Warn Riak backend users for possible Python 3.7 incompatibilities. Contributed by **George Psarakis** - **Python Runtime**: Added Python 3.7 support. Contributed by **Omer Katz** & **Asif Saif Uddin** - **Auth Serializer**: Revamped the auth serializer. The auth serializer received a complete overhaul. It was previously horribly broken. We now depend on cryptography instead of pyOpenSSL for this serializer. - **Command Line**: `celery report`{.interpreted-text role="program"} now reports kernel version along with other platform details. Contributed by **Omer Katz** - **Canvas**: Fixed chords with chains which include sub chords in a group. Celery now correctly executes the last task in these types of canvases: ``` python c = chord( group([ chain( dummy.si(), chord( group([dummy.si(), dummy.si()]), dummy.si(), ), ), chain( dummy.si(), chord( group([dummy.si(), dummy.si()]), dummy.si(), ), ), ]), dummy.si() ) c.delay().get() ``` Contributed by **Maximilien Cuony** - **Canvas**: Complex canvases with error callbacks no longer raises an `AttributeError`{.interpreted-text role="class"}. Very complex canvases such as [this](https://github.com/merchise/xopgi.base/blob/6634819ad5c701c04bc9baa5c527449070843b71/xopgi/xopgi_cdr/cdr_agent.py#L181) no longer raise an `AttributeError`{.interpreted-text role="class"} which prevents constructing them. We do not know why this bug occurs yet. Contributed by **Manuel Vázquez Acosta** - **Command Line**: Added proper error messages in cases where app cannot be loaded. Previously, celery crashed with an exception. We now print a proper error message. Contributed by **Omer Katz** - **Task**: Added the `task_default_priority`{.interpreted-text role="setting"} setting. You can now set the default priority of a task using the `task_default_priority`{.interpreted-text role="setting"} setting. The setting\'s value will be used if no priority is provided for a specific task. Contributed by **:github_user:\`madprogrammer\`** - **Dependencies**: Bump minimum required version of Kombu to 4.3 and Billiard to 3.6. Contributed by **Asif Saif Uddin** - **Result Backend**: Fix memory leak. We reintroduced weak references to bound methods for AsyncResult callback promises, after adding full weakref support for Python 2 in [vine](https://github.com/celery/vine/tree/v1.2.0). More details can be found in [celery/celery#4839](https://github.com/celery/celery/pull/4839). Contributed by **George Psarakis** and **:github_user:\`monsterxx03\`**. - **Task Execution**: Fixed roundtrip serialization for eager tasks. When doing the roundtrip serialization for eager tasks, the task serializer will always be JSON unless the [serializer]{.title-ref} argument is present in the call to `celery.app.task.Task.apply_async`{.interpreted-text role="meth"}. If the serializer argument is present but is [\'pickle\']{.title-ref}, an exception will be raised as pickle-serialized objects cannot be deserialized without specifying to [serialization.loads]{.title-ref} what content types should be accepted. The Producer\'s [serializer]{.title-ref} seems to be set to [None]{.title-ref}, causing the default to JSON serialization. We now continue to use (in order) the [serializer]{.title-ref} argument to `celery.app.task.Task.apply_async`{.interpreted-text role="meth"}, if present, or the [Producer]{.title-ref}\'s serializer if not [None]{.title-ref}. If the [Producer]{.title-ref}\'s serializer is [None]{.title-ref}, it will use the Celery app\'s [task_serializer]{.title-ref} configuration entry as the serializer. Contributed by **Brett Jackson** - **Redis Result Backend**: The `celery.backends.redis.ResultConsumer`{.interpreted-text role="class"} class no longer assumes `celery.backends.redis.ResultConsumer.start`{.interpreted-text role="meth"} to be called before `celery.backends.redis.ResultConsumer.drain_events`{.interpreted-text role="meth"}. This fixes a race condition when using the Gevent workers pool. Contributed by **Noam Kush** - **Task**: Added the `task_inherit_parent_priority`{.interpreted-text role="setting"} setting. Setting the `task_inherit_parent_priority`{.interpreted-text role="setting"} configuration option to [True]{.title-ref} will make Celery tasks inherit the priority of the previous task linked to it. Examples: ``` python c = celery.chain( add.s(2), # priority=None add.s(3).set(priority=5), # priority=5 add.s(4), # priority=5 add.s(5).set(priority=3), # priority=3 add.s(6), # priority=3 ) ``` ``` python @app.task(bind=True) def child_task(self): pass @app.task(bind=True) def parent_task(self): child_task.delay() # child_task will also have priority=5 parent_task.apply_async(args=[], priority=5) ``` Contributed by **:github_user:\`madprogrammer\`** - **Canvas**: Added the `result_chord_join_timeout`{.interpreted-text role="setting"} setting. Previously, `celery.result.GroupResult.join`{.interpreted-text role="meth"} had a fixed timeout of 3 seconds. The `result_chord_join_timeout`{.interpreted-text role="setting"} setting now allows you to change it. Contributed by **:github_user:\`srafehi\`** Code Cleanups, Test Coverage & CI Improvements by: > - **Jon Dufresne** > - **Asif Saif Uddin** > - **Omer Katz** > - **Brett Jackson** > - **Bruno Alla** > - **:github_user:\`tothegump\`** > - **Bojan Jovanovic** > - **Florian Chardin** > - **:github_user:\`walterqian\`** > - **Fabian Becker** > - **Lars Rinn** > - **:github_user:\`madprogrammer\`** > - **Ciaran Courtney** Documentation Fixes by: > - **Lewis M. Kabui** > - **Dash Winterson** > - **Shanavas M** > - **Brett Randall** > - **Przemysław Suliga** > - **Joshua Schmid** > - **Asif Saif Uddin** > - **Xiaodong** > - **Vikas Prasad** > - **Jamie Alessio** > - **Lars Kruse** > - **Guilherme Caminha** > - **Andrea Rabbaglietti** > - **Itay Bittan** > - **Noah Hall** > - **Peng Weikang** > - **Mariatta Wijaya** > - **Ed Morley** > - **Paweł Adamczak** > - **:github_user:\`CoffeeExpress\`** > - **:github_user:\`aviadatsnyk\`** > - **Brian Schrader** > - **Josue Balandrano Coronel** > - **Tom Clancy** > - **Sebastian Wojciechowski** > - **Meysam Azad** > - **Willem Thiart** > - **Charles Chan** > - **Omer Katz** > - **Milind Shakya** --- # Change history {#changelog-4.4} This document contains change notes for bugfix & new features in the 4.4.x series, please see `whatsnew-4.4`{.interpreted-text role="ref"} for an overview of what\'s new in Celery 4.4. ## 4.4.7 release-date : 2020-07-31 11.45 P.M UTC+6:00 release-by : Asif Saif Uddin - Add task_received, task_rejected and task_unknown to signals module. - \[ES backend\] add 401 as safe for retry. - treat internal errors as failure. - Remove redis fanout caveats. - FIX: -A and \--args should behave the same. (#6223) - Class-based tasks autoretry (#6233) - Preserve order of group results with Redis result backend (#6218) - Replace future with celery.five Fixes #6250, and use raise_with_context instead of reraise - Fix REMAP_SIGTERM=SIGQUIT not working - (Fixes#6258) MongoDB: fix for serialization issue (#6259) - Make use of ordered sets in Redis opt-in - Test, CI, Docker & style and minor doc improvements. ## 4.4.6 release-date : 2020-06-24 2.40 P.M UTC+6:00 release-by : Asif Saif Uddin - Remove autoscale force_scale methods (#6085). - Fix autoscale test - Pass ping destination to request - chord: merge init options with run options - Put back KeyValueStoreBackend.set method without state - Added \--range-prefix option to [celery multi]{.title-ref} (#6180) - Added as_list function to AsyncResult class (#6179) - Fix CassandraBackend error in threads or gevent pool (#6147) - Kombu 4.6.11 ## 4.4.5 release-date : 2020-06-08 12.15 P.M UTC+6:00 release-by : Asif Saif Uddin - Add missing dependency on future (#6146). - ElasticSearch: Retry index if document was deleted between index - fix windows build - Customize the retry interval of chord_unlock tasks - fix multi tests in local ## 4.4.4 release-date : 2020-06-03 11.00 A.M UTC+6:00 release-by : Asif Saif Uddin - Fix autoretry_for with explicit retry (#6138). - Kombu 4.6.10 - Use Django DB max age connection setting (fixes #4116). - Add retry on recoverable exception for the backend (#6122). - Fix random distribution of jitter for exponential backoff. - ElasticSearch: add setting to save meta as json. - fix #6136. celery 4.4.3 always trying create /var/run/celery directory. - Add task_internal_error signal (#6049). ## 4.4.3 release-date : 2020-06-01 4.00 P.M UTC+6:00 release-by : Asif Saif Uddin - Fix backend utf-8 encoding in s3 backend . - Kombu 4.6.9 - Task class definitions can have retry attributes (#5869) - Upgraded pycurl to the latest version that supports wheel. - Add uptime to the stats inspect command - Fixing issue #6019: unable to use mysql SSL parameters when getting - Clean TraceBack to reduce memory leaks for exception task (#6024) - exceptions: NotRegistered: fix up language - Give up sending a worker-offline message if transport is not connected - Add Task to \_\_all\_\_ in celery.\_\_init\_\_.py - Ensure a single chain object in a chain does not raise MaximumRecursion - Fix autoscale when prefetch_multiplier is 1 - Allow start_worker to function without ping task - Update celeryd.conf - Fix correctly handle configuring the serializer for always_eager mode. - Remove doubling of prefetch_count increase when prefetch_multiplier - Fix eager function not returning result after retries - return retry result if not throw and is_eager - Always requeue while worker lost regardless of the redelivered flag - Allow relative paths in the filesystem backend (#6070) - \[Fixed Issue #6017\] - Avoid race condition due to task duplication. - Exceptions must be old-style classes or derived from BaseException - Fix windows build (#6104) - Add encode to meta task in base.py (#5894) - Update time.py to solve the microsecond issues (#5199) - Change backend \_ensure_not_eager error to warning - Add priority support for \'celery.chord_unlock\' task (#5766) - Change eager retry behaviour - Avoid race condition in elasticsearch backend - backends base get_many pass READY_STATES arg - Add integration tests for Elasticsearch and fix \_update - feat(backend): Adds cleanup to ArangoDB backend - remove jython check - fix filesystem backend cannot not be serialized by picked ## 4.4.0 release-date : 2019-12-16 9.45 A.M UTC+6:00 release-by : Asif Saif Uddin - This version is officially supported on CPython 2.7, 3.5, 3.6, 3.7 & 3.8 and is also supported on PyPy2 & PyPy3. - Kombu 4.6.7 - Task class definitions can have retry attributes (#5869) ## 4.4.0rc5 release-date : 2019-12-07 21.05 A.M UTC+6:00 release-by : Asif Saif Uddin - Kombu 4.6.7 - Events bootstep disabled if no events (#5807) - SQS - Reject on failure (#5843) - Add a concurrency model with ThreadPoolExecutor (#5099) - Add auto expiry for DynamoDB backend (#5805) - Store extending result in all backends (#5661) - Fix a race condition when publishing a very large chord header (#5850) - Improve docs and test matrix ## 4.4.0rc4 release-date : 2019-11-11 00.45 A.M UTC+6:00 release-by : Asif Saif Uddin - Kombu 4.6.6 - Py-AMQP 2.5.2 - Python 3.8 - Numerious bug fixes - PyPy 7.2 ## 4.4.0rc3 release-date : 2019-08-14 23.00 P.M UTC+6:00 release-by : Asif Saif Uddin - Kombu 4.6.4 - Billiard 3.6.1 - Py-AMQP 2.5.1 - Avoid serializing datetime (#5606) - Fix: (group() \| group()) not equals single group (#5574) - Revert \"Broker connection uses the heartbeat setting from app config. - Additional file descriptor safety checks. - fixed call for null args (#5631) - Added generic path for cache backend. - Fix Nested group(chain(group)) fails (#5638) - Use self.run() when overriding \_\_call\_\_ (#5652) - Fix termination of asyncloop (#5671) - Fix migrate task to work with both v1 and v2 of the message protocol. - Updating task_routes config during runtime now have effect. ## 4.4.0rc2 release-date : 2019-06-15 4:00 A.M UTC+6:00 release-by : Asif Saif Uddin - Many bugs and regressions fixed. - Kombu 4.6.3 ## 4.4.0rc1 release-date : 2019-06-06 1:00 P.M UTC+6:00 release-by : Asif Saif Uddin - Python 3.4 drop - Kombu 4.6.1 - Replace deprecated PyMongo methods usage (#5443) - Pass task request when calling update_state (#5474) - Fix bug in remaining time calculation in case of DST time change (#5411) - Fix missing task name when requesting extended result (#5439) - Fix [collections]{.title-ref} import issue on Python 2.7 (#5428) - handle [AttributeError]{.title-ref} in base backend exception deserializer (#5435) - Make [AsynPool]{.title-ref}\'s [proc_alive_timeout]{.title-ref} configurable (#5476) - AMQP Support for extended result (#5495) - Fix SQL Alchemy results backend to work with extended result (#5498) - Fix restoring of exceptions with required param (#5500) - Django: Re-raise exception if [ImportError]{.title-ref} not caused by missing tasks module (#5211) - Django: fixed a regression putting DB connections in invalid state when [CONN_MAX_AGE != 0]{.title-ref} (#5515) - Fixed [OSError]{.title-ref} leading to lost connection to broker (#4457) - Fixed an issue with inspect API unable get details of Request - Fix mogodb backend authentication (#5527) - Change column type for Extended Task Meta args/kwargs to LargeBinary - Handle http_auth in Elasticsearch backend results (#5545) - Fix task serializer being ignored with [task_always_eager=True]{.title-ref} (#5549) - Fix [task.replace]{.title-ref} to work in [.apply() as well as ]{.title-ref}.apply_async()\` (#5540) - Fix sending of [worker_process_init]{.title-ref} signal for solo worker (#5562) - Fix exception message upacking (#5565) - Add delay parameter function to beat_schedule (#5558) - Multiple documentation updates ## 4.3.0 release-date : 2019-03-31 7:00 P.M UTC+3:00 release-by : Omer Katz - Added support for broadcasting using a regular expression pattern or a glob pattern to multiple Pidboxes. This allows you to inspect or ping multiple workers at once. Contributed by **Dmitry Malinovsky** & **Jason Held** - Added support for PEP 420 namespace packages. This allows you to load tasks from namespace packages. Contributed by **Colin Watson** - Added `acks_on_failure_or_timeout`{.interpreted-text role="setting"} as a setting instead of a task only option. This was missing from the original PR but now added for completeness. Contributed by **Omer Katz** - Added the `task_received`{.interpreted-text role="signal"} signal. Contributed by **Omer Katz** - Fixed a crash of our CLI that occurred for everyone using Python \< 3.6. The crash was introduced in [acd6025](https://github.com/celery/celery/commit/acd6025b7dc4db112a31020686fc8b15e1722c67) by using the `ModuleNotFoundError`{.interpreted-text role="class"} exception which was introduced in Python 3.6. Contributed by **Omer Katz** - Fixed a crash that occurred when using the Redis result backend while the `result_expires`{.interpreted-text role="setting"} is set to None. Contributed by **Toni Ruža** & **Omer Katz** - Added support the [DNS seedlist connection format](https://docs.mongodb.com/manual/reference/connection-string/#dns-seedlist-connection-format) for the MongoDB result backend. This requires the [dnspython]{.title-ref} package which will be installed by default when installing the dependencies for the MongoDB result backend. Contributed by **George Psarakis** - Bump the minimum eventlet version to 0.24.1. Contributed by **George Psarakis** - Replace the [msgpack-python]{.title-ref} package with [msgpack]{.title-ref}. We\'re no longer using the deprecated package. See our `important notes `{.interpreted-text role="ref"} for this release for further details on how to upgrade. Contributed by **Daniel Hahler** - Allow scheduling error handlers which are not registered tasks in the current worker. These kind of error handlers are now possible: ``` python from celery import Signature Signature( 'bar', args=['foo'], link_error=Signature('msg.err', queue='msg') ).apply_async() ``` - Additional fixes and enhancements to the SSL support of the Redis broker and result backend. Contributed by **Jeremy Cohen** Code Cleanups, Test Coverage & CI Improvements by: > - **Omer Katz** > - **Florian Chardin** Documentation Fixes by: > - **Omer Katz** > - **Samuel Huang** > - **Amir Hossein Saeid Mehr** > - **Dmytro Litvinov** ## 4.3.0 RC2 release-date : 2019-03-03 9:30 P.M UTC+2:00 release-by : Omer Katz - **Filesystem Backend**: Added meaningful error messages for filesystem backend. Contributed by **Lars Rinn** - **New Result Backend**: Added the ArangoDB backend. Contributed by **Dilip Vamsi Moturi** - **Django**: Prepend current working directory instead of appending so that the project directory will have precedence over system modules as expected. Contributed by **Antonin Delpeuch** - Bump minimum py-redis version to 3.2.0. Due to multiple bugs in earlier versions of py-redis that were causing issues for Celery, we were forced to bump the minimum required version to 3.2.0. Contributed by **Omer Katz** - **Dependencies**: Bump minimum required version of Kombu to 4.4 Contributed by **Omer Katz** ## 4.3.0 RC1 release-date : 2019-02-20 5:00 PM IST release-by : Omer Katz - **Canvas**: `celery.chain.apply`{.interpreted-text role="meth"} does not ignore keyword arguments anymore when applying the chain. Contributed by **Korijn van Golen** - **Result Set**: Don\'t attempt to cache results in a `celery.result.ResultSet`{.interpreted-text role="class"}. During a join, the results cache was populated using `celery.result.ResultSet.get`{.interpreted-text role="meth"}, if one of the results contains an exception, joining unexpectedly failed. The results cache is now removed. Contributed by **Derek Harland** - **Application**: `celery.Celery.autodiscover_tasks`{.interpreted-text role="meth"} now attempts to import the package itself when the [related_name]{.title-ref} keyword argument is [None]{.title-ref}. Contributed by **Alex Ioannidis** - **Windows Support**: On Windows 10, stale PID files prevented celery beat to run. We now remove them when a `SystemExit`{.interpreted-text role="class"} is raised. Contributed by **:github_user:\`na387\`** - **Task**: Added the new `task_acks_on_failure_or_timeout`{.interpreted-text role="setting"} setting. Acknowledging SQS messages on failure or timing out makes it impossible to use dead letter queues. We introduce the new option acks_on_failure_or_timeout, to ensure we can totally fallback on native SQS message lifecycle, using redeliveries for retries (in case of slow processing or failure) and transitions to dead letter queue after defined number of times. Contributed by **Mario Kostelac** - **RabbitMQ Broker**: Adjust HA headers to work on RabbitMQ 3.x. This change also means we\'re ending official support for RabbitMQ 2.x. Contributed by **Asif Saif Uddin** - **Command Line**: Improve `celery update`{.interpreted-text role="program"} error handling. Contributed by **Federico Bond** - **Canvas**: Support chords with `task_always_eager`{.interpreted-text role="setting"} set to [True]{.title-ref}. Contributed by **Axel Haustant** - **Result Backend**: Optionally store task properties in result backend. Setting the `result_extended`{.interpreted-text role="setting"} configuration option to [True]{.title-ref} enables storing additional task properties in the result backend. Contributed by **John Arnold** - **Couchbase Result Backend**: Allow the Couchbase result backend to automatically detect the serialization format. Contributed by **Douglas Rohde** - **New Result Backend**: Added the Azure Block Blob Storage result backend. The backend is implemented on top of the azure-storage library which uses Azure Blob Storage for a scalable low-cost PaaS backend. The backend was load tested via a simple nginx/gunicorn/sanic app hosted on a DS4 virtual machine (4 vCores, 16 GB RAM) and was able to handle 600+ concurrent users at \~170 RPS. The commit also contains a live end-to-end test to facilitate verification of the backend functionality. The test is activated by setting the [AZUREBLOCKBLOB_URL]{.title-ref} environment variable to [azureblockblob://{ConnectionString}]{.title-ref} where the value for [ConnectionString]{.title-ref} can be found in the [Access Keys]{.title-ref} pane of a Storage Account resources in the Azure Portal. Contributed by **Clemens Wolff** - **Task**: `celery.app.task.update_state`{.interpreted-text role="meth"} now accepts keyword arguments. This allows passing extra fields to the result backend. These fields are unused by default but custom result backends can use them to determine how to store results. Contributed by **Christopher Dignam** - Gracefully handle consumer `kombu.exceptions.DecodeError`{.interpreted-text role="class"}. When using the v2 protocol the worker no longer crashes when the consumer encounters an error while decoding a message. Contributed by **Steven Sklar** - **Deployment**: Fix init.d service stop. Contributed by **Marcus McHale** - **Django**: Drop support for Django \< 1.11. Contributed by **Asif Saif Uddin** - **Django**: Remove old djcelery loader. Contributed by **Asif Saif Uddin** - **Result Backend**: `celery.worker.request.Request`{.interpreted-text role="class"} now passes `celery.app.task.Context`{.interpreted-text role="class"} to the backend\'s store_result functions. Since the class currently passes [self]{.title-ref} to these functions, revoking a task resulted in corrupted task result data when django-celery-results was used. Contributed by **Kiyohiro Yamaguchi** - **Worker**: Retry if the heartbeat connection dies. Previously, we keep trying to write to the broken connection. This results in a memory leak because the event dispatcher will keep appending the message to the outbound buffer. Contributed by **Raf Geens** - **Celery Beat**: Handle microseconds when scheduling. Contributed by **K Davis** - **Asynpool**: Fixed deadlock when closing socket. Upon attempting to close a socket, `celery.concurrency.asynpool.AsynPool`{.interpreted-text role="class"} only removed the queue writer from the hub but did not remove the reader. This led to a deadlock on the file descriptor and eventually the worker stopped accepting new tasks. We now close both the reader and the writer file descriptors in a single loop iteration which prevents the deadlock. Contributed by **Joshua Engelman** - **Celery Beat**: Correctly consider timezone when calculating timestamp. Contributed by **:github_user:\`yywing\`** - **Celery Beat**: `celery.beat.Scheduler.schedules_equal`{.interpreted-text role="meth"} can now handle either arguments being a [None]{.title-ref} value. Contributed by **:github_user:\` ratson\`** - **Documentation/Sphinx**: Fixed Sphinx support for shared_task decorated functions. Contributed by **Jon Banafato** - **New Result Backend**: Added the CosmosDB result backend. This change adds a new results backend. The backend is implemented on top of the pydocumentdb library which uses Azure CosmosDB for a scalable, globally replicated, high-performance, low-latency and high-throughput PaaS backend. Contributed by **Clemens Wolff** - **Application**: Added configuration options to allow separate multiple apps to run on a single RabbitMQ vhost. The newly added `event_exchange`{.interpreted-text role="setting"} and `control_exchange`{.interpreted-text role="setting"} configuration options allow users to use separate Pidbox exchange and a separate events exchange. This allow different Celery applications to run separately on the same vhost. Contributed by **Artem Vasilyev** - **Result Backend**: Forget parent result metadata when forgetting a result. Contributed by **:github_user:\`tothegump\`** - **Task** Store task arguments inside `celery.exceptions.MaxRetriesExceededError`{.interpreted-text role="class"}. Contributed by **Anthony Ruhier** - **Result Backend**: Added the `result_accept_content`{.interpreted-text role="setting"} setting. This feature allows to configure different accepted content for the result backend. A special serializer ([auth]{.title-ref}) is used for signed messaging, however the result_serializer remains in json, because we don\'t want encrypted content in our result backend. To accept unsigned content from the result backend, we introduced this new configuration option to specify the accepted content from the backend. - **Canvas**: Fixed error callback processing for class based tasks. Contributed by **Victor Mireyev** - **New Result Backend**: Added the S3 result backend. Contributed by **Florian Chardin** - **Task**: Added support for Cythonized Celery tasks. Contributed by **Andrey Skabelin** - **Riak Result Backend**: Warn Riak backend users for possible Python 3.7 incompatibilities. Contributed by **George Psarakis** - **Python Runtime**: Added Python 3.7 support. Contributed by **Omer Katz** & **Asif Saif Uddin** - **Auth Serializer**: Revamped the auth serializer. The auth serializer received a complete overhaul. It was previously horribly broken. We now depend on cryptography instead of pyOpenSSL for this serializer. - **Command Line**: `celery report`{.interpreted-text role="program"} now reports kernel version along with other platform details. Contributed by **Omer Katz** - **Canvas**: Fixed chords with chains which include sub chords in a group. Celery now correctly executes the last task in these types of canvases: ``` python c = chord( group([ chain( dummy.si(), chord( group([dummy.si(), dummy.si()]), dummy.si(), ), ), chain( dummy.si(), chord( group([dummy.si(), dummy.si()]), dummy.si(), ), ), ]), dummy.si() ) c.delay().get() ``` Contributed by **Maximilien Cuony** - **Canvas**: Complex canvases with error callbacks no longer raises an `AttributeError`{.interpreted-text role="class"}. Very complex canvases such as [this](https://github.com/merchise/xopgi.base/blob/6634819ad5c701c04bc9baa5c527449070843b71/xopgi/xopgi_cdr/cdr_agent.py#L181) no longer raise an `AttributeError`{.interpreted-text role="class"} which prevents constructing them. We do not know why this bug occurs yet. Contributed by **Manuel Vázquez Acosta** - **Command Line**: Added proper error messages in cases where app cannot be loaded. Previously, celery crashed with an exception. We now print a proper error message. Contributed by **Omer Katz** - **Task**: Added the `task_default_priority`{.interpreted-text role="setting"} setting. You can now set the default priority of a task using the `task_default_priority`{.interpreted-text role="setting"} setting. The setting\'s value will be used if no priority is provided for a specific task. Contributed by **:github_user:\`madprogrammer\`** - **Dependencies**: Bump minimum required version of Kombu to 4.3 and Billiard to 3.6. Contributed by **Asif Saif Uddin** - **Result Backend**: Fix memory leak. We reintroduced weak references to bound methods for AsyncResult callback promises, after adding full weakref support for Python 2 in [vine](https://github.com/celery/vine/tree/v1.2.0). More details can be found in [celery/celery#4839](https://github.com/celery/celery/pull/4839). Contributed by **George Psarakis** and **:github_user:\`monsterxx03\`**. - **Task Execution**: Fixed roundtrip serialization for eager tasks. When doing the roundtrip serialization for eager tasks, the task serializer will always be JSON unless the [serializer]{.title-ref} argument is present in the call to `celery.app.task.Task.apply_async`{.interpreted-text role="meth"}. If the serializer argument is present but is [\'pickle\']{.title-ref}, an exception will be raised as pickle-serialized objects cannot be deserialized without specifying to [serialization.loads]{.title-ref} what content types should be accepted. The Producer\'s [serializer]{.title-ref} seems to be set to [None]{.title-ref}, causing the default to JSON serialization. We now continue to use (in order) the [serializer]{.title-ref} argument to `celery.app.task.Task.apply_async`{.interpreted-text role="meth"}, if present, or the [Producer]{.title-ref}\'s serializer if not [None]{.title-ref}. If the [Producer]{.title-ref}\'s serializer is [None]{.title-ref}, it will use the Celery app\'s [task_serializer]{.title-ref} configuration entry as the serializer. Contributed by **Brett Jackson** - **Redis Result Backend**: The `celery.backends.redis.ResultConsumer`{.interpreted-text role="class"} class no longer assumes `celery.backends.redis.ResultConsumer.start`{.interpreted-text role="meth"} to be called before `celery.backends.redis.ResultConsumer.drain_events`{.interpreted-text role="meth"}. This fixes a race condition when using the Gevent workers pool. Contributed by **Noam Kush** - **Task**: Added the `task_inherit_parent_priority`{.interpreted-text role="setting"} setting. Setting the `task_inherit_parent_priority`{.interpreted-text role="setting"} configuration option to [True]{.title-ref} will make Celery tasks inherit the priority of the previous task linked to it. Examples: ``` python c = celery.chain( add.s(2), # priority=None add.s(3).set(priority=5), # priority=5 add.s(4), # priority=5 add.s(5).set(priority=3), # priority=3 add.s(6), # priority=3 ) ``` ``` python @app.task(bind=True) def child_task(self): pass @app.task(bind=True) def parent_task(self): child_task.delay() # child_task will also have priority=5 parent_task.apply_async(args=[], priority=5) ``` Contributed by **:github_user:\`madprogrammer\`** - **Canvas**: Added the `result_chord_join_timeout`{.interpreted-text role="setting"} setting. Previously, `celery.result.GroupResult.join`{.interpreted-text role="meth"} had a fixed timeout of 3 seconds. The `result_chord_join_timeout`{.interpreted-text role="setting"} setting now allows you to change it. Contributed by **:github_user:\`srafehi\`** Code Cleanups, Test Coverage & CI Improvements by: > - **Jon Dufresne** > - **Asif Saif Uddin** > - **Omer Katz** > - **Brett Jackson** > - **Bruno Alla** > - **:github_user:\`tothegump\`** > - **Bojan Jovanovic** > - **Florian Chardin** > - **:github_user:\`walterqian\`** > - **Fabian Becker** > - **Lars Rinn** > - **:github_user:\`madprogrammer\`** > - **Ciaran Courtney** Documentation Fixes by: > - **Lewis M. Kabui** > - **Dash Winterson** > - **Shanavas M** > - **Brett Randall** > - **Przemysław Suliga** > - **Joshua Schmid** > - **Asif Saif Uddin** > - **Xiaodong** > - **Vikas Prasad** > - **Jamie Alessio** > - **Lars Kruse** > - **Guilherme Caminha** > - **Andrea Rabbaglietti** > - **Itay Bittan** > - **Noah Hall** > - **Peng Weikang** > - **Mariatta Wijaya** > - **Ed Morley** > - **Paweł Adamczak** > - **:github_user:\`CoffeeExpress\`** > - **:github_user:\`aviadatsnyk\`** > - **Brian Schrader** > - **Josue Balandrano Coronel** > - **Tom Clancy** > - **Sebastian Wojciechowski** > - **Meysam Azad** > - **Willem Thiart** > - **Charles Chan** > - **Omer Katz** > - **Milind Shakya** --- # Change history This document contains change notes for bugfix & new features in the 5.0.x , please see `whatsnew-5.0`{.interpreted-text role="ref"} for an overview of what\'s new in Celery 5.0. ## 5.0.6 {#version-5.0.6} release-date : 2021-06-28 3.00 P.M UTC+3:00 release-by : Omer Katz - Inspect commands accept arguments again (#6710). - The `worker_pool`{.interpreted-text role="setting"} setting is now respected correctly (#6711). - Ensure AMQPContext exposes an app attribute (#6741). - Exit celery with non zero exit value if failing (#6602). - \--quiet flag now actually makes celery avoid producing logs (#6599). - pass_context for handle_preload_options decorator (#6583). - Fix \--pool=threads support in command line options parsing (#6787). - Fix the behavior of our json serialization which regressed in 5.0 (#6561). - celery -A app events -c camera now works as expected (#6774). ## 5.0.5 {#version-5.0.5} release-date : 2020-12-16 5.35 P.M UTC+2:00 release-by : Omer Katz - Ensure keys are strings when deleting results from S3 (#6537). - Fix a regression breaking [celery \--help]{.title-ref} and [celery events]{.title-ref} (#6543). ## 5.0.4 {#version-5.0.4} release-date : 2020-12-08 2.40 P.M UTC+2:00 release-by : Omer Katz - DummyClient of cache+memory:// backend now shares state between threads (#6524). This fixes a problem when using our pytest integration with the in memory result backend. Because the state wasn\'t shared between threads, #6416 results in test suites hanging on [result.get()]{.title-ref}. ## 5.0.3 {#version-5.0.3} release-date : 2020-12-03 6.30 P.M UTC+2:00 release-by : Omer Katz - Make [\--workdir]{.title-ref} eager for early handling (#6457). - When using the MongoDB backend, don\'t cleanup if result_expires is 0 or None (#6462). - Fix passing queues into purge command (#6469). - Restore [app.start()]{.title-ref} and [app.worker_main()]{.title-ref} (#6481). - Detaching no longer creates an extra log file (#6426). - Result backend instances are now thread local to ensure thread safety (#6416). - Don\'t upgrade click to 8.x since click-repl doesn\'t support it yet. - Restore preload options (#6516). ## 5.0.2 {#version-5.0.2} release-date : 2020-11-02 8.00 P.M UTC+2:00 release-by : Omer Katz - Fix \_autodiscover_tasks_from_fixups (#6424). - Flush worker prints, notably the banner (#6432). - **Breaking Change**: Remove [ha_policy]{.title-ref} from queue definition. (#6440) > This argument has no effect since RabbitMQ 3.0. Therefore, We feel > comfortable dropping it in a patch release. - Python 3.9 support (#6418). - **Regression**: When using the prefork pool, pick the fair scheduling strategy by default (#6447). - Preserve callbacks when replacing a task with a chain (#6189). - Fix max_retries override on [self.retry()]{.title-ref} (#6436). - Raise proper error when replacing with an empty chain (#6452) ## 5.0.1 {#version-5.0.1} release-date : 2020-10-18 1.00 P.M UTC+3:00 release-by : Omer Katz - Specify UTF-8 as the encoding for log files (#6357). - Custom headers now propagate when using the protocol 1 hybrid messages (#6374). - Retry creating the database schema for the database results backend in case of a race condition (#6298). - When using the Redis results backend, awaiting for a chord no longer hangs when setting `result_expires`{.interpreted-text role="setting"} to 0 (#6373). - When a user tries to specify the app as an option for the subcommand, a custom error message is displayed (#6363). - Fix the [\--without-gossip]{.title-ref}, [\--without-mingle]{.title-ref}, and [\--without-heartbeat]{.title-ref} options which now work as expected. (#6365) - Provide a clearer error message when the application cannot be loaded. - Avoid printing deprecation warnings for settings when they are loaded from Django settings (#6385). - Allow lowercase log levels for the [\--loglevel]{.title-ref} option (#6388). - Detaching now works as expected (#6401). - Restore broadcasting messages from [celery control]{.title-ref} (#6400). - Pass back real result for single task chains (#6411). - Ensure group tasks a deeply serialized (#6342). - Fix chord element counting (#6354). - Restore the [celery shell]{.title-ref} command (#6421). ## 5.0.0 {#version-5.0.0} release-date : 2020-09-24 6.00 P.M UTC+3:00 release-by : Omer Katz - **Breaking Change** Remove AMQP result backend (#6360). - Warn when deprecated settings are used (#6353). - Expose retry_policy for Redis result backend (#6330). - Prepare Celery to support the yet to be released Python 3.9 (#6328). ## 5.0.0rc3 release-date : 2020-09-07 4.00 P.M UTC+3:00 release-by : Omer Katz - More cleanups of leftover Python 2 support (#6338). ## 5.0.0rc2 release-date : 2020-09-01 6.30 P.M UTC+3:00 release-by : Omer Katz - Bump minimum required eventlet version to 0.26.1. - Update Couchbase Result backend to use SDK V3. - Restore monkeypatching when gevent or eventlet are used. ## 5.0.0rc1 release-date : 2020-08-24 9.00 P.M UTC+3:00 release-by : Omer Katz - Allow to opt out of ordered group results when using the Redis result backend (#6290). - **Breaking Change** Remove the deprecated celery.utils.encoding module. ## 5.0.0b1 release-date : 2020-08-19 8.30 P.M UTC+3:00 release-by : Omer Katz - **Breaking Change** Drop support for the Riak result backend (#5686). - **Breaking Change** pytest plugin is no longer enabled by default (#6288). Install pytest-celery to enable it. - **Breaking Change** Brand new CLI based on Click (#5718). ## 5.0.0a2 release-date : 2020-08-05 7.15 P.M UTC+3:00 release-by : Omer Katz - Bump Kombu version to 5.0 (#5686). ## 5.0.0a1 release-date : 2020-08-02 9.30 P.M UTC+3:00 release-by : Omer Katz - Removed most of the compatibility code that supports Python 2 (#5686). - Modernized code to work on Python 3.6 and above (#5686). --- # Change history {#changelog-5.1} This document contains change notes for bugfix & new features in the & 5.1.x series, please see `whatsnew-5.1`{.interpreted-text role="ref"} for an overview of what\'s new in Celery 5.1. ## 5.1.2 release-date : 2021-06-28 16.15 P.M UTC+3:00 release-by : Omer Katz - When chords fail, correctly call errbacks. (#6814) > We had a special case for calling errbacks when a chord failed > which assumed they were old style. This change ensures that we > call the proper errback dispatch method which understands new and > old style errbacks, and adds test to confirm that things behave as > one might expect now. - Avoid using the `Event.isSet()` deprecated alias. (#6824) - Reintroduce sys.argv default behaviour for `Celery.start()`. (#6825) ## 5.1.1 release-date : 2021-06-17 16.10 P.M UTC+3:00 release-by : Omer Katz - Fix `--pool=threads` support in command line options parsing. (#6787) - Fix `LoggingProxy.write()` return type. (#6791) - Couchdb key is now always coerced into a string. (#6781) - grp is no longer imported unconditionally. (#6804) : This fixes a regression in 5.1.0 when running Celery in non-unix systems. - Ensure regen utility class gets marked as done when concertised. (#6789) - Preserve call/errbacks of replaced tasks. (#6770) - Use single-lookahead for regen consumption. (#6799) - Revoked tasks are no longer incorrectly marked as retried. (#6812, #6816) ## 5.1.0 release-date : 2021-05-23 19.20 P.M UTC+3:00 release-by : Omer Katz - `celery -A app events -c camera` now works as expected. (#6774) - Bump minimum required Kombu version to 5.1.0. ## 5.1.0rc1 {#version-5.1.0rc1} release-date : 2021-05-02 16.06 P.M UTC+3:00 release-by : Omer Katz - Celery Mailbox accept and serializer parameters are initialized from configuration. (#6757) - Error propagation and errback calling for group-like signatures now works as expected. (#6746) - Fix sanitization of passwords in sentinel URIs. (#6765) - Add LOG_RECEIVED to customize logging. (#6758) ## 5.1.0b2 {#version-5.1.0b2} release-date : 2021-05-02 16.06 P.M UTC+3:00 release-by : Omer Katz - Fix the behavior of our json serialization which regressed in 5.0. (#6561) - Add support for SQLAlchemy 1.4. (#6709) - Safeguard against schedule entry without kwargs. (#6619) - `task.apply_async(ignore_result=True)` now avoids persisting the results. (#6713) - Update systemd tmpfiles path. (#6688) - Ensure AMQPContext exposes an app attribute. (#6741) - Inspect commands accept arguments again (#6710). - Chord counting of group children is now accurate. (#6733) - Add a setting `worker_cancel_long_running_tasks_on_connection_loss`{.interpreted-text role="setting"} to terminate tasks with late acknowledgement on connection loss. (#6654) - The `task-revoked` event and the `task_revoked` signal are not duplicated when `Request.on_failure` is called. (#6654) - Restore pickling support for `Retry`. (#6748) - Add support in the redis result backend for authenticating with a username. (#6750) - The `worker_pool`{.interpreted-text role="setting"} setting is now respected correctly. (#6711) ## 5.1.0b1 {#version-5.1.0b1} release-date : 2021-04-02 10.25 P.M UTC+6:00 release-by : Asif Saif Uddin - Add sentinel_kwargs to Redis Sentinel docs. - Depend on the maintained python-consul2 library. (#6544). - Use result_chord_join_timeout instead of hardcoded default value. - Upgrade AzureBlockBlob storage backend to use Azure blob storage library v12 (#6580). - Improved integration tests. - pass_context for handle_preload_options decorator (#6583). - Makes regen less greedy (#6589). - Pytest worker shutdown timeout (#6588). - Exit celery with non zero exit value if failing (#6602). - Raise BackendStoreError when set value is too large for Redis. - Trace task optimizations are now set via Celery app instance. - Make trace_task_ret and fast_trace_task public. - reset_worker_optimizations and create_request_cls has now app as optional parameter. - Small refactor in exception handling of on_failure (#6633). - Fix for issue #5030 \"Celery Result backend on Windows OS\". - Add store_eager_result setting so eager tasks can store result on the result backend (#6614). - Allow heartbeats to be sent in tests (#6632). - Fixed default visibility timeout note in sqs documentation. - Support Redis Sentinel with SSL. - Simulate more exhaustive delivery info in apply(). - Start chord header tasks as soon as possible (#6576). - Forward shadow option for retried tasks (#6655). - \--quiet flag now actually makes celery avoid producing logs (#6599). - Update platforms.py \"superuser privileges\" check (#6600). - Remove unused property [autoregister]{.title-ref} from the Task class (#6624). - fnmatch.translate() already translates globs for us. (#6668). - Upgrade some syntax to Python 3.6+. - Add [azureblockblob_base_path]{.title-ref} config (#6669). - Fix checking expiration of X.509 certificates (#6678). - Drop the lzma extra. - Fix JSON decoding errors when using MongoDB as backend (#6675). - Allow configuration of RedisBackend\'s health_check_interval (#6666). - Safeguard against schedule entry without kwargs (#6619). - Docs only - SQS broker - add STS support (#6693) through kombu. - Drop fun_accepts_kwargs backport. - Tasks can now have required kwargs at any order (#6699). - Min py-amqp 5.0.6. - min billiard is now 3.6.4.0. - Minimum kombu now is5.1.0b1. - Numerous docs fixes. - Moved CI to github action. - Updated deployment scripts. - Updated docker. - Initial support of python 3.9 added. --- # Change history {#changelog-5.3} This document contains change notes for bugfix & new features in the & 5.3.x series, please see `whatsnew-5.3`{.interpreted-text role="ref"} for an overview of what\'s new in Celery 5.3. ## 5.3.6 release-date : 2023-11-22 9:15 P.M GMT+6 release-by : Asif Saif Uddin This release is focused mainly to fix AWS SQS new feature comatibility issue and old regressions. The code changes are mostly fix for regressions. More details can be found below. - Increased docker-build CI job timeout from 30m -\> 60m (#8635) - Incredibly minor spelling fix. (#8649) - Fix non-zero exit code when receiving remote shutdown (#8650) - Update task.py get_custom_headers missing \'compression\' key (#8633) - Update kombu\>=5.3.4 to fix SQS request compatibility with boto JSON serializer (#8646) - test requirements version update (#8655) - Update elasticsearch version (#8656) - Propagates more ImportErrors during autodiscovery (#8632) ## 5.3.5 release-date : 2023-11-10 7:15 P.M GMT+6 release-by : Asif Saif Uddin - Update test.txt versions (#8481) - fix os.getcwd() FileNotFoundError (#8448) - Fix typo in CONTRIBUTING.rst (#8494) - typo(doc): configuration.rst (#8484) - assert before raise (#8495) - Update GHA checkout version (#8496) - Fixed replaced_task_nesting (#8500) - Fix code indentation for route_task() example (#8502) - support redis 5.x (#8504) - Fix typos in test_canvas.py (#8498) - Marked flaky tests (#8508) - Fix typos in calling.rst (#8506) - Added support for replaced_task_nesting in chains (#8501) - Fix typos in canvas.rst (#8509) - Patch Version Release Checklist (#8488) - Added Python 3.11 support to Dockerfile (#8511) - Dependabot (Celery) (#8510) - Bump actions/checkout from 3 to 4 (#8512) - Update ETA example to include timezone (#8516) - Replaces datetime.fromisoformat with the more lenient dateutil parser (#8507) - Fixed indentation in Dockerfile for Python 3.11 (#8527) - Fix git bug in Dockerfile (#8528) - Tox lint upgrade from Python 3.9 to Python 3.11 (#8526) - Document gevent concurrency (#8520) - Update test.txt (#8530) - Celery Docker Upgrades (#8531) - pyupgrade upgrade v3.11.0 -\> v3.13.0 (#8535) - Update msgpack.txt (#8548) - Update auth.txt (#8547) - Update msgpack.txt to fix build issues (#8552) - Basic ElasticSearch / ElasticClient 8.x Support (#8519) - Fix eager tasks does not populate name field (#8486) - Fix typo in celery.app.control (#8563) - Update solar.txt ephem (#8566) - Update test.txt pytest-timeout (#8565) - Correct some mypy errors (#8570) - Update elasticsearch.txt (#8573) - Update test.txt deps (#8574) - Update test.txt (#8590) - Improved the \"Next steps\" documentation (#8561). (#8600) - Disabled couchbase tests due to broken package breaking main (#8602) - Update elasticsearch deps (#8605) - Update cryptography==41.0.5 (#8604) - Update pytest==7.4.3 (#8606) - test initial support of python 3.12.x (#8549) - updated new versions to fix CI (#8607) - Update zstd.txt (#8609) - Fixed CI Support with Python 3.12 (#8611) - updated CI, docs and classifier for next release (#8613) - updated dockerfile to add python 3.12 (#8614) - lint,mypy,docker-unit-tests -\> Python 3.12 (#8617) - Correct type of [request]{.title-ref} in [task_revoked]{.title-ref} documentation (#8616) - update docs docker image (#8618) - Fixed RecursionError caused by giving [config_from_object]{.title-ref} nested mod... (#8619) - Fix: serialization error when gossip working (#6566) - \[documentation\] broker_connection_max_retries of 0 does not mean \"retry forever\" (#8626) - added 2 debian package for better stability in Docker (#8629) ## 5.3.4 release-date : 2023-09-03 10:10 P.M GMT+2 release-by : Tomer Nosrati ::: warning ::: title Warning ::: This version has reverted the breaking changes introduced in 5.3.2 and 5.3.3: - Revert \"store children with database backend\" (#8475) - Revert \"Fix eager tasks does not populate name field\" (#8476) ::: - Bugfix: Removed unecessary stamping code from \_chord.run() (#8339) - User guide fix (hotfix for #1755) (#8342) - store children with database backend (#8338) - Stamping bugfix with group/chord header errback linking (#8347) - Use argsrepr and kwargsrepr in LOG_RECEIVED (#8301) - Fixing minor typo in code example in calling.rst (#8366) - add documents for timeout settings (#8373) - fix: copyright year (#8380) - setup.py: enable include_package_data (#8379) - Fix eager tasks does not populate name field (#8383) - Update test.txt dependencies (#8389) - Update auth.txt deps (#8392) - Fix backend.get_task_meta ignores the result_extended config parameter in mongodb backend (#8391) - Support preload options for shell and purge commands (#8374) - Implement safer ArangoDB queries (#8351) - integration test: cleanup worker after test case (#8361) - Added \"Tomer Nosrati\" to CONTRIBUTORS.txt (#8400) - Update README.rst (#8404) - Update README.rst (#8408) - fix(canvas): add group index when unrolling tasks (#8427) - fix(beat): debug statement should only log AsyncResult.id if it exists (#8428) - Lint fixes & pre-commit autoupdate (#8414) - Update auth.txt (#8435) - Update mypy on test.txt (#8438) - added missing kwargs arguments in some cli cmd (#8049) - Fix #8431: Set format_date to False when calling \_get_result_meta on mongo backend (#8432) - Docs: rewrite out-of-date code (#8441) - Limit redis client to 4.x since 5.x fails the test suite (#8442) - Limit tox to \< 4.9 (#8443) - Fixed issue: Flags broker_connection_retry_on_startup & broker_connection_retry aren't reliable (#8446) - doc update from #7651 (#8451) - Remove tox version limit (#8464) - Fixed AttributeError: \'str\' object has no attribute (#8463) - Upgraded Kombu from 5.3.1 -\> 5.3.2 (#8468) - Document need for [CELERY]() prefix on CLI env vars (#8469) - Use string value for CELERY_SKIP_CHECKS envvar (#8462) - Revert \"store children with database backend\" (#8475) - Revert \"Fix eager tasks does not populate name field\" (#8476) - Update Changelog (#8474) - Remove as it seems to be buggy. (#8340) - Revert \"Add Semgrep to CI\" (#8477) - Revert \"Revert \"Add Semgrep to CI\"\" (#8478) ## 5.3.3 (Yanked) release-date : 2023-08-31 1:47 P.M GMT+2 release-by : Tomer Nosrati ::: warning ::: title Warning ::: This version has been yanked due to breaking API changes. The breaking changes include: - Store children with database backend (#8338) - Fix eager tasks does not populate name field (#8383) ::: - Fixed changelog for 5.3.2 release docs. ## 5.3.2 (Yanked) release-date : 2023-08-31 1:30 P.M GMT+2 release-by : Tomer Nosrati ::: warning ::: title Warning ::: This version has been yanked due to breaking API changes. The breaking changes include: - Store children with database backend (#8338) - Fix eager tasks does not populate name field (#8383) ::: - Bugfix: Removed unecessary stamping code from \_chord.run() (#8339) - User guide fix (hotfix for #1755) (#8342) - Store children with database backend (#8338) - Stamping bugfix with group/chord header errback linking (#8347) - Use argsrepr and kwargsrepr in LOG_RECEIVED (#8301) - Fixing minor typo in code example in calling.rst (#8366) - Add documents for timeout settings (#8373) - Fix: copyright year (#8380) - Setup.py: enable include_package_data (#8379) - Fix eager tasks does not populate name field (#8383) - Update test.txt dependencies (#8389) - Update auth.txt deps (#8392) - Fix backend.get_task_meta ignores the result_extended config parameter in mongodb backend (#8391) - Support preload options for shell and purge commands (#8374) - Implement safer ArangoDB queries (#8351) - Integration test: cleanup worker after test case (#8361) - Added \"Tomer Nosrati\" to CONTRIBUTORS.txt (#8400) - Update README.rst (#8404) - Update README.rst (#8408) - Fix(canvas): add group index when unrolling tasks (#8427) - Fix(beat): debug statement should only log AsyncResult.id if it exists (#8428) - Lint fixes & pre-commit autoupdate (#8414) - Update auth.txt (#8435) - Update mypy on test.txt (#8438) - Added missing kwargs arguments in some cli cmd (#8049) - Fix #8431: Set format_date to False when calling \_get_result_meta on mongo backend (#8432) - Docs: rewrite out-of-date code (#8441) - Limit redis client to 4.x since 5.x fails the test suite (#8442) - Limit tox to \< 4.9 (#8443) - Fixed issue: Flags broker_connection_retry_on_startup & broker_connection_retry aren't reliable (#8446) - Doc update from #7651 (#8451) - Remove tox version limit (#8464) - Fixed AttributeError: \'str\' object has no attribute (#8463) - Upgraded Kombu from 5.3.1 -\> 5.3.2 (#8468) ## 5.3.1 release-date : 2023-06-18 8:15 P.M GMT+6 release-by : Asif Saif Uddin - Upgrade to latest pycurl release (#7069). - Limit librabbitmq\>=2.0.0; python_version \< \'3.11\' (#8302). - Added initial support for python 3.11 (#8304). - ChainMap observers fix (#8305). - Revert optimization CLI flag behaviour back to original. - Restrict redis 4.5.5 as it has severe bugs (#8317). - Tested pypy 3.10 version in CI (#8320). - Bump new version of kombu to 5.3.1 (#8323). - Fixed a small float value of retry_backoff (#8295). - Limit pyro4 up to python 3.10 only as it is (#8324). ## 5.3.0 release-date : 2023-06-06 12:00 P.M GMT+6 release-by : Asif Saif Uddin - Test kombu 5.3.0 & minor doc update (#8294). - Update librabbitmq.txt \> 2.0.0 (#8292). - Upgrade syntax to py3.8 (#8281). ## 5.3.0rc2 release-date : 2023-05-31 9:00 P.M GMT+6 release-by : Asif Saif Uddin - Add missing dependency. - Fix exc_type being the exception instance rather. - Fixed revoking tasks by stamped headers (#8269). - Support sqlalchemy 2.0 in tests (#8271). - Fix docker (#8275). - Update redis.txt to 4.5 (#8278). - Update kombu\>=5.3.0rc2. ## 5.3.0rc1 release-date : 2023-05-11 4:24 P.M GMT+2 release-by : Tomer Nosrati - fix functiom name by \@cuishuang in #8087 - Update CELERY_TASK_EAGER setting in user guide by \@thebalaa in #8085 - Stamping documentation fixes & cleanups by \@Nusnus in #8092 - switch to maintained pyro5 by \@auvipy in #8093 - udate dependencies of tests by \@auvipy in #8095 - cryptography==39.0.1 by \@auvipy in #8096 - Annotate celery/security/certificate.py by \@Kludex in #7398 - Deprecate parse_iso8601 in favor of fromisoformat by \@stumpylog in #8098 - pytest==7.2.2 by \@auvipy in #8106 - Type annotations for celery/utils/text.py by \@max-muoto in #8107 - Update web framework URLs by \@sblondon in #8112 - Fix contribution URL by \@sblondon in #8111 - Trying to clarify CERT_REQUIRED by \@pamelafox in #8113 - Fix potential AttributeError on \'stamps\' by \@Darkheir in #8115 - Type annotations for celery/apps/beat.py by \@max-muoto in #8108 - Fixed bug where retrying a task loses its stamps by \@Nusnus in #8120 - Type hints for celery/schedules.py by \@max-muoto in #8114 - Reference Gopher Celery in README by \@marselester in #8131 - Update sqlalchemy.txt by \@auvipy in #8136 - azure-storage-blob 12.15.0 by \@auvipy in #8137 - test kombu 5.3.0b3 by \@auvipy in #8138 - fix: add expire string parse. by \@Bidaya0 in #8134 - Fix worker crash on un-pickleable exceptions by \@youtux in #8133 - CLI help output: avoid text rewrapping by click by \@woutdenolf in #8152 - Warn when an unnamed periodic task override another one. by \@iurisilvio in #8143 - Fix Task.handle_ignore not wrapping exceptions properly by \@youtux in #8149 - Hotfix for (#8120) - Stamping bug with retry by \@Nusnus in #8158 - Fix integration test by \@youtux in #8156 - Fixed bug in revoke_by_stamped_headers where impl did not match doc by \@Nusnus in #8162 - Align revoke and revoke_by_stamped_headers return values (terminate=True) by \@Nusnus in #8163 - Update & simplify GHA pip caching by \@stumpylog in #8164 - Update auth.txt by \@auvipy in #8167 - Update test.txt versions by \@auvipy in #8173 - remove extra = from test.txt by \@auvipy in #8179 - Update sqs.txt kombu\[sqs\]\>=5.3.0b3 by \@auvipy in #8174 - Added signal triggered before fork by \@jaroslawporada in #8177 - Update documentation on SQLAlchemy by \@max-muoto in #8188 - Deprecate pytz and use zoneinfo by \@max-muoto in #8159 - Update dev.txt by \@auvipy in #8192 - Update test.txt by \@auvipy in #8193 - Update test-integration.txt by \@auvipy in #8194 - Update zstd.txt by \@auvipy in #8195 - Update s3.txt by \@auvipy in #8196 - Update msgpack.txt by \@auvipy in #8199 - Update solar.txt by \@auvipy in #8198 - Add Semgrep to CI by \@Nusnus in #8201 - Added semgrep to README.rst by \@Nusnus in #8202 - Update django.txt by \@auvipy in #8197 - Update redis.txt 4.3.6 by \@auvipy in #8161 - start removing codecov from pypi by \@auvipy in #8206 - Update test.txt dependencies by \@auvipy in #8205 - Improved doc for: worker_deduplicate_successful_tasks by \@Nusnus in #8209 - Renamed revoked_headers to revoked_stamps by \@Nusnus in #8210 - Ensure argument for map is JSON serializable by \@candleindark in #8229 ## 5.3.0b2 release-date : 2023-02-19 1:47 P.M GMT+2 release-by : Asif Saif Uddin - BLM-2: Adding unit tests to chord clone by \@Nusnus in #7668 - Fix unknown task error typo by \@dcecile in #7675 - rename redis integration test class so that tests are executed by \@wochinge in #7684 - Check certificate/private key type when loading them by \@qrmt in #7680 - Added integration test_chord_header_id_duplicated_on_rabbitmq_msg_duplication() by \@Nusnus in #7692 - New feature flag: allow_error_cb_on_chord_header - allowing setting an error callback on chord header by \@Nusnus in #7712 - Update README.rst sorting Python/Celery versions by \@andrebr in #7714 - Fixed a bug where stamping a chord body would not use the correct stamping method by \@Nusnus in #7722 - Fixed doc duplication typo for Signature.stamp() by \@Nusnus in #7725 - Fix issue 7726: variable used in finally block may not be instantiated by \@woutdenolf in #7727 - Fixed bug in chord stamping with another chord as a body + unit test by \@Nusnus in #7730 - Use \"describe_table\" not \"create_table\" to check for existence of DynamoDB table by \@maxfirman in #7734 - Enhancements for task_allow_error_cb_on_chord_header tests and docs by \@Nusnus in #7744 - Improved custom stamping visitor documentation by \@Nusnus in #7745 - Improved the coverage of test_chord_stamping_body_chord() by \@Nusnus in #7748 - billiard \>= 3.6.3.0,\<5.0 for rpm by \@auvipy in #7764 - Fixed memory leak with ETA tasks at connection error when worker_cancel_long_running_tasks_on_connection_loss is enabled by \@Nusnus in #7771 - Fixed bug where a chord with header of type tuple was not supported in the link_error flow for task_allow_error_cb_on_chord_header flag by \@Nusnus in #7772 - Scheduled weekly dependency update for week 38 by \@pyup-bot in #7767 - recreate_module: set spec to the new module by \@skshetry in #7773 - Override integration test config using integration-tests-config.json by \@thedrow in #7778 - Fixed error handling bugs due to upgrade to a newer version of billiard by \@Nusnus in #7781 - Do not recommend using easy_install anymore by \@jugmac00 in #7789 - GitHub Workflows security hardening by \@sashashura in #7768 - Update ambiguous acks_late doc by \@Zhong-z in #7728 - billiard \>=4.0.2,\<5.0 by \@auvipy in #7720 - importlib_metadata remove deprecated entry point interfaces by \@woutdenolf in #7785 - Scheduled weekly dependency update for week 41 by \@pyup-bot in #7798 - pyzmq\>=22.3.0 by \@auvipy in #7497 - Remove amqp from the BACKEND_ALISES list by \@Kludex in #7805 - Replace print by logger.debug by \@Kludex in #7809 - Ignore coverage on except ImportError by \@Kludex in #7812 - Add mongodb dependencies to test.txt by \@Kludex in #7810 - Fix grammar typos on the whole project by \@Kludex in #7815 - Remove isatty wrapper function by \@Kludex in #7814 - Remove unused variable \_range by \@Kludex in #7813 - Add type annotation on concurrency/threads.py by \@Kludex in #7808 - Fix linter workflow by \@Kludex in #7816 - Scheduled weekly dependency update for week 42 by \@pyup-bot in #7821 - Remove .cookiecutterrc by \@Kludex in #7830 - Remove .coveragerc file by \@Kludex in #7826 - kombu\>=5.3.0b2 by \@auvipy in #7834 - Fix readthedocs build failure by \@woutdenolf in #7835 - Fixed bug in group, chord, chain stamp() method, where the visitor overrides the previously stamps in tasks of these objects by \@Nusnus in #7825 - Stabilized test_mutable_errback_called_by_chord_from_group_fail_multiple by \@Nusnus in #7837 - Use SPDX license expression in project metadata by \@RazerM in #7845 - New control command revoke_by_stamped_headers by \@Nusnus in #7838 - Clarify wording in Redis priority docs by \@strugee in #7853 - Fix non working example of using celery_worker pytest fixture by \@paradox-lab in #7857 - Removed the mandatory requirement to include stamped_headers key when implementing on_signature() by \@Nusnus in #7856 - Update serializer docs by \@sondrelg in #7858 - Remove reference to old Python version by \@Kludex in #7829 - Added on_replace() to Task to allow manipulating the replaced sig with custom changes at the end of the task.replace() by \@Nusnus in #7860 - Add clarifying information to completed_count documentation by \@hankehly in #7873 - Stabilized test_revoked_by_headers_complex_canvas by \@Nusnus in #7877 - StampingVisitor will visit the callbacks and errbacks of the signature by \@Nusnus in #7867 - Fix \"rm: no operand\" error in clean-pyc script by \@hankehly in #7878 - Add \--skip-checks flag to bypass django core checks by \@mudetz in #7859 - Scheduled weekly dependency update for week 44 by \@pyup-bot in #7868 - Added two new unit tests to callback stamping by \@Nusnus in #7882 - Sphinx extension: use inspect.signature to make it Python 3.11 compatible by \@mathiasertl in #7879 - cryptography==38.0.3 by \@auvipy in #7886 - Canvas.py doc enhancement by \@Nusnus in #7889 - Fix typo by \@sondrelg in #7890 - fix typos in optional tests by \@hsk17 in #7876 - Canvas.py doc enhancement by \@Nusnus in #7891 - Fix revoke by headers tests stability by \@Nusnus in #7892 - feat: add global keyprefix for backend result keys by \@kaustavb12 in #7620 - Canvas.py doc enhancement by \@Nusnus in #7897 - fix(sec): upgrade sqlalchemy to 1.2.18 by \@chncaption in #7899 - Canvas.py doc enhancement by \@Nusnus in #7902 - Fix test warnings by \@ShaheedHaque in #7906 - Support for out-of-tree worker pool implementations by \@ShaheedHaque in #7880 - Canvas.py doc enhancement by \@Nusnus in #7907 - Use bound task in base task example. Closes #7909 by \@WilliamDEdwards in #7910 - Allow the stamping visitor itself to set the stamp value type instead of casting it to a list by \@Nusnus in #7914 - Stamping a task left the task properties dirty by \@Nusnus in #7916 - Fixed bug when chaining a chord with a group by \@Nusnus in #7919 - Fixed bug in the stamping visitor mechanism where the request was lacking the stamps in the \'stamps\' property by \@Nusnus in #7928 - Fixed bug in task_accepted() where the request was not added to the requests but only to the active_requests by \@Nusnus in #7929 - Fix bug in TraceInfo.\_log_error() where the real exception obj was hiding behind \'ExceptionWithTraceback\' by \@Nusnus in #7930 - Added integration test: test_all_tasks_of_canvas_are_stamped() by \@Nusnus in #7931 - Added new example for the stamping mechanism: examples/stamping by \@Nusnus in #7933 - Fixed a bug where replacing a stamped task and stamping it again by \@Nusnus in #7934 - Bugfix for nested group stamping on task replace by \@Nusnus in #7935 - Added integration test test_stamping_example_canvas() by \@Nusnus in #7937 - Fixed a bug in losing chain links when unchaining an inner chain with links by \@Nusnus in #7938 - Removing as not mandatory by \@auvipy in #7885 - Housekeeping for Canvas.py by \@Nusnus in #7942 - Scheduled weekly dependency update for week 50 by \@pyup-bot in #7954 - try pypy 3.9 in CI by \@auvipy in #7956 - sqlalchemy==1.4.45 by \@auvipy in #7943 - billiard\>=4.1.0,\<5.0 by \@auvipy in #7957 - feat(typecheck): allow changing type check behavior on the app level; by \@moaddib666 in #7952 - Add broker_channel_error_retry option by \@nkns165 in #7951 - Add beat_cron_starting_deadline_seconds to prevent unwanted cron runs by \@abs25 in #7945 - Scheduled weekly dependency update for week 51 by \@pyup-bot in #7965 - Added doc to \"retry_errors\" newly supported field of \"publish_retry_policy\" of the task namespace by \@Nusnus in #7967 - Renamed from master to main in the docs and the CI workflows by \@Nusnus in #7968 - Fix docs for the exchange to use with worker_direct by \@alessio-b2c2 in #7973 - Pin redis==4.3.4 by \@auvipy in #7974 - return list of nodes to make sphinx extension compatible with Sphinx 6.0 by \@mathiasertl in #7978 - use version range redis\>=4.2.2,\<4.4.0 by \@auvipy in #7980 - Scheduled weekly dependency update for week 01 by \@pyup-bot in #7987 - Add annotations to minimise differences with celery-aio-pool\'s tracer.py. by \@ShaheedHaque in #7925 - Fixed bug where linking a stamped task did not add the stamp to the link\'s options by \@Nusnus in #7992 - sqlalchemy==1.4.46 by \@auvipy in #7995 - pytz by \@auvipy in #8002 - Fix few typos, provide configuration + workflow for codespell to catch any new by \@yarikoptic in #8023 - RabbitMQ links update by \@arnisjuraga in #8031 - Ignore files generated by tests by \@Kludex in #7846 - Revert \"sqlalchemy==1.4.46 (#7995)\" by \@Nusnus in #8033 - Fixed bug with replacing a stamped task with a chain or a group (inc. links/errlinks) by \@Nusnus in #8034 - Fixed formatting in setup.cfg that caused flake8 to misbehave by \@Nusnus in #8044 - Removed duplicated import Iterable by \@Nusnus in #8046 - Fix docs by \@Nusnus in #8047 - Document \--logfile default by \@strugee in #8057 - Stamping Mechanism Refactoring by \@Nusnus in #8045 - result_backend_thread_safe config shares backend across threads by \@CharlieTruong in #8058 - Fix cronjob that use day of month and negative UTC timezone by \@pkyosx in #8053 - Stamping Mechanism Examples Refactoring by \@Nusnus in #8060 - Fixed bug in Task.on_stamp_replaced() by \@Nusnus in #8061 - Stamping Mechanism Refactoring 2 by \@Nusnus in #8064 - Changed default append_stamps from True to False (meaning duplicates ... by \@Nusnus in #8068 - typo in comment: mailicious =\> malicious by \@yanick in #8072 - Fix command for starting flower with specified broker URL by \@ShukantPal in #8071 - Improve documentation on ETA/countdown tasks (#8069) by \@norbertcyran in #8075 ## 5.3.0b1 release-date : 2022-08-01 5:15 P.M UTC+6:00 release-by : Asif Saif Uddin - Canvas Header Stamping (#7384). - async chords should pass it\'s kwargs to the group/body. - beat: Suppress banner output with the quiet option (#7608). - Fix honor Django\'s TIME_ZONE setting. - Don\'t warn about DEBUG=True for Django. - Fixed the on_after_finalize cannot access tasks due to deadlock. - Bump kombu\>=5.3.0b1,\<6.0. - Make default worker state limits configurable (#7609). - Only clear the cache if there are no active writers. - Billiard 4.0.1 ## 5.3.0a1 release-date : 2022-06-29 5:15 P.M UTC+6:00 release-by : Asif Saif Uddin - Remove Python 3.4 compatibility code. - call ping to set connection attr for avoiding redis parse_response error. - Use importlib instead of deprecated pkg_resources. - fix #7245 uid duplicated in command params. - Fix subscribed_to maybe empty (#7232). - Fix: Celery beat sleeps 300 seconds sometimes even when it should run a task within a few seconds (e.g. 13 seconds) #7290. - Add security_key_password option (#7292). - Limit elasticsearch support to below version 8.0. - try new major release of pytest 7 (#7330). - broker_connection_retry should no longer apply on startup (#7300). - Remove \_\_ne\_\_ methods (#7257). - fix #7200 uid and gid. - Remove exception-throwing from the signal handler. - Add mypy to the pipeline (#7383). - Expose more debugging information when receiving unknown tasks. (#7405) - Avoid importing buf_t from billiard\'s compat module as it was removed. - Avoid negating a constant in a loop. (#7443) - Ensure expiration is of float type when migrating tasks (#7385). - load_extension_class_names - correct module_name (#7406) - Bump pymongo\[srv\]\>=4.0.2. - Use inspect.getgeneratorstate in asynpool.gen_not_started (#7476). - Fix test with missing .get() (#7479). - azure-storage-blob\>=12.11.0 - Make start_worker, setup_default_app reusable outside of pytest. - Ensure a proper error message is raised when id for key is empty (#7447). - Crontab string representation does not match UNIX crontab expression. - Worker should exit with ctx.exit to get the right exitcode for non-zero. - Fix expiration check (#7552). - Use callable built-in. - Include dont_autoretry_for option in tasks. (#7556) - fix: Syntax error in arango query. - Fix custom headers propagation on task retries (#7555). - Silence backend warning when eager results are stored. - Reduce prefetch count on restart and gradually restore it (#7350). - Improve workflow primitive subclassing (#7593). - test kombu\>=5.3.0a1,\<6.0 (#7598). - Canvas Header Stamping (#7384). --- # Change history {#changelog-5.4} This document contains change notes for bugfix & new features in the & 5.4.x series, please see `whatsnew-5.4`{.interpreted-text role="ref"} for an overview of what\'s new in Celery 5.4. ## 5.4.0 release-date : 2024-04-17 release-by : Tomer Nosrati Celery v5.4.0 and v5.3.x have consistently focused on enhancing the overall QA, both internally and externally. This effort led to the new pytest-celery v1.0.0 release, developed concurrently with v5.3.0 & v5.4.0. This release introduces two significant QA enhancements: - **Smoke Tests**: A new layer of automatic tests has been added to Celery\'s standard CI. These tests are designed to handle production scenarios and complex conditions efficiently. While new contributions will not be halted due to the lack of smoke tests, we will request smoke tests for advanced changes where appropriate. - [Standalone Bug Report Script](https://docs.celeryq.dev/projects/pytest-celery/en/latest/userguide/celery-bug-report.html): The new pytest-celery plugin now allows for encapsulating a complete Celery dockerized setup within a single pytest script. Incorporating these into new bug reports will enable us to reproduce reported bugs deterministically, potentially speeding up the resolution process. Contrary to the positive developments above, there have been numerous reports about issues with the Redis broker malfunctioning upon restarts and disconnections. Our initial attempts to resolve this were not successful (#8796). With our enhanced QA capabilities, we are now prepared to address the core issue with Redis (as a broker) again. The rest of the changes for this release are grouped below, with the changes from the latest release candidate listed at the end. ### Changes - Add a Task class specialised for Django (#8491) - Add Google Cloud Storage (GCS) backend (#8868) - Added documentation to the smoke tests infra (#8970) - Added a checklist item for using pytest-celery in a bug report (#8971) - Bugfix: Missing id on chain (#8798) - Bugfix: Worker not consuming tasks after Redis broker restart (#8796) - Catch UnicodeDecodeError when opening corrupt beat-schedule.db (#8806) - chore(ci): Enhance CI with [workflow_dispatch]{.title-ref} for targeted debugging and testing (#8826) - Doc: Enhance \"Testing with Celery\" section (#8955) - Docfix: pip install celery\[sqs\] -\> pip install \"celery\[sqs\]\" (#8829) - Enable efficient [chord]{.title-ref} when using dynamicdb as backend store (#8783) - feat(daemon): allows daemonization options to be fetched from app settings (#8553) - Fix DeprecationWarning: datetime.datetime.utcnow() (#8726) - Fix recursive result parents on group in middle of chain (#8903) - Fix typos and grammar (#8915) - Fixed version documentation tag from #8553 in configuration.rst (#8802) - Hotfix: Smoke tests didn\'t allow customizing the worker\'s command arguments, now it does (#8937) - Make custom remote control commands available in CLI (#8489) - Print safe_say() to stdout for non-error flows (#8919) - Support moto 5.0 (#8838) - Update contributing guide to use ssh upstream url (#8881) - Update optimizing.rst (#8945) - Updated concurrency docs page. (#8753) ### Dependencies Updates - Bump actions/setup-python from 4 to 5 (#8701) - Bump codecov/codecov-action from 3 to 4 (#8831) - Bump isort from 5.12.0 to 5.13.2 (#8772) - Bump msgpack from 1.0.7 to 1.0.8 (#8885) - Bump mypy from 1.8.0 to 1.9.0 (#8898) - Bump pre-commit to 3.6.1 (#8839) - Bump pre-commit/action from 3.0.0 to 3.0.1 (#8835) - Bump pytest from 8.0.2 to 8.1.1 (#8901) - Bump pytest-celery to v1.0.0 (#8962) - Bump pytest-cov to 5.0.0 (#8924) - Bump pytest-order from 1.2.0 to 1.2.1 (#8941) - Bump pytest-subtests from 0.11.0 to 0.12.1 (#8896) - Bump pytest-timeout from 2.2.0 to 2.3.1 (#8894) - Bump python-memcached from 1.59 to 1.61 (#8776) - Bump sphinx-click from 4.4.0 to 5.1.0 (#8774) - Update cryptography to 42.0.5 (#8869) - Update elastic-transport requirement from \<=8.12.0 to \<=8.13.0 (#8933) - Update elasticsearch requirement from \<=8.12.1 to \<=8.13.0 (#8934) - Upgraded Sphinx from v5.3.0 to v7.x.x (#8803) ### Changes since 5.4.0rc2 - Update elastic-transport requirement from \<=8.12.0 to \<=8.13.0 (#8933) - Update elasticsearch requirement from \<=8.12.1 to \<=8.13.0 (#8934) - Hotfix: Smoke tests didn\'t allow customizing the worker\'s command arguments, now it does (#8937) - Bump pytest-celery to 1.0.0rc3 (#8946) - Update optimizing.rst (#8945) - Doc: Enhance \"Testing with Celery\" section (#8955) - Bump pytest-celery to v1.0.0 (#8962) - Bump pytest-order from 1.2.0 to 1.2.1 (#8941) - Added documentation to the smoke tests infra (#8970) - Added a checklist item for using pytest-celery in a bug report (#8971) - Added changelog for v5.4.0 (#8973) - Bump version: 5.4.0rc2 → 5.4.0 (#8974) ## 5.4.0rc2 release-date : 2024-03-27 release-by : Tomer Nosrati - feat(daemon): allows daemonization options to be fetched from app settings (#8553) - Fixed version documentation tag from #8553 in configuration.rst (#8802) - Upgraded Sphinx from v5.3.0 to v7.x.x (#8803) - Update elasticsearch requirement from \<=8.11.1 to \<=8.12.0 (#8810) - Update elastic-transport requirement from \<=8.11.0 to \<=8.12.0 (#8811) - Update cryptography to 42.0.0 (#8814) - Catch UnicodeDecodeError when opening corrupt beat-schedule.db (#8806) - Update cryptography to 42.0.1 (#8817) - Limit moto to \<5.0.0 until the breaking issues are fixed (#8820) - Enable efficient [chord]{.title-ref} when using dynamicdb as backend store (#8783) - Add a Task class specialised for Django (#8491) - Sync kombu versions in requirements and setup.cfg (#8825) - chore(ci): Enhance CI with [workflow_dispatch]{.title-ref} for targeted debugging and testing (#8826) - Update cryptography to 42.0.2 (#8827) - Docfix: pip install celery\[sqs\] -\> pip install \"celery\[sqs\]\" (#8829) - Bump pre-commit/action from 3.0.0 to 3.0.1 (#8835) - Support moto 5.0 (#8838) - Another fix for [link_error]{.title-ref} signatures being [dict\`s instead of \`Signature]{.title-ref} s (#8841) - Bump codecov/codecov-action from 3 to 4 (#8831) - Upgrade from pytest-celery v1.0.0b1 -\> v1.0.0b2 (#8843) - Bump pytest from 7.4.4 to 8.0.0 (#8823) - Update pre-commit to 3.6.1 (#8839) - Update cryptography to 42.0.3 (#8854) - Bump pytest from 8.0.0 to 8.0.1 (#8855) - Update cryptography to 42.0.4 (#8864) - Update pytest to 8.0.2 (#8870) - Update cryptography to 42.0.5 (#8869) - Update elasticsearch requirement from \<=8.12.0 to \<=8.12.1 (#8867) - Eliminate consecutive chords generated by group \| task upgrade (#8663) - Make custom remote control commands available in CLI (#8489) - Add Google Cloud Storage (GCS) backend (#8868) - Bump msgpack from 1.0.7 to 1.0.8 (#8885) - Update pytest to 8.1.0 (#8886) - Bump pytest-timeout from 2.2.0 to 2.3.1 (#8894) - Bump pytest-subtests from 0.11.0 to 0.12.1 (#8896) - Bump mypy from 1.8.0 to 1.9.0 (#8898) - Update pytest to 8.1.1 (#8901) - Update contributing guide to use ssh upstream url (#8881) - Fix recursive result parents on group in middle of chain (#8903) - Bump pytest-celery to 1.0.0b4 (#8899) - Adjusted smoke tests CI time limit (#8907) - Update pytest-rerunfailures to 14.0 (#8910) - Use the \"all\" extra for pytest-celery (#8911) - Fix typos and grammar (#8915) - Bump pytest-celery to 1.0.0rc1 (#8918) - Print safe_say() to stdout for non-error flows (#8919) - Update pytest-cov to 5.0.0 (#8924) - Bump pytest-celery to 1.0.0rc2 (#8928) ## 5.4.0rc1 release-date : 2024-01-17 7:00 P.M GMT+2 release-by : Tomer Nosrati Celery v5.4 continues our effort to provide improved stability in production environments. The release candidate version is available for testing. The official release is planned for March-April 2024. - New Config: worker_enable_prefetch_count_reduction (#8581) - Added \"Serverless\" section to Redis doc (redis.rst) (#8640) - Upstash\'s Celery example repo link fix (#8665) - Update mypy version (#8679) - Update cryptography dependency to 41.0.7 (#8690) - Add type annotations to celery/utils/nodenames.py (#8667) - Issue 3426. Adding myself to the contributors. (#8696) - Bump actions/setup-python from 4 to 5 (#8701) - Fixed bug where chord.link_error() throws an exception on a dict type errback object (#8702) - Bump github/codeql-action from 2 to 3 (#8725) - Fixed multiprocessing integration tests not running on Mac (#8727) - Added make docker-docs (#8729) - Fix DeprecationWarning: datetime.datetime.utcnow() (#8726) - Remove [new]{.title-ref} adjective in docs (#8743) - add type annotation to celery/utils/sysinfo.py (#8747) - add type annotation to celery/utils/iso8601.py (#8750) - Change type annotation to celery/utils/iso8601.py (#8752) - Update test deps (#8754) - Mark flaky: test_asyncresult_get_cancels_subscription() (#8757) - change \_read_as_base64 (b64encode returns bytes) on celery/utils/term.py (#8759) - Replace string concatenation with fstring on celery/utils/term.py (#8760) - Add type annotation to celery/utils/term.py (#8755) - Skipping test_tasks::test_task_accepted (#8761) - Updated concurrency docs page. (#8753) - Changed pyup -\> dependabot for updating dependencies (#8764) - Bump isort from 5.12.0 to 5.13.2 (#8772) - Update elasticsearch requirement from \<=8.11.0 to \<=8.11.1 (#8775) - Bump sphinx-click from 4.4.0 to 5.1.0 (#8774) - Bump python-memcached from 1.59 to 1.61 (#8776) - Update elastic-transport requirement from \<=8.10.0 to \<=8.11.0 (#8780) - python-memcached==1.61 -\> python-memcached\>=1.61 (#8787) - Remove usage of utcnow (#8791) - Smoke Tests (#8793) - Moved smoke tests to their own workflow (#8797) - Bugfix: Worker not consuming tasks after Redis broker restart (#8796) - Bugfix: Missing id on chain (#8798) --- # Change history {#changelog-5.5} This document contains change notes for bugfix & new features in the main branch & 5.5.x series, please see `whatsnew-5.5`{.interpreted-text role="ref"} for an overview of what\'s new in Celery 5.5. ## 5.5.3 {#version-5.5.3} release-date : 2025-06-01 release-by : Tomer Nosrati ### What\'s Changed - make the tests run on python 3.13 for gcs backend (#9677) - Added DeepWiki to README (#9683) - Limit redis to \<=v5.2.1 to match Kombu (#9693) - Use EX_OK instead of literal zero (#9684) - Make wheel metadata reproducible (#9687) - let celery install from kombu dependencies for better align (#9696) - Fix stamping documentation to clarify stamped_headers key is optional in visitor methods (#9697) - Support apply_async without queue argument on quorum queues (#9686) - Updated rabbitmq doc about using quorum queues with task routes (#9707) - Add: Dumper Unit Test (#9711) - Add unit test for event.group_from (#9709) - refactor: add beat_cron_starting_deadline documentation warning (#9712) - fix: resolve issue #9569 by supporting distinct broker transport options for workers (#9695) - Fixes issue with retry callback arguments in DelayedDelivery (#9708) - get_exchange-unit-test (#9710) - ISSUE-9704: Update documentation of result_expires, filesystem backend is supported (#9716) - update to blacksmith ubuntu 24.04 (#9717) - Added unit tests for celery.utils.iso8601 (#9725) - Update introduction.rst docs (#9728) - Prepare for release: v5.5.3 (#9732) ## 5.5.2 {#version-5.5.2} release-date : 2025-04-25 release-by : Tomer Nosrati ### What\'s Changed - Fix calculating remaining time across DST changes (#9669) - Remove [setup_logger]{.title-ref} from COMPAT_MODULES (#9668) - Fix mongodb bullet and fix github links in contributions section (#9672) - Prepare for release: v5.5.2 (#9675) ## 5.5.1 {#version-5.5.1} release-date : 2025-04-08 release-by : Tomer Nosrati ### What\'s Changed - Fixed \"AttributeError: list object has no attribute strip\" with quorum queues and failover brokers (#9657) - Prepare for release: v5.5.1 (#9660) ## 5.5.0 {#version-5.5.0} release-date : 2025-03-31 release-by : Tomer Nosrati Celery v5.5.0 is now available. ### Key Highlights See `whatsnew-5.5`{.interpreted-text role="ref"} for a complete overview or read the main highlights below. #### Redis Broker Stability Improvements Long-standing disconnection issues with the Redis broker have been identified and resolved in Kombu 5.5.0, which is included with this release. These improvements significantly enhance stability when using Redis as a broker. Additionally, the Redis backend now has better exception handling with the new `exception_safe_to_retry` feature, which improves resilience during temporary Redis connection issues. See `conf-redis-result-backend`{.interpreted-text role="ref"} for complete documentation. Contributed by [\@drienkop](https://github.com/drienkop) in [#9614](https://github.com/celery/celery/pull/9614). #### `pycurl` replaced with `urllib3` Replaced the `pycurl`{.interpreted-text role="pypi"} dependency with `urllib3`{.interpreted-text role="pypi"}. We\'re monitoring the performance impact of this change and welcome feedback from users who notice any significant differences in their environments. Contributed by [\@spawn-guy](https://github.com/spawn-guy) in Kombu [#2134](https://github.com/celery/kombu/pull/2134) and integrated in Celery via [#9526](https://github.com/celery/celery/pull/9526). #### RabbitMQ Quorum Queues Support Added support for RabbitMQ\'s new [Quorum Queues](https://www.rabbitmq.com/docs/quorum-queues) feature, including compatibility with ETA tasks. This implementation has some limitations compared to classic queues, so please refer to the documentation for details. [Native Delayed Delivery](https://docs.particular.net/transports/rabbitmq/delayed-delivery) is automatically enabled when quorum queues are detected to implement the ETA mechanism. See `using-quorum-queues`{.interpreted-text role="ref"} for complete documentation. Configuration options: - `broker_native_delayed_delivery_queue_type`{.interpreted-text role="setting"}: Specifies the queue type for delayed delivery (default: `quorum`) - `task_default_queue_type`{.interpreted-text role="setting"}: Sets the default queue type for tasks (default: `classic`) - `worker_detect_quorum_queues`{.interpreted-text role="setting"}: Controls automatic detection of quorum queues (default: `True`) Contributed in [#9207](https://github.com/celery/celery/pull/9207), [#9121](https://github.com/celery/celery/pull/9121), and [#9599](https://github.com/celery/celery/pull/9599). For details regarding the 404 errors, see [New Year\'s Security Incident](https://github.com/celery/celery/discussions/9525). #### Soft Shutdown Mechanism Soft shutdown is a time limited warm shutdown, initiated just before the cold shutdown. The worker will allow `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} seconds for all currently executing tasks to finish before it terminates. If the time limit is reached, the worker will initiate a cold shutdown and cancel all currently executing tasks. This feature is particularly valuable when using brokers with visibility timeout mechanisms, such as Redis or SQS. It allows the worker enough time to re-queue tasks that were not completed before exiting, preventing task loss during worker shutdown. See `worker-stopping`{.interpreted-text role="ref"} for complete documentation on worker shutdown types. Configuration options: - `worker_soft_shutdown_timeout`{.interpreted-text role="setting"}: Sets the duration in seconds for the soft shutdown period (default: `0.0`, disabled) - `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"}: Controls whether soft shutdown should be enabled even when the worker is idle (default: `False`) Contributed by [\@Nusnus](https://github.com/Nusnus) in [#9213](https://github.com/celery/celery/pull/9213), [#9231](https://github.com/celery/celery/pull/9231), and [#9238](https://github.com/celery/celery/pull/9238). #### Pydantic Support New native support for Pydantic models in tasks. This integration allows you to leverage Pydantic\'s powerful data validation and serialization capabilities directly in your Celery tasks. Example usage: ``` python from pydantic import BaseModel from celery import Celery app = Celery('tasks') class ArgModel(BaseModel): value: int class ReturnModel(BaseModel): value: str @app.task(pydantic=True) def x(arg: ArgModel) -> ReturnModel: # args/kwargs type hinted as Pydantic model will be converted assert isinstance(arg, ArgModel) # The returned model will be converted to a dict automatically return ReturnModel(value=f"example: {arg.value}") ``` See `task-pydantic`{.interpreted-text role="ref"} for complete documentation. Configuration options: - `pydantic=True`: Enables Pydantic integration for the task - `pydantic_strict=True/False`: Controls whether strict validation is enabled (default: `False`) - `pydantic_context={...}`: Provides additional context for validation - `pydantic_dump_kwargs={...}`: Customizes serialization behavior Contributed by [\@mathiasertl](https://github.com/mathiasertl) in [#9023](https://github.com/celery/celery/pull/9023), [#9319](https://github.com/celery/celery/pull/9319), and [#9393](https://github.com/celery/celery/pull/9393). #### Google Pub/Sub Transport New support for Google Cloud Pub/Sub as a message transport, expanding Celery\'s cloud integration options. See `broker-gcpubsub`{.interpreted-text role="ref"} for complete documentation. For the Google Pub/Sub support you have to install additional dependencies: ``` console $ pip install "celery[gcpubsub]" ``` Then configure your Celery application to use the Google Pub/Sub transport: ``` python broker_url = 'gcpubsub://projects/project-id' ``` Contributed by [\@haimjether](https://github.com/haimjether) in [#9351](https://github.com/celery/celery/pull/9351). #### Python 3.13 Support Official support for Python 3.13. All core dependencies have been updated to ensure compatibility, including Kombu and py-amqp. This release maintains compatibility with Python 3.8 through 3.13, as well as PyPy 3.10+. Contributed by [\@Nusnus](https://github.com/Nusnus) in [#9309](https://github.com/celery/celery/pull/9309) and [#9350](https://github.com/celery/celery/pull/9350). #### REMAP_SIGTERM Support The \"REMAP_SIGTERM\" feature, previously undocumented, has been tested, documented, and is now officially supported. This feature allows you to remap the SIGTERM signal to SIGQUIT, enabling you to initiate a soft or cold shutdown using TERM instead of QUIT. This is particularly useful in containerized environments where SIGTERM is the standard signal for graceful termination. See `Cold Shutdown documentation `{.interpreted-text role="ref"} for more info. To enable this feature, set the environment variable: ``` bash export REMAP_SIGTERM="SIGQUIT" ``` Contributed by [\@Nusnus](https://github.com/Nusnus) in [#9461](https://github.com/celery/celery/pull/9461). #### Database Backend Improvements New `create_tables_at_setup` option for the database backend. This option controls when database tables are created, allowing for non-lazy table creation. By default (`create_tables_at_setup=True`), tables are created during backend initialization. Setting this to `False` defers table creation until they are actually needed, which can be useful in certain deployment scenarios where you want more control over database schema management. See `conf-database-result-backend`{.interpreted-text role="ref"} for complete documentation. Configuration: ``` python app.conf.result_backend = 'db+sqlite:///results.db' app.conf.database_create_tables_at_setup = False ``` Contributed by [\@MarcBresson](https://github.com/MarcBresson) in [#9228](https://github.com/celery/celery/pull/9228). ### What\'s Changed - (docs): use correct version celery v.5.4.x (#8975) - Update mypy to 1.10.0 (#8977) - Limit pymongo\<4.7 when Python \<= 3.10 due to breaking changes in 4.7 (#8988) - Bump pytest from 8.1.1 to 8.2.0 (#8987) - Update README to Include FastAPI in Framework Integration Section (#8978) - Clarify return values of \...\_on_commit methods (#8984) - add kafka broker docs (#8935) - Limit pymongo\<4.7 regardless of Python version (#8999) - Update pymongo\[srv\] requirement from \<4.7,\>=4.0.2 to \>=4.0.2,\<4.8 (#9000) - Update elasticsearch requirement from \<=8.13.0 to \<=8.13.1 (#9004) - security: SecureSerializer: support generic low-level serializers (#8982) - don\'t kill if pid same as file (#8997) (#8998) - Update cryptography to 42.0.6 (#9005) - Bump cryptography from 42.0.6 to 42.0.7 (#9009) - don\'t kill if pid same as file (#8997) (#8998) (#9007) - Added -vv to unit, integration and smoke tests (#9014) - SecuritySerializer: ensure pack separator will not be conflicted with serialized fields (#9010) - Update sphinx-click to 5.2.2 (#9025) - Bump sphinx-click from 5.2.2 to 6.0.0 (#9029) - Fix a typo to display the help message in first-steps-with-django (#9036) - Pinned requests to v2.31.0 due to docker-py bug #3256 (#9039) - Fix certificate validity check (#9037) - Revert \"Pinned requests to v2.31.0 due to docker-py bug #3256\" (#9043) - Bump pytest from 8.2.0 to 8.2.1 (#9035) - Update elasticsearch requirement from \<=8.13.1 to \<=8.13.2 (#9045) - Fix detection of custom task set as class attribute with Django (#9038) - Update elastic-transport requirement from \<=8.13.0 to \<=8.13.1 (#9050) - Bump pycouchdb from 1.14.2 to 1.16.0 (#9052) - Update pytest to 8.2.2 (#9060) - Bump cryptography from 42.0.7 to 42.0.8 (#9061) - Update elasticsearch requirement from \<=8.13.2 to \<=8.14.0 (#9069) - \[enhance feature\] Crontab schedule: allow using month names (#9068) - Enhance tox environment: \[testenv:clean\] (#9072) - Clarify docs about Reserve one task at a time (#9073) - GCS docs fixes (#9075) - Use hub.remove_writer instead of hub.remove for write fds (#4185) (#9055) - Class method to process crontab string (#9079) - Fixed smoke tests env bug when using integration tasks that rely on Redis (#9090) - Bugfix - a task will run multiple times when chaining chains with groups (#9021) - Bump mypy from 1.10.0 to 1.10.1 (#9096) - Don\'t add a separator to global_keyprefix if it already has one (#9080) - Update pymongo\[srv\] requirement from \<4.8,\>=4.0.2 to \>=4.0.2,\<4.9 (#9111) - Added missing import in examples for Django (#9099) - Bump Kombu to v5.4.0rc1 (#9117) - Removed skipping Redis in t/smoke/tests/test_consumer.py tests (#9118) - Update pytest-subtests to 0.13.0 (#9120) - Increased smoke tests CI timeout (#9122) - Bump Kombu to v5.4.0rc2 (#9127) - Update zstandard to 0.23.0 (#9129) - Update pytest-subtests to 0.13.1 (#9130) - Changed retry to tenacity in smoke tests (#9133) - Bump mypy from 1.10.1 to 1.11.0 (#9135) - Update cryptography to 43.0.0 (#9138) - Update pytest to 8.3.1 (#9137) - Added support for Quorum Queues (#9121) - Bump Kombu to v5.4.0rc3 (#9139) - Cleanup in Changelog.rst (#9141) - Update Django docs for CELERY_CACHE_BACKEND (#9143) - Added missing docs to previous releases (#9144) - Fixed a few documentation build warnings (#9145) - docs(README): link invalid (#9148) - Prepare for (pre) release: v5.5.0b1 (#9146) - Bump pytest from 8.3.1 to 8.3.2 (#9153) - Remove setuptools deprecated test command from setup.py (#9159) - Pin pre-commit to latest version 3.8.0 from Python 3.9 (#9156) - Bump mypy from 1.11.0 to 1.11.1 (#9164) - Change \"docker-compose\" to \"docker compose\" in Makefile (#9169) - update python versions and docker compose (#9171) - Add support for Pydantic model validation/serialization (fixes #8751) (#9023) - Allow local dynamodb to be installed on another host than localhost (#8965) - Terminate job implementation for gevent concurrency backend (#9083) - Bump Kombu to v5.4.0 (#9177) - Add check for soft_time_limit and time_limit values (#9173) - Prepare for (pre) release: v5.5.0b2 (#9178) - Added SQS (localstack) broker to canvas smoke tests (#9179) - Pin elastic-transport to \<= latest version 8.15.0 (#9182) - Update elasticsearch requirement from \<=8.14.0 to \<=8.15.0 (#9186) - improve formatting (#9188) - Add basic helm chart for celery (#9181) - Update kafka.rst (#9194) - Update pytest-order to 1.3.0 (#9198) - Update mypy to 1.11.2 (#9206) - all added to routes (#9204) - Fix typos discovered by codespell (#9212) - Use tzdata extras with zoneinfo backports (#8286) - Use [docker compose]{.title-ref} in Contributing\'s doc build section (#9219) - Failing test for issue #9119 (#9215) - Fix date_done timezone issue (#8385) - CI Fixes to smoke tests (#9223) - fix: passes current request context when pushing to request_stack (#9208) - Fix broken link in the Using RabbitMQ docs page (#9226) - Added Soft Shutdown Mechanism (#9213) - Added worker_enable_soft_shutdown_on_idle (#9231) - Bump cryptography from 43.0.0 to 43.0.1 (#9233) - Added docs regarding the relevancy of soft shutdown and ETA tasks (#9238) - Show broker_connection_retry_on_startup warning only if it evaluates as False (#9227) - Fixed docker-docs CI failure (#9240) - Added docker cleanup auto-fixture to improve smoke tests stability (#9243) - print is not thread-safe, so should not be used in signal handler (#9222) - Prepare for (pre) release: v5.5.0b3 (#9244) - Correct the error description in exception message when validate soft_time_limit (#9246) - Update msgpack to 1.1.0 (#9249) - chore(utils/time.py): rename [\_is_ambigious]{.title-ref} -\> [\_is_ambiguous]{.title-ref} (#9248) - Reduced Smoke Tests to min/max supported python (3.8/3.12) (#9252) - Update pytest to 8.3.3 (#9253) - Update elasticsearch requirement from \<=8.15.0 to \<=8.15.1 (#9255) - update mongodb without deprecated [\[srv\]]{.title-ref} extra requirement (#9258) - blacksmith.sh: Migrate workflows to Blacksmith (#9261) - Fixes #9119: inject dispatch_uid for retry-wrapped receivers (#9247) - Run all smoke tests CI jobs together (#9263) - Improve documentation on visibility timeout (#9264) - Bump pytest-celery to 1.1.2 (#9267) - Added missing \"app.conf.visibility_timeout\" in smoke tests (#9266) - Improved stability with t/smoke/tests/test_consumer.py (#9268) - Improved Redis container stability in the smoke tests (#9271) - Disabled EXHAUST_MEMORY tests in Smoke-tasks (#9272) - Marked xfail for test_reducing_prefetch_count with Redis - flaky test (#9273) - Fixed pypy unit tests random failures in the CI (#9275) - Fixed more pypy unit tests random failures in the CI (#9278) - Fix Redis container from aborting randomly (#9276) - Run Integration & Smoke CI tests together after unit tests passes (#9280) - Added \"loglevel verbose\" to Redis containers in smoke tests (#9282) - Fixed Redis error in the smoke tests: \"Possible SECURITY ATTACK detected\" (#9284) - Refactored the smoke tests github workflow (#9285) - Increased \--reruns 3-\>4 in smoke tests (#9286) - Improve stability of smoke tests (CI and Local) (#9287) - Fixed Smoke tests CI \"test-case\" lables (specific instead of general) (#9288) - Use assert_log_exists instead of wait_for_log in worker smoke tests (#9290) - Optimized t/smoke/tests/test_worker.py (#9291) - Enable smoke tests dockers check before each test starts (#9292) - Relaxed smoke tests flaky tests mechanism (#9293) - Updated quorum queue detection to handle multiple broker instances (#9294) - Non-lazy table creation for database backend (#9228) - Pin pymongo to latest version 4.9 (#9297) - Bump pymongo from 4.9 to 4.9.1 (#9298) - Bump Kombu to v5.4.2 (#9304) - Use rabbitmq:3 in stamping smoke tests (#9307) - Bump pytest-celery to 1.1.3 (#9308) - Added Python 3.13 Support (#9309) - Add log when global qos is disabled (#9296) - Added official release docs (whatsnew) for v5.5 (#9312) - Enable Codespell autofix (#9313) - Pydantic typehints: Fix optional, allow generics (#9319) - Prepare for (pre) release: v5.5.0b4 (#9322) - Added Blacksmith.sh to the Sponsors section in the README (#9323) - Revert \"Added Blacksmith.sh to the Sponsors section in the README\" (#9324) - Added Blacksmith.sh to the Sponsors section in the README (#9325) - Added missing \" " in README (#9326) - Use Blacksmith SVG logo (#9327) - Updated Blacksmith SVG logo (#9328) - Revert \"Updated Blacksmith SVG logo\" (#9329) - Update pymongo to 4.10.0 (#9330) - Update pymongo to 4.10.1 (#9332) - Update user guide to recommend delay_on_commit (#9333) - Pin pre-commit to latest version 4.0.0 (Python 3.9+) (#9334) - Update ephem to 4.1.6 (#9336) - Updated Blacksmith SVG logo (#9337) - Prepare for (pre) release: v5.5.0rc1 (#9341) - Fix: Treat dbm.error as a corrupted schedule file (#9331) - Pin pre-commit to latest version 4.0.1 (#9343) - Added Python 3.13 to Dockerfiles (#9350) - Skip test_pool_restart_import_modules on PyPy due to test issue (#9352) - Update elastic-transport requirement from \<=8.15.0 to \<=8.15.1 (#9347) - added dragonfly logo (#9353) - Update README.rst (#9354) - Update README.rst (#9355) - Update mypy to 1.12.0 (#9356) - Bump Kombu to v5.5.0rc1 (#9357) - Fix [celery \--loader]{.title-ref} option parsing (#9361) - Add support for Google Pub/Sub transport (#9351) - Add native incr support for GCSBackend (#9302) - fix(perform_pending_operations): prevent task duplication on shutdown... (#9348) - Update grpcio to 1.67.0 (#9365) - Update google-cloud-firestore to 2.19.0 (#9364) - Annotate celery/utils/timer2.py (#9362) - Update cryptography to 43.0.3 (#9366) - Update mypy to 1.12.1 (#9368) - Bump mypy from 1.12.1 to 1.13.0 (#9373) - Pass timeout and confirm_timeout to producer.publish() (#9374) - Bump Kombu to v5.5.0rc2 (#9382) - Bump pytest-cov from 5.0.0 to 6.0.0 (#9388) - default strict to False for pydantic tasks (#9393) - Only log that global QoS is disabled if using amqp (#9395) - chore: update sponsorship logo (#9398) - Allow custom hostname for celery_worker in celery.contrib.pytest / celery.contrib.testing.worker (#9405) - Removed docker-docs from CI (optional job, malfunctioning) (#9406) - Added a utility to format changelogs from the auto-generated GitHub release notes (#9408) - Bump codecov/codecov-action from 4 to 5 (#9412) - Update elasticsearch requirement from \<=8.15.1 to \<=8.16.0 (#9410) - Native Delayed Delivery in RabbitMQ (#9207) - Prepare for (pre) release: v5.5.0rc2 (#9416) - Document usage of broker_native_delayed_delivery_queue_type (#9419) - Adjust section in what\'s new document regarding quorum queues support (#9420) - Update pytest-rerunfailures to 15.0 (#9422) - Document group unrolling (#9421) - fix small typo acces -\> access (#9434) - Update cryptography to 44.0.0 (#9437) - Added pypy to Dockerfile (#9438) - Skipped flaky tests on pypy (all pass after \~10 reruns) (#9439) - Allowing managed credentials for azureblockblob (#9430) - Allow passing Celery objects to the Click entry point (#9426) - support Request termination for gevent (#9440) - Prevent event_mask from being overwritten. (#9432) - Update pytest to 8.3.4 (#9444) - Prepare for (pre) release: v5.5.0rc3 (#9450) - Bugfix: SIGQUIT not initiating cold shutdown when [task_acks_late=False]{.title-ref} (#9461) - Fixed pycurl dep with Python 3.8 (#9471) - Update elasticsearch requirement from \<=8.16.0 to \<=8.17.0 (#9469) - Bump pytest-subtests from 0.13.1 to 0.14.1 (#9459) - documentation: Added a type annotation to the periodic task example (#9473) - Prepare for (pre) release: v5.5.0rc4 (#9474) - Bump mypy from 1.13.0 to 1.14.0 (#9476) - Fix cassandra backend port settings not working (#9465) - Unroll group when a group with a single item is chained using the \| operator (#9456) - fix(django): catch the right error when trying to close db connection (#9392) - Replacing a task with a chain which contains a group now returns a result instead of hanging (#9484) - Avoid using a group of one as it is now unrolled into a chain (#9510) - Link to the correct IRC network (#9509) - Bump pytest-github-actions-annotate-failures from 0.2.0 to 0.3.0 (#9504) - Update canvas.rst to fix output result from chain object (#9502) - Unauthorized Changes Cleanup (#9528) - \[RE-APPROVED\] fix(django): catch the right error when trying to close db connection (#9529) - \[RE-APPROVED\] Link to the correct IRC network (#9531) - \[RE-APPROVED\] Update canvas.rst to fix output result from chain object (#9532) - Update test-ci-base.txt (#9539) - Update install-pyenv.sh (#9540) - Update elasticsearch requirement from \<=8.17.0 to \<=8.17.1 (#9518) - Bump google-cloud-firestore from 2.19.0 to 2.20.0 (#9493) - Bump mypy from 1.14.0 to 1.14.1 (#9483) - Update elastic-transport requirement from \<=8.15.1 to \<=8.17.0 (#9490) - Update Dockerfile by adding missing Python version 3.13 (#9549) - Fix typo for default of sig (#9495) - fix(crontab): resolve constructor type conflicts (#9551) - worker_max_memory_per_child: kilobyte is 1024 bytes (#9553) - Fix formatting in quorum queue docs (#9555) - Bump cryptography from 44.0.0 to 44.0.1 (#9556) - Fix the send_task method when detecting if the native delayed delivery approach is available (#9552) - Reverted PR #7814 & minor code improvement (#9494) - Improved donation and sponsorship visibility (#9558) - Updated the Getting Help section, replacing deprecated with new resources (#9559) - Fixed django example (#9562) - Bump Kombu to v5.5.0rc3 (#9564) - Bump ephem from 4.1.6 to 4.2 (#9565) - Bump pytest-celery to v1.2.0 (#9568) - Remove dependency on [pycurl]{.title-ref} (#9526) - Set TestWorkController.\_\_test\_\_ (#9574) - Fixed bug when revoking by stamped headers a stamp that does not exist (#9575) - Canvas Stamping Doc Fixes (#9578) - Bugfix: Chord with a chord in header doesn\'t invoke error callback on inner chord header failure (default config) (#9580) - Prepare for (pre) release: v5.5.0rc5 (#9582) - Bump google-cloud-firestore from 2.20.0 to 2.20.1 (#9584) - Fix tests with Click 8.2 (#9590) - Bump cryptography from 44.0.1 to 44.0.2 (#9591) - Update elasticsearch requirement from \<=8.17.1 to \<=8.17.2 (#9594) - Bump pytest from 8.3.4 to 8.3.5 (#9598) - Refactored and Enhanced DelayedDelivery bootstep (#9599) - Improve docs about acks_on_failure_or_timeout (#9577) - Update SECURITY.md (#9609) - remove flake8plus as not needed anymore (#9610) - remove \[bdist_wheel\] universal = 0 from setup.cfg as not needed (#9611) - remove importlib-metadata as not needed in python3.8 anymore (#9612) - feat: define exception_safe_to_retry for redisbackend (#9614) - Bump Kombu to v5.5.0 (#9615) - Update elastic-transport requirement from \<=8.17.0 to \<=8.17.1 (#9616) - \[docs\] fix first-steps (#9618) - Revert \"Improve docs about acks_on_failure_or_timeout\" (#9606) - Improve CI stability and performance (#9624) - Improved explanation for Database transactions at user guide for tasks (#9617) - update tests to use python 3.8 codes only (#9627) - #9597: Ensure surpassing Hard Timeout limit when task_acks_on_failure_or_timeout is False rejects the task (#9626) - Lock Kombu to v5.5.x (using urllib3 instead of pycurl) (#9632) - Lock pytest-celery to v1.2.x (using urllib3 instead of pycurl) (#9633) - Add Codecov Test Analytics (#9635) - Bump Kombu to v5.5.2 (#9643) - Prepare for release: v5.5.0 (#9644) ## 5.5.0rc5 {#version-5.5.0rc5} release-date : 2025-02-25 release-by : Tomer Nosrati Celery v5.5.0 Release Candidate 5 is now available for testing. Please help us test this version and report any issues. ### Key Highlights See `whatsnew-5.5`{.interpreted-text role="ref"} or read the main highlights below. #### Using Kombu 5.5.0rc3 The minimum required Kombu version has been bumped to 5.5.0. Kombu is currently at 5.5.0rc3. #### Complete Quorum Queues Support A completely new ETA mechanism was developed to allow full support with RabbitMQ Quorum Queues. After upgrading to this version, please share your feedback on the quorum queues support. Relevant Issues: [#9207](https://github.com/celery/celery/discussions/9207), [#6067](https://github.com/celery/celery/discussions/6067) - New `documentation `{.interpreted-text role="ref"}. - New `broker_native_delayed_delivery_queue_type`{.interpreted-text role="setting"} configuration option. #### New support for Google Pub/Sub transport After upgrading to this version, please share your feedback on the Google Pub/Sub transport support. Relevant Issues: [#9351](https://github.com/celery/celery/pull/9351) #### Python 3.13 Improved Support Additional dependencies have been migrated successfully to Python 3.13, including Kombu and py-amqp. #### Soft Shutdown The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited \"warm shutdown\" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting. The soft shutdown is disabled by default, and can be enabled by setting the new configuration option `worker_soft_shutdown_timeout`{.interpreted-text role="setting"}. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways. The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like `Redis `{.interpreted-text role="ref"} or `SQS `{.interpreted-text role="ref"}, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., `Restoring 1 unacknowledged message(s)`) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely. After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism. Relevant Issues: [#9213](https://github.com/celery/celery/pull/9213), [#9231](https://github.com/celery/celery/pull/9231), [#9238](https://github.com/celery/celery/pull/9238) - New `documentation `{.interpreted-text role="ref"} for each shutdown type. - New `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} configuration option. - New `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} configuration option. #### REMAP_SIGTERM The `REMAP_SIGTERM` \"hidden feature\" has been tested, `documented `{.interpreted-text role="ref"} and is now officially supported. This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using `TERM`{.interpreted-text role="sig"} instead of `QUIT`{.interpreted-text role="sig"}. #### Pydantic Support This release introduces support for Pydantic models in Celery tasks. For more info, see the new pydantic example and PR [#9023](https://github.com/celery/celery/pull/9023) by \@mathiasertl. After upgrading to this version, please share your feedback on the new Pydantic support. #### Redis Broker Stability Improvements The root cause of the Redis broker instability issue has been [identified and resolved](https://github.com/celery/kombu/pull/2007) in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements. After upgrading to this version, please share your feedback on the Redis broker stability. Relevant Issues: [#7276](https://github.com/celery/celery/discussions/7276), [#8091](https://github.com/celery/celery/discussions/8091), [#8030](https://github.com/celery/celery/discussions/8030), [#8384](https://github.com/celery/celery/discussions/8384) #### Quorum Queues Initial Support This release introduces the initial support for Quorum Queues with Celery. See new configuration options for more details: - `task_default_queue_type`{.interpreted-text role="setting"} - `worker_detect_quorum_queues`{.interpreted-text role="setting"} After upgrading to this version, please share your feedback on the Quorum Queues support. Relevant Issues: [#6067](https://github.com/celery/celery/discussions/6067), [#9121](https://github.com/celery/celery/discussions/9121) ### What\'s Changed - Bump mypy from 1.13.0 to 1.14.0 (#9476) - Fix cassandra backend port settings not working (#9465) - Unroll group when a group with a single item is chained using the \| operator (#9456) - fix(django): catch the right error when trying to close db connection (#9392) - Replacing a task with a chain which contains a group now returns a result instead of hanging (#9484) - Avoid using a group of one as it is now unrolled into a chain (#9510) - Link to the correct IRC network (#9509) - Bump pytest-github-actions-annotate-failures from 0.2.0 to 0.3.0 (#9504) - Update canvas.rst to fix output result from chain object (#9502) - Unauthorized Changes Cleanup (#9528) - \[RE-APPROVED\] fix(django): catch the right error when trying to close db connection (#9529) - \[RE-APPROVED\] Link to the correct IRC network (#9531) - \[RE-APPROVED\] Update canvas.rst to fix output result from chain object (#9532) - Update test-ci-base.txt (#9539) - Update install-pyenv.sh (#9540) - Update elasticsearch requirement from \<=8.17.0 to \<=8.17.1 (#9518) - Bump google-cloud-firestore from 2.19.0 to 2.20.0 (#9493) - Bump mypy from 1.14.0 to 1.14.1 (#9483) - Update elastic-transport requirement from \<=8.15.1 to \<=8.17.0 (#9490) - Update Dockerfile by adding missing Python version 3.13 (#9549) - Fix typo for default of sig (#9495) - fix(crontab): resolve constructor type conflicts (#9551) - worker_max_memory_per_child: kilobyte is 1024 bytes (#9553) - Fix formatting in quorum queue docs (#9555) - Bump cryptography from 44.0.0 to 44.0.1 (#9556) - Fix the send_task method when detecting if the native delayed delivery approach is available (#9552) - Reverted PR #7814 & minor code improvement (#9494) - Improved donation and sponsorship visibility (#9558) - Updated the Getting Help section, replacing deprecated with new resources (#9559) - Fixed django example (#9562) - Bump Kombu to v5.5.0rc3 (#9564) - Bump ephem from 4.1.6 to 4.2 (#9565) - Bump pytest-celery to v1.2.0 (#9568) - Remove dependency on [pycurl]{.title-ref} (#9526) - Set TestWorkController.\_\_test\_\_ (#9574) - Fixed bug when revoking by stamped headers a stamp that does not exist (#9575) - Canvas Stamping Doc Fixes (#9578) - Bugfix: Chord with a chord in header doesn\'t invoke error callback on inner chord header failure (default config) (#9580) - Prepare for (pre) release: v5.5.0rc5 (#9582) ## 5.5.0rc4 {#version-5.5.0rc4} release-date : 2024-12-19 release-by : Tomer Nosrati Celery v5.5.0 Release Candidate 4 is now available for testing. Please help us test this version and report any issues. ### Key Highlights See `whatsnew-5.5`{.interpreted-text role="ref"} or read the main highlights below. #### Using Kombu 5.5.0rc2 The minimum required Kombu version has been bumped to 5.5.0. Kombu is current at 5.5.0rc2. #### Complete Quorum Queues Support A completely new ETA mechanism was developed to allow full support with RabbitMQ Quorum Queues. After upgrading to this version, please share your feedback on the quorum queues support. Relevant Issues: [#9207](https://github.com/celery/celery/discussions/9207), [#6067](https://github.com/celery/celery/discussions/6067) - New `documentation `{.interpreted-text role="ref"}. - New `broker_native_delayed_delivery_queue_type`{.interpreted-text role="setting"} configuration option. #### New support for Google Pub/Sub transport After upgrading to this version, please share your feedback on the Google Pub/Sub transport support. Relevant Issues: [#9351](https://github.com/celery/celery/pull/9351) #### Python 3.13 Improved Support Additional dependencies have been migrated successfully to Python 3.13, including Kombu and py-amqp. #### Soft Shutdown The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited \"warm shutdown\" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting. The soft shutdown is disabled by default, and can be enabled by setting the new configuration option `worker_soft_shutdown_timeout`{.interpreted-text role="setting"}. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways. The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like `Redis `{.interpreted-text role="ref"} or `SQS `{.interpreted-text role="ref"}, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., `Restoring 1 unacknowledged message(s)`) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely. After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism. Relevant Issues: [#9213](https://github.com/celery/celery/pull/9213), [#9231](https://github.com/celery/celery/pull/9231), [#9238](https://github.com/celery/celery/pull/9238) - New `documentation `{.interpreted-text role="ref"} for each shutdown type. - New `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} configuration option. - New `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} configuration option. #### REMAP_SIGTERM The `REMAP_SIGTERM` \"hidden feature\" has been tested, `documented `{.interpreted-text role="ref"} and is now officially supported. This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using `TERM`{.interpreted-text role="sig"} instead of `QUIT`{.interpreted-text role="sig"}. #### Pydantic Support This release introduces support for Pydantic models in Celery tasks. For more info, see the new pydantic example and PR [#9023](https://github.com/celery/celery/pull/9023) by \@mathiasertl. After upgrading to this version, please share your feedback on the new Pydantic support. #### Redis Broker Stability Improvements The root cause of the Redis broker instability issue has been [identified and resolved](https://github.com/celery/kombu/pull/2007) in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements. After upgrading to this version, please share your feedback on the Redis broker stability. Relevant Issues: [#7276](https://github.com/celery/celery/discussions/7276), [#8091](https://github.com/celery/celery/discussions/8091), [#8030](https://github.com/celery/celery/discussions/8030), [#8384](https://github.com/celery/celery/discussions/8384) #### Quorum Queues Initial Support This release introduces the initial support for Quorum Queues with Celery. See new configuration options for more details: - `task_default_queue_type`{.interpreted-text role="setting"} - `worker_detect_quorum_queues`{.interpreted-text role="setting"} After upgrading to this version, please share your feedback on the Quorum Queues support. Relevant Issues: [#6067](https://github.com/celery/celery/discussions/6067), [#9121](https://github.com/celery/celery/discussions/9121) ### What\'s Changed - Bugfix: SIGQUIT not initiating cold shutdown when [task_acks_late=False]{.title-ref} (#9461) - Fixed pycurl dep with Python 3.8 (#9471) - Update elasticsearch requirement from \<=8.16.0 to \<=8.17.0 (#9469) - Bump pytest-subtests from 0.13.1 to 0.14.1 (#9459) - documentation: Added a type annotation to the periodic task example (#9473) - Prepare for (pre) release: v5.5.0rc4 (#9474) ## 5.5.0rc3 {#version-5.5.0rc3} release-date : 2024-12-03 release-by : Tomer Nosrati Celery v5.5.0 Release Candidate 3 is now available for testing. Please help us test this version and report any issues. ### Key Highlights See `whatsnew-5.5`{.interpreted-text role="ref"} or read the main highlights below. #### Using Kombu 5.5.0rc2 The minimum required Kombu version has been bumped to 5.5.0. Kombu is current at 5.5.0rc2. #### Complete Quorum Queues Support A completely new ETA mechanism was developed to allow full support with RabbitMQ Quorum Queues. After upgrading to this version, please share your feedback on the quorum queues support. Relevant Issues: [#9207](https://github.com/celery/celery/discussions/9207), [#6067](https://github.com/celery/celery/discussions/6067) - New `documentation `{.interpreted-text role="ref"}. - New `broker_native_delayed_delivery_queue_type`{.interpreted-text role="setting"} configuration option. #### New support for Google Pub/Sub transport After upgrading to this version, please share your feedback on the Google Pub/Sub transport support. Relevant Issues: [#9351](https://github.com/celery/celery/pull/9351) #### Python 3.13 Improved Support Additional dependencies have been migrated successfully to Python 3.13, including Kombu and py-amqp. #### Soft Shutdown The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited \"warm shutdown\" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting. The soft shutdown is disabled by default, and can be enabled by setting the new configuration option `worker_soft_shutdown_timeout`{.interpreted-text role="setting"}. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways. The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like `Redis `{.interpreted-text role="ref"} or `SQS `{.interpreted-text role="ref"}, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., `Restoring 1 unacknowledged message(s)`) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely. After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism. Relevant Issues: [#9213](https://github.com/celery/celery/pull/9213), [#9231](https://github.com/celery/celery/pull/9231), [#9238](https://github.com/celery/celery/pull/9238) - New `documentation `{.interpreted-text role="ref"} for each shutdown type. - New `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} configuration option. - New `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} configuration option. #### REMAP_SIGTERM The `REMAP_SIGTERM` \"hidden feature\" has been tested, `documented `{.interpreted-text role="ref"} and is now officially supported. This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using `TERM`{.interpreted-text role="sig"} instead of `QUIT`{.interpreted-text role="sig"}. #### Pydantic Support This release introduces support for Pydantic models in Celery tasks. For more info, see the new pydantic example and PR [#9023](https://github.com/celery/celery/pull/9023) by \@mathiasertl. After upgrading to this version, please share your feedback on the new Pydantic support. #### Redis Broker Stability Improvements The root cause of the Redis broker instability issue has been [identified and resolved](https://github.com/celery/kombu/pull/2007) in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements. After upgrading to this version, please share your feedback on the Redis broker stability. Relevant Issues: [#7276](https://github.com/celery/celery/discussions/7276), [#8091](https://github.com/celery/celery/discussions/8091), [#8030](https://github.com/celery/celery/discussions/8030), [#8384](https://github.com/celery/celery/discussions/8384) #### Quorum Queues Initial Support This release introduces the initial support for Quorum Queues with Celery. See new configuration options for more details: - `task_default_queue_type`{.interpreted-text role="setting"} - `worker_detect_quorum_queues`{.interpreted-text role="setting"} After upgrading to this version, please share your feedback on the Quorum Queues support. Relevant Issues: [#6067](https://github.com/celery/celery/discussions/6067), [#9121](https://github.com/celery/celery/discussions/9121) ### What\'s Changed - Document usage of broker_native_delayed_delivery_queue_type (#9419) - Adjust section in what\'s new document regarding quorum queues support (#9420) - Update pytest-rerunfailures to 15.0 (#9422) - Document group unrolling (#9421) - fix small typo acces -\> access (#9434) - Update cryptography to 44.0.0 (#9437) - Added pypy to Dockerfile (#9438) - Skipped flaky tests on pypy (all pass after \~10 reruns) (#9439) - Allowing managed credentials for azureblockblob (#9430) - Allow passing Celery objects to the Click entry point (#9426) - support Request termination for gevent (#9440) - Prevent event_mask from being overwritten. (#9432) - Update pytest to 8.3.4 (#9444) - Prepare for (pre) release: v5.5.0rc3 (#9450) ## 5.5.0rc2 {#version-5.5.0rc2} release-date : 2024-11-18 release-by : Tomer Nosrati Celery v5.5.0 Release Candidate 2 is now available for testing. Please help us test this version and report any issues. ### Key Highlights See `whatsnew-5.5`{.interpreted-text role="ref"} or read the main highlights below. #### Using Kombu 5.5.0rc2 The minimum required Kombu version has been bumped to 5.5.0. Kombu is current at 5.5.0rc2. #### Complete Quorum Queues Support A completely new ETA mechanism was developed to allow full support with RabbitMQ Quorum Queues. After upgrading to this version, please share your feedback on the quorum queues support. Relevant Issues: [#9207](https://github.com/celery/celery/discussions/9207), [#6067](https://github.com/celery/celery/discussions/6067) - New `documentation `{.interpreted-text role="ref"}. - New `broker_native_delayed_delivery_queue_type`{.interpreted-text role="setting"} configuration option. #### New support for Google Pub/Sub transport After upgrading to this version, please share your feedback on the Google Pub/Sub transport support. Relevant Issues: [#9351](https://github.com/celery/celery/pull/9351) #### Python 3.13 Improved Support Additional dependencies have been migrated successfully to Python 3.13, including Kombu and py-amqp. ### Previous Pre-release Highlights #### Python 3.13 Initial Support This release introduces the initial support for Python 3.13 with Celery. After upgrading to this version, please share your feedback on the Python 3.13 support. #### Soft Shutdown The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited \"warm shutdown\" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting. The soft shutdown is disabled by default, and can be enabled by setting the new configuration option `worker_soft_shutdown_timeout`{.interpreted-text role="setting"}. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways. The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like `Redis `{.interpreted-text role="ref"} or `SQS `{.interpreted-text role="ref"}, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., `Restoring 1 unacknowledged message(s)`) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely. After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism. Relevant Issues: [#9213](https://github.com/celery/celery/pull/9213), [#9231](https://github.com/celery/celery/pull/9231), [#9238](https://github.com/celery/celery/pull/9238) - New `documentation `{.interpreted-text role="ref"} for each shutdown type. - New `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} configuration option. - New `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} configuration option. #### REMAP_SIGTERM The `REMAP_SIGTERM` \"hidden feature\" has been tested, `documented `{.interpreted-text role="ref"} and is now officially supported. This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using `TERM`{.interpreted-text role="sig"} instead of `QUIT`{.interpreted-text role="sig"}. #### Pydantic Support This release introduces support for Pydantic models in Celery tasks. For more info, see the new pydantic example and PR [#9023](https://github.com/celery/celery/pull/9023) by \@mathiasertl. After upgrading to this version, please share your feedback on the new Pydantic support. #### Redis Broker Stability Improvements The root cause of the Redis broker instability issue has been [identified and resolved](https://github.com/celery/kombu/pull/2007) in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements. After upgrading to this version, please share your feedback on the Redis broker stability. Relevant Issues: [#7276](https://github.com/celery/celery/discussions/7276), [#8091](https://github.com/celery/celery/discussions/8091), [#8030](https://github.com/celery/celery/discussions/8030), [#8384](https://github.com/celery/celery/discussions/8384) #### Quorum Queues Initial Support This release introduces the initial support for Quorum Queues with Celery. See new configuration options for more details: - `task_default_queue_type`{.interpreted-text role="setting"} - `worker_detect_quorum_queues`{.interpreted-text role="setting"} After upgrading to this version, please share your feedback on the Quorum Queues support. Relevant Issues: [#6067](https://github.com/celery/celery/discussions/6067), [#9121](https://github.com/celery/celery/discussions/9121) ### What\'s Changed - Fix: Treat dbm.error as a corrupted schedule file (#9331) - Pin pre-commit to latest version 4.0.1 (#9343) - Added Python 3.13 to Dockerfiles (#9350) - Skip test_pool_restart_import_modules on PyPy due to test issue (#9352) - Update elastic-transport requirement from \<=8.15.0 to \<=8.15.1 (#9347) - added dragonfly logo (#9353) - Update README.rst (#9354) - Update README.rst (#9355) - Update mypy to 1.12.0 (#9356) - Bump Kombu to v5.5.0rc1 (#9357) - Fix [celery \--loader]{.title-ref} option parsing (#9361) - Add support for Google Pub/Sub transport (#9351) - Add native incr support for GCSBackend (#9302) - fix(perform_pending_operations): prevent task duplication on shutdown... (#9348) - Update grpcio to 1.67.0 (#9365) - Update google-cloud-firestore to 2.19.0 (#9364) - Annotate celery/utils/timer2.py (#9362) - Update cryptography to 43.0.3 (#9366) - Update mypy to 1.12.1 (#9368) - Bump mypy from 1.12.1 to 1.13.0 (#9373) - Pass timeout and confirm_timeout to producer.publish() (#9374) - Bump Kombu to v5.5.0rc2 (#9382) - Bump pytest-cov from 5.0.0 to 6.0.0 (#9388) - default strict to False for pydantic tasks (#9393) - Only log that global QoS is disabled if using amqp (#9395) - chore: update sponsorship logo (#9398) - Allow custom hostname for celery_worker in celery.contrib.pytest / celery.contrib.testing.worker (#9405) - Removed docker-docs from CI (optional job, malfunctioning) (#9406) - Added a utility to format changelogs from the auto-generated GitHub release notes (#9408) - Bump codecov/codecov-action from 4 to 5 (#9412) - Update elasticsearch requirement from \<=8.15.1 to \<=8.16.0 (#9410) - Native Delayed Delivery in RabbitMQ (#9207) - Prepare for (pre) release: v5.5.0rc2 (#9416) ## 5.5.0rc1 {#version-5.5.0rc1} release-date : 2024-10-08 release-by : Tomer Nosrati Celery v5.5.0 Release Candidate 1 is now available for testing. Please help us test this version and report any issues. ### Key Highlights See `whatsnew-5.5`{.interpreted-text role="ref"} or read main highlights below. #### Python 3.13 Initial Support This release introduces the initial support for Python 3.13 with Celery. After upgrading to this version, please share your feedback on the Python 3.13 support. #### Soft Shutdown The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited \"warm shutdown\" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting. The soft shutdown is disabled by default, and can be enabled by setting the new configuration option `worker_soft_shutdown_timeout`{.interpreted-text role="setting"}. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways. The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like `Redis `{.interpreted-text role="ref"} or `SQS `{.interpreted-text role="ref"}, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., `Restoring 1 unacknowledged message(s)`) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely. After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism. Relevant Issues: [#9213](https://github.com/celery/celery/pull/9213), [#9231](https://github.com/celery/celery/pull/9231), [#9238](https://github.com/celery/celery/pull/9238) - New `documentation `{.interpreted-text role="ref"} for each shutdown type. - New `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} configuration option. - New `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} configuration option. #### REMAP_SIGTERM The `REMAP_SIGTERM` \"hidden feature\" has been tested, `documented `{.interpreted-text role="ref"} and is now officially supported. This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using `TERM`{.interpreted-text role="sig"} instead of `QUIT`{.interpreted-text role="sig"}. #### Pydantic Support This release introduces support for Pydantic models in Celery tasks. For more info, see the new pydantic example and PR [#9023](https://github.com/celery/celery/pull/9023) by \@mathiasertl. After upgrading to this version, please share your feedback on the new Pydantic support. #### Redis Broker Stability Improvements The root cause of the Redis broker instability issue has been [identified and resolved](https://github.com/celery/kombu/pull/2007) in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements. After upgrading to this version, please share your feedback on the Redis broker stability. Relevant Issues: [#7276](https://github.com/celery/celery/discussions/7276), [#8091](https://github.com/celery/celery/discussions/8091), [#8030](https://github.com/celery/celery/discussions/8030), [#8384](https://github.com/celery/celery/discussions/8384) #### Quorum Queues Initial Support This release introduces the initial support for Quorum Queues with Celery. See new configuration options for more details: - `task_default_queue_type`{.interpreted-text role="setting"} - `worker_detect_quorum_queues`{.interpreted-text role="setting"} After upgrading to this version, please share your feedback on the Quorum Queues support. Relevant Issues: [#6067](https://github.com/celery/celery/discussions/6067), [#9121](https://github.com/celery/celery/discussions/9121) ### What\'s Changed - Added Blacksmith.sh to the Sponsors section in the README (#9323) - Revert \"Added Blacksmith.sh to the Sponsors section in the README\" (#9324) - Added Blacksmith.sh to the Sponsors section in the README (#9325) - Added missing \" " in README (#9326) - Use Blacksmith SVG logo (#9327) - Updated Blacksmith SVG logo (#9328) - Revert \"Updated Blacksmith SVG logo\" (#9329) - Update pymongo to 4.10.0 (#9330) - Update pymongo to 4.10.1 (#9332) - Update user guide to recommend delay_on_commit (#9333) - Pin pre-commit to latest version 4.0.0 (Python 3.9+) (#9334) - Update ephem to 4.1.6 (#9336) - Updated Blacksmith SVG logo (#9337) - Prepare for (pre) release: v5.5.0rc1 (#9341) ## 5.5.0b4 {#version-5.5.0b4} release-date : 2024-09-30 release-by : Tomer Nosrati Celery v5.5.0 Beta 4 is now available for testing. Please help us test this version and report any issues. ### Key Highlights #### Python 3.13 Initial Support This release introduces the initial support for Python 3.13 with Celery. After upgrading to this version, please share your feedback on the Python 3.13 support. ### Previous Pre-release Highlights #### Soft Shutdown The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited \"warm shutdown\" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting. The soft shutdown is disabled by default, and can be enabled by setting the new configuration option `worker_soft_shutdown_timeout`{.interpreted-text role="setting"}. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways. The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like `Redis `{.interpreted-text role="ref"} or `SQS `{.interpreted-text role="ref"}, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., `Restoring 1 unacknowledged message(s)`) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely. After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism. Relevant Issues: [#9213](https://github.com/celery/celery/pull/9213), [#9231](https://github.com/celery/celery/pull/9231), [#9238](https://github.com/celery/celery/pull/9238) - New `documentation `{.interpreted-text role="ref"} for each shutdown type. - New `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} configuration option. - New `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} configuration option. #### REMAP_SIGTERM The `REMAP_SIGTERM` \"hidden feature\" has been tested, `documented `{.interpreted-text role="ref"} and is now officially supported. This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using `TERM`{.interpreted-text role="sig"} instead of `QUIT`{.interpreted-text role="sig"}. #### Pydantic Support This release introduces support for Pydantic models in Celery tasks. For more info, see the new pydantic example and PR [#9023](https://github.com/celery/celery/pull/9023) by \@mathiasertl. After upgrading to this version, please share your feedback on the new Pydantic support. #### Redis Broker Stability Improvements The root cause of the Redis broker instability issue has been [identified and resolved](https://github.com/celery/kombu/pull/2007) in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements. After upgrading to this version, please share your feedback on the Redis broker stability. Relevant Issues: [#7276](https://github.com/celery/celery/discussions/7276), [#8091](https://github.com/celery/celery/discussions/8091), [#8030](https://github.com/celery/celery/discussions/8030), [#8384](https://github.com/celery/celery/discussions/8384) #### Quorum Queues Initial Support This release introduces the initial support for Quorum Queues with Celery. See new configuration options for more details: - `task_default_queue_type`{.interpreted-text role="setting"} - `worker_detect_quorum_queues`{.interpreted-text role="setting"} After upgrading to this version, please share your feedback on the Quorum Queues support. Relevant Issues: [#6067](https://github.com/celery/celery/discussions/6067), [#9121](https://github.com/celery/celery/discussions/9121) ### What\'s Changed - Correct the error description in exception message when validate soft_time_limit (#9246) - Update msgpack to 1.1.0 (#9249) - chore(utils/time.py): rename [\_is_ambigious]{.title-ref} -\> [\_is_ambiguous]{.title-ref} (#9248) - Reduced Smoke Tests to min/max supported python (3.8/3.12) (#9252) - Update pytest to 8.3.3 (#9253) - Update elasticsearch requirement from \<=8.15.0 to \<=8.15.1 (#9255) - Update mongodb without deprecated [\[srv\]]{.title-ref} extra requirement (#9258) - blacksmith.sh: Migrate workflows to Blacksmith (#9261) - Fixes #9119: inject dispatch_uid for retry-wrapped receivers (#9247) - Run all smoke tests CI jobs together (#9263) - Improve documentation on visibility timeout (#9264) - Bump pytest-celery to 1.1.2 (#9267) - Added missing \"app.conf.visibility_timeout\" in smoke tests (#9266) - Improved stability with t/smoke/tests/test_consumer.py (#9268) - Improved Redis container stability in the smoke tests (#9271) - Disabled EXHAUST_MEMORY tests in Smoke-tasks (#9272) - Marked xfail for test_reducing_prefetch_count with Redis - flaky test (#9273) - Fixed pypy unit tests random failures in the CI (#9275) - Fixed more pypy unit tests random failures in the CI (#9278) - Fix Redis container from aborting randomly (#9276) - Run Integration & Smoke CI tests together after unit tests pass (#9280) - Added \"loglevel verbose\" to Redis containers in smoke tests (#9282) - Fixed Redis error in the smoke tests: \"Possible SECURITY ATTACK detected\" (#9284) - Refactored the smoke tests github workflow (#9285) - Increased \--reruns 3-\>4 in smoke tests (#9286) - Improve stability of smoke tests (CI and Local) (#9287) - Fixed Smoke tests CI \"test-case\" labels (specific instead of general) (#9288) - Use assert_log_exists instead of wait_for_log in worker smoke tests (#9290) - Optimized t/smoke/tests/test_worker.py (#9291) - Enable smoke tests dockers check before each test starts (#9292) - Relaxed smoke tests flaky tests mechanism (#9293) - Updated quorum queue detection to handle multiple broker instances (#9294) - Non-lazy table creation for database backend (#9228) - Pin pymongo to latest version 4.9 (#9297) - Bump pymongo from 4.9 to 4.9.1 (#9298) - Bump Kombu to v5.4.2 (#9304) - Use rabbitmq:3 in stamping smoke tests (#9307) - Bump pytest-celery to 1.1.3 (#9308) - Added Python 3.13 Support (#9309) - Add log when global qos is disabled (#9296) - Added official release docs (whatsnew) for v5.5 (#9312) - Enable Codespell autofix (#9313) - Pydantic typehints: Fix optional, allow generics (#9319) - Prepare for (pre) release: v5.5.0b4 (#9322) ## 5.5.0b3 {#version-5.5.0b3} release-date : 2024-09-08 release-by : Tomer Nosrati Celery v5.5.0 Beta 3 is now available for testing. Please help us test this version and report any issues. ### Key Highlights #### Soft Shutdown The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited \"warm shutdown\" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting. The soft shutdown is disabled by default, and can be enabled by setting the new configuration option `worker_soft_shutdown_timeout`{.interpreted-text role="setting"}. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways. The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like `Redis `{.interpreted-text role="ref"} or `SQS `{.interpreted-text role="ref"}, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., `Restoring 1 unacknowledged message(s)`) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely. After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism. Relevant Issues: [#9213](https://github.com/celery/celery/pull/9213), [#9231](https://github.com/celery/celery/pull/9231), [#9238](https://github.com/celery/celery/pull/9238) - New `documentation `{.interpreted-text role="ref"} for each shutdown type. - New `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} configuration option. - New `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} configuration option. #### REMAP_SIGTERM The `REMAP_SIGTERM` \"hidden feature\" has been tested, `documented `{.interpreted-text role="ref"} and is now officially supported. This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using `TERM`{.interpreted-text role="sig"} instead of `QUIT`{.interpreted-text role="sig"}. ### Previous Pre-release Highlights #### Pydantic Support This release introduces support for Pydantic models in Celery tasks. For more info, see the new pydantic example and PR [#9023](https://github.com/celery/celery/pull/9023) by \@mathiasertl. After upgrading to this version, please share your feedback on the new Pydantic support. #### Redis Broker Stability Improvements The root cause of the Redis broker instability issue has been [identified and resolved](https://github.com/celery/kombu/pull/2007) in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements. After upgrading to this version, please share your feedback on the Redis broker stability. Relevant Issues: [#7276](https://github.com/celery/celery/discussions/7276), [#8091](https://github.com/celery/celery/discussions/8091), [#8030](https://github.com/celery/celery/discussions/8030), [#8384](https://github.com/celery/celery/discussions/8384) #### Quorum Queues Initial Support This release introduces the initial support for Quorum Queues with Celery. See new configuration options for more details: - `task_default_queue_type`{.interpreted-text role="setting"} - `worker_detect_quorum_queues`{.interpreted-text role="setting"} After upgrading to this version, please share your feedback on the Quorum Queues support. Relevant Issues: [#6067](https://github.com/celery/celery/discussions/6067), [#9121](https://github.com/celery/celery/discussions/9121) ### What\'s Changed - Added SQS (localstack) broker to canvas smoke tests (#9179) - Pin elastic-transport to \<= latest version 8.15.0 (#9182) - Update elasticsearch requirement from \<=8.14.0 to \<=8.15.0 (#9186) - Improve formatting (#9188) - Add basic helm chart for celery (#9181) - Update kafka.rst (#9194) - Update pytest-order to 1.3.0 (#9198) - Update mypy to 1.11.2 (#9206) - All added to routes (#9204) - Fix typos discovered by codespell (#9212) - Use tzdata extras with zoneinfo backports (#8286) - Use [docker compose]{.title-ref} in Contributing\'s doc build section (#9219) - Failing test for issue #9119 (#9215) - Fix date_done timezone issue (#8385) - CI Fixes to smoke tests (#9223) - Fix: passes current request context when pushing to request_stack (#9208) - Fix broken link in the Using RabbitMQ docs page (#9226) - Added Soft Shutdown Mechanism (#9213) - Added worker_enable_soft_shutdown_on_idle (#9231) - Bump cryptography from 43.0.0 to 43.0.1 (#9233) - Added docs regarding the relevancy of soft shutdown and ETA tasks (#9238) - Show broker_connection_retry_on_startup warning only if it evaluates as False (#9227) - Fixed docker-docs CI failure (#9240) - Added docker cleanup auto-fixture to improve smoke tests stability (#9243) - print is not thread-safe, so should not be used in signal handler (#9222) - Prepare for (pre) release: v5.5.0b3 (#9244) ## 5.5.0b2 {#version-5.5.0b2} release-date : 2024-08-06 release-by : Tomer Nosrati Celery v5.5.0 Beta 2 is now available for testing. Please help us test this version and report any issues. ### Key Highlights #### Pydantic Support This release introduces support for Pydantic models in Celery tasks. For more info, see the new pydantic example and PR [#9023](https://github.com/celery/celery/pull/9023) by \@mathiasertl. After upgrading to this version, please share your feedback on the new Pydantic support. ### Previous Beta Highlights #### Redis Broker Stability Improvements The root cause of the Redis broker instability issue has been [identified and resolved](https://github.com/celery/kombu/pull/2007) in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements. After upgrading to this version, please share your feedback on the Redis broker stability. Relevant Issues: [#7276](https://github.com/celery/celery/discussions/7276), [#8091](https://github.com/celery/celery/discussions/8091), [#8030](https://github.com/celery/celery/discussions/8030), [#8384](https://github.com/celery/celery/discussions/8384) #### Quorum Queues Initial Support This release introduces the initial support for Quorum Queues with Celery. See new configuration options for more details: - `task_default_queue_type`{.interpreted-text role="setting"} - `worker_detect_quorum_queues`{.interpreted-text role="setting"} After upgrading to this version, please share your feedback on the Quorum Queues support. Relevant Issues: [#6067](https://github.com/celery/celery/discussions/6067), [#9121](https://github.com/celery/celery/discussions/9121) ### What\'s Changed - Bump pytest from 8.3.1 to 8.3.2 (#9153) - Remove setuptools deprecated test command from setup.py (#9159) - Pin pre-commit to latest version 3.8.0 from Python 3.9 (#9156) - Bump mypy from 1.11.0 to 1.11.1 (#9164) - Change \"docker-compose\" to \"docker compose\" in Makefile (#9169) - update python versions and docker compose (#9171) - Add support for Pydantic model validation/serialization (fixes #8751) (#9023) - Allow local dynamodb to be installed on another host than localhost (#8965) - Terminate job implementation for gevent concurrency backend (#9083) - Bump Kombu to v5.4.0 (#9177) - Add check for soft_time_limit and time_limit values (#9173) - Prepare for (pre) release: v5.5.0b2 (#9178) ## 5.5.0b1 {#version-5.5.0b1} release-date : 2024-07-24 release-by : Tomer Nosrati Celery v5.5.0 Beta 1 is now available for testing. Please help us test this version and report any issues. ### Key Highlights #### Redis Broker Stability Improvements The root cause of the Redis broker instability issue has been [identified and resolved](https://github.com/celery/kombu/pull/2007) in the release-candidate for Kombu v5.4.0. This beta release has been upgraded to use the new Kombu RC version, which should resolve the disconnections bug and offer additional improvements. After upgrading to this version, please share your feedback on the Redis broker stability. Relevant Issues: [#7276](https://github.com/celery/celery/discussions/7276), [#8091](https://github.com/celery/celery/discussions/8091), [#8030](https://github.com/celery/celery/discussions/8030), [#8384](https://github.com/celery/celery/discussions/8384) #### Quorum Queues Initial Support This release introduces the initial support for Quorum Queues with Celery. See new configuration options for more details: - `task_default_queue_type`{.interpreted-text role="setting"} - `worker_detect_quorum_queues`{.interpreted-text role="setting"} After upgrading to this version, please share your feedback on the Quorum Queues support. Relevant Issues: [#6067](https://github.com/celery/celery/discussions/6067), [#9121](https://github.com/celery/celery/discussions/9121) ### What\'s Changed - (docs): use correct version celery v.5.4.x (#8975) - Update mypy to 1.10.0 (#8977) - Limit pymongo\<4.7 when Python \<= 3.10 due to breaking changes in 4.7 (#8988) - Bump pytest from 8.1.1 to 8.2.0 (#8987) - Update README to Include FastAPI in Framework Integration Section (#8978) - Clarify return values of \...\_on_commit methods (#8984) - add kafka broker docs (#8935) - Limit pymongo\<4.7 regardless of Python version (#8999) - Update pymongo\[srv\] requirement from \<4.7,\>=4.0.2 to \>=4.0.2,\<4.8 (#9000) - Update elasticsearch requirement from \<=8.13.0 to \<=8.13.1 (#9004) - security: SecureSerializer: support generic low-level serializers (#8982) - don\'t kill if pid same as file (#8997) (#8998) - Update cryptography to 42.0.6 (#9005) - Bump cryptography from 42.0.6 to 42.0.7 (#9009) - Added -vv to unit, integration and smoke tests (#9014) - SecuritySerializer: ensure pack separator will not be conflicted with serialized fields (#9010) - Update sphinx-click to 5.2.2 (#9025) - Bump sphinx-click from 5.2.2 to 6.0.0 (#9029) - Fix a typo to display the help message in first-steps-with-django (#9036) - Pinned requests to v2.31.0 due to docker-py bug #3256 (#9039) - Fix certificate validity check (#9037) - Revert \"Pinned requests to v2.31.0 due to docker-py bug #3256\" (#9043) - Bump pytest from 8.2.0 to 8.2.1 (#9035) - Update elasticsearch requirement from \<=8.13.1 to \<=8.13.2 (#9045) - Fix detection of custom task set as class attribute with Django (#9038) - Update elastic-transport requirement from \<=8.13.0 to \<=8.13.1 (#9050) - Bump pycouchdb from 1.14.2 to 1.16.0 (#9052) - Update pytest to 8.2.2 (#9060) - Bump cryptography from 42.0.7 to 42.0.8 (#9061) - Update elasticsearch requirement from \<=8.13.2 to \<=8.14.0 (#9069) - \[enhance feature\] Crontab schedule: allow using month names (#9068) - Enhance tox environment: \[testenv:clean\] (#9072) - Clarify docs about Reserve one task at a time (#9073) - GCS docs fixes (#9075) - Use hub.remove_writer instead of hub.remove for write fds (#4185) (#9055) - Class method to process crontab string (#9079) - Fixed smoke tests env bug when using integration tasks that rely on Redis (#9090) - Bugfix - a task will run multiple times when chaining chains with groups (#9021) - Bump mypy from 1.10.0 to 1.10.1 (#9096) - Don\'t add a separator to global_keyprefix if it already has one (#9080) - Update pymongo\[srv\] requirement from \<4.8,\>=4.0.2 to \>=4.0.2,\<4.9 (#9111) - Added missing import in examples for Django (#9099) - Bump Kombu to v5.4.0rc1 (#9117) - Removed skipping Redis in t/smoke/tests/test_consumer.py tests (#9118) - Update pytest-subtests to 0.13.0 (#9120) - Increased smoke tests CI timeout (#9122) - Bump Kombu to v5.4.0rc2 (#9127) - Update zstandard to 0.23.0 (#9129) - Update pytest-subtests to 0.13.1 (#9130) - Changed retry to tenacity in smoke tests (#9133) - Bump mypy from 1.10.1 to 1.11.0 (#9135) - Update cryptography to 43.0.0 (#9138) - Update pytest to 8.3.1 (#9137) - Added support for Quorum Queues (#9121) - Bump Kombu to v5.4.0rc3 (#9139) - Cleanup in Changelog.rst (#9141) - Update Django docs for CELERY_CACHE_BACKEND (#9143) - Added missing docs to previous releases (#9144) - Fixed a few documentation build warnings (#9145) - docs(README): link invalid (#9148) - Prepare for (pre) release: v5.5.0b1 (#9146) --- # Change history {#changelog-5.6} This document contains change notes for bugfix & new features in the main branch & 5.6.x series, please see `whatsnew-5.6`{.interpreted-text role="ref"} for an overview of what\'s new in Celery 5.6. ## 5.6.1 {#version-5.6.1} release-date : 2025-12-29 release-by : Tomer Nosrati ### What\'s Changed - Fix Redis Sentinel ACL authentication support (#10013) - Fix: Broker heartbeats not sent during graceful shutdown (#9986) - docs #5410 \-- Document confirm_publish broker transport option (#10016) - close DB pools only in prefork mode (#10020) - Fix: Avoid unnecessary Django database connection creation during cleanup (#10015) - reliable prefork detection (#10023) - better coverage (#10029) - Docs: clarify [result_extended]{.title-ref} vs periodic task metadata and show [headers\[\"periodic_task_name\"\]]{.title-ref} example (#10030) - Stop importing pytest_subtests (#10032) - Only use exceptiongroup backport for Python \< 3.11 (#10033) - Prepare for release: v5.6.1 (#10037) ## 5.6.0 {#version-5.6.0} release-date : 2025-11-30 release-by : Tomer Nosrati Celery v5.6.0 is now available. ### Key Highlights See `whatsnew-5.6`{.interpreted-text role="ref"} for a complete overview or read the main highlights below. #### Python 3.9 Minimum Version Celery 5.6.0 drops support for Python 3.8 (EOL). The minimum required Python version is now 3.9. Users still on Python 3.8 must upgrade their Python version before upgrading to Celery 5.6.0. Additionally, this release includes initial support for Python 3.14. #### SQS: Reverted to `pycurl` from `urllib3` The switch from `pycurl` to `urllib3` for the SQS transport (introduced in Celery 5.5.0 via Kombu) has been reverted due to critical issues affecting SQS users: - Processing throughput dropped from \~100 tasks/sec to \~3/sec in some environments - `UnknownOperationException` errors causing container crash loops - Silent message processing failures with no error logs Users of the SQS transport must ensure `pycurl` is installed. If you removed `pycurl` after upgrading to Celery 5.5.0, you will need to reinstall it. Contributed by [\@auvipy](https://github.com/auvipy) in [#9620](https://github.com/celery/celery/pull/9620). #### Security Fix: Broker Credential Leak Prevention Fixed a security issue where broker URLs containing passwords were being logged in plaintext by the delayed delivery mechanism. Broker credentials are now properly sanitized in all log output. Contributed by [\@giancarloromeo](https://github.com/giancarloromeo) in [#9997](https://github.com/celery/celery/pull/9997). #### Memory Leak Fixes Two significant memory leaks have been fixed in this release: **Exception Handling Memory Leak**: Fixed a critical memory leak in task exception handling that was particularly severe on Python 3.11+ due to enhanced traceback data. The fix properly breaks reference cycles in tracebacks to allow garbage collection. Contributed by [\@jaiganeshs21](https://github.com/jaiganeshs21) in [#9799](https://github.com/celery/celery/pull/9799). **Pending Result Memory Leak**: Fixed a memory leak where `AsyncResult` subscriptions were not being cleaned up when results were forgotten. Contributed by [\@tsoos99dev](https://github.com/tsoos99dev) in [#9806](https://github.com/celery/celery/pull/9806). #### ETA Task Memory Limit New configuration option `worker_eta_task_limit`{.interpreted-text role="setting"} to prevent out-of-memory crashes when workers fetch large numbers of ETA or countdown tasks. Previously, workers could exhaust available memory when the broker contained many scheduled tasks. Example usage: ``` python app.conf.worker_eta_task_limit = 1000 ``` Contributed by [\@sashu2310](https://github.com/sashu2310) in [#9853](https://github.com/celery/celery/pull/9853). #### Queue Type Selection for Auto-created Queues New configuration options allow specifying the queue type and exchange type when Celery auto-creates missing queues. This is particularly useful for RabbitMQ users who want to use quorum queues with auto-created queues. Configuration options: - `task_create_missing_queue_type`{.interpreted-text role="setting"}: Sets the queue type for auto-created queues (e.g., `quorum`, `classic`) - `task_create_missing_queue_exchange_type`{.interpreted-text role="setting"}: Sets the exchange type for auto-created queues Example usage: ``` python app.conf.task_create_missing_queue_type = 'quorum' ``` Contributed by [\@ghirailghiro](https://github.com/ghirailghiro) in [#9815](https://github.com/celery/celery/pull/9815). ### What\'s Changed - Prepare for release: v5.6.0 (#10010) ## 5.6.0rc2 {#version-5.6.0rc2} release-date : 2025-11-22 release-by : Tomer Nosrati Celery v5.6.0 Release Candidate 2 is now available for testing. Please help us test this version and report any issues. ### What\'s Changed - Remove Python 4.0 version condition for pytest dependencies (#9993) - Sanitize broker URL in delayed delivery logs (avoid leaking credentials) (#9997) - Don\'t fail task on timeout during cold shutdown (#9678) - Add Py39-314t to CI (#9999) - asynpool: Don\'t return from inside a finally block (#10000) - Prepare for (pre) release: v5.6.0rc2 (#10005) ## 5.6.0rc1 {#version-5.6.0rc1} release-date : 2025-11-02 release-by : Tomer Nosrati Celery v5.6.0 Release Candidate 1 is now available for testing. Please help us test this version and report any issues. ### What\'s Changed - Add support for Django Connection pool (#9953) - Pin tblib to ==3.1.0 (#9967) - fix(worker): continue to attempt to bind other queues after a native delayed delivery binding failure has occurred (#9959) - Handle UnpicklingError in persistent scheduler initialization (#9952) - Bug Fix: Nested Chords Fail When Using django-celery-results with a Redis Backend (#9950) - Add support pymongo 4.12 (#9665) - Make tests compatible with pymongo \>= 4.14 (#9968) - tblib updated from 3.1.0 to 3.2.0 (#9970) - Fix remaining function typing and docstring (#9971) - Fix regex pattern in version parsing and remove duplicate entry in \_\_all\_\_ (#9978) - Bump Kombu to v5.6.0 and removed \<5.7 limit on kombu (#9981) - Prepare for (pre) release: v5.6.0rc1 (#9982) ## 5.6.0b2 {#version-5.6.0b2} release-date : 2025-10-20 release-by : Tomer Nosrati Celery v5.6.0 Beta 2 is now available for testing. Please help us test this version and report any issues. ### What\'s Changed - GitHub Actions: Test on Python 3.14 release candidate 2 (#9891) - Update pypy to python 3.11 (#9896) - Feature: Add support credential_provider to Redis Backend (#9879) - Celery.timezone: try tzlocal.get_localzone() before using LocalTimezone (#9862) - Run integration tests on Python 3.14 (#9903) - Fix arithmetic overflow for MSSQL result backend (#9904) - Add documentation for task_id param for apply_async function (#9906) - Support redis client name (#9900) - Bump Kombu to v5.6.0rc1 (#9918) - Fix broker connection retry attempt counter in the error log (#9911) - fix: restrict disable-prefetch feature to Redis brokers only (#9919) - fix(): preserve group order in replaced signature (#9910) - Remove Python 3.8 from CI workflow (#9930) - Update default Python versions in integration tests (#9931) - Update tox.ini to remove Python 3.8 (#9932) - Remove Python 3.8 from Dockerfile (#9933) - Update Python version requirement to 3.9 (#9935) - Update pypy version from 3.10 to 3.11 in Dockerfile (#9934) - Flake8 fixes (#9955) - Remove test-pypy3.txt from Dockerfile dependencies (#9939) - Remove backports.zoneinfo for Python 3.9 compatibility (#9956) - Update pytest-cov version for Python compatibility (#9957) - Update pytest-rerunfailures and pre-commit versions (#9958) - Prepare for (pre) release: v5.6.0b2 (#9938) ## 5.6.0b1 {#version-5.6.0b1} release-date : 2025-09-15 release-by : Tomer Nosrati Celery v5.6.0 Beta 1 is now available for testing. Please help us test this version and report any issues. ### What\'s Changed - docs: mention of json serializer recursive reference message size blowup (#5000) (#9743) - docs: typo in canvas.rst (#9744) - Makes \_on_retry return a float as required to be used as errback on retry_over_time (#9741) - Update canvas.rst doc calculation order for callback (#9758) - Updated Blacksmith logo (#9763) - Made the Sponsors logos link to their website (#9764) - add missing cloudamqp logo (#9767) - Improve sponsor visibility (#9768) - fix: (#9773) task_id must not be empty with chain as body of a chord (#9774) - Update setup.py to fix deprecation warning (#9771) - Adds integration test for chord_unlock bug when routed to quorum/topic queue (#9766) - Add xfail test for default queue/exchange fallback ignoring [task_default]()\* settings (#9765) - Add xfail test for RabbitMQ quorum queue global QoS race condition (#9770) - fix: (#8786) time out when chord header fails with group body (#9788) - Fix #9738 : Add root_id and parent_id to .apply() (#9784) - Replace DelayedDelivery connection creation to use context manger (#9793) - Fix #9794: Pydantic integration fails with \_\_future\_\_.annotations. (#9795) - add go and rust implementation in docs (#9800) - Fix memory leak in exception handling (Issue #8882) (#9799) - Fix handlers docs (Issue #9787) (#9804) - Remove importlib_metadata leftovers (#9791) - Update timeout minutes for smoke tests CI (#9807) - Revert \"Remove dependency on [pycurl]{.title-ref}\" (#9620) - Add Blacksmith Docker layer caching to all Docker builds (#9840) - Bump Kombu to v5.6.0b1 (#9839) - Disable pytest-xdist for smoke tests and increase retries (CI ONLY) (#9842) - Fix Python 3.13 compatibility in events dumper (#9826) - Dockerfile Build Optimizations (#9733) - Migrated from to in the CI (#9846) - Remove incorrect example (#9854) - Revert \"Use Django DB max age connection setting\" (#9824) - Fix pending_result memory leak (#9806) - Update python-package.yml (#9856) - Bump Kombu to v5.6.0b2 (#9858) - Refactor integration and smoke tests CI (#9855) - Fix [AsyncResult.forget()]{.title-ref} with couchdb backend method raises [TypeError: a bytes-like object is required, not \'str\']{.title-ref} (#9865) - Improve Docs for SQS Authentication (#9868) - Added [.github/copilot-instructions.md]{.title-ref} for GitHub Copilot (#9874) - misc: credit removal (#9877) - Choose queue type and exchange type when creating missing queues (fix #9671) (#9815) - fix: prevent celery from hanging due to spawned greenlet errors in greenlet drainers (#9371) - Feature/disable prefetch fixes (#9863) - Add worker_eta_task_limit configuration to manage ETA task memory usage (#9853) - Update runner version in Docker workflow (#9884) - Prepare for (pre) release: v5.6.0b1 (#9890) --- # History This section contains historical change histories, for the latest version please visit `changelog`{.interpreted-text role="ref"}. Release : Date : ::: {.toctree maxdepth="2"} whatsnew-5.6 changelog-5.6 whatsnew-5.5 changelog-5.5 whatsnew-5.4 changelog-5.4 whatsnew-5.3 changelog-5.3 whatsnew-5.1 changelog-5.1 whatsnew-5.0 changelog-5.0 whatsnew-4.4 changelog-4.4 whatsnew-4.3 changelog-4.3 whatsnew-4.2 changelog-4.2 whatsnew-4.1 changelog-4.1 whatsnew-4.0 changelog-4.0 whatsnew-3.1 changelog-3.1 whatsnew-3.0 changelog-3.0 whatsnew-2.5 changelog-2.5 changelog-2.4 changelog-2.3 changelog-2.2 changelog-2.1 changelog-2.0 changelog-1.0 ::: --- # What\'s new in Celery 2.5 {#whatsnew-2.5} Celery aims to be a flexible and reliable, best-of-breed solution to process vast amounts of messages in a distributed fashion, while providing operations with the tools to maintain such a system. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. To read more about Celery you should visit our [website](http://celeryproject.org/). While this version is backward compatible with previous versions it\'s important that you read the following section. If you use Celery in combination with Django you must also read the [django-celery changelog \]{.title-ref} and upgrade to `django-celery 2.5 `{.interpreted-text role="pypi"}. This version is officially supported on CPython 2.5, 2.6, 2.7, 3.2 and 3.3, as well as PyPy and Jython. ::: {.contents local=""} ::: ## Important Notes {#v250-important} ### Broker connection pool now enabled by default The default limit is 10 connections, if you have many threads/green-threads using connections at the same time you may want to tweak this limit to avoid contention. See the `BROKER_POOL_LIMIT`{.interpreted-text role="setting"} setting for more information. Also note that publishing tasks will be retried by default, to change this default or the default retry policy see `CELERY_TASK_PUBLISH_RETRY`{.interpreted-text role="setting"} and `CELERY_TASK_PUBLISH_RETRY_POLICY`{.interpreted-text role="setting"}. ### Rabbit Result Backend: Exchange is no longer *auto delete* The exchange used for results in the Rabbit (AMQP) result backend used to have the *auto_delete* flag set, which could result in a race condition leading to an annoying warning. ::: admonition For RabbitMQ users Old exchanges created with the *auto_delete* flag enabled has to be removed. The `camqadm`{.interpreted-text role="program"} command can be used to delete the previous exchange: ``` console $ camqadm exchange.delete celeryresults ``` As an alternative to deleting the old exchange you can configure a new name for the exchange: CELERY_RESULT_EXCHANGE = 'celeryresults2' But you have to make sure that all clients and workers use this new setting, so they\'re updated to use the same exchange name. ::: ### Solution for hanging workers (but must be manually enabled) The [CELERYD_FORCE_EXECV]{.title-ref} setting has been added to solve a problem with deadlocks that originate when threads and fork is mixed together: ``` python CELERYD_FORCE_EXECV = True ``` This setting is recommended for all users using the prefork pool, but especially users also using time limits or a max tasks per child setting. - See [Python Issue 6721](http://bugs.python.org/issue6721#msg140215) to read more about this issue, and why resorting to `~os.execv`{.interpreted-text role="func"}\` is the only safe solution. Enabling this option will result in a slight performance penalty when new child worker processes are started, and it will also increase memory usage (but many platforms are optimized, so the impact may be minimal). Considering that it ensures reliability when replacing lost worker processes, it should be worth it. - It\'s already the default behavior on Windows. - It will be the default behavior for all platforms in a future version. ## Optimization {#v250-optimizations} - The code path used when the worker executes a task has been heavily optimized, meaning the worker is able to process a great deal more tasks/second compared to previous versions. As an example the solo pool can now process up to 15000 tasks/second on a 4 core MacBook Pro when using the `pylibrabbitmq`{.interpreted-text role="pypi"} transport, where it previously could only do 5000 tasks/second. - The task error tracebacks are now much shorter. - Fixed a noticeable delay in task processing when rate limits are enabled. ## Deprecation Time-line Changes {#v250-deprecations} ### Removals - The old `TaskSet`{.interpreted-text role="class"} signature of `(task_name, list_of_tasks)` can no longer be used (originally scheduled for removal in 2.4). The deprecated `.task_name` and `.task` attributes has also been removed. - The functions `celery.execute.delay_task`, `celery.execute.apply`, and `celery.execute.apply_async` has been removed (originally) scheduled for removal in 2.3). - The built-in `ping` task has been removed (originally scheduled for removal in 2.3). Please use the ping broadcast command instead. - It\'s no longer possible to import `subtask` and `TaskSet` from `celery.task.base`{.interpreted-text role="mod"}, please import them from `celery.task`{.interpreted-text role="mod"} instead (originally scheduled for removal in 2.4). ### Deprecated modules - The `celery.decorators`{.interpreted-text role="mod"} module has changed status from pending deprecation to deprecated, and is scheduled for removal in version 4.0. The `celery.task` module must be used instead. ## News {#v250-news} ### Timezone support Celery can now be configured to treat all incoming and outgoing dates as UTC, and the local timezone can be configured. This isn\'t yet enabled by default, since enabling time zone support means workers running versions pre-2.5 will be out of sync with upgraded workers. To enable UTC you have to set `CELERY_ENABLE_UTC`{.interpreted-text role="setting"}: CELERY_ENABLE_UTC = True When UTC is enabled, dates and times in task messages will be converted to UTC, and then converted back to the local timezone when received by a worker. You can change the local timezone using the `CELERY_TIMEZONE`{.interpreted-text role="setting"} setting. Installing the `pytz`{.interpreted-text role="pypi"} library is recommended when using a custom timezone, to keep timezone definition up-to-date, but it will fallback to a system definition of the timezone if available. UTC will enabled by default in version 3.0. ::: note ::: title Note ::: `django-celery`{.interpreted-text role="pypi"} will use the local timezone as specified by the `TIME_ZONE` setting, it will also honor the new [USE_TZ](https://docs.djangoproject.com/en/dev/topics/i18n/timezones/) setting introduced in Django 1.4. ::: ### New security serializer using cryptographic signing A new serializer has been added that signs and verifies the signature of messages. The name of the new serializer is `auth`, and needs additional configuration to work (see `conf-security`{.interpreted-text role="ref"}). ::: seealso `guide-security`{.interpreted-text role="ref"} ::: Contributed by Mher Movsisyan. ### New `CELERY_ANNOTATIONS`{.interpreted-text role="setting"} setting This new setting enables the configuration to modify task classes and their attributes. The setting can be a dict, or a list of annotation objects that filter for tasks and return a map of attributes to change. As an example, this is an annotation to change the `rate_limit` attribute for the `tasks.add` task: ``` python CELERY_ANNOTATIONS = {'tasks.add': {'rate_limit': '10/s'}} ``` or change the same for all tasks: ``` python CELERY_ANNOTATIONS = {'*': {'rate_limit': '10/s'}} ``` You can change methods too, for example the `on_failure` handler: ``` python def my_on_failure(self, exc, task_id, args, kwargs, einfo): print('Oh no! Task failed: %r' % (exc,)) CELERY_ANNOTATIONS = {'*': {'on_failure': my_on_failure}} ``` If you need more flexibility then you can also create objects that filter for tasks to annotate: ``` python class MyAnnotate(object): def annotate(self, task): if task.name.startswith('tasks.'): return {'rate_limit': '10/s'} CELERY_ANNOTATIONS = (MyAnnotate(), {other_annotations,}) ``` ### `current` provides the currently executing task The new `celery.task.current`{.interpreted-text role="data"} proxy will always give the currently executing task. **Example**: ``` python from celery.task import current, task @task def update_twitter_status(auth, message): twitter = Twitter(auth) try: twitter.update_status(message) except twitter.FailWhale, exc: # retry in 10 seconds. current.retry(countdown=10, exc=exc) ``` Previously you\'d\'ve to type `update_twitter_status.retry(…)` here, which can be annoying for long task names. ::: note ::: title Note ::: This won\'t work if the task function is called directly (i.e., `update_twitter_status(a, b)`). For that to work `apply` must be used: `update_twitter_status.apply((a, b))`. ::: ### In Other News - Now depends on Kombu 2.1.0. - Efficient Chord support for the Memcached backend (Issue #533) > This means Memcached joins Redis in the ability to do non-polling > chords. > > Contributed by Dan McGee. - Adds Chord support for the Rabbit result backend (amqp) > The Rabbit result backend can now use the fallback chord solution. - Sending `QUIT`{.interpreted-text role="sig"} to `celeryd` will now cause it cold terminate. > That is, it won\'t finish executing the tasks it\'s currently > working on. > > Contributed by Alec Clowes. - New \"detailed\" mode for the Cassandra backend. > Allows to have a \"detailed\" mode for the Cassandra backend. > Basically the idea is to keep all states using Cassandra wide > columns. New states are then appended to the row as new columns, > the last state being the last column. > > See the `CASSANDRA_DETAILED_MODE`{.interpreted-text > role="setting"} setting. > > Contributed by Steeve Morin. - The Crontab parser now matches Vixie Cron behavior when parsing ranges with steps (e.g., 1-59/2). > Contributed by Daniel Hepper. - `celerybeat` can now be configured on the command-line like `celeryd`. Additional configuration must be added at the end of the argument list followed by `--`, for example: ``` console $ celerybeat -l info -- celerybeat.max_loop_interval=10.0 ``` - Now limits the number of frames in a traceback so that `celeryd` doesn\'t crash on maximum recursion limit exceeded exceptions (Issue #615). > The limit is set to the current recursion limit divided by 8 > (which is 125 by default). > > To get or set the current recursion limit use > `sys.getrecursionlimit`{.interpreted-text role="func"} and > `sys.setrecursionlimit`{.interpreted-text role="func"}. - More information is now preserved in the pickleable traceback. > This has been added so that Sentry can show more details. > > Contributed by Sean O\'Connor. - CentOS init-script has been updated and should be more flexible. > Contributed by Andrew McFague. - MongoDB result backend now supports `forget()`. > Contributed by Andrew McFague - `task.retry()` now re-raises the original exception keeping the original stack trace. > Suggested by `ojii`{.interpreted-text role="github_user"}. - The [\--uid]{.title-ref} argument to daemons now uses `initgroups()` to set groups to all the groups the user is a member of. > Contributed by Łukasz Oleś. - `celeryctl`: Added `shell` command. > The shell will have the current_app (`celery`) and all tasks > automatically added to locals. - `celeryctl`: Added `migrate` command. > The migrate command moves all tasks from one broker to another. > Note that this is experimental and you should have a backup of the > data before proceeding. > > **Examples**: > > ``` console > $ celeryctl migrate redis://localhost amqp://localhost > $ celeryctl migrate amqp://localhost//v1 amqp://localhost//v2 > $ python manage.py celeryctl migrate django:// redis:// > ``` - Routers can now override the `exchange` and `routing_key` used to create missing queues (Issue #577). > By default this will always use the name of the queue, but you can > now have a router return exchange and routing_key keys to set > them. > > This is useful when using routing classes which decides a > destination at run-time. > > Contributed by Akira Matsuzaki. - Redis result backend: Adds support for a `max_connections` parameter. > It\'s now possible to configure the maximum number of simultaneous > connections in the Redis connection pool used for results. > > The default max connections setting can be configured using the > `CELERY_REDIS_MAX_CONNECTIONS`{.interpreted-text role="setting"} > setting, or it can be changed individually by > `RedisBackend(max_connections=int)`. > > Contributed by Steeve Morin. - Redis result backend: Adds the ability to wait for results without polling. > Contributed by Steeve Morin. - MongoDB result backend: Now supports save and restore `taskset`. > Contributed by Julien Poissonnier. - There\'s a new `guide-security`{.interpreted-text role="ref"} guide in the documentation. - The init-scripts have been updated, and many bugs fixed. > Contributed by Chris Streeter. - User (tilde) is now expanded in command-line arguments. - Can now configure `CELERYCTL`{.interpreted-text role="envvar"} environment variable in `/etc/default/celeryd`{.interpreted-text role="file"}. > While not necessary for operation, `celeryctl`{.interpreted-text > role="program"} is used for the `celeryd status` command, and the > path to `celeryctl`{.interpreted-text role="program"} must be > configured for that to work. > > The daemonization cookbook contains examples. > > Contributed by Jude Nagurney. - The MongoDB result backend can now use Replica Sets. > Contributed by Ivan Metzlar. - gevent: Now supports autoscaling (Issue #599). > Contributed by Mark Lavin. - multiprocessing: Mediator thread is now always enabled, even though rate limits are disabled, as the pool semaphore is known to block the main thread, causing broadcast commands and shutdown to depend on the semaphore being released. ## Fixes - Exceptions that are re-raised with a new exception object now keeps the original stack trace. - Windows: Fixed the `no handlers found for multiprocessing` warning. - Windows: The `celeryd` program can now be used. > Previously Windows users had to launch `celeryd` using > `python -m celery.bin.celeryd`. - Redis result backend: Now uses `SETEX` command to set result key, and expiry atomically. > Suggested by `yaniv-aknin`{.interpreted-text role="github_user"}. - `celeryd`: Fixed a problem where shutdown hanged when `Control-c`{.interpreted-text role="kbd"} was used to terminate. - `celeryd`: No longer crashes when channel errors occur. > Fix contributed by Roger Hu. - Fixed memory leak in the eventlet pool, caused by the use of `greenlet.getcurrent`. > Fix contributed by Ignas Mikalajūnas. - Cassandra backend: No longer uses `pycassa.connect`{.interpreted-text role="func"} which is deprecated since `pycassa`{.interpreted-text role="pypi"} 1.4. > Fix contributed by Jeff Terrace. - Fixed unicode decode errors that could occur while sending error emails. > Fix contributed by Seong Wun Mun. - `celery.bin` programs now always defines `__package__` as recommended by PEP-366. - `send_task` now emits a warning when used in combination with `CELERY_ALWAYS_EAGER`{.interpreted-text role="setting"} (Issue #581). > Contributed by Mher Movsisyan. - `apply_async` now forwards the original keyword arguments to `apply` when `CELERY_ALWAYS_EAGER`{.interpreted-text role="setting"} is enabled. - `celeryev` now tries to re-establish the connection if the connection to the broker is lost (Issue #574). - `celeryev`: Fixed a crash occurring if a task has no associated worker information. > Fix contributed by Matt Williamson. - The current date and time is now consistently taken from the current loaders `now` method. - Now shows helpful error message when given a configuration module ending in `.py` that can\'t be imported. - `celeryctl`: The `--expires `{.interpreted-text role="option"} and `--eta `{.interpreted-text role="option"} arguments to the apply command can now be an ISO-8601 formatted string. - `celeryctl` now exits with exit status `EX_UNAVAILABLE` (69) if no replies have been received. --- # What\'s new in Celery 3.0 (Chiastic Slide) {#whatsnew-3.0} Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. To read more about Celery you should go read the `introduction `{.interpreted-text role="ref"}. While this version is backward compatible with previous versions it\'s important that you read the following section. If you use Celery in combination with Django you must also read the [django-celery changelog](https://github.com/celery/django-celery/tree/master/Changelog) and upgrade to `django-celery 3.0 `{.interpreted-text role="pypi"}. This version is officially supported on CPython 2.5, 2.6, 2.7, 3.2 and 3.3, as well as PyPy and Jython. ## Highlights ::: topic **Overview** - A new and improved API, that\'s both simpler and more powerful. > Everyone must read the new `first-steps`{.interpreted-text > role="ref"} tutorial, and the new `next-steps`{.interpreted-text > role="ref"} tutorial. Oh, and why not reread the user guide while > you\'re at it :) > > There are no current plans to deprecate the old API, so you don\'t > have to be in a hurry to port your applications. - The worker is now thread-less, giving great performance improvements. - The new \"Canvas\" makes it easy to define complex work-flows. > Ever wanted to chain tasks together? This is possible, but not > just that, now you can even chain together groups and chords, or > even combine multiple chains. > > Read more in the `Canvas `{.interpreted-text > role="ref"} user guide. - All of Celery\'s command-line programs are now available from a single `celery`{.interpreted-text role="program"} umbrella command. - This is the last version to support Python 2.5. > Starting with Celery 3.1, Python 2.6 or later is required. - Support for the new `librabbitmq`{.interpreted-text role="pypi"} C client. > Celery will automatically use the `librabbitmq`{.interpreted-text > role="pypi"} module if installed, which is a very fast and > memory-optimized replacement for the `amqp`{.interpreted-text > role="pypi"} module. - Redis support is more reliable with improved ack emulation. - Celery now always uses UTC - Over 600 commits, 30k additions/36k deletions. > In comparison 1.0➝ 2.0 had 18k additions/8k deletions. ::: ::: {.contents local="" depth="2"} ::: ## Important Notes {#v300-important} ### Broadcast exchanges renamed The workers remote control command exchanges has been renamed (a new `pidbox`{.interpreted-text role="term"} name), this is because the `auto_delete` flag on the exchanges has been removed, and that makes it incompatible with earlier versions. You can manually delete the old exchanges if you want, using the `celery amqp`{.interpreted-text role="program"} command (previously called `camqadm`): ``` console $ celery amqp exchange.delete celeryd.pidbox $ celery amqp exchange.delete reply.celeryd.pidbox ``` ### Event-loop The worker is now running *without threads* when used with RabbitMQ (AMQP), or Redis as a broker, resulting in: - Much better overall performance. - Fixes several edge case race conditions. - Sub-millisecond timer precision. - Faster shutdown times. The transports supported are: `py-amqp` `librabbitmq`, `redis`, and `amqplib`. Hopefully this can be extended to include additional broker transports in the future. For increased reliability the `CELERY_FORCE_EXECV`{.interpreted-text role="setting"} setting is enabled by default if the event-loop isn\'t used. ### New `celery` umbrella command All Celery\'s command-line programs are now available from a single `celery`{.interpreted-text role="program"} umbrella command. You can see a list of sub-commands and options by running: ``` console $ celery help ``` Commands include: - `celery worker` (previously `celeryd`). - `celery beat` (previously `celerybeat`). - `celery amqp` (previously `camqadm`). The old programs are still available (`celeryd`, `celerybeat`, etc), but you\'re discouraged from using them. ### Now depends on `billiard`{.interpreted-text role="pypi"} Billiard is a fork of the multiprocessing containing the no-execv patch by `sbt` (), and also contains the pool improvements previously located in Celery. This fork was necessary as changes to the C extension code was required for the no-execv patch to work. - Issue #625 - Issue #627 - Issue #640 - [django-celery #122 \ Reducing the possibility of data loss. > > Acks are now implemented by storing a copy of the message when the > message is consumed. The copy isn\'t removed until the consumer > acknowledges or rejects it. > > This means that unacknowledged messages will be redelivered either > when the connection is closed, or when the visibility timeout is > exceeded. > > - Visibility timeout > > > This is a timeout for acks, so that if the consumer doesn\'t ack > > the message within this time limit, the message is redelivered > > to another consumer. > > > > The timeout is set to one hour by default, but can be changed by > > configuring a transport option: > > > > BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 18000} # 5 hours > > ::: note > ::: title > Note > ::: > > Messages that haven\'t been acked will be redelivered if the > visibility timeout is exceeded, for Celery users this means that > ETA/countdown tasks that are scheduled to execute with a time that > exceeds the visibility timeout will be executed twice (or more). If > you plan on using long ETA/countdowns you should tweak the visibility > timeout accordingly. > ::: > > Setting a long timeout means that it\'ll take a long time for messages > to be redelivered in the event of a power failure, but if so happens > you could temporarily set the visibility timeout lower to flush out > messages when you start up the systems again. ## News {#v300-news} ### Chaining Tasks Tasks can now have callbacks and errbacks, and dependencies are recorded - The task message format have been updated with two new extension keys > Both keys can be empty/undefined or a list of subtasks. > > - `callbacks` > > > Applied if the task exits successfully, with the result of > > the task as an argument. > > - `errbacks` > > > Applied if an error occurred while executing the task, with > > the uuid of the task as an argument. Since it may not be > > possible to serialize the exception instance, it passes the > > uuid of the task instead. The uuid can then be used to > > retrieve the exception and traceback of the task from the > > result backend. > > - `link` and `link_error` keyword arguments has been added to > `apply_async`. > > > These add callbacks and errbacks to the task, and you can > > read more about them at `calling-links`{.interpreted-text > > role="ref"}. > > - We now track what subtasks a task sends, and some result > backends supports retrieving this information. > > > - task.request.children > > > > > Contains the result instances of the subtasks the > > > currently executing task has applied. > > > > - AsyncResult.children > > > > > Returns the tasks dependencies, as a list of > > > `AsyncResult`/`ResultSet` instances. > > > > - AsyncResult.iterdeps > > > > > Recursively iterates over the tasks dependencies, > > > yielding [(parent, node)]{.title-ref} tuples. > > > > > > Raises IncompleteStream if any of the dependencies > > > hasn\'t returned yet. > > > > - AsyncResult.graph > > > > > A > > > `~celery.utils.graph.DependencyGraph`{.interpreted-text > > > role="class"} of the tasks dependencies. With this you > > > can also convert to dot format: > > > > > > ``` python > > > with open('graph.dot') as fh: > > > result.graph.to_dot(fh) > > > ``` > > > > > > then produce an image of the graph: > > > > > > ``` console > > > $ dot -Tpng graph.dot -o graph.png > > > ``` - A new special subtask called `chain` is also included: > ``` pycon > >>> from celery import chain > > # (2 + 2) * 8 / 2 > >>> res = chain(add.subtask((2, 2)), > mul.subtask((8,)), > div.subtask((2,))).apply_async() > >>> res.get() == 16 > > >>> res.parent.get() == 32 > > >>> res.parent.parent.get() == 4 > ``` - Adds `AsyncResult.get_leaf`{.interpreted-text role="meth"} > Waits and returns the result of the leaf subtask. That\'s the last > node found when traversing the graph, but this means that the > graph can be 1-dimensional only (in effect a list). - Adds `subtask.link(subtask)` + `subtask.link_error(subtask)` > Shortcut to `s.options.setdefault('link', []).append(subtask)` - Adds `subtask.flatten_links()` > Returns a flattened list of all dependencies (recursively) ### Redis: Priority support The message\'s `priority` field is now respected by the Redis transport by having multiple lists for each named queue. The queues are then consumed by in order of priority. The priority field is a number in the range of 0 - 9, where 0 is the default and highest priority. The priority range is collapsed into four steps by default, since it is unlikely that nine steps will yield more benefit than using four steps. The number of steps can be configured by setting the `priority_steps` transport option, which must be a list of numbers in **sorted order**: ``` pycon >>> BROKER_TRANSPORT_OPTIONS = { ... 'priority_steps': [0, 2, 4, 6, 8, 9], ... } ``` Priorities implemented in this way isn\'t as reliable as priorities on the server side, which is why the feature is nicknamed \"quasi-priorities\"; **Using routing is still the suggested way of ensuring quality of service**, as client implemented priorities fall short in a number of ways, for example if the worker is busy with long running tasks, has prefetched many messages, or the queues are congested. Still, it is possible that using priorities in combination with routing can be more beneficial than using routing or priorities alone. Experimentation and monitoring should be used to prove this. Contributed by Germán M. Bravo. ### Redis: Now cycles queues so that consuming is fair This ensures that a very busy queue won\'t block messages from other queues, and ensures that all queues have an equal chance of being consumed from. This used to be the case before, but the behavior was accidentally changed while switching to using blocking pop. ### [group]{.title-ref}/[chord]{.title-ref}/[chain]{.title-ref} are now subtasks - group is no longer an alias to `TaskSet`, but new all together, since it was very difficult to migrate the `TaskSet` class to become a subtask. - A new shortcut has been added to tasks: > ``` pycon > >>> task.s(arg1, arg2, kw=1) > ``` > > as a shortcut to: > > ``` pycon > >>> task.subtask((arg1, arg2), {'kw': 1}) > ``` - Tasks can be chained by using the `|` operator: > ``` pycon > >>> (add.s(2, 2), pow.s(2)).apply_async() > ``` - Subtasks can be \"evaluated\" using the `~` operator: > ``` pycon > >>> ~add.s(2, 2) > 4 > > >>> ~(add.s(2, 2) | pow.s(2)) > ``` > > is the same as: > > ``` pycon > >>> chain(add.s(2, 2), pow.s(2)).apply_async().get() > ``` - A new subtask_type key has been added to the subtask dictionary. > This can be the string `"chord"`, `"group"`, `"chain"`, > `"chunks"`, `"xmap"`, or `"xstarmap"`. - maybe_subtask now uses subtask_type to reconstruct the object, to be used when using non-pickle serializers. - The logic for these operations have been moved to dedicated tasks celery.chord, celery.chain and celery.group. - subtask no longer inherits from AttributeDict. > It\'s now a pure dict subclass with properties for attribute > access to the relevant keys. - The repr\'s now outputs how the sequence would like imperatively: > ``` pycon > >>> from celery import chord > > >>> (chord([add.s(i, i) for i in xrange(10)], xsum.s()) > | pow.s(2)) > tasks.xsum([tasks.add(0, 0), > tasks.add(1, 1), > tasks.add(2, 2), > tasks.add(3, 3), > tasks.add(4, 4), > tasks.add(5, 5), > tasks.add(6, 6), > tasks.add(7, 7), > tasks.add(8, 8), > tasks.add(9, 9)]) | tasks.pow(2) > ``` ### New remote control commands These commands were previously experimental, but they\'ve proven stable and is now documented as part of the official API. - `add_consumer`{.interpreted-text role="control"}/`cancel_consumer`{.interpreted-text role="control"} > Tells workers to consume from a new queue, or cancel consuming > from a queue. This command has also been changed so that the > worker remembers the queues added, so that the change will persist > even if the connection is re-connected. > > These commands are available programmatically as > `@control.add_consumer`{.interpreted-text role="meth"} / > `@control.cancel_consumer`{.interpreted-text role="meth"}: > > ``` pycon > >>> celery.control.add_consumer(queue_name, > ... destination=['w1.example.com']) > >>> celery.control.cancel_consumer(queue_name, > ... destination=['w1.example.com']) > ``` > > or using the `celery control`{.interpreted-text role="program"} > command: > > ``` console > $ celery control -d w1.example.com add_consumer queue > $ celery control -d w1.example.com cancel_consumer queue > ``` > > ::: note > ::: title > Note > ::: > > Remember that a control command without *destination* will be sent > to **all workers**. > ::: - `autoscale`{.interpreted-text role="control"} > Tells workers with `--autoscale` enabled to change autoscale > max/min concurrency settings. > > This command is available programmatically as > `@control.autoscale`{.interpreted-text role="meth"}: > > ``` pycon > >>> celery.control.autoscale(max=10, min=5, > ... destination=['w1.example.com']) > ``` > > or using the `celery control`{.interpreted-text role="program"} > command: > > ``` console > $ celery control -d w1.example.com autoscale 10 5 > ``` - `pool_grow`{.interpreted-text role="control"}/`pool_shrink`{.interpreted-text role="control"} > Tells workers to add or remove pool processes. > > These commands are available programmatically as > `@control.pool_grow`{.interpreted-text role="meth"} / > `@control.pool_shrink`{.interpreted-text role="meth"}: > > ``` pycon > >>> celery.control.pool_grow(2, destination=['w1.example.com']) > >>> celery.control.pool_shrink(2, destination=['w1.example.com']) > ``` > > or using the `celery control`{.interpreted-text role="program"} > command: > > ``` console > $ celery control -d w1.example.com pool_grow 2 > $ celery control -d w1.example.com pool_shrink 2 > ``` - `celery control`{.interpreted-text role="program"} now supports `rate_limit`{.interpreted-text role="control"} and `time_limit`{.interpreted-text role="control"} commands. > See `celery control --help` for details. ### Crontab now supports Day of Month, and Month of Year arguments See the updated list of examples at `beat-crontab`{.interpreted-text role="ref"}. ### Immutable subtasks `subtask`\'s can now be immutable, which means that the arguments won\'t be modified when calling callbacks: ``` pycon >>> chain(add.s(2, 2), clear_static_electricity.si()) ``` means it\'ll not receive the argument of the parent task, and `.si()` is a shortcut to: ``` pycon >>> clear_static_electricity.subtask(immutable=True) ``` ### Logging Improvements Logging support now conforms better with best practices. - Classes used by the worker no longer uses app.get_default_logger, but uses [celery.utils.log.get_logger]{.title-ref} which simply gets the logger not setting the level, and adds a NullHandler. - Loggers are no longer passed around, instead every module using logging defines a module global logger that\'s used throughout. - All loggers inherit from a common logger called \"celery\". - Before `task.get_logger` would setup a new logger for every task, and even set the log level. This is no longer the case. > - Instead all task loggers now inherit from a common > \"celery.task\" logger that\'s set up when programs call > [setup_logging_subsystem]{.title-ref}. > - Instead of using LoggerAdapter to augment the formatter with > the task_id and task_name field, the task base logger now use > a special formatter adding these values at run-time from the > currently executing task. - In fact, `task.get_logger` is no longer recommended, it is better to add a module-level logger to your tasks module. > For example, like this: > > ``` python > from celery.utils.log import get_task_logger > > logger = get_task_logger(__name__) > > @celery.task > def add(x, y): > logger.debug('Adding %r + %r' % (x, y)) > return x + y > ``` > > The resulting logger will then inherit from the `"celery.task"` > logger so that the current task name and id is included in logging > output. - Redirected output from stdout/stderr is now logged to a \"celery.redirected\" logger. - In addition a few warnings.warn have been replaced with logger.warn. - Now avoids the \'no handlers for logger multiprocessing\' warning ### Task registry no longer global Every Celery instance now has its own task registry. You can make apps share registries by specifying it: ``` pycon >>> app1 = Celery() >>> app2 = Celery(tasks=app1.tasks) ``` Note that tasks are shared between registries by default, so that tasks will be added to every subsequently created task registry. As an alternative tasks can be private to specific task registries by setting the `shared` argument to the `@task` decorator: ``` python @celery.task(shared=False) def add(x, y): return x + y ``` ### Abstract tasks are now lazily bound The `~celery.task.Task`{.interpreted-text role="class"} class is no longer bound to an app by default, it will first be bound (and configured) when a concrete subclass is created. This means that you can safely import and make task base classes, without also initializing the app environment: ``` python from celery.task import Task class DebugTask(Task): abstract = True def __call__(self, *args, **kwargs): print('CALLING %r' % (self,)) return self.run(*args, **kwargs) ``` ``` pycon >>> DebugTask >>> @celery1.task(base=DebugTask) ... def add(x, y): ... return x + y >>> add.__class__ > ``` ### Lazy task decorators The `@task` decorator is now lazy when used with custom apps. That is, if `accept_magic_kwargs` is enabled (her by called \"compat mode\"), the task decorator executes inline like before, however for custom apps the \@task decorator now returns a special PromiseProxy object that\'s only evaluated on access. All promises will be evaluated when `@finalize`{.interpreted-text role="meth"} is called, or implicitly when the task registry is first used. ### Smart [\--app]{.title-ref} option The `--app `{.interpreted-text role="option"} option now \'auto-detects\' > - If the provided path is a module it tries to get an attribute > named \'celery\'. > - If the provided path is a package it tries to import a sub module > named celery\', and get the celery attribute from that module. For example, if you have a project named `proj` where the celery app is located in `from proj.celery import app`, then the following will be equivalent: ``` console $ celery worker --app=proj $ celery worker --app=proj.celery: $ celery worker --app=proj.celery:app ``` ### In Other News - New `CELERYD_WORKER_LOST_WAIT`{.interpreted-text role="setting"} to control the timeout in seconds before `billiard.WorkerLostError`{.interpreted-text role="exc"} is raised when a worker can\'t be signaled (Issue #595). > Contributed by Brendon Crawford. - Redis event monitor queues are now automatically deleted (Issue #436). - App instance factory methods have been converted to be cached descriptors that creates a new subclass on access. > For example, this means that `app.Worker` is an actual class and > will work as expected when: > > ``` python > class Worker(app.Worker): > ... > ``` - New signal: `task_success`{.interpreted-text role="signal"}. - Multiprocessing logs are now only emitted if the `MP_LOG`{.interpreted-text role="envvar"} environment variable is set. - The Celery instance can now be created with a broker URL > ``` python > app = Celery(broker='redis://') > ``` - Result backends can now be set using a URL > Currently only supported by redis. Example use: > > ``` python > CELERY_RESULT_BACKEND = 'redis://localhost/1' > ``` - Heartbeat frequency now every 5s, and frequency sent with event > The heartbeat frequency is now available in the worker event > messages, so that clients can decide when to consider workers > offline based on this value. - Module celery.actors has been removed, and will be part of cl instead. - Introduces new `celery` command, which is an entry-point for all other commands. > The main for this command can be run by calling `celery.start()`. - Annotations now supports decorators if the key starts with \'@\'. > For example: > > ``` python > def debug_args(fun): > > @wraps(fun) > def _inner(*args, **kwargs): > print('ARGS: %r' % (args,)) > return _inner > > CELERY_ANNOTATIONS = { > 'tasks.add': {'@__call__': debug_args}, > } > ``` > > Also tasks are now always bound by class so that annotated methods > end up being bound. - Bug-report now available as a command and broadcast command > - Get it from a Python REPL: > > > ``` pycon > > >>> import celery > > >>> print(celery.bugreport()) > > ``` > > - Using the `celery` command line program: > > > ``` console > > $ celery report > > ``` > > - Get it from remote workers: > > > ``` console > > $ celery inspect report > > ``` - Module `celery.log` moved to `celery.app.log`{.interpreted-text role="mod"}. - Module `celery.task.control` moved to `celery.app.control`{.interpreted-text role="mod"}. - New signal: `task_revoked`{.interpreted-text role="signal"} > Sent in the main process when the task is revoked or terminated. - `AsyncResult.task_id` renamed to `AsyncResult.id` - `TasksetResult.taskset_id` renamed to `.id` - `xmap(task, sequence)` and `xstarmap(task, sequence)` > Returns a list of the results applying the task function to every > item in the sequence. > > Example: > > ``` pycon > >>> from celery import xstarmap > > >>> xstarmap(add, zip(range(10), range(10)).apply_async() > [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] > ``` - `chunks(task, sequence, chunksize)` - `group.skew(start=, stop=, step=)` > Skew will skew the countdown for the individual tasks in a group > \-- for example with this group: > > ``` pycon > >>> g = group(add.s(i, i) for i in xrange(10)) > ``` Skewing the tasks from 0 seconds to 10 seconds: > ``` pycon > >>> g.skew(stop=10) > ``` Will have the first task execute in 0 seconds, the second in 1 second, the third in 2 seconds and so on. - 99% test Coverage - `CELERY_QUEUES`{.interpreted-text role="setting"} can now be a list/tuple of `~kombu.Queue`{.interpreted-text role="class"} instances. > Internally `@amqp.queues`{.interpreted-text role="attr"} is now a > mapping of name/Queue instances, instead of converting on the fly. - Can now specify connection for `@control.inspect`{.interpreted-text role="class"}. > ``` python > from kombu import Connection > > i = celery.control.inspect(connection=Connection('redis://')) > i.active_queues() > ``` - `CELERY_FORCE_EXECV`{.interpreted-text role="setting"} is now enabled by default. > If the old behavior is wanted the setting can be set to False, or > the new [\--no-execv]{.title-ref} option to > `celery worker`{.interpreted-text role="program"}. - Deprecated module `celery.conf` has been removed. - The `CELERY_TIMEZONE`{.interpreted-text role="setting"} now always require the `pytz`{.interpreted-text role="pypi"} library to be installed (except if the timezone is set to [UTC]{.title-ref}). - The Tokyo Tyrant backend has been removed and is no longer supported. - Now uses `~kombu.common.maybe_declare`{.interpreted-text role="func"} to cache queue declarations. - There\'s no longer a global default for the `CELERYBEAT_MAX_LOOP_INTERVAL`{.interpreted-text role="setting"} setting, it is instead set by individual schedulers. - Worker: now truncates very long message bodies in error reports. - No longer deep-copies exceptions when trying to serialize errors. - `CELERY_BENCH`{.interpreted-text role="envvar"} environment variable, will now also list memory usage statistics at worker shutdown. - Worker: now only ever use a single timer for all timing needs, and instead set different priorities. - An exceptions arguments are now safely pickled > Contributed by Matt Long. - Worker/Beat no longer logs the start-up banner. > Previously it would be logged with severity warning, now it\'s > only written to stdout. - The `contrib/` directory in the distribution has been renamed to `extra/`. - New signal: `task_revoked`{.interpreted-text role="signal"} - `celery.contrib.migrate`{.interpreted-text role="mod"}: Many improvements, including; filtering, queue migration, and support for acking messages on the broker migrating from. > Contributed by John Watson. - Worker: Prefetch count increments are now optimized and grouped together. - Worker: No longer calls `consume` on the remote control command queue twice. > Probably didn\'t cause any problems, but was unnecessary. ### Internals - `app.broker_connection` is now `app.connection` > Both names still work. - Compatibility modules are now generated dynamically upon use. > These modules are `celery.messaging`, `celery.log`, > `celery.decorators` and `celery.registry`. - `celery.utils`{.interpreted-text role="mod"} refactored into multiple modules: > `celery.utils.text`{.interpreted-text role="mod"} > `celery.utils.imports`{.interpreted-text role="mod"} > `celery.utils.functional`{.interpreted-text role="mod"} - Now using `kombu.utils.encoding`{.interpreted-text role="mod"} instead of `celery.utils.encoding`{.interpreted-text role="mod"}. - Renamed module `celery.routes` -\> `celery.app.routes`{.interpreted-text role="mod"}. - Renamed package `celery.db` -\> `celery.backends.database`{.interpreted-text role="mod"}. - Renamed module `celery.abstract` -\> `celery.worker.bootsteps`{.interpreted-text role="mod"}. - Command line docs are now parsed from the module docstrings. - Test suite directory has been reorganized. - `setup.py`{.interpreted-text role="program"} now reads docs from the `requirements/`{.interpreted-text role="file"} directory. - Celery commands no longer wraps output (Issue #700). > Contributed by Thomas Johansson. ## Experimental {#v300-experimental} ### `celery.contrib.methods`{.interpreted-text role="mod"}: Task decorator for methods This is an experimental module containing a task decorator, and a task decorator filter, that can be used to create tasks out of methods: from celery.contrib.methods import task_method class Counter(object): def __init__(self): self.value = 1 @celery.task(name='Counter.increment', filter=task_method) def increment(self, n=1): self.value += 1 return self.value See `celery.contrib.methods`{.interpreted-text role="mod"} for more information. ## Unscheduled Removals {#v300-unscheduled-removals} Usually we don\'t make backward incompatible removals, but these removals should have no major effect. - The following settings have been renamed: > - `CELERYD_ETA_SCHEDULER` -\> `CELERYD_TIMER` > - `CELERYD_ETA_SCHEDULER_PRECISION` -\> > `CELERYD_TIMER_PRECISION` ## Deprecation Time-line Changes {#v300-deprecations} See the `deprecation-timeline`{.interpreted-text role="ref"}. - The `celery.backends.pyredis` compat module has been removed. > Use `celery.backends.redis`{.interpreted-text role="mod"} instead! - The following undocumented API\'s has been moved: > - `control.inspect.add_consumer` -\> > `@control.add_consumer`{.interpreted-text role="meth"}. > - `control.inspect.cancel_consumer` -\> > `@control.cancel_consumer`{.interpreted-text role="meth"}. > - `control.inspect.enable_events` -\> > `@control.enable_events`{.interpreted-text role="meth"}. > - `control.inspect.disable_events` -\> > `@control.disable_events`{.interpreted-text role="meth"}. > > This way `inspect()` is only used for commands that don\'t modify > anything, while idempotent control commands that make changes are > on the control objects. ## Fixes - Retry SQLAlchemy backend operations on DatabaseError/OperationalError (Issue #634) - Tasks that called `retry` wasn\'t acknowledged if acks late was enabled > Fix contributed by David Markey. - The message priority argument wasn\'t properly propagated to Kombu (Issue #708). > Fix contributed by Eran Rundstein --- # What\'s new in Celery 3.1 (Cipater) {#whatsnew-3.1} Author : Ask Solem (`ask at celeryproject.org`) ::: sidebar **Change history** What\'s new documents describe the changes in major versions, we also have a `changelog`{.interpreted-text role="ref"} that lists the changes in bugfix releases (0.0.x), while older series are archived under the `history`{.interpreted-text role="ref"} section. ::: Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. To read more about Celery you should go read the `introduction `{.interpreted-text role="ref"}. While this version is backward compatible with previous versions it\'s important that you read the following section. This version is officially supported on CPython 2.6, 2.7, and 3.3, and also supported on PyPy. ::: topic **Table of Contents** Make sure you read the important notes before upgrading to this version. ::: ::: {.contents local="" depth="2"} ::: ## Preface Deadlocks have long plagued our workers, and while uncommon they\'re not acceptable. They\'re also infamous for being extremely hard to diagnose and reproduce, so to make this job easier I wrote a stress test suite that bombards the worker with different tasks in an attempt to break it. What happens if thousands of worker child processes are killed every second? what if we also kill the broker connection every 10 seconds? These are examples of what the stress test suite will do to the worker, and it reruns these tests using different configuration combinations to find edge case bugs. The end result was that I had to rewrite the prefork pool to avoid the use of the POSIX semaphore. This was extremely challenging, but after months of hard work the worker now finally passes the stress test suite. There\'s probably more bugs to find, but the good news is that we now have a tool to reproduce them, so should you be so unlucky to experience a bug then we\'ll write a test for it and squash it! Note that I\'ve also moved many broker transports into experimental status: the only transports recommended for production use today is RabbitMQ and Redis. I don\'t have the resources to maintain all of them, so bugs are left unresolved. I wish that someone will step up and take responsibility for these transports or donate resources to improve them, but as the situation is now I don\'t think the quality is up to date with the rest of the code-base so I cannot recommend them for production use. The next version of Celery 4.0 will focus on performance and removing rarely used parts of the library. Work has also started on a new message protocol, supporting multiple languages and more. The initial draft can be found `here `{.interpreted-text role="ref"}. This has probably been the hardest release I\'ve worked on, so no introduction to this changelog would be complete without a massive thank you to everyone who contributed and helped me test it! Thank you for your support! *--- Ask Solem* ## Important Notes {#v310-important} ### Dropped support for Python 2.5 Celery now requires Python 2.6 or later. The new dual code base runs on both Python 2 and 3, without requiring the `2to3` porting tool. ::: note ::: title Note ::: This is also the last version to support Python 2.6! From Celery 4.0 and on-wards Python 2.7 or later will be required. ::: ### Last version to enable Pickle by default {#last-version-to-enable-pickle} Starting from Celery 4.0 the default serializer will be json. If you depend on pickle being accepted you should be prepared for this change by explicitly allowing your worker to consume pickled messages using the `CELERY_ACCEPT_CONTENT`{.interpreted-text role="setting"} setting: ``` python CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml'] ``` Make sure you only select the serialization formats you\'ll actually be using, and make sure you\'ve properly secured your broker from unwanted access (see the `Security Guide `{.interpreted-text role="ref"}). The worker will emit a deprecation warning if you don\'t define this setting. ::: topic **for Kombu users** Kombu 3.0 no longer accepts pickled messages by default, so if you use Kombu directly then you have to configure your consumers: see the `Kombu 3.0 Changelog `{.interpreted-text role="ref"} for more information. ::: ### Old command-line programs removed and deprecated Everyone should move to the new `celery`{.interpreted-text role="program"} umbrella command, so we\'re incrementally deprecating the old command names. In this version we\'ve removed all commands that aren\'t used in init-scripts. The rest will be removed in 4.0. ------------------------------------------------------------------------------- Program New Status Replacement ------------------- -------------- -------------------------------------------- `celeryd` *DEPRECATED* `celery worker`{.interpreted-text role="program"} `celerybeat` *DEPRECATED* `celery beat`{.interpreted-text role="program"} `celeryd-multi` *DEPRECATED* `celery multi`{.interpreted-text role="program"} `celeryctl` **REMOVED** `celery inspect|control`{.interpreted-text role="program"} `celeryev` **REMOVED** `celery events`{.interpreted-text role="program"} `camqadm` **REMOVED** `celery amqp`{.interpreted-text role="program"} ------------------------------------------------------------------------------- If this isn\'t a new installation then you may want to remove the old commands: ``` console $ pip uninstall celery $ # repeat until it fails # ... $ pip uninstall celery $ pip install celery ``` Please run `celery --help`{.interpreted-text role="program"} for help using the umbrella command. ## News {#v310-news} ### Prefork Pool Improvements These improvements are only active if you use an async capable transport. This means only RabbitMQ (AMQP) and Redis are supported at this point and other transports will still use the thread-based fallback implementation. - Pool is now using one IPC queue per child process. > Previously the pool shared one queue between all child processes, > using a POSIX semaphore as a mutex to achieve exclusive read and > write access. > > The POSIX semaphore has now been removed and each child process > gets a dedicated queue. This means that the worker will require > more file descriptors (two descriptors per process), but it also > means that performance is improved and we can send work to > individual child processes. > > POSIX semaphores aren\'t released when a process is killed, so > killing processes could lead to a deadlock if it happened while > the semaphore was acquired. There\'s no good solution to fix this, > so the best option was to remove the semaphore. - Asynchronous write operations > The pool now uses async I/O to send work to the child processes. - Lost process detection is now immediate. > If a child process is killed or exits mysteriously the pool > previously had to wait for 30 seconds before marking the task with > a `~celery.exceptions.WorkerLostError`{.interpreted-text > role="exc"}. It had to do this because the out-queue was shared > between all processes, and the pool couldn\'t be certain whether > the process completed the task or not. So an arbitrary timeout of > 30 seconds was chosen, as it was believed that the out-queue > would\'ve been drained by this point. > > This timeout is no longer necessary, and so the task can be marked > as failed as soon as the pool gets the notification that the > process exited. - Rare race conditions fixed > Most of these bugs were never reported to us, but were discovered > while running the new stress test suite. #### Caveats ::: topic **Long running tasks** The new pool will send tasks to a child process as long as the process in-queue is writable, and since the socket is buffered this means that the processes are, in effect, prefetching tasks. This benefits performance but it also means that other tasks may be stuck waiting for a long running task to complete: -> send T1 to Process A # A executes T1 -> send T2 to Process B # B executes T2 <- T2 complete -> send T3 to Process A # A still executing T1, T3 stuck in local buffer and # won't start until T1 returns The buffer size varies based on the operating system: some may have a buffer as small as 64KB but on recent Linux versions the buffer size is 1MB (can only be changed system wide). You can disable this prefetching behavior by enabling the `-Ofair `{.interpreted-text role="option"} worker option: ``` console $ celery -A proj worker -l info -Ofair ``` With this option enabled the worker will only write to workers that are available for work, disabling the prefetch behavior. ::: ::: topic **Max tasks per child** If a process exits and pool prefetch is enabled the worker may have already written many tasks to the process in-queue, and these tasks must then be moved back and rewritten to a new process. This is very expensive if you have the `--max-tasks-per-child `{.interpreted-text role="option"} option set to a low value (e.g., less than 10), you should not be using the `-Ofast `{.interpreted-text role="option"} scheduler option. ::: ### Django supported out of the box Celery 3.0 introduced a shiny new API, but unfortunately didn\'t have a solution for Django users. The situation changes with this version as Django is now supported in core and new Django users coming to Celery are now expected to use the new API directly. The Django community has a convention where there\'s a separate `django-x` package for every library, acting like a bridge between Django and the library. Having a separate project for Django users has been a pain for Celery, with multiple issue trackers and multiple documentation sources, and then lastly since 3.0 we even had different APIs. With this version we challenge that convention and Django users will use the same library, the same API and the same documentation as everyone else. There\'s no rush to port your existing code to use the new API, but if you\'d like to experiment with it you should know that: - You need to use a Celery application instance. > The new Celery API introduced in 3.0 requires users to instantiate > the library by creating an application: > > ``` python > from celery import Celery > > app = Celery() > ``` - You need to explicitly integrate Celery with Django > Celery won\'t automatically use the Django settings, so you can > either configure Celery separately or you can tell it to use the > Django settings with: > > ``` python > app.config_from_object('django.conf:settings') > ``` > > Neither will it automatically traverse your installed apps to find > task modules. If you want this behavior, you must explicitly pass > a list of Django instances to the Celery app: > > ``` python > from django.conf import settings > app.autodiscover_tasks(lambda: settings.INSTALLED_APPS) > ``` - You no longer use `manage.py` > Instead you use the `celery`{.interpreted-text role="program"} > command directly: > > ``` console > $ celery -A proj worker -l info > ``` > > For this to work your app module must store the > `DJANGO_SETTINGS_MODULE`{.interpreted-text role="envvar"} > environment variable, see the example in the `Django > guide `{.interpreted-text role="ref"}. To get started with the new API you should first read the `first-steps`{.interpreted-text role="ref"} tutorial, and then you should read the Django-specific instructions in `django-first-steps`{.interpreted-text role="ref"}. The fixes and improvements applied by the `django-celery`{.interpreted-text role="pypi"} library are now automatically applied by core Celery when it detects that the `DJANGO_SETTINGS_MODULE`{.interpreted-text role="envvar"} environment variable is set. The distribution ships with a new example project using Django in `examples/django`{.interpreted-text role="file"}: Some features still require the `django-celery`{.interpreted-text role="pypi"} library: > - Celery doesn\'t implement the Django database or cache result > backends. > > - > > Celery doesn\'t ship with the database-based periodic task > > : scheduler. ::: note ::: title Note ::: If you\'re still using the old API when you upgrade to Celery 3.1 then you must make sure that your settings module contains the `djcelery.setup_loader()` line, since this will no longer happen as a side-effect of importing the `django-celery`{.interpreted-text role="pypi"} module. New users (or if you\'ve ported to the new API) don\'t need the `setup_loader` line anymore, and must make sure to remove it. ::: ### Events are now ordered using logical time Keeping physical clocks in perfect sync is impossible, so using time-stamps to order events in a distributed system isn\'t reliable. Celery event messages have included a logical clock value for some time, but starting with this version that field is also used to order them. Also, events now record timezone information by including a new `utcoffset` field in the event message. This is a signed integer telling the difference from UTC time in hours, so for example, an event sent from the Europe/London timezone in daylight savings time will have an offset of 1. `@events.Receiver`{.interpreted-text role="class"} will automatically convert the time-stamps to the local timezone. ::: note ::: title Note ::: The logical clock is synchronized with other nodes in the same cluster (neighbors), so this means that the logical epoch will start at the point when the first worker in the cluster starts. If all of the workers are shutdown the clock value will be lost and reset to 0. To protect against this, you should specify the `celery worker --statedb`{.interpreted-text role="option"} option such that the worker can persist the clock value at shutdown. You may notice that the logical clock is an integer value and increases very rapidly. Don\'t worry about the value overflowing though, as even in the most busy clusters it may take several millennium before the clock exceeds a 64 bits value. ::: ### New worker node name format (`name@host`) Node names are now constructed by two elements: name and host-name separated by \'@\'. This change was made to more easily identify multiple instances running on the same machine. If a custom name isn\'t specified then the worker will use the name \'celery\' by default, resulting in a fully qualified node name of \'\': ``` console $ celery worker -n example.com celery@example.com ``` To also set the name you must include the @: ``` console $ celery worker -n worker1@example.com worker1@example.com ``` The worker will identify itself using the fully qualified node name in events and broadcast messages, so where before a worker would identify itself as \'worker1.example.com\', it\'ll now use \'\'. Remember that the `-n `{.interpreted-text role="option"} argument also supports simple variable substitutions, so if the current host-name is *george.example.com* then the `%h` macro will expand into that: ``` console $ celery worker -n worker1@%h worker1@george.example.com ``` The available substitutions are as follows: -------------------------------------------------------- Variable Substitution --------------- ---------------------------------------- `%h` Full host-name (including domain name) `%d` Domain name only `%n` Host-name only (without domain name) `%%` The character `%` -------------------------------------------------------- ### Bound tasks The task decorator can now create \"bound tasks\", which means that the task will receive the `self` argument. ``` python @app.task(bind=True) def send_twitter_status(self, oauth, tweet): try: twitter = Twitter(oauth) twitter.update_status(tweet) except (Twitter.FailWhaleError, Twitter.LoginError) as exc: raise self.retry(exc=exc) ``` Using *bound tasks* is now the recommended approach whenever you need access to the task instance or request context. Previously one would\'ve to refer to the name of the task instead (`send_twitter_status.retry`), but this could lead to problems in some configurations. ### Mingle: Worker synchronization The worker will now attempt to synchronize with other workers in the same cluster. Synchronized data currently includes revoked tasks and logical clock. This only happens at start-up and causes a one second start-up delay to collect broadcast responses from other workers. You can disable this bootstep using the `celery worker --without-mingle`{.interpreted-text role="option"} option. ### Gossip: Worker \<-\> Worker communication Workers are now passively subscribing to worker related events like heartbeats. This means that a worker knows what other workers are doing and can detect if they go offline. Currently this is only used for clock synchronization, but there are many possibilities for future additions and you can write extensions that take advantage of this already. Some ideas include consensus protocols, reroute task to best worker (based on resource usage or data locality) or restarting workers when they crash. We believe that although this is a small addition, it opens amazing possibilities. You can disable this bootstep using the `celery worker --without-gossip`{.interpreted-text role="option"} option. ### Bootsteps: Extending the worker By writing bootsteps you can now easily extend the consumer part of the worker to add additional features, like custom message consumers. The worker has been using bootsteps for some time, but these were never documented. In this version the consumer part of the worker has also been rewritten to use bootsteps and the new `guide-extending`{.interpreted-text role="ref"} guide documents examples extending the worker, including adding custom message consumers. See the `guide-extending`{.interpreted-text role="ref"} guide for more information. ::: note ::: title Note ::: Bootsteps written for older versions won\'t be compatible with this version, as the API has changed significantly. The old API was experimental and internal but should you be so unlucky to use it then please contact the mailing-list and we\'ll help you port the bootstep to the new API. ::: ### New RPC result backend This new experimental version of the `amqp` result backend is a good alternative to use in classical RPC scenarios, where the process that initiates the task is always the process to retrieve the result. It uses Kombu to send and retrieve results, and each client uses a unique queue for replies to be sent to. This avoids the significant overhead of the original amqp result backend which creates one queue per task. By default results sent using this backend won\'t persist, so they won\'t survive a broker restart. You can enable the `CELERY_RESULT_PERSISTENT`{.interpreted-text role="setting"} setting to change that. ``` python CELERY_RESULT_BACKEND = 'rpc' CELERY_RESULT_PERSISTENT = True ``` Note that chords are currently not supported by the RPC backend. ### Time limits can now be set by the client Two new options have been added to the Calling API: `time_limit` and `soft_time_limit`: ``` pycon >>> res = add.apply_async((2, 2), time_limit=10, soft_time_limit=8) >>> res = add.subtask((2, 2), time_limit=10, soft_time_limit=8).delay() >>> res = add.s(2, 2).set(time_limit=10, soft_time_limit=8).delay() ``` Contributed by Mher Movsisyan. ### Redis: Broadcast messages and virtual hosts Broadcast messages are currently seen by all virtual hosts when using the Redis transport. You can now fix this by enabling a prefix to all channels so that the messages are separated: ``` python BROKER_TRANSPORT_OPTIONS = {'fanout_prefix': True} ``` Note that you\'ll not be able to communicate with workers running older versions or workers that doesn\'t have this setting enabled. This setting will be the default in a future version. Related to Issue #1490. ### `pytz`{.interpreted-text role="pypi"} replaces `python-dateutil`{.interpreted-text role="pypi"} dependency Celery no longer depends on the `python-dateutil`{.interpreted-text role="pypi"} library, but instead a new dependency on the `pytz`{.interpreted-text role="pypi"} library was added. The `pytz`{.interpreted-text role="pypi"} library was already recommended for accurate timezone support. This also means that dependencies are the same for both Python 2 and Python 3, and that the `requirements/default-py3k.txt`{.interpreted-text role="file"} file has been removed. ### Support for `setuptools`{.interpreted-text role="pypi"} extra requirements Pip now supports the `setuptools`{.interpreted-text role="pypi"} extra requirements format, so we\'ve removed the old bundles concept, and instead specify setuptools extras. You install extras by specifying them inside brackets: ``` console $ pip install celery[redis,mongodb] ``` The above will install the dependencies for Redis and MongoDB. You can list as many extras as you want. ::: warning ::: title Warning ::: You can\'t use the `celery-with-*` packages anymore, as these won\'t be updated to use Celery 3.1. ::: ------------------------------------------------------------------- Extension Requirement entry Type ------------- ------------------------- --------------------------- Redis `celery[redis]` transport, result backend MongoDB `celery[mongodb]` transport, result backend CouchDB `celery[couchdb]` transport Beanstalk `celery[beanstalk]` transport ZeroMQ `celery[zeromq]` transport Zookeeper `celery[zookeeper]` transport SQLAlchemy `celery[sqlalchemy]` transport, result backend librabbitmq `celery[librabbitmq]` transport (C amqp client) ------------------------------------------------------------------- The complete list with examples is found in the `bundles`{.interpreted-text role="ref"} section. ### `subtask.__call__()` now executes the task directly A misunderstanding led to `Signature.__call__` being an alias of `.delay` but this doesn\'t conform to the calling API of `Task` which calls the underlying task method. This means that: ``` python @app.task def add(x, y): return x + y add.s(2, 2)() ``` now does the same as calling the task directly: ``` pycon >>> add(2, 2) ``` ### In Other News - Now depends on `Kombu 3.0 `{.interpreted-text role="ref"}. - Now depends on `billiard`{.interpreted-text role="pypi"} version 3.3. - Worker will now crash if running as the root user with pickle enabled. - Canvas: `group.apply_async` and `chain.apply_async` no longer starts separate task. > That the group and chord primitives supported the \"calling API\" > like other subtasks was a nice idea, but it was useless in > practice and often confused users. If you still want this behavior > you can define a task to do it for you. - New method `Signature.freeze()` can be used to \"finalize\" signatures/subtask. > Regular signature: > > ``` pycon > >>> s = add.s(2, 2) > >>> result = s.freeze() > >>> result > > >>> s.delay() > > ``` > > Group: > > ``` pycon > >>> g = group(add.s(2, 2), add.s(4, 4)) > >>> result = g.freeze() > 70c0fb3d-b60e-4b22-8df7-aa25b9abc86d, > 58fcd260-2e32-4308-a2ea-f5be4a24f7f4]> > >>> g() > > ``` - Chord exception behavior defined (Issue #1172). > From this version the chord callback will change state to FAILURE > when a task part of a chord raises an exception. > > See more at `chord-errors`{.interpreted-text role="ref"}. - New ability to specify additional command line options to the worker and beat programs. > The `@user_options`{.interpreted-text role="attr"} attribute can > be used to add additional command-line arguments, and expects > `optparse`{.interpreted-text role="mod"}-style options: > > ``` python > from celery import Celery > from celery.bin import Option > > app = Celery() > app.user_options['worker'].add( > Option('--my-argument'), > ) > ``` > > See the `guide-extending`{.interpreted-text role="ref"} guide for > more information. - All events now include a `pid` field, which is the process id of the process that sent the event. - Event heartbeats are now calculated based on the time when the event was received by the monitor, and not the time reported by the worker. > This means that a worker with an out-of-sync clock will no longer > show as \'Offline\' in monitors. > > A warning is now emitted if the difference between the senders > time and the internal time is greater than 15 seconds, suggesting > that the clocks are out of sync. - Monotonic clock support. > A monotonic clock is now used for timeouts and scheduling. > > The monotonic clock function is built-in starting from Python 3.4, > but we also have fallback implementations for Linux and macOS. - `celery worker`{.interpreted-text role="program"} now supports a new `--detach `{.interpreted-text role="option"} argument to start the worker as a daemon in the background. - `@events.Receiver`{.interpreted-text role="class"} now sets a `local_received` field for incoming events, which is set to the time of when the event was received. - `@events.Dispatcher`{.interpreted-text role="class"} now accepts a `groups` argument which decides a white-list of event groups that\'ll be sent. > The type of an event is a string separated by \'-\', where the > part before the first \'-\' is the group. Currently there are only > two groups: `worker` and `task`. > > A dispatcher instantiated as follows: > > ``` pycon > >>> app.events.Dispatcher(connection, groups=['worker']) > ``` > > will only send worker related events and silently drop any > attempts to send events related to any other group. - New `BROKER_FAILOVER_STRATEGY`{.interpreted-text role="setting"} setting. > This setting can be used to change the transport fail-over > strategy, can either be a callable returning an iterable or the > name of a Kombu built-in failover strategy. Default is > \"round-robin\". > > Contributed by Matt Wise. - `Result.revoke` will no longer wait for replies. > You can add the `reply=True` argument if you really want to wait > for responses from the workers. - Better support for link and link_error tasks for chords. > Contributed by Steeve Morin. - Worker: Now emits warning if the `CELERYD_POOL`{.interpreted-text role="setting"} setting is set to enable the eventlet/gevent pools. > The [-P]{.title-ref} option should always be used to select the > eventlet/gevent pool to ensure that the patches are applied as > early as possible. > > If you start the worker in a wrapper (like Django\'s > `manage.py`{.interpreted-text role="file"}) then you must apply > the patches manually, for example by creating an alternative > wrapper that monkey patches at the start of the program before > importing any other modules. - There\'s a now an \'inspect clock\' command which will collect the current logical clock value from workers. - [celery inspect stats]{.title-ref} now contains the process id of the worker\'s main process. > Contributed by Mher Movsisyan. - New remote control command to dump a workers configuration. > Example: > > ``` console > $ celery inspect conf > ``` > > Configuration values will be converted to values supported by JSON > where possible. > > Contributed by Mher Movsisyan. - New settings `CELERY_EVENT_QUEUE_TTL`{.interpreted-text role="setting"} and `CELERY_EVENT_QUEUE_EXPIRES`{.interpreted-text role="setting"}. > These control when a monitors event queue is deleted, and for how > long events published to that queue will be visible. Only > supported on RabbitMQ. - New Couchbase result backend. > This result backend enables you to store and retrieve task results > using [Couchbase](). > > See `conf-couchbase-result-backend`{.interpreted-text role="ref"} > for more information about configuring this result backend. > > Contributed by Alain Masiero. - CentOS init-script now supports starting multiple worker instances. > See the script header for details. > > Contributed by Jonathan Jordan. - `AsyncResult.iter_native` now sets default interval parameter to 0.5 > Fix contributed by Idan Kamara - New setting `BROKER_LOGIN_METHOD`{.interpreted-text role="setting"}. > This setting can be used to specify an alternate login method for > the AMQP transports. > > Contributed by Adrien Guinet - The `dump_conf` remote control command will now give the string representation for types that aren\'t JSON compatible. - Function [celery.security.setup_security]{.title-ref} is now `@setup_security`{.interpreted-text role="func"}. - Task retry now propagates the message expiry value (Issue #980). > The value is forwarded at is, so the expiry time won\'t change. To > update the expiry time you\'d\'ve to pass a new expires argument > to `retry()`. - Worker now crashes if a channel error occurs. > Channel errors are transport specific and is the list of > exceptions returned by `Connection.channel_errors`. For RabbitMQ > this means that Celery will crash if the equivalence checks for > one of the queues in `CELERY_QUEUES`{.interpreted-text > role="setting"} mismatches, which makes sense since this is a > scenario where manual intervention is required. - Calling `AsyncResult.get()` on a chain now propagates errors for previous tasks (Issue #1014). - The parent attribute of `AsyncResult` is now reconstructed when using JSON serialization (Issue #1014). - Worker disconnection logs are now logged with severity warning instead of error. > Contributed by Chris Adams. - `events.State` no longer crashes when it receives unknown event types. - SQLAlchemy Result Backend: New `CELERY_RESULT_DB_TABLENAMES`{.interpreted-text role="setting"} setting can be used to change the name of the database tables used. > Contributed by Ryan Petrello. - SQLAlchemy Result Backend: Now calls `enginge.dispose` after fork : (Issue #1564). > If you create your own SQLAlchemy engines then you must also > make sure that these are closed after fork in the worker: > > ``` python > from multiprocessing.util import register_after_fork > > engine = create_engine(*engine_args) > register_after_fork(engine, engine.dispose) > ``` - A stress test suite for the Celery worker has been written. > This is located in the `funtests/stress` directory in the git > repository. There\'s a README file there to get you started. - The logger named `celery.concurrency` has been renamed to `celery.pool`. - New command line utility `celery graph`. > This utility creates graphs in GraphViz dot format. > > You can create graphs from the currently installed bootsteps: > > ``` console > # Create graph of currently installed bootsteps in both the worker > # and consumer name-spaces. > $ celery graph bootsteps | dot -T png -o steps.png > > # Graph of the consumer name-space only. > $ celery graph bootsteps consumer | dot -T png -o consumer_only.png > > # Graph of the worker name-space only. > $ celery graph bootsteps worker | dot -T png -o worker_only.png > ``` > > Or graphs of workers in a cluster: > > ``` console > # Create graph from the current cluster > $ celery graph workers | dot -T png -o workers.png > > # Create graph from a specified list of workers > $ celery graph workers nodes:w1,w2,w3 | dot -T png workers.png > > # also specify the number of threads in each worker > $ celery graph workers nodes:w1,w2,w3 threads:2,4,6 > > # …also specify the broker and backend URLs shown in the graph > $ celery graph workers broker:amqp:// backend:redis:// > > # …also specify the max number of workers/threads shown (wmax/tmax), > # enumerating anything that exceeds that number. > $ celery graph workers wmax:10 tmax:3 > ``` - Changed the way that app instances are pickled. > Apps can now define a `__reduce_keys__` method that\'s used > instead of the old `AppPickler` attribute. For example, if your > app defines a custom \'foo\' attribute that needs to be preserved > when pickling you can define a `__reduce_keys__` as such: > > ``` python > import celery > > class Celery(celery.Celery): > > def __init__(self, *args, **kwargs): > super(Celery, self).__init__(*args, **kwargs) > self.foo = kwargs.get('foo') > > def __reduce_keys__(self): > return super(Celery, self).__reduce_keys__().update( > foo=self.foo, > ) > ``` > > This is a much more convenient way to add support for pickling > custom attributes. The old `AppPickler` is still supported but its > use is discouraged and we would like to remove it in a future > version. - Ability to trace imports for debugging purposes. > The `C_IMPDEBUG`{.interpreted-text role="envvar"} can be set to > trace imports as they occur: > > ``` console > $ C_IMDEBUG=1 celery worker -l info > ``` > > ``` console > $ C_IMPDEBUG=1 celery shell > ``` - Message headers now available as part of the task request. > Example adding and retrieving a header value: > > ``` python > @app.task(bind=True) > def t(self): > return self.request.headers.get('sender') > > >>> t.apply_async(headers={'sender': 'George Costanza'}) > ``` - New `before_task_publish`{.interpreted-text role="signal"} signal dispatched before a task message is sent and can be used to modify the final message fields (Issue #1281). - New `after_task_publish`{.interpreted-text role="signal"} signal replaces the old `task_sent`{.interpreted-text role="signal"} signal. > The `task_sent`{.interpreted-text role="signal"} signal is now > deprecated and shouldn\'t be used. - New `worker_process_shutdown`{.interpreted-text role="signal"} signal is dispatched in the prefork pool child processes as they exit. > Contributed by Daniel M Taub. - `celery.platforms.PIDFile` renamed to `celery.platforms.Pidfile`{.interpreted-text role="class"}. - MongoDB Backend: Can now be configured using a URL: - MongoDB Backend: No longer using deprecated `pymongo.Connection`. - MongoDB Backend: Now disables `auto_start_request`. - MongoDB Backend: Now enables `use_greenlets` when eventlet/gevent is used. - `subtask()` / `maybe_subtask()` renamed to `signature()`/`maybe_signature()`. > Aliases still available for backwards compatibility. - The `correlation_id` message property is now automatically set to the id of the task. - The task message `eta` and `expires` fields now includes timezone information. - All result backends `store_result`/`mark_as_*` methods must now accept a `request` keyword argument. - Events now emit warning if the broken `yajl` library is used. - The `celeryd_init`{.interpreted-text role="signal"} signal now takes an extra keyword argument: `option`. > This is the mapping of parsed command line arguments, and can be > used to prepare new preload arguments > (`app.user_options['preload']`). - New callback: `@on_configure`{.interpreted-text role="meth"}. > This callback is called when an app is about to be configured (a > configuration key is required). - Worker: No longer forks on `HUP`{.interpreted-text role="sig"}. > This means that the worker will reuse the same pid for better > support with external process supervisors. > > Contributed by Jameel Al-Aziz. - Worker: The log message `Got task from broker …` was changed to `Received task …`. - Worker: The log message `Skipping revoked task …` was changed to `Discarding revoked task …`. - Optimization: Improved performance of `ResultSet.join_native()`. > Contributed by Stas Rudakou. - The `task_revoked`{.interpreted-text role="signal"} signal now accepts new `request` argument (Issue #1555). > The revoked signal is dispatched after the task request is removed > from the stack, so it must instead use the > `~celery.worker.request.Request`{.interpreted-text role="class"} > object to get information about the task. - Worker: New `-X `{.interpreted-text role="option"} command line argument to exclude queues (Issue #1399). > The `-X `{.interpreted-text role="option"} > argument is the inverse of the > `-Q `{.interpreted-text role="option"} argument > and accepts a list of queues to exclude (not consume from): > > ``` console > # Consume from all queues in CELERY_QUEUES, but not the 'foo' queue. > $ celery worker -A proj -l info -X foo > ``` - Adds `C_FAKEFORK`{.interpreted-text role="envvar"} environment variable for simple init-script/`celery multi`{.interpreted-text role="program"} debugging. > This means that you can now do: > > ``` console > $ C_FAKEFORK=1 celery multi start 10 > ``` > > or: > > ``` console > $ C_FAKEFORK=1 /etc/init.d/celeryd start > ``` > > to avoid the daemonization step to see errors that aren\'t visible > due to missing stdout/stderr. > > A `dryrun` command has been added to the generic init-script that > enables this option. - New public API to push and pop from the current task stack: > `celery.app.push_current_task`{.interpreted-text role="func"} and > `celery.app.pop_current_task`{.interpreted-text role="func"}\`. - `RetryTaskError` has been renamed to `~celery.exceptions.Retry`{.interpreted-text role="exc"}. > The old name is still available for backwards compatibility. - New semi-predicate exception `~celery.exceptions.Reject`{.interpreted-text role="exc"}. > This exception can be raised to `reject`/`requeue` the task > message, see `task-semipred-reject`{.interpreted-text role="ref"} > for examples. - `Semipredicates `{.interpreted-text role="ref"} documented: (Retry/Ignore/Reject). ## Scheduled Removals {#v310-removals} - The `BROKER_INSIST` setting and the `insist` argument to `~@connection` is no longer supported. - The `CELERY_AMQP_TASK_RESULT_CONNECTION_MAX` setting is no longer supported. > Use `BROKER_POOL_LIMIT`{.interpreted-text role="setting"} instead. - The `CELERY_TASK_ERROR_WHITELIST` setting is no longer supported. > You should set the > `~celery.utils.mail.ErrorMail`{.interpreted-text role="class"} > attribute of the task class instead. You can also do this using > `CELERY_ANNOTATIONS`{.interpreted-text role="setting"}: > > > ``` python > > from celery import Celery > > from celery.utils.mail import ErrorMail > > > > class MyErrorMail(ErrorMail): > > whitelist = (KeyError, ImportError) > > > > def should_send(self, context, exc): > > return isinstance(exc, self.whitelist) > > > > app = Celery() > > app.conf.CELERY_ANNOTATIONS = { > > '*': { > > 'ErrorMail': MyErrorMails, > > } > > } > > ``` - Functions that creates a broker connections no longer supports the `connect_timeout` argument. > This can now only be set using the > `BROKER_CONNECTION_TIMEOUT`{.interpreted-text role="setting"} > setting. This is because functions no longer create connections > directly, but instead get them from the connection pool. - The `CELERY_AMQP_TASK_RESULT_EXPIRES` setting is no longer supported. > Use `CELERY_TASK_RESULT_EXPIRES`{.interpreted-text role="setting"} > instead. ## Deprecation Time-line Changes {#v310-deprecations} See the `deprecation-timeline`{.interpreted-text role="ref"}. ## Fixes {#v310-fixes} - AMQP Backend: join didn\'t convert exceptions when using the json serializer. - Non-abstract task classes are now shared between apps (Issue #1150). > Note that non-abstract task classes shouldn\'t be used in the new > API. You should only create custom task classes when you use them > as a base class in the `@task` decorator. > > This fix ensure backwards compatibility with older Celery versions > so that non-abstract task classes works even if a module is > imported multiple times so that the app is also instantiated > multiple times. - Worker: Workaround for Unicode errors in logs (Issue #427). - Task methods: `.apply_async` now works properly if args list is None (Issue #1459). - Eventlet/gevent/solo/threads pools now properly handles `BaseException`{.interpreted-text role="exc"} errors raised by tasks. - `autoscale`{.interpreted-text role="control"} and `pool_grow`{.interpreted-text role="control"}/`pool_shrink`{.interpreted-text role="control"} remote control commands will now also automatically increase and decrease the consumer prefetch count. > Fix contributed by Daniel M. Taub. - `celery control pool_` commands didn\'t coerce string arguments to int. - Redis/Cache chords: Callback result is now set to failure if the group disappeared from the database (Issue #1094). - Worker: Now makes sure that the shutdown process isn\'t initiated more than once. - Programs: `celery multi`{.interpreted-text role="program"} now properly handles both `-f` and `--logfile `{.interpreted-text role="option"} options (Issue #1541). ## Internal changes {#v310-internal} - Module `celery.task.trace` has been renamed to `celery.app.trace`{.interpreted-text role="mod"}. - Module `celery.concurrency.processes` has been renamed to `celery.concurrency.prefork`{.interpreted-text role="mod"}. - Classes that no longer fall back to using the default app: > - Result backends > (`celery.backends.base.BaseBackend`{.interpreted-text > role="class"}) > - `celery.worker.WorkController`{.interpreted-text role="class"} > - `celery.worker.Consumer`{.interpreted-text role="class"} > - `celery.worker.request.Request`{.interpreted-text > role="class"} > > This means that you have to pass a specific app when instantiating > these classes. - `EventDispatcher.copy_buffer` renamed to `@events.Dispatcher.extend_buffer`{.interpreted-text role="meth"}. - Removed unused and never documented global instance `celery.events.state.state`. - `@events.Receiver`{.interpreted-text role="class"} is now a `kombu.mixins.ConsumerMixin`{.interpreted-text role="class"} subclass. - `celery.apps.worker.Worker`{.interpreted-text role="class"} has been refactored as a subclass of `celery.worker.WorkController`{.interpreted-text role="class"}. > This removes a lot of duplicate functionality. - The `Celery.with_default_connection` method has been removed in favor of `with app.connection_or_acquire` (`@connection_or_acquire`{.interpreted-text role="meth"}) - The `celery.results.BaseDictBackend` class has been removed and is replaced by `celery.results.BaseBackend`{.interpreted-text role="class"}. --- # What\'s new in Celery 4.0 (latentcall) {#whatsnew-4.0} Author : Ask Solem (`ask at celeryproject.org`) ::: sidebar **Change history** What\'s new documents describe the changes in major versions, we also have a `changelog`{.interpreted-text role="ref"} that lists the changes in bugfix releases (0.0.x), while older series are archived under the `history`{.interpreted-text role="ref"} section. ::: Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. To read more about Celery you should go read the `introduction `{.interpreted-text role="ref"}. While this version is backward compatible with previous versions it\'s important that you read the following section. This version is officially supported on CPython 2.7, 3.4, and 3.5. and also supported on PyPy. ::: topic **Table of Contents** Make sure you read the important notes before upgrading to this version. ::: ::: {.contents local="" depth="3"} ::: ## Preface Welcome to Celery 4! This is a massive release with over two years of changes. Not only does it come with many new features, but it also fixes a massive list of bugs, so in many ways you could call it our \"Snow Leopard\" release. The next major version of Celery will support Python 3.5 only, where we are planning to take advantage of the new asyncio library. This release would not have been possible without the support of my employer, [Robinhood](https://robinhood.com) (we\'re hiring!). - Ask Solem Dedicated to Sebastian \"Zeb\" Bjørnerud (RIP), with special thanks to [Ty Wilkins](http://tywilkins.com), for designing our new logo, all the contributors who help make this happen, and my colleagues at [Robinhood](https://robinhood.com). ### Wall of Contributors Aaron McMillin, Adam Chainz, Adam Renberg, Adriano Martins de Jesus, Adrien Guinet, Ahmet Demir, Aitor Gómez-Goiri, Alan Justino, Albert Wang, Alex Koshelev, Alex Rattray, Alex Williams, Alexander Koshelev, Alexander Lebedev, Alexander Oblovatniy, Alexey Kotlyarov, Ali Bozorgkhan, Alice Zoë Bevan--McGregor, Allard Hoeve, Alman One, Amir Rustamzadeh, Andrea Rabbaglietti, Andrea Rosa, Andrei Fokau, Andrew Rodionoff, Andrew Stewart, Andriy Yurchuk, Aneil Mallavarapu, Areski Belaid, Armenak Baburyan, Arthur Vuillard, Artyom Koval, Asif Saifuddin Auvi, Ask Solem, Balthazar Rouberol, Batiste Bieler, Berker Peksag, Bert Vanderbauwhede, Brendan Smithyman, Brian Bouterse, Bryce Groff, Cameron Will, ChangBo Guo, Chris Clark, Chris Duryee, Chris Erway, Chris Harris, Chris Martin, Chillar Anand, Colin McIntosh, Conrad Kramer, Corey Farwell, Craig Jellick, Cullen Rhodes, Dallas Marlow, Daniel Devine, Daniel Wallace, Danilo Bargen, Davanum Srinivas, Dave Smith, David Baumgold, David Harrigan, David Pravec, Dennis Brakhane, Derek Anderson, Dmitry Dygalo, Dmitry Malinovsky, Dongweiming, Dudás Ádám, Dustin J. Mitchell, Ed Morley, Edward Betts, Éloi Rivard, Emmanuel Cazenave, Fahad Siddiqui, Fatih Sucu, Feanil Patel, Federico Ficarelli, Felix Schwarz, Felix Yan, Fernando Rocha, Flavio Grossi, Frantisek Holop, Gao Jiangmiao, George Whewell, Gerald Manipon, Gilles Dartiguelongue, Gino Ledesma, Greg Wilbur, Guillaume Seguin, Hank John, Hogni Gylfason, Ilya Georgievsky, Ionel Cristian Mărieș, Ivan Larin, James Pulec, Jared Lewis, Jason Veatch, Jasper Bryant-Greene, Jeff Widman, Jeremy Tillman, Jeremy Zafran, Jocelyn Delalande, Joe Jevnik, Joe Sanford, John Anderson, John Barham, John Kirkham, John Whitlock, Jonathan Vanasco, Joshua Harlow, João Ricardo, Juan Carlos Ferrer, Juan Rossi, Justin Patrin, Kai Groner, Kevin Harvey, Kevin Richardson, Komu Wairagu, Konstantinos Koukopoulos, Kouhei Maeda, Kracekumar Ramaraju, Krzysztof Bujniewicz, Latitia M. Haskins, Len Buckens, Lev Berman, lidongming, Lorenzo Mancini, Lucas Wiman, Luke Pomfrey, Luyun Xie, Maciej Obuchowski, Manuel Kaufmann, Marat Sharafutdinov, Marc Sibson, Marcio Ribeiro, Marin Atanasov Nikolov, Mathieu Fenniak, Mark Parncutt, Mauro Rocco, Maxime Beauchemin, Maxime Vdb, Mher Movsisyan, Michael Aquilina, Michael Duane Mooring, Michael Permana, Mickaël Penhard, Mike Attwood, Mitchel Humpherys, Mohamed Abouelsaoud, Morris Tweed, Morton Fox, Môshe van der Sterre, Nat Williams, Nathan Van Gheem, Nicolas Unravel, Nik Nyby, Omer Katz, Omer Korner, Ori Hoch, Paul Pearce, Paulo Bu, Pavlo Kapyshin, Philip Garnero, Pierre Fersing, Piotr Kilczuk, Piotr Maślanka, Quentin Pradet, Radek Czajka, Raghuram Srinivasan, Randy Barlow, Raphael Michel, Rémy Léone, Robert Coup, Robert Kolba, Rockallite Wulf, Rodolfo Carvalho, Roger Hu, Romuald Brunet, Rongze Zhu, Ross Deane, Ryan Luckie, Rémy Greinhofer, Samuel Giffard, Samuel Jaillet, Sergey Azovskov, Sergey Tikhonov, Seungha Kim, Simon Peeters, Spencer E. Olson, Srinivas Garlapati, Stephen Milner, Steve Peak, Steven Sklar, Stuart Axon, Sukrit Khera, Tadej Janež, Taha Jahangir, Takeshi Kanemoto, Tayfun Sen, Tewfik Sadaoui, Thomas French, Thomas Grainger, Tomas Machalek, Tobias Schottdorf, Tocho Tochev, Valentyn Klindukh, Vic Kumar, Vladimir Bolshakov, Vladimir Gorbunov, Wayne Chang, Wieland Hoffmann, Wido den Hollander, Wil Langford, Will Thompson, William King, Yury Selivanov, Vytis Banaitis, Zoran Pavlovic, Xin Li, 許邱翔, `allenling`{.interpreted-text role="github_user"}, `alzeih`{.interpreted-text role="github_user"}, `bastb`{.interpreted-text role="github_user"}, `bee-keeper`{.interpreted-text role="github_user"}, `ffeast`{.interpreted-text role="github_user"}, `firefly4268`{.interpreted-text role="github_user"}, `flyingfoxlee`{.interpreted-text role="github_user"}, `gdw2`{.interpreted-text role="github_user"}, `gitaarik`{.interpreted-text role="github_user"}, `hankjin`{.interpreted-text role="github_user"}, `lvh`{.interpreted-text role="github_user"}, `m-vdb`{.interpreted-text role="github_user"}, `kindule`{.interpreted-text role="github_user"}, `mdk`{.interpreted-text role="github_user"}:, `michael-k`{.interpreted-text role="github_user"}, `mozillazg`{.interpreted-text role="github_user"}, `nokrik`{.interpreted-text role="github_user"}, `ocean1`{.interpreted-text role="github_user"}, `orlo666`{.interpreted-text role="github_user"}, `raducc`{.interpreted-text role="github_user"}, `wanglei`{.interpreted-text role="github_user"}, `worldexception`{.interpreted-text role="github_user"}, `xBeAsTx`{.interpreted-text role="github_user"}. ::: note ::: title Note ::: This wall was automatically generated from git history, so sadly it doesn\'t not include the people who help with more important things like answering mailing-list questions. ::: ## Upgrading from Celery 3.1 ### Step 1: Upgrade to Celery 3.1.25 If you haven\'t already, the first step is to upgrade to Celery 3.1.25. This version adds forward compatibility to the new message protocol, so that you can incrementally upgrade from 3.1 to 4.0. Deploy the workers first by upgrading to 3.1.25, this means these workers can process messages sent by clients using both 3.1 and 4.0. After the workers are upgraded you can upgrade the clients (e.g. web servers). ### Step 2: Update your configuration with the new setting names This version radically changes the configuration setting names, to be more consistent. The changes are fully backwards compatible, so you have the option to wait until the old setting names are deprecated, but to ease the transition we have included a command-line utility that rewrites your settings automatically. See `v400-upgrade-settings`{.interpreted-text role="ref"} for more information. ### Step 3: Read the important notes in this document Make sure you are not affected by any of the important upgrade notes mentioned in the following section. An especially important note is that Celery now checks the arguments you send to a task by matching it to the signature (`v400-typing`{.interpreted-text role="ref"}). ### Step 4: Upgrade to Celery 4.0 At this point you can upgrade your workers and clients with the new version. ## Important Notes {#v400-important} ### Dropped support for Python 2.6 Celery now requires Python 2.7 or later, and also drops support for Python 3.3 so supported versions are: - CPython 2.7 - CPython 3.4 - CPython 3.5 - PyPy 5.4 (`pypy2`) - PyPy 5.5-alpha (`pypy3`) ### Last major version to support Python 2 Starting from Celery 5.0 only Python 3.5+ will be supported. To make sure you\'re not affected by this change you should pin the Celery version in your requirements file, either to a specific version: `celery==4.0.0`, or a range: `celery>=4.0,<5.0`. Dropping support for Python 2 will enable us to remove massive amounts of compatibility code, and going with Python 3.5 allows us to take advantage of typing, async/await, asyncio, and similar concepts there\'s no alternative for in older versions. Celery 4.x will continue to work on Python 2.7, 3.4, 3.5; just as Celery 3.x still works on Python 2.6. ### Django support Celery 4.x requires Django 1.8 or later, but we really recommend using at least Django 1.9 for the new `transaction.on_commit` feature. A common problem when calling tasks from Django is when the task is related to a model change, and you wish to cancel the task if the transaction is rolled back, or ensure the task is only executed after the changes have been written to the database. `transaction.atomic` enables you to solve this problem by adding the task as a callback to be called only when the transaction is committed. Example usage: ``` python from functools import partial from django.db import transaction from .models import Article, Log from .tasks import send_article_created_notification def create_article(request): with transaction.atomic(): article = Article.objects.create(**request.POST) # send this task only if the rest of the transaction succeeds. transaction.on_commit(partial( send_article_created_notification.delay, article_id=article.pk)) Log.objects.create(type=Log.ARTICLE_CREATED, object_pk=article.pk) ``` ### Removed features - Microsoft Windows is no longer supported. The test suite is passing, and Celery seems to be working with Windows, but we make no guarantees as we are unable to diagnose issues on this platform. If you are a company requiring support on this platform, please get in touch. - Jython is no longer supported. #### Features removed for simplicity - Webhook task machinery (`celery.task.http`) has been removed. > Nowadays it\'s easy to use the `requests`{.interpreted-text > role="pypi"} module to write webhook tasks manually. We would love > to use requests but we are simply unable to as there\'s a very > vocal \'anti-dependency\' mob in the Python community > > If you need backwards compatibility you can simply copy + paste > the 3.1 version of the module and make sure it\'s imported by the > worker: > - Tasks no longer sends error emails. > This also removes support for `app.mail_admins`, and any > functionality related to sending emails. - `celery.contrib.batches` has been removed. > This was an experimental feature, so not covered by our > deprecation timeline guarantee. > > You can copy and pass the existing batches code for use within > your projects: > #### Features removed for lack of funding We announced with the 3.1 release that some transports were moved to experimental status, and that there\'d be no official support for the transports. As this subtle hint for the need of funding failed we\'ve removed them completely, breaking backwards compatibility. - Using the Django ORM as a broker is no longer supported. > You can still use the Django ORM as a result backend: see > `django-celery-results`{.interpreted-text role="ref"} section for > more information. - Using SQLAlchemy as a broker is no longer supported. > You can still use SQLAlchemy as a result backend. - Using CouchDB as a broker is no longer supported. > You can still use CouchDB as a result backend. - Using IronMQ as a broker is no longer supported. - Using Beanstalk as a broker is no longer supported. In addition some features have been removed completely so that attempting to use them will raise an exception: - The `--autoreload` feature has been removed. This was an experimental feature, and not covered by our deprecation timeline guarantee. The flag is removed completely so the worker will crash at startup when present. Luckily this flag isn\'t used in production systems. - The experimental `threads` pool is no longer supported and has been removed. - The `force_execv` feature is no longer supported. > The `celery worker` command now ignores the `--no-execv`, > `--force-execv`, and the `CELERYD_FORCE_EXECV` setting. > > This flag will be removed completely in 5.0 and the worker will > raise an error. - The old legacy \"amqp\" result backend has been deprecated, and will be removed in Celery 5.0. > Please use the `rpc` result backend for RPC-style calls, and a > persistent result backend for multi-consumer results. We think most of these can be fixed without considerable effort, so if you\'re interested in getting any of these features back, please get in touch. **Now to the good news**\... ### New Task Message Protocol This version introduces a brand new task message protocol, the first major change to the protocol since the beginning of the project. The new protocol is enabled by default in this version and since the new version isn\'t backwards compatible you have to be careful when upgrading. The 3.1.25 version was released to add compatibility with the new protocol so the easiest way to upgrade is to upgrade to that version first, then upgrade to 4.0 in a second deployment. If you wish to keep using the old protocol you may also configure the protocol version number used: ``` python app = Celery() app.conf.task_protocol = 1 ``` Read more about the features available in the new protocol in the news section found later in this document. ### Lowercase setting names {#v400-upgrade-settings} In the pursuit of beauty all settings are now renamed to be in all lowercase and some setting names have been renamed for consistency. This change is fully backwards compatible so you can still use the uppercase setting names, but we would like you to upgrade as soon as possible and you can do this automatically using the `celery upgrade settings`{.interpreted-text role="program"} command: ``` console $ celery upgrade settings proj/settings.py ``` This command will modify your module in-place to use the new lower-case names (if you want uppercase with a \"`CELERY`\" prefix see block below), and save a backup in `proj/settings.py.orig`{.interpreted-text role="file"}. > If you\'re loading Celery configuration from the Django settings > module then you\'ll want to keep using the uppercase names. > > You also want to use a `CELERY_` prefix so that no Celery settings > collide with Django settings used by other apps. > > To do this, you\'ll first need to convert your settings file to use > the new consistent naming scheme, and add the prefix to all Celery > related settings: > > ``` console > $ celery upgrade settings proj/settings.py --django > ``` > > After upgrading the settings file, you need to set the prefix > explicitly in your `proj/celery.py` module: > > ``` python > app.config_from_object('django.conf:settings', namespace='CELERY') > ``` > > You can find the most up to date Django Celery integration example > here: `django-first-steps`{.interpreted-text role="ref"}. > > ::: note > ::: title > Note > ::: > > This will also add a prefix to settings that didn\'t previously have > one, for example `BROKER_URL` should be written `CELERY_BROKER_URL` > with a namespace of `CELERY` `CELERY_BROKER_URL`. > ::: > > Luckily you don\'t have to manually change the files, as the > `celery upgrade settings --django`{.interpreted-text role="program"} > program should do the right thing. The loader will try to detect if your configuration is using the new format, and act accordingly, but this also means you\'re not allowed to mix and match new and old setting names, that\'s unless you provide a value for both alternatives. The major difference between previous versions, apart from the lower case names, are the renaming of some prefixes, like `celerybeat_` to `beat_`, `celeryd_` to `worker_`. The `celery_` prefix has also been removed, and task related settings from this name-space is now prefixed by `task_`, worker related settings with `worker_`. Apart from this most of the settings will be the same in lowercase, apart from a few special ones: **Setting name** **Replace with** ------------------------------------ -------------------------------------------------------------------------------------------------------------- `CELERY_MAX_CACHED_RESULTS` `result_cache_max`{.interpreted-text role="setting"} `CELERY_MESSAGE_COMPRESSION` `result_compression`{.interpreted-text role="setting"}/`task_compression`{.interpreted-text role="setting"}. `CELERY_TASK_RESULT_EXPIRES` `result_expires`{.interpreted-text role="setting"} `CELERY_RESULT_DBURI` `result_backend`{.interpreted-text role="setting"} `CELERY_RESULT_ENGINE_OPTIONS` `database_engine_options`{.interpreted-text role="setting"} `-*-_DB_SHORT_LIVED_SESSIONS` `database_short_lived_sessions`{.interpreted-text role="setting"} `CELERY_RESULT_DB_TABLE_NAMES` `database_db_names`{.interpreted-text role="setting"} `CELERY_ACKS_LATE` `task_acks_late`{.interpreted-text role="setting"} `CELERY_ALWAYS_EAGER` `task_always_eager`{.interpreted-text role="setting"} `CELERY_ANNOTATIONS` `task_annotations`{.interpreted-text role="setting"} `CELERY_MESSAGE_COMPRESSION` `task_compression`{.interpreted-text role="setting"} `CELERY_CREATE_MISSING_QUEUES` `task_create_missing_queues`{.interpreted-text role="setting"} `CELERY_DEFAULT_DELIVERY_MODE` `task_default_delivery_mode`{.interpreted-text role="setting"} `CELERY_DEFAULT_EXCHANGE` `task_default_exchange`{.interpreted-text role="setting"} `CELERY_DEFAULT_EXCHANGE_TYPE` `task_default_exchange_type`{.interpreted-text role="setting"} `CELERY_DEFAULT_QUEUE` `task_default_queue`{.interpreted-text role="setting"} `CELERY_DEFAULT_RATE_LIMIT` `task_default_rate_limit`{.interpreted-text role="setting"} `CELERY_DEFAULT_ROUTING_KEY` `task_default_routing_key`{.interpreted-text role="setting"} `-"-_EAGER_PROPAGATES_EXCEPTIONS` `task_eager_propagates`{.interpreted-text role="setting"} `CELERY_IGNORE_RESULT` `task_ignore_result`{.interpreted-text role="setting"} `CELERY_TASK_PUBLISH_RETRY` `task_publish_retry`{.interpreted-text role="setting"} `CELERY_TASK_PUBLISH_RETRY_POLICY` `task_publish_retry_policy`{.interpreted-text role="setting"} `CELERY_QUEUES` `task_queues`{.interpreted-text role="setting"} `CELERY_ROUTES` `task_routes`{.interpreted-text role="setting"} `CELERY_SEND_TASK_SENT_EVENT` `task_send_sent_event`{.interpreted-text role="setting"} `CELERY_TASK_SERIALIZER` `task_serializer`{.interpreted-text role="setting"} `CELERYD_TASK_SOFT_TIME_LIMIT` `task_soft_time_limit`{.interpreted-text role="setting"} `CELERYD_TASK_TIME_LIMIT` `task_time_limit`{.interpreted-text role="setting"} `CELERY_TRACK_STARTED` `task_track_started`{.interpreted-text role="setting"} `CELERY_DISABLE_RATE_LIMITS` `worker_disable_rate_limits`{.interpreted-text role="setting"} `CELERY_ENABLE_REMOTE_CONTROL` `worker_enable_remote_control`{.interpreted-text role="setting"} `CELERYD_SEND_EVENTS` `worker_send_task_events`{.interpreted-text role="setting"} You can see a full table of the changes in `conf-old-settings-map`{.interpreted-text role="ref"}. ### Json is now the default serializer The time has finally come to end the reign of `pickle`{.interpreted-text role="mod"} as the default serialization mechanism, and json is the default serializer starting from this version. This change was `announced with the release of Celery 3.1 `{.interpreted-text role="ref"}. If you\'re still depending on `pickle`{.interpreted-text role="mod"} being the default serializer, then you have to configure your app before upgrading to 4.0: ``` python task_serializer = 'pickle' result_serializer = 'pickle' accept_content = {'pickle'} ``` The Json serializer now also supports some additional types: - `~datetime.datetime`{.interpreted-text role="class"}, `~datetime.time`{.interpreted-text role="class"}, `~datetime.date`{.interpreted-text role="class"} > Converted to json text, in ISO-8601 format. - `~decimal.Decimal`{.interpreted-text role="class"} > Converted to json text. - `django.utils.functional.Promise`{.interpreted-text role="class"} > Django only: Lazy strings used for translation etc., are evaluated > and conversion to a json type is attempted. - `uuid.UUID`{.interpreted-text role="class"} > Converted to json text. You can also define a `__json__` method on your custom classes to support JSON serialization (must return a json compatible type): ``` python class Person: first_name = None last_name = None address = None def __json__(self): return { 'first_name': self.first_name, 'last_name': self.last_name, 'address': self.address, } ``` ### The Task base class no longer automatically register tasks The `~@Task`{.interpreted-text role="class"} class is no longer using a special meta-class that automatically registers the task in the task registry. Instead this is now handled by the `@task`{.interpreted-text role="class"} decorators. If you\'re still using class based tasks, then you need to register these manually: ``` python class CustomTask(Task): def run(self): print('running') CustomTask = app.register_task(CustomTask()) ``` The best practice is to use custom task classes only for overriding general behavior, and then using the task decorator to realize the task: ``` python @app.task(bind=True, base=CustomTask) def custom(self): print('running') ``` This change also means that the `abstract` attribute of the task no longer has any effect. ### Task argument checking {#v400-typing} The arguments of the task are now verified when calling the task, even asynchronously: ``` pycon >>> @app.task ... def add(x, y): ... return x + y >>> add.delay(8, 8) >>> add.delay(8) Traceback (most recent call last): File "", line 1, in File "celery/app/task.py", line 376, in delay return self.apply_async(args, kwargs) File "celery/app/task.py", line 485, in apply_async check_arguments(*(args or ()), **(kwargs or {})) TypeError: add() takes exactly 2 arguments (1 given) ``` You can disable the argument checking for any task by setting its `~@Task.typing`{.interpreted-text role="attr"} attribute to `False`{.interpreted-text role="const"}: ``` pycon >>> @app.task(typing=False) ... def add(x, y): ... return x + y ``` Or if you would like to disable this completely for all tasks you can pass `strict_typing=False` when creating the app: ``` python app = Celery(..., strict_typing=False) ``` ### Redis Events not backward compatible The Redis `fanout_patterns` and `fanout_prefix` transport options are now enabled by default. Workers/monitors without these flags enabled won\'t be able to see workers with this flag disabled. They can still execute tasks, but they cannot receive each others monitoring messages. You can upgrade in a backward compatible manner by first configuring your 3.1 workers and monitors to enable the settings, before the final upgrade to 4.0: ``` python BROKER_TRANSPORT_OPTIONS = { 'fanout_patterns': True, 'fanout_prefix': True, } ``` ### Redis Priorities Reversed Priority 0 is now lowest, 9 is highest. This change was made to make priority support consistent with how it works in AMQP. Contributed by **Alex Koshelev**. ### Django: Auto-discover now supports Django app configurations The `autodiscover_tasks()` function can now be called without arguments, and the Django handler will automatically find your installed apps: ``` python app.autodiscover_tasks() ``` The Django integration `example in the documentation `{.interpreted-text role="ref"} has been updated to use the argument-less call. This also ensures compatibility with the new, ehm, `AppConfig` stuff introduced in recent Django versions. ### Worker direct queues no longer use auto-delete Workers/clients running 4.0 will no longer be able to send worker direct messages to workers running older versions, and vice versa. If you\'re relying on worker direct messages you should upgrade your 3.x workers and clients to use the new routing settings first, by replacing `celery.utils.worker_direct`{.interpreted-text role="func"} with this implementation: ``` python from kombu import Exchange, Queue worker_direct_exchange = Exchange('C.dq2') def worker_direct(hostname): return Queue( '{hostname}.dq2'.format(hostname), exchange=worker_direct_exchange, routing_key=hostname, ) ``` This feature closed Issue #2492. ### Old command-line programs removed Installing Celery will no longer install the `celeryd`, `celerybeat` and `celeryd-multi` programs. This was announced with the release of Celery 3.1, but you may still have scripts pointing to the old names, so make sure you update these to use the new umbrella command: ----------------------------------------------------------------------- Program New Status Replacement ------------------- -------------- ------------------------------------ `celeryd` **REMOVED** `celery worker`{.interpreted-text role="program"} `celerybeat` **REMOVED** `celery beat`{.interpreted-text role="program"} `celeryd-multi` **REMOVED** `celery multi`{.interpreted-text role="program"} ----------------------------------------------------------------------- ## News {#v400-news} ### New protocol highlights The new protocol fixes many problems with the old one, and enables some long-requested features: - Most of the data are now sent as message headers, instead of being serialized with the message body. > In version 1 of the protocol the worker always had to deserialize > the message to be able to read task meta-data like the task id, > name, etc. This also meant that the worker was forced to > double-decode the data, first deserializing the message on > receipt, serializing the message again to send to child process, > then finally the child process deserializes the message again. > > Keeping the meta-data fields in the message headers means the > worker doesn\'t actually have to decode the payload before > delivering the task to the child process, and also that it\'s now > possible for the worker to reroute a task written in a language > different from Python to a different worker. - A new `lang` message header can be used to specify the programming language the task is written in. - Worker stores results for internal errors like `ContentDisallowed`, and other deserialization errors. - Worker stores results and sends monitoring events for unregistered task errors. - Worker calls callbacks/errbacks even when the result is sent by the parent process (e.g., `WorkerLostError`{.interpreted-text role="exc"} when a child process terminates, deserialization errors, unregistered tasks). - A new `origin` header contains information about the process sending the task (worker node-name, or PID and host-name information). - A new `shadow` header allows you to modify the task name used in logs. > This is useful for dispatch like patterns, like a task that calls > any function using pickle (don\'t do this at home): > > ``` python > from celery import Task > from celery.utils.imports import qualname > > class call_as_task(Task): > > def shadow_name(self, args, kwargs, options): > return 'call_as_task:{0}'.format(qualname(args[0])) > > def run(self, fun, *args, **kwargs): > return fun(*args, **kwargs) > call_as_task = app.register_task(call_as_task()) > ``` - New `argsrepr` and `kwargsrepr` fields contain textual representations of the task arguments (possibly truncated) for use in logs, monitors, etc. > This means the worker doesn\'t have to deserialize the message > payload to display the task arguments for informational purposes. - Chains now use a dedicated `chain` field enabling support for chains of thousands and more tasks. - New `parent_id` and `root_id` headers adds information about a tasks relationship with other tasks. > - `parent_id` is the task id of the task that called this task > - `root_id` is the first task in the work-flow. > > These fields can be used to improve monitors like flower to group > related messages together (like chains, groups, chords, complete > work-flows, etc). - `app.TaskProducer` replaced by `@amqp.create_task_message`{.interpreted-text role="meth"} and `@amqp.send_task_message`{.interpreted-text role="meth"}. > Dividing the responsibilities into creating and sending means that > people who want to send messages using a Python AMQP client > directly, don\'t have to implement the protocol. > > The `@amqp.create_task_message`{.interpreted-text role="meth"} > method calls either `@amqp.as_task_v2`{.interpreted-text > role="meth"}, or `@amqp.as_task_v1`{.interpreted-text role="meth"} > depending on the configured task protocol, and returns a special > `~celery.app.amqp.task_message`{.interpreted-text role="class"} > tuple containing the headers, properties and body of the task > message. ::: seealso The new task protocol is documented in full here: `message-protocol-task-v2`{.interpreted-text role="ref"}. ::: ### Prefork Pool Improvements #### Tasks now log from the child process Logging of task success/failure now happens from the child process executing the task. As a result logging utilities, like Sentry can get full information about tasks, including variables in the traceback stack. #### `-Ofair` is now the default scheduling strategy To re-enable the default behavior in 3.1 use the `-Ofast` command-line option. There\'s been lots of confusion about what the `-Ofair` command-line option does, and using the term \"prefetch\" in explanations have probably not helped given how confusing this terminology is in AMQP. When a Celery worker using the prefork pool receives a task, it needs to delegate that task to a child process for execution. The prefork pool has a configurable number of child processes (`--concurrency`) that can be used to execute tasks, and each child process uses pipes/sockets to communicate with the parent process: - inqueue (pipe/socket): parent sends task to the child process - outqueue (pipe/socket): child sends result/return value to the parent. In Celery 3.1 the default scheduling mechanism was simply to send the task to the first `inqueue` that was writable, with some heuristics to make sure we round-robin between them to ensure each child process would receive the same amount of tasks. This means that in the default scheduling strategy, a worker may send tasks to the same child process that is already executing a task. If that task is long running, it may block the waiting task for a long time. Even worse, hundreds of short-running tasks may be stuck behind a long running task even when there are child processes free to do work. The `-Ofair` scheduling strategy was added to avoid this situation, and when enabled it adds the rule that no task should be sent to the a child process that is already executing a task. The fair scheduling strategy may perform slightly worse if you have only short running tasks. #### Limit child process resident memory size You can now limit the maximum amount of memory allocated per prefork pool child process by setting the worker `--max-memory-per-child `{.interpreted-text role="option"} option, or the `worker_max_memory_per_child`{.interpreted-text role="setting"} setting. The limit is for RSS/resident memory size and is specified in kilobytes. A child process having exceeded the limit will be terminated and replaced with a new process after the currently executing task returns. See `worker-max-memory-per-child`{.interpreted-text role="ref"} for more information. Contributed by **Dave Smith**. #### One log-file per child process Init-scrips and `celery multi`{.interpreted-text role="program"} now uses the [%I]{.title-ref} log file format option (e.g., `/var/log/celery/%n%I.log`{.interpreted-text role="file"}). This change was necessary to ensure each child process has a separate log file after moving task logging to the child process, as multiple processes writing to the same log file can cause corruption. You\'re encouraged to upgrade your init-scripts and `celery multi`{.interpreted-text role="program"} arguments to use this new option. ### Transports #### RabbitMQ priority queue support See `routing-options-rabbitmq-priorities`{.interpreted-text role="ref"} for more information. Contributed by **Gerald Manipon**. #### Configure broker URL for read/write separately New `broker_read_url`{.interpreted-text role="setting"} and `broker_write_url`{.interpreted-text role="setting"} settings have been added so that separate broker URLs can be provided for connections used for consuming/publishing. In addition to the configuration options, two new methods have been added the app API: > - `app.connection_for_read()` > - `app.connection_for_write()` These should now be used in place of `app.connection()` to specify the intent of the required connection. ::: note ::: title Note ::: Two connection pools are available: `app.pool` (read), and `app.producer_pool` (write). The latter doesn\'t actually give connections but full `kombu.Producer`{.interpreted-text role="class"} instances. ``` python def publish_some_message(app, producer=None): with app.producer_or_acquire(producer) as producer: ... def consume_messages(app, connection=None): with app.connection_or_acquire(connection) as connection: ... ``` ::: #### RabbitMQ queue extensions support Queue declarations can now set a message TTL and queue expiry time directly, by using the `message_ttl` and `expires` arguments New arguments have been added to `~kombu.Queue`{.interpreted-text role="class"} that lets you directly and conveniently configure RabbitMQ queue extensions in queue declarations: - `Queue(expires=20.0)` > Set queue expiry time in float seconds. > > See `kombu.Queue.expires`{.interpreted-text role="attr"}. - `Queue(message_ttl=30.0)` > Set queue message time-to-live float seconds. > > See `kombu.Queue.message_ttl`{.interpreted-text role="attr"}. - `Queue(max_length=1000)` > Set queue max length (number of messages) as int. > > See `kombu.Queue.max_length`{.interpreted-text role="attr"}. - `Queue(max_length_bytes=1000)` > Set queue max length (message size total in bytes) as int. > > See `kombu.Queue.max_length_bytes`{.interpreted-text role="attr"}. - `Queue(max_priority=10)` > Declare queue to be a priority queue that routes messages based on > the `priority` field of the message. > > See `kombu.Queue.max_priority`{.interpreted-text role="attr"}. #### Amazon SQS transport now officially supported The SQS broker transport has been rewritten to use async I/O and as such joins RabbitMQ, Redis and QPid as officially supported transports. The new implementation also takes advantage of long polling, and closes several issues related to using SQS as a broker. This work was sponsored by Nextdoor. #### Apache QPid transport now officially supported Contributed by **Brian Bouterse**. ### Redis: Support for Sentinel You can point the connection to a list of sentinel URLs like: ``` text sentinel://0.0.0.0:26379;sentinel://0.0.0.0:26380/... ``` where each sentinel is separated by a [;]{.title-ref}. Multiple sentinels are handled by `kombu.Connection`{.interpreted-text role="class"} constructor, and placed in the alternative list of servers to connect to in case of connection failure. Contributed by **Sergey Azovskov**, and **Lorenzo Mancini**. ### Tasks #### Task Auto-retry Decorator Writing custom retry handling for exception events is so common that we now have built-in support for it. For this a new `autoretry_for` argument is now supported by the task decorators, where you can specify a tuple of exceptions to automatically retry for: ``` python from twitter.exceptions import FailWhaleError @app.task(autoretry_for=(FailWhaleError,)) def refresh_timeline(user): return twitter.refresh_timeline(user) ``` See `task-autoretry`{.interpreted-text role="ref"} for more information. Contributed by **Dmitry Malinovsky**. #### `Task.replace` Improvements - `self.replace(signature)` can now replace any task, chord or group, and the signature to replace with can be a chord, group or any other type of signature. - No longer inherits the callbacks and errbacks of the existing task. > If you replace a node in a tree, then you wouldn\'t expect the new > node to inherit the children of the old node. - `Task.replace_in_chord` has been removed, use `.replace` instead. - If the replacement is a group, that group will be automatically converted to a chord, where the callback \"accumulates\" the results of the group tasks. > A new built-in task ([celery.accumulate]{.title-ref} was added for > this purpose) Contributed by **Steeve Morin**, and **Ask Solem**. #### Remote Task Tracebacks The new `task_remote_tracebacks`{.interpreted-text role="setting"} will make task tracebacks more useful by injecting the stack of the remote worker. This feature requires the additional `tblib`{.interpreted-text role="pypi"} library. Contributed by **Ionel Cristian Mărieș**. #### Handling task connection errors Connection related errors occurring while sending a task is now re-raised as a `kombu.exceptions.OperationalError`{.interpreted-text role="exc"} error: ``` pycon >>> try: ... add.delay(2, 2) ... except add.OperationalError as exc: ... print('Could not send task %r: %r' % (add, exc)) ``` See `calling-connection-errors`{.interpreted-text role="ref"} for more information. #### Gevent/Eventlet: Dedicated thread for consuming results When using `gevent`{.interpreted-text role="pypi"}, or `eventlet`{.interpreted-text role="pypi"} there is now a single thread responsible for consuming events. This means that if you have many calls retrieving results, there will be a dedicated thread for consuming them: ``` python result = add.delay(2, 2) # this call will delegate to the result consumer thread: # once the consumer thread has received the result this greenlet can # continue. value = result.get(timeout=3) ``` This makes performing RPC calls when using gevent/eventlet perform much better. #### `AsyncResult.then(on_success, on_error)` The AsyncResult API has been extended to support the `~vine.promise`{.interpreted-text role="class"} protocol. This currently only works with the RPC (amqp) and Redis result backends, but lets you attach callbacks to when tasks finish: ``` python import gevent.monkey monkey.patch_all() import time from celery import Celery app = Celery(broker='amqp://', backend='rpc') @app.task def add(x, y): return x + y def on_result_ready(result): print('Received result for id %r: %r' % (result.id, result.result,)) add.delay(2, 2).then(on_result_ready) time.sleep(3) # run gevent event loop for a while. ``` Demonstrated using `gevent`{.interpreted-text role="pypi"} here, but really this is an API that\'s more useful in callback-based event loops like `twisted`{.interpreted-text role="pypi"}, or `tornado`{.interpreted-text role="pypi"}. #### New Task Router API The `task_routes`{.interpreted-text role="setting"} setting can now hold functions, and map routes now support glob patterns and regexes. Instead of using router classes you can now simply define a function: ``` python def route_for_task(name, args, kwargs, options, task=None, **kwargs): from proj import tasks if name == tasks.add.name: return {'queue': 'hipri'} ``` If you don\'t need the arguments you can use start arguments, just make sure you always also accept star arguments so that we have the ability to add more features in the future: ``` python def route_for_task(name, *args, **kwargs): from proj import tasks if name == tasks.add.name: return {'queue': 'hipri', 'priority': 9} ``` Both the `options` argument and the new `task` keyword argument are new to the function-style routers, and will make it easier to write routers based on execution options, or properties of the task. The optional `task` keyword argument won\'t be set if a task is called by name using `@send_task`{.interpreted-text role="meth"}. For more examples, including using glob/regexes in routers please see `task_routes`{.interpreted-text role="setting"} and `routing-automatic`{.interpreted-text role="ref"}. #### Canvas Refactor The canvas/work-flow implementation have been heavily refactored to fix some long outstanding issues. - Error callbacks can now take real exception and traceback instances (Issue #2538). > ``` pycon > >>> add.s(2, 2).on_error(log_error.s()).delay() > ``` > > Where `log_error` could be defined as: > > ``` python > @app.task > def log_error(request, exc, traceback): > with open(os.path.join('/var/errors', request.id), 'a') as fh: > print('--\n\n{0} {1} {2}'.format( > task_id, exc, traceback), file=fh) > ``` > > See `guide-canvas`{.interpreted-text role="ref"} for more > examples. - `chain(a, b, c)` now works the same as `a | b | c`. > This means chain may no longer return an instance of `chain`, > instead it may optimize the workflow so that e.g. two groups > chained together becomes one group. - Now unrolls groups within groups into a single group (Issue #1509). - chunks/map/starmap tasks now routes based on the target task - chords and chains can now be immutable. - Fixed bug where serialized signatures weren\'t converted back into signatures (Issue #2078) > Fix contributed by **Ross Deane**. - Fixed problem where chains and groups didn\'t work when using JSON serialization (Issue #2076). > Fix contributed by **Ross Deane**. - Creating a chord no longer results in multiple values for keyword argument \'task_id\' (Issue #2225). > Fix contributed by **Aneil Mallavarapu**. - Fixed issue where the wrong result is returned when a chain contains a chord as the penultimate task. > Fix contributed by **Aneil Mallavarapu**. - Special case of `group(A.s() | group(B.s() | C.s()))` now works. - Chain: Fixed bug with incorrect id set when a subtask is also a chain. - `group | group` is now flattened into a single group (Issue #2573). - Fixed issue where `group | task` wasn\'t upgrading correctly to chord (Issue #2922). - Chords now properly sets `result.parent` links. - `chunks`/`map`/`starmap` are now routed based on the target task. - `Signature.link` now works when argument is scalar (not a list) : (Issue #2019). - `group()` now properly forwards keyword arguments (Issue #3426). > Fix contributed by **Samuel Giffard**. - A `chord` where the header group only consists of a single task is now turned into a simple chain. - Passing a `link` argument to `group.apply_async()` now raises an error (Issue #3508). - `chord | sig` now attaches to the chord callback (Issue #3356). ### Periodic Tasks #### New API for configuring periodic tasks This new API enables you to use signatures when defining periodic tasks, removing the chance of mistyping task names. An example of the new API is `here `{.interpreted-text role="ref"}. #### Optimized Beat implementation The `celery beat`{.interpreted-text role="program"} implementation has been optimized for millions of periodic tasks by using a heap to schedule entries. Contributed by **Ask Solem** and **Alexander Koshelev**. #### Schedule tasks based on sunrise, sunset, dawn and dusk See `beat-solar`{.interpreted-text role="ref"} for more information. Contributed by **Mark Parncutt**. ### Result Backends #### RPC Result Backend matured Lots of bugs in the previously experimental RPC result backend have been fixed and can now be considered to production use. Contributed by **Ask Solem**, **Morris Tweed**. #### Redis: Result backend optimizations ##### `result.get()` is now using pub/sub for streaming task results Calling `result.get()` when using the Redis result backend used to be extremely expensive as it was using polling to wait for the result to become available. A default polling interval of 0.5 seconds didn\'t help performance, but was necessary to avoid a spin loop. The new implementation is using Redis Pub/Sub mechanisms to publish and retrieve results immediately, greatly improving task round-trip times. Contributed by **Yaroslav Zhavoronkov** and **Ask Solem**. ##### New optimized chord join implementation This was an experimental feature introduced in Celery 3.1, that could only be enabled by adding `?new_join=1` to the result backend URL configuration. We feel that the implementation has been tested thoroughly enough to be considered stable and enabled by default. The new implementation greatly reduces the overhead of chords, and especially with larger chords the performance benefit can be massive. #### New Riak result backend introduced See `conf-riak-result-backend`{.interpreted-text role="ref"} for more information. Contributed by **Gilles Dartiguelongue**, **Alman One** and **NoKriK**. #### New CouchDB result backend introduced See `conf-couchdb-result-backend`{.interpreted-text role="ref"} for more information. Contributed by **Nathan Van Gheem**. #### New Consul result backend introduced Add support for Consul as a backend using the Key/Value store of Consul. Consul has an HTTP API where through you can store keys with their values. The backend extends KeyValueStoreBackend and implements most of the methods. Mainly to set, get and remove objects. This allows Celery to store Task results in the K/V store of Consul. Consul also allows to set a TTL on keys using the Sessions from Consul. This way the backend supports auto expiry of Task results. For more information on Consul visit The backend uses `python-consul`{.interpreted-text role="pypi"} for talking to the HTTP API. This package is fully Python 3 compliant just as this backend is: ``` console $ pip install python-consul ``` That installs the required package to talk to Consul\'s HTTP API from Python. You can also specify consul as an extension in your dependency on Celery: ``` console $ pip install celery[consul] ``` See `bundles`{.interpreted-text role="ref"} for more information. Contributed by **Wido den Hollander**. #### Brand new Cassandra result backend A brand new Cassandra backend utilizing the new `cassandra-driver`{.interpreted-text role="pypi"} library is replacing the old result backend using the older `pycassa`{.interpreted-text role="pypi"} library. See `conf-cassandra-result-backend`{.interpreted-text role="ref"} for more information. To depend on Celery with Cassandra as the result backend use: ``` console $ pip install celery[cassandra] ``` You can also combine multiple extension requirements, please see `bundles`{.interpreted-text role="ref"} for more information. #### New Elasticsearch result backend introduced See `conf-elasticsearch-result-backend`{.interpreted-text role="ref"} for more information. To depend on Celery with Elasticsearch as the result backend use: ``` console $ pip install celery[elasticsearch] ``` You can also combine multiple extension requirements, please see `bundles`{.interpreted-text role="ref"} for more information. Contributed by **Ahmet Demir**. #### New File-system result backend introduced See `conf-filesystem-result-backend`{.interpreted-text role="ref"} for more information. Contributed by **Môshe van der Sterre**. ### Event Batching Events are now buffered in the worker and sent as a list, reducing the overhead required to send monitoring events. For authors of custom event monitors there will be no action required as long as you\'re using the Python Celery helpers (`~@events.Receiver`{.interpreted-text role="class"}) to implement your monitor. However, if you\'re parsing raw event messages you must now account for batched event messages, as they differ from normal event messages in the following way: - The routing key for a batch of event messages will be set to `.multi` where the only batched event group is currently `task` (giving a routing key of `task.multi`). - The message body will be a serialized list-of-dictionaries instead of a dictionary. Each item in the list can be regarded as a normal event message body. ### In Other News\... #### Requirements - Now depends on `Kombu 4.0 `{.interpreted-text role="ref"}. - Now depends on `billiard`{.interpreted-text role="pypi"} version 3.5. - No longer depends on `anyjson`{.interpreted-text role="pypi"}. Good-bye old friend :( #### Tasks - The \"anon-exchange\" is now used for simple name-name direct routing. This increases performance as it completely bypasses the routing table, in addition it also improves reliability for the Redis broker transport. - An empty ResultSet now evaluates to True. > Fix contributed by **Colin McIntosh**. - The default routing key (`task_default_routing_key`{.interpreted-text role="setting"}) and exchange name (`task_default_exchange`{.interpreted-text role="setting"}) is now taken from the `task_default_queue`{.interpreted-text role="setting"} setting. > This means that to change the name of the default queue, you now > only have to set a single setting. - New `task_reject_on_worker_lost`{.interpreted-text role="setting"} setting, and `~@Task.reject_on_worker_lost`{.interpreted-text role="attr"} task attribute decides what happens when the child worker process executing a late ack task is terminated. > Contributed by **Michael Permana**. - `Task.subtask` renamed to `Task.signature` with alias. - `Task.subtask_from_request` renamed to `Task.signature_from_request` with alias. - The `delivery_mode` attribute for `kombu.Queue`{.interpreted-text role="class"} is now respected (Issue #1953). - Routes in `task-routes`{.interpreted-text role="setting"} can now specify a `~kombu.Queue`{.interpreted-text role="class"} instance directly. > Example: > > ``` python > task_routes = {'proj.tasks.add': {'queue': Queue('add')}} > ``` - `AsyncResult` now raises `ValueError`{.interpreted-text role="exc"} if task_id is None. (Issue #1996). - Retried tasks didn\'t forward expires setting (Issue #3297). - `result.get()` now supports an `on_message` argument to set a callback to be called for every message received. - New abstract classes added: > - `~celery.utils.abstract.CallableTask`{.interpreted-text > role="class"} > > > Looks like a task. > > - `~celery.utils.abstract.CallableSignature`{.interpreted-text > role="class"} > > > Looks like a task signature. - `Task.replace` now properly forwards callbacks (Issue #2722). > Fix contributed by **Nicolas Unravel**. - `Task.replace`: Append to chain/chord (Closes #3232) > Fixed issue #3232, adding the signature to the chain (if there\'s > any). Fixed the chord suppress if the given signature contains > one. > > Fix contributed by `honux`{.interpreted-text role="github_user"}. - Task retry now also throws in eager mode. > Fix contributed by **Feanil Patel**. #### Beat - Fixed crontab infinite loop with invalid date. > When occurrence can never be reached (example, April, 31th), > trying to reach the next occurrence would trigger an infinite > loop. > > Try fixing that by raising a `RuntimeError`{.interpreted-text > role="exc"} after 2,000 iterations > > (Also added a test for crontab leap years in the process) > > Fix contributed by **Romuald Brunet**. - Now ensures the program exits with a non-zero exit code when an exception terminates the service. > Fix contributed by **Simon Peeters**. #### App - Dates are now always timezone aware even if `enable_utc`{.interpreted-text role="setting"} is disabled (Issue #943). > Fix contributed by **Omer Katz**. - **Config**: App preconfiguration is now also pickled with the configuration. > Fix contributed by **Jeremy Zafran**. - The application can now change how task names are generated using : the `~@gen_task_name`{.interpreted-text role="meth"} method. Contributed by **Dmitry Malinovsky**. - App has new `app.current_worker_task` property that returns the task that\'s currently being worked on (or `None`{.interpreted-text role="const"}). (Issue #2100). #### Logging - `~celery.utils.log.get_task_logger`{.interpreted-text role="func"} now raises an exception if trying to use the name \"celery\" or \"celery.task\" (Issue #3475). #### Execution Pools - **Eventlet/Gevent**: now enables AMQP heartbeat (Issue #3338). - **Eventlet/Gevent**: Fixed race condition leading to \"simultaneous read\" errors (Issue #2755). - **Prefork**: Prefork pool now uses `poll` instead of `select` where available (Issue #2373). - **Prefork**: Fixed bug where the pool would refuse to shut down the worker (Issue #2606). - **Eventlet**: Now returns pool size in `celery inspect stats`{.interpreted-text role="program"} command. > Contributed by **Alexander Oblovatniy**. ### Testing - Celery is now a `pytest`{.interpreted-text role="pypi"} plugin, including fixtures useful for unit and integration testing. > See the `testing user guide `{.interpreted-text > role="ref"} for more information. #### Transports - `amqps://` can now be specified to require SSL. - **Redis Transport**: The Redis transport now supports the `broker_use_ssl`{.interpreted-text role="setting"} option. > Contributed by **Robert Kolba**. - JSON serializer now calls `obj.__json__` for unsupported types. > This means you can now define a `__json__` method for custom types > that can be reduced down to a built-in json type. > > Example: > > ``` python > class Person: > first_name = None > last_name = None > address = None > > def __json__(self): > return { > 'first_name': self.first_name, > 'last_name': self.last_name, > 'address': self.address, > } > ``` - JSON serializer now handles datetime\'s, Django promise, UUID and Decimal. - New `Queue.consumer_arguments` can be used for the ability to set consumer priority via `x-priority`. See Example: ``` python consumer = Consumer(channel, consumer_arguments={'x-priority': 3}) ``` - Queue/Exchange: `no_declare` option added (also enabled for internal amq. exchanges). #### Programs - Celery is now using `argparse`{.interpreted-text role="mod"}, instead of `optparse`{.interpreted-text role="mod"}. - All programs now disable colors if the controlling terminal is not a TTY. - `celery worker`{.interpreted-text role="program"}: The `-q` argument now disables the startup banner. - `celery worker`{.interpreted-text role="program"}: The \"worker ready\" message is now logged using severity info, instead of warn. - `celery multi`{.interpreted-text role="program"}: `%n` format for is now synonym with `%N` to be consistent with `celery worker`{.interpreted-text role="program"}. - `celery inspect`{.interpreted-text role="program"}/`celery control`{.interpreted-text role="program"}: now supports a new `--json `{.interpreted-text role="option"} option to give output in json format. - `celery inspect registered`{.interpreted-text role="program"}: now ignores built-in tasks. - `celery purge`{.interpreted-text role="program"} now takes `-Q` and `-X` options used to specify what queues to include and exclude from the purge. - New `celery logtool`{.interpreted-text role="program"}: Utility for filtering and parsing celery worker log-files - `celery multi`{.interpreted-text role="program"}: now passes through [%i]{.title-ref} and [%I]{.title-ref} log file formats. - General: `%p` can now be used to expand to the full worker node-name in log-file/pid-file arguments. - A new command line option : `--executable `{.interpreted-text role="option"} is now available for daemonizing programs (`celery worker`{.interpreted-text role="program"} and `celery beat`{.interpreted-text role="program"}). > Contributed by **Bert Vanderbauwhede**. - `celery worker`{.interpreted-text role="program"}: supports new `--prefetch-multiplier `{.interpreted-text role="option"} option. > Contributed by **Mickaël Penhard**. - The `--loader` argument is now always effective even if an app argument is set (Issue #3405). - inspect/control now takes commands from registry > This means user remote-control commands can also be used from the > command-line. > > Note that you need to specify the arguments/and type of arguments > for the arguments to be correctly passed on the command-line. > > There are now two decorators, which use depends on the type of > command: [\@inspect_command]{.title-ref} + \`@control_command\`: > > ``` python > from celery.worker.control import control_command > > @control_command( > args=[('n', int)] > signature='[N=1]', > ) > def something(state, n=1, **kwargs): > ... > ``` > > Here `args` is a list of args supported by the command. The list > must contain tuples of `(argument_name, type)`. > > `signature` is just the command-line help used in e.g. > `celery -A proj control --help`. > > Commands also support [variadic]{.title-ref} arguments, which > means that any arguments left over will be added to a single > variable. Here demonstrated by the `terminate` command which takes > a signal argument and a variable number of task_ids: > > ``` python > from celery.worker.control import control_command > > @control_command( > args=[('signal', str)], > signature=' [id1, [id2, [..., [idN]]]]', > variadic='ids', > ) > def terminate(state, signal, ids, **kwargs): > ... > ``` > > This command can now be called using: > > ``` console > $ celery -A proj control terminate SIGKILL id1 id2 id3` > ``` > > See `worker-custom-control-commands`{.interpreted-text role="ref"} > for more information. #### Worker - Improvements and fixes for `~celery.utils.collections.LimitedSet`{.interpreted-text role="class"}. > Getting rid of leaking memory + adding `minlen` size of the set: > the minimal residual size of the set after operating for some > time. `minlen` items are kept, even if they should\'ve been > expired. > > Problems with older and even more old code: > > 1. Heap would tend to grow in some scenarios (like adding an item > multiple times). > 2. Adding many items fast wouldn\'t clean them soon enough (if > ever). > 3. When talking to other workers, revoked.\_data was sent, but it > was processed on the other side as iterable. That means giving > those keys new (current) time-stamp. By doing this workers > could recycle items forever. Combined with 1) and 2), this > means that in large set of workers, you\'re getting out of > memory soon. > > All those problems should be fixed now. > > This should fix issues #3095, #3086. > > Contributed by **David Pravec**. - New settings to control remote control command queues. > - `control_queue_expires`{.interpreted-text role="setting"} > > > Set queue expiry time for both remote control command > > queues, and remote control reply queues. > > - `control_queue_ttl`{.interpreted-text role="setting"} > > > Set message time-to-live for both remote control command > > queues, and remote control reply queues. > > Contributed by **Alan Justino**. - The `worker_shutdown`{.interpreted-text role="signal"} signal is now always called during shutdown. > Previously it would not be called if the worker instance was > collected by gc first. - Worker now only starts the remote control command consumer if the broker transport used actually supports them. - Gossip now sets `x-message-ttl` for event queue to heartbeat_interval s. (Issue #2005). - Now preserves exit code (Issue #2024). - Now rejects messages with an invalid ETA value (instead of ack, which means they will be sent to the dead-letter exchange if one is configured). - Fixed crash when the `-purge` argument was used. - Log\--level for unrecoverable errors changed from `error` to `critical`. - Improved rate limiting accuracy. - Account for missing timezone information in task expires field. > Fix contributed by **Albert Wang**. - The worker no longer has a `Queues` bootsteps, as it is now : superfluous. - Now emits the \"Received task\" line even for revoked tasks. (Issue #3155). - Now respects `broker_connection_retry`{.interpreted-text role="setting"} setting. > Fix contributed by **Nat Williams**. - New `control_queue_ttl`{.interpreted-text role="setting"} and `control_queue_expires`{.interpreted-text role="setting"} settings now enables you to configure remote control command message TTLs, and queue expiry time. > Contributed by **Alan Justino**. - New `celery.worker.state.requests`{.interpreted-text role="data"} enables O(1) loookup of active/reserved tasks by id. - Auto-scale didn\'t always update keep-alive when scaling down. > Fix contributed by **Philip Garnero**. - Fixed typo `options_list` -\> `option_list`. > Fix contributed by **Greg Wilbur**. - Some worker command-line arguments and `Worker()` class arguments have been renamed for consistency. > All of these have aliases for backward compatibility. > > - `--send-events` -\> `--task-events` > - `--schedule` -\> `--schedule-filename` > - `--maxtasksperchild` -\> `--max-tasks-per-child` > - `Beat(scheduler_cls=)` -\> `Beat(scheduler=)` > - `Worker(send_events=True)` -\> `Worker(task_events=True)` > - `Worker(task_time_limit=)` -\> `Worker(time_limit=`) > - `Worker(task_soft_time_limit=)` -\> `Worker(soft_time_limit=)` > - `Worker(state_db=)` -\> `Worker(statedb=)` > - `Worker(working_directory=)` -\> `Worker(workdir=)` #### Debugging Utilities - `celery.contrib.rdb`{.interpreted-text role="mod"}: Changed remote debugger banner so that you can copy and paste the address easily (no longer has a period in the address). > Contributed by **Jonathan Vanasco**. - Fixed compatibility with recent `psutil`{.interpreted-text role="pypi"} versions (Issue #3262). #### Signals - **App**: New signals for app configuration/finalization: > - `app.on_configure <@on_configure>`{.interpreted-text > role="data"} > - `app.on_after_configure <@on_after_configure>`{.interpreted-text > role="data"} > - `app.on_after_finalize <@on_after_finalize>`{.interpreted-text > role="data"} - **Task**: New task signals for rejected task messages: > - `celery.signals.task_rejected`{.interpreted-text role="data"}. > - `celery.signals.task_unknown`{.interpreted-text role="data"}. - **Worker**: New signal for when a heartbeat event is sent. > - `celery.signals.heartbeat_sent`{.interpreted-text role="data"} > > > Contributed by **Kevin Richardson**. #### Events - Event messages now uses the RabbitMQ `x-message-ttl` option to ensure older event messages are discarded. > The default is 5 seconds, but can be changed using the > `event_queue_ttl`{.interpreted-text role="setting"} setting. - `Task.send_event` now automatically retries sending the event on connection failure, according to the task publish retry settings. - Event monitors now sets the `event_queue_expires`{.interpreted-text role="setting"} setting by default. > The queues will now expire after 60 seconds after the monitor > stops consuming from it. - Fixed a bug where a None value wasn\'t handled properly. > Fix contributed by **Dongweiming**. - New `event_queue_prefix`{.interpreted-text role="setting"} setting can now be used to change the default `celeryev` queue prefix for event receiver queues. > Contributed by **Takeshi Kanemoto**. - `State.tasks_by_type` and `State.tasks_by_worker` can now be used as a mapping for fast access to this information. #### Deployment - Generic init-scripts now support `CELERY_SU`{.interpreted-text role="envvar"} and `CELERYD_SU_ARGS`{.interpreted-text role="envvar"} environment variables to set the path and arguments for `su`{.interpreted-text role="command"} (`su(1)`{.interpreted-text role="manpage"}). - Generic init-scripts now better support FreeBSD and other BSD systems by searching `/usr/local/etc/`{.interpreted-text role="file"} for the configuration file. > Contributed by **Taha Jahangir**. - Generic init-script: Fixed strange bug for `celerybeat` where restart didn\'t always work (Issue #3018). - The systemd init script now uses a shell when executing services. > Contributed by **Tomas Machalek**. #### Result Backends - Redis: Now has a default socket timeout of 120 seconds. > The default can be changed using the new > `redis_socket_timeout`{.interpreted-text role="setting"} setting. > > Contributed by **Raghuram Srinivasan**. - RPC Backend result queues are now auto delete by default (Issue #2001). - RPC Backend: Fixed problem where exception wasn\'t deserialized properly with the json serializer (Issue #2518). > Fix contributed by **Allard Hoeve**. - CouchDB: The backend used to double-json encode results. > Fix contributed by **Andrew Stewart**. - CouchDB: Fixed typo causing the backend to not be found (Issue #3287). > Fix contributed by **Andrew Stewart**. - MongoDB: Now supports setting the `result_serialzier`{.interpreted-text role="setting"} setting to `bson` to use the MongoDB libraries own serializer. > Contributed by **Davide Quarta**. - MongoDB: URI handling has been improved to use : database name, user and password from the URI if provided. Contributed by **Samuel Jaillet**. - SQLAlchemy result backend: Now ignores all result engine options when using NullPool (Issue #1930). - SQLAlchemy result backend: Now sets max char size to 155 to deal with brain damaged MySQL Unicode implementation (Issue #1748). - **General**: All Celery exceptions/warnings now inherit from common `~celery.exceptions.CeleryError`{.interpreted-text role="class"}/`~celery.exceptions.CeleryWarning`{.interpreted-text role="class"}. (Issue #2643). #### Documentation Improvements Contributed by: - Adam Chainz - Amir Rustamzadeh - Arthur Vuillard - Batiste Bieler - Berker Peksag - Bryce Groff - Daniel Devine - Edward Betts - Jason Veatch - Jeff Widman - Maciej Obuchowski - Manuel Kaufmann - Maxime Beauchemin - Mitchel Humpherys - Pavlo Kapyshin - Pierre Fersing - Rik - Steven Sklar - Tayfun Sen - Wieland Hoffmann ## Reorganization, Deprecations, and Removals ### Incompatible changes - Prefork: Calling `result.get()` or joining any result from within a task now raises `RuntimeError`{.interpreted-text role="exc"}. > In previous versions this would emit a warning. - `celery.worker.consumer`{.interpreted-text role="mod"} is now a package, not a module. - Module `celery.worker.job` renamed to `celery.worker.request`{.interpreted-text role="mod"}. - Beat: `Scheduler.Publisher`/`.publisher` renamed to `.Producer`/`.producer`. - Result: The task_name argument/attribute of `@AsyncResult`{.interpreted-text role="class"} was removed. > This was historically a field used for `pickle`{.interpreted-text > role="mod"} compatibility, but is no longer needed. - Backends: Arguments named `status` renamed to `state`. - Backends: `backend.get_status()` renamed to `backend.get_state()`. - Backends: `backend.maybe_reraise()` renamed to `.maybe_throw()` > The promise API uses .throw(), so this change was made to make it > more consistent. > > There\'s an alias available, so you can still use maybe_reraise > until Celery 5.0. ### Unscheduled Removals {#v400-unscheduled-removals} - The experimental `celery.contrib.methods`{.interpreted-text role="mod"} feature has been removed, as there were far many bugs in the implementation to be useful. - The CentOS init-scripts have been removed. > These didn\'t really add any features over the generic > init-scripts, so you\'re encouraged to use them instead, or > something like `supervisor`{.interpreted-text role="pypi"}. ### Reorganization Deprecations {#v400-deprecations-reorg} These symbols have been renamed, and while there\'s an alias available in this version for backward compatibility, they will be removed in Celery 5.0, so make sure you rename these ASAP to make sure it won\'t break for that release. Chances are that you\'ll only use the first in this list, but you never know: - `celery.utils.worker_direct` -\> `celery.utils.nodenames.worker_direct`{.interpreted-text role="meth"}. - `celery.utils.nodename` -\> `celery.utils.nodenames.nodename`{.interpreted-text role="meth"}. - `celery.utils.anon_nodename` -\> `celery.utils.nodenames.anon_nodename`{.interpreted-text role="meth"}. - `celery.utils.nodesplit` -\> `celery.utils.nodenames.nodesplit`{.interpreted-text role="meth"}. - `celery.utils.default_nodename` -\> `celery.utils.nodenames.default_nodename`{.interpreted-text role="meth"}. - `celery.utils.node_format` -\> `celery.utils.nodenames.node_format`{.interpreted-text role="meth"}. - `celery.utils.host_format` -\> `celery.utils.nodenames.host_format`{.interpreted-text role="meth"}. ### Scheduled Removals {#v400-removals} #### Modules - Module `celery.worker.job` has been renamed to `celery.worker.request`{.interpreted-text role="mod"}. > This was an internal module so shouldn\'t have any effect. It\'s > now part of the public API so must not change again. - Module `celery.task.trace` has been renamed to `celery.app.trace` as the `celery.task` package is being phased out. The module will be removed in version 5.0 so please change any import from: from celery.task.trace import X to: from celery.app.trace import X - Old compatibility aliases in the `celery.loaders`{.interpreted-text role="mod"} module has been removed. > - Removed `celery.loaders.current_loader()`, use: > `current_app.loader` > - Removed `celery.loaders.load_settings()`, use: > `current_app.conf` #### Result - `AsyncResult.serializable()` and `celery.result.from_serializable` : has been removed: Use instead: ``` pycon >>> tup = result.as_tuple() >>> from celery.result import result_from_tuple >>> result = result_from_tuple(tup) ``` - Removed `BaseAsyncResult`, use `AsyncResult` for instance checks instead. - Removed `TaskSetResult`, use `GroupResult` instead. > - `TaskSetResult.total` -\> `len(GroupResult)` > - `TaskSetResult.taskset_id` -\> `GroupResult.id` - Removed `ResultSet.subtasks`, use `ResultSet.results` instead. #### TaskSet TaskSet has been removed, as it was replaced by the `group` construct in Celery 3.0. If you have code like this: ``` pycon >>> from celery.task import TaskSet >>> TaskSet(add.subtask((i, i)) for i in xrange(10)).apply_async() ``` You need to replace that with: ``` pycon >>> from celery import group >>> group(add.s(i, i) for i in xrange(10))() ``` #### Events - Removals for class `celery.events.state.Worker`{.interpreted-text role="class"}: > - `Worker._defaults` attribute. > > > Use `{k: getattr(worker, k) for k in worker._fields}`. > > - `Worker.update_heartbeat` > > > Use `Worker.event(None, timestamp, received)` > > - `Worker.on_online` > > > Use `Worker.event('online', timestamp, received, fields)` > > - `Worker.on_offline` > > > Use `Worker.event('offline', timestamp, received, fields)` > > - `Worker.on_heartbeat` > > > Use `Worker.event('heartbeat', timestamp, received, fields)` - Removals for class `celery.events.state.Task`{.interpreted-text role="class"}: > - `Task._defaults` attribute. > > > Use `{k: getattr(task, k) for k in task._fields}`. > > - `Task.on_sent` > > > Use `Worker.event('sent', timestamp, received, fields)` > > - `Task.on_received` > > > Use `Task.event('received', timestamp, received, fields)` > > - `Task.on_started` > > > Use `Task.event('started', timestamp, received, fields)` > > - `Task.on_failed` > > > Use `Task.event('failed', timestamp, received, fields)` > > - `Task.on_retried` > > > Use `Task.event('retried', timestamp, received, fields)` > > - `Task.on_succeeded` > > > Use `Task.event('succeeded', timestamp, received, fields)` > > - `Task.on_revoked` > > > Use `Task.event('revoked', timestamp, received, fields)` > > - `Task.on_unknown_event` > > > Use `Task.event(short_type, timestamp, received, fields)` > > - `Task.update` > > > Use `Task.event(short_type, timestamp, received, fields)` > > - `Task.merge` > > > Contact us if you need this. #### Magic keyword arguments Support for the very old magic keyword arguments accepted by tasks is finally removed in this version. If you\'re still using these you have to rewrite any task still using the old `celery.decorators` module and depending on keyword arguments being passed to the task, for example: from celery.decorators import task @task() def add(x, y, task_id=None): print('My task id is %r' % (task_id,)) should be rewritten into: from celery import task @task(bind=True) def add(self, x, y): print('My task id is {0.request.id}'.format(self)) ### Removed Settings The following settings have been removed, and is no longer supported: #### Logging Settings **Setting name** **Replace with** ------------------------ ------------------------------------------------------------- `CELERYD_LOG_LEVEL` `celery worker --loglevel`{.interpreted-text role="option"} `CELERYD_LOG_FILE` `celery worker --logfile`{.interpreted-text role="option"} `CELERYBEAT_LOG_LEVEL` `celery beat --loglevel`{.interpreted-text role="option"} `CELERYBEAT_LOG_FILE` `celery beat --logfile`{.interpreted-text role="option"} `CELERYMON_LOG_LEVEL` celerymon is deprecated, use flower `CELERYMON_LOG_FILE` celerymon is deprecated, use flower `CELERYMON_LOG_FORMAT` celerymon is deprecated, use flower #### Task Settings **Setting name** **Replace with** --------------------------- ------------------ `CELERY_CHORD_PROPAGATES` N/A ### Changes to internal API - Module `celery.datastructures` renamed to `celery.utils.collections`{.interpreted-text role="mod"}. - Module `celery.utils.timeutils` renamed to `celery.utils.time`{.interpreted-text role="mod"}. - `celery.utils.datastructures.DependencyGraph` moved to `celery.utils.graph`{.interpreted-text role="mod"}. - `celery.utils.jsonify` is now `celery.utils.serialization.jsonify`{.interpreted-text role="func"}. - `celery.utils.strtobool` is now `celery.utils.serialization.strtobool`{.interpreted-text role="func"}. - `celery.utils.is_iterable` has been removed. > Instead use: > > ``` python > isinstance(x, collections.Iterable) > ``` - `celery.utils.lpmerge` is now `celery.utils.collections.lpmerge`{.interpreted-text role="func"}. - `celery.utils.cry` is now `celery.utils.debug.cry`{.interpreted-text role="func"}. - `celery.utils.isatty` is now `celery.platforms.isatty`{.interpreted-text role="func"}. - `celery.utils.gen_task_name` is now `celery.utils.imports.gen_task_name`{.interpreted-text role="func"}. - `celery.utils.deprecated` is now `celery.utils.deprecated.Callable`{.interpreted-text role="func"} - `celery.utils.deprecated_property` is now `celery.utils.deprecated.Property`{.interpreted-text role="func"}. - `celery.utils.warn_deprecated` is now `celery.utils.deprecated.warn`{.interpreted-text role="func"} ## Deprecation Time-line Changes {#v400-deprecations} See the `deprecation-timeline`{.interpreted-text role="ref"}. --- # What\'s new in Celery 4.1 (latentcall) {#whatsnew-4.1} Author : Omer Katz (`omer.drow at gmail.com`) ::: sidebar **Change history** What\'s new documents describe the changes in major versions, we also have a `changelog`{.interpreted-text role="ref"} that lists the changes in bugfix releases (0.0.x), while older series are archived under the `history`{.interpreted-text role="ref"} section. ::: Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. To read more about Celery you should go read the `introduction `{.interpreted-text role="ref"}. While this version is backward compatible with previous versions it\'s important that you read the following section. This version is officially supported on CPython 2.7, 3.4, 3.5 & 3.6 and is also supported on PyPy. ::: topic **Table of Contents** Make sure you read the important notes before upgrading to this version. ::: ::: {.contents local="" depth="2"} ::: ## Preface The 4.1.0 release continues to improve our efforts to provide you with the best task execution platform for Python. This release is mainly a bug fix release, ironing out some issues and regressions found in Celery 4.0.0. We added official support for Python 3.6 and PyPy 5.8.0. This is the first time we release without Ask Solem as an active contributor. We\'d like to thank him for his hard work in creating and maintaining Celery over the years. Since Ask Solem was not involved there were a few kinks in the release process which we promise to resolve in the next release. This document was missing when we did release Celery 4.1.0. Also, we did not update the release codename as we should have. We apologize for the inconvenience. For the time being, I, Omer Katz will be the release manager. Thank you for your support! *--- Omer Katz* ### Wall of Contributors Acey \<\> Acey9 \<\> Alan Hamlett \<\> Alan Justino da Silva \<\> Alejandro Pernin \<\> Alli \<\> Andreas Pelme \<\> Andrew de Quincey \<\> Anthony Lukach \<\> Arcadiy Ivanov \<\> Arnaud Rocher \<\> Arthur Vigil \<\> Asif Saifuddin Auvi \<\> Ask Solem \<\> BLAGA Razvan-Paul \<\> Brendan MacDonell \<\> Brian Luan \<\> Brian May \<\> Bruno Alla \<\> Chris Kuehl \<\> Christian \<\> Christopher Hoskin \<\> Daniel Hahler \<\> Daniel Huang \<\> Derek Harland \<\> Dmytro Petruk \<\> Ed Morley \<\> Eric Poelke \<\> Felipe \<\> François Voron \<\> GDR! \<\> George Psarakis \<\> J Alan Brogan \<\> James Michael DuPont \<\> Jamie Alessio \<\> Javier Domingo Cansino \<\> Jay McGrath \<\> Jian Yu \<\> Joey Wilhelm \<\> Jon Dufresne \<\> Kalle Bronsen \<\> Kirill Romanov \<\> Laurent Peuch \<\> Luke Plant \<\> Marat Sharafutdinov \<\> Marc Gibbons \<\> Marc Hörsken \<\> Michael \<\> Michael Howitz \<\> Michal Kuffa \<\> Mike Chen \<\> Mike Helmick \<\> Morgan Doocy \<\> Moussa Taifi \<\> Omer Katz \<\> Patrick Cloke \<\> Peter Bittner \<\> Preston Moore \<\> Primož Kerin \<\> Pysaoke \<\> Rick Wargo \<\> Rico Moorman \<\> Roman Sichny \<\> Ross Patterson \<\> Ryan Hiebert \<\> Rémi Marenco \<\> Salvatore Rinchiera \<\> Samuel Dion-Girardeau \<\> Sergey Fursov \<\> Simon Legner \<\> Simon Schmidt \<\> Slam \<<3lnc.slam@gmail.com>\> Static \<\> Steffen Allner \<\> Steven \<\> Steven Johns \<\> Tamer Sherif \<\> Tao Qingyun \<<845767657@qq.com>\> Tayfun Sen \<\> Taylor C. Richberger \<\> Thierry RAMORASOAVINA \<\> Tom \'Biwaa\' Riat \<\> Viktor Holmqvist \<\> Viraj \<\> Vivek Anand \<\> Will \<\> Wojciech Żywno \<\> Yoichi NAKAYAMA \<\> YuLun Shih \<\> Yuhannaa \<\> abhinav nilaratna \<\> aydin \<\> csfeathers \<\> georgepsarakis \<\> orf \<\> shalev67 \<\> sww \<\> tnir \<\> 何翔宇(Sean Ho) \<\> ::: note ::: title Note ::: This wall was automatically generated from git history, so sadly it doesn\'t not include the people who help with more important things like answering mailing-list questions. ::: ## Important Notes {#v410-important} ### Added support for Python 3.6 & PyPy 5.8.0 We now run our unit test suite and integration test suite on Python 3.6.x and PyPy 5.8.0. We expect newer versions of PyPy to work but unfortunately we do not have the resources to test PyPy with those versions. The supported Python Versions are: - CPython 2.7 - CPython 3.4 - CPython 3.5 - CPython 3.6 - PyPy 5.8 (`pypy2`) ## News {#v410-news} ### Result Backends #### New DynamoDB Results Backend We added a new results backend for those of you who are using DynamoDB. If you are interested in using this results backend, refer to `conf-dynamodb-result-backend`{.interpreted-text role="ref"} for more information. #### Elasticsearch The Elasticsearch results backend is now more robust and configurable. See `conf-elasticsearch-result-backend`{.interpreted-text role="ref"} for more information about the new configuration options. #### Redis The Redis results backend can now use TLS to encrypt the communication with the Redis database server. See `conf-redis-result-backend`{.interpreted-text role="ref"}. #### MongoDB The MongoDB results backend can now handle binary-encoded task results. This was a regression from 4.0.0 which resulted in a problem using serializers such as MsgPack or Pickle in conjunction with the MongoDB results backend. ### Periodic Tasks The task schedule now updates automatically when new tasks are added. Now if you use the Django database scheduler, you can add and remove tasks from the schedule without restarting Celery beat. ### Tasks The `disable_sync_subtasks` argument was added to allow users to override disabling synchronous subtasks. See `task-synchronous-subtasks`{.interpreted-text role="ref"} ### Canvas Multiple bugs were resolved resulting in a much smoother experience when using Canvas. --- # What\'s new in Celery 4.2 (windowlicker) {#whatsnew-4.2} Author : Omer Katz (`omer.drow at gmail.com`) ::: sidebar **Change history** What\'s new documents describe the changes in major versions, we also have a `changelog`{.interpreted-text role="ref"} that lists the changes in bugfix releases (0.0.x), while older series are archived under the `history`{.interpreted-text role="ref"} section. ::: Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. To read more about Celery you should go read the `introduction `{.interpreted-text role="ref"}. While this version is backward compatible with previous versions it\'s important that you read the following section. This version is officially supported on CPython 2.7, 3.4, 3.5 & 3.6 and is also supported on PyPy. ::: topic **Table of Contents** Make sure you read the important notes before upgrading to this version. ::: ::: {.contents local="" depth="2"} ::: ## Preface The 4.2.0 release continues to improve our efforts to provide you with the best task execution platform for Python. This release is mainly a bug fix release, ironing out some issues and regressions found in Celery 4.0.0. Traditionally, releases were named after [Autechre](https://en.wikipedia.org/wiki/Autechre)\'s track names. This release continues this tradition in a slightly different way. Each major version of Celery will use a different artist\'s track names as codenames. From now on, the 4.x series will be codenamed after [Aphex Twin](https://en.wikipedia.org/wiki/Aphex_Twin)\'s track names. This release is codenamed after his very famous track, [Windowlicker](https://youtu.be/UBS4Gi1y_nc?t=4m). Thank you for your support! *--- Omer Katz* ### Wall of Contributors Aaron Harnly \<\> Aaron Harnly \<\> Aaron McMillin \<\> Aaron Ross \<\> Aaron Ross \<\> Aaron Schumacher \<\> abecciu \<\> abhinav nilaratna \<\> Acey9 \<\> Acey \<\> aclowes \<\> Adam Chainz \<\> Adam DePue \<\> Adam Endicott \<\> Adam Renberg \<\> Adam Venturella \<\> Adaptification \<\> Adrian \<\> adriano petrich \<\> Adrian Rego \<\> Adrien Guinet \<\> Agris Ameriks \<\> Ahmet Demir \<\> air-upc \<\> Aitor Gómez-Goiri \<\> Akira Matsuzaki \<\> Akshar Raaj \<\> Alain Masiero \<\> Alan Hamlett \<\> Alan Hamlett \<\> Alan Justino \<\> Alan Justino da Silva \<\> Albert Wang \<\> Alcides Viamontes Esquivel \<\> Alec Clowes \<\> Alejandro Pernin \<\> Alejandro Varas \<\> Aleksandr Kuznetsov \<\> Ales Zoulek \<\> Alexander \<\> Alexander A. Sosnovskiy \<\> Alexander Koshelev \<\> Alexander Koval \<\> Alexander Oblovatniy \<\> Alexander Oblovatniy \<\> Alexander Ovechkin \<\> Alexander Smirnov \<\> Alexandru Chirila \<\> Alexey Kotlyarov \<\> Alexey Zatelepin \<\> Alex Garel \<\> Alex Hill \<\> Alex Kiriukha \<\> Alex Koshelev \<\> Alex Rattray \<\> Alex Williams \<\> Alex Zaitsev \<\> Ali Bozorgkhan \<\> Allan Caffee \<\> Allard Hoeve \<\> allenling \<\> Alli \<\> Alman One \<\> Alman One \<\> alman-one \<\> Amir Rustamzadeh \<\> \<\> Anarchist666 \<\> Anders Pearson \<\> Andrea Rabbaglietti \<\> Andreas Pelme \<\> Andreas Savvides \<\> Andrei Fokau \<\> Andrew de Quincey \<\> Andrew Kittredge \<\> Andrew McFague \<\> Andrew Stewart \<\> Andrew Watts \<\> Andrew Wong \<\> Andrey Voronov \<\> Andriy Yurchuk \<\> Aneil Mallavarapu \<\> anentropic \<\> anh \<\> Ankur Dedania \<\> Anthony Lukach \<\> antlegrand \<<2t.antoine@gmail.com>\> Antoine Legrand \<\> Anton \<\> Anton Gladkov \<\> Antonin Delpeuch \<\> Arcadiy Ivanov \<\> areski \<\> Armenak Baburyan \<\> Armin Ronacher \<\> armo \<\> Arnaud Rocher \<\> arpanshah29 \<\> Arsenio Santos \<\> Arthur Vigil \<\> Arthur Vuillard \<\> Ashish Dubey \<\> Asif Saifuddin Auvi \<\> Asif Saifuddin Auvi \<\> ask \<\> Ask Solem \<\> Ask Solem \<\> Ask Solem Hoel \<\> aydin \<\> baeuml \<\> Balachandran C \<\> Balthazar Rouberol \<\> Balthazar Rouberol \<\> bartloop \<<38962178+bartloop@users.noreply.github.com>\> Bartosz Ptaszynski \<\> Batiste Bieler \<\> bee-keeper \<\> Bence Tamas \<\> Ben Firshman \<\> Ben Welsh \<\> Berker Peksag \<\> Bert Vanderbauwhede \<\> Bert Vanderbauwhede \<\> BLAGA Razvan-Paul \<\> bobbybeever \<\> bobby \<\> Bobby Powers \<\> Bohdan Rybak \<\> Brad Jasper \<\> Branko Čibej \<\> BR \<\> Brendan MacDonell \<\> Brendon Crawford \<\> Brent Watson \<\> Brian Bouterse \<\> Brian Dixon \<\> Brian Luan \<\> Brian May \<\> Brian Peiris \<\> Brian Rosner \<\> Brodie Rao \<\> Bruno Alla \<\> Bryan Berg \<\> Bryan Berg \<\> Bryan Bishop \<\> Bryan Helmig \<\> Bryce Groff \<\> Caleb Mingle \<\> Carlos Garcia-Dubus \<\> Catalin Iacob \<\> Charles McLaughlin \<\> Chase Seibert \<\> ChillarAnand \<\> Chris Adams \<\> Chris Angove \<\> Chris Chamberlin \<\> chrisclark \<\> Chris Harris \<\> Chris Kuehl \<\> Chris Martin \<\> Chris Mitchell \<\> Chris Rose \<\> Chris St. Pierre \<\> Chris Streeter \<\> Christian \<\> Christoph Burgmer \<\> Christopher Hoskin \<\> Christopher Lee \<\> Christopher Peplin \<\> Christopher Peplin \<\> Christoph Krybus \<\> clayg \<\> Clay Gerrard \<.(none)\> Clemens Wolff \<\> cmclaughlin \<\> Codeb Fan \<\> Colin McIntosh \<\> Conrad Kramer \<\> Corey Farwell \<\> Craig Younkins \<\> csfeathers \<\> Cullen Rhodes \<\> daftshady \<\> Dan \<\> Dan Hackner \<\> Daniel Devine \<\> Daniele Procida \<\> Daniel Hahler \<\> Daniel Hepper \<\> Daniel Huang \<\> Daniel Lundin \<\> Daniel Lundin \<\> Daniel Watkins \<\> Danilo Bargen \<\> Dan McGee \<\> Dan McGee \<\> Dan Wilson \<\> Daodao \<\> Dave Smith \<\> Dave Smith \<\> David Arthur \<\> David Arthur \<\> David Baumgold \<\> David Cramer \<\> David Davis \<\> David Harrigan \<\> David Harrigan \<\> David Markey \<\> David Miller \<\> David Miller \<\> David Pravec \<\> David Pravec \<\> David Strauss \<\> David White \<\> DDevine \<\> Denis Podlesniy \<\> Denis Shirokov \<\> Dennis Brakhane \<\> Derek Harland \<\> derek_kim \<\> dessant \<\> Dieter Adriaenssens \<\> Dima Kurguzov \<\> dimka665 \<\> dimlev \<\> dmarkey \<\> Dmitry Malinovsky \<\> Dmitry Malinovsky \<\> dmollerm \<\> Dmytro Petruk \<\> dolugen \<\> dongweiming \<\> dongweiming \<\> Dongweiming \<\> dtheodor \<\> Dudás Ádám \<\> Dustin J. Mitchell \<\> D. Yu \<\> Ed Morley \<\> Eduardo Ramírez \<\> Edward Betts \<\> Emil Stanchev \<\> Eran Rundstein \<.(none)\> ergo \<\> Eric Poelke \<\> Eric Zarowny \<\> ernop \<\> Evgeniy \<\> evildmp \<\> fatihsucu \<\> Fatih Sucu \<\> Feanil Patel \<\> Felipe \<\> Felipe Godói Rosário \<\> Felix Berger \<\> Fengyuan Chen \<\> Fernando Rocha \<\> ffeast \<\> Flavio Percoco Premoli \<\> Florian Apolloner \<\> Florian Apolloner \<.(none)\> Florian Demmer \<\> flyingfoxlee \<\> Francois Visconte \<\> François Voron \<\> Frédéric Junod \<\> fredj \<\> frol \<\> Gabriel \<\> Gao Jiangmiao \<\> GDR! \<\> GDvalle \<\> Geoffrey Bauduin \<\> georgepsarakis \<\> George Psarakis \<\> George Sibble \<\> George Tantiras \<\> Georgy Cheshkov \<\> Gerald Manipon \<\> German M. Bravo \<\> Gert Van Gool \<\> Gilles Dartiguelongue \<\> Gino Ledesma \<\> gmanipon \<\> Grant Thomas \<\> Greg Haskins \<\> gregoire \<\> Greg Taylor \<\> Greg Wilbur \<\> Guillaume Gauvrit \<\> Guillaume Gendre \<\> Gun.io Whitespace Robot \<\> Gunnlaugur Thor Briem \<\> harm \<\> Harm Verhagen \<\> Harry Moreno \<\> hclihn \<<23141651+hclihn@users.noreply.github.com>\> hekevintran \<\> honux \<\> Honza Kral \<\> Honza Král \<\> Hooksie \<\> Hsiaoming Yang \<\> Huang Huang \<\> Hynek Schlawack \<\> Hynek Schlawack \<\> Ian Dees \<\> Ian McCracken \<\> Ian Wilson \<\> Idan Kamara \<\> Ignas Mikalajūnas \<\> Igor Kasianov \<\> illes \<\> Ilya \<<4beast@gmail.com>\> Ilya Georgievsky \<\> Ionel Cristian Mărieș \<\> Ionel Maries Cristian \<\> Ionut Turturica \<\> Iurii Kriachko \<\> Ivan Metzlar \<\> Ivan Virabyan \<\> j0hnsmith \<\> Jackie Leng \<\> J Alan Brogan \<\> Jameel Al-Aziz \<\> James M. Allen \<\> James Michael DuPont \<\> James Pulec \<\> James Remeika \<\> Jamie Alessio \<\> Jannis Leidel \<\> Jared Biel \<\> Jason Baker \<\> Jason Baker \<\> Jason Veatch \<\> Jasper Bryant-Greene \<\> Javier Domingo Cansino \<\> Javier Martin Montull \<\> Jay Farrimond \<\> Jay McGrath \<\> jbiel \<\> jbochi \<\> Jed Smith \<\> Jeff Balogh \<\> Jeff Balogh \<\> Jeff Terrace \<\> Jeff Widman \<\> Jelle Verstraaten \<\> Jeremy Cline \<\> Jeremy Zafran \<\> jerry \<\> Jerzy Kozera \<\> Jerzy Kozera \<\> jespern \<\> Jesper Noehr \<\> Jesse \<\> jess \<\> Jess Johnson \<\> Jian Yu \<\> JJ \<\> João Ricardo \<\> Jocelyn Delalande \<\> JocelynDelalande \<\> Joe Jevnik \<\> Joe Sanford \<\> Joe Sanford \<\> Joey Wilhelm \<\> John Anderson \<\> John Arnold \<\> John Barham \<\> John Watson \<\> John Watson \<\> John Watson \<\> John Whitlock \<\> Jonas Haag \<\> Jonas Obrist \<\> Jonatan Heyman \<\> Jonathan Jordan \<\> Jonathan Sundqvist \<\> jonathan vanasco \<\> Jon Chen \<\> Jon Dufresne \<\> Josh \<\> Josh Kupershmidt \<\> Joshua \"jag\" Ginsberg \<\> Josue Balandrano Coronel \<\> Jozef \<\> jpellerin \<.(none)\> jpellerin \<\> JP \<\> JTill \<\> Juan Gutierrez \<\> Juan Ignacio Catalano \<\> Juan Rossi \<\> Juarez Bochi \<\> Jude Nagurney \<\> Julien Deniau \<\> julienp \<\> Julien Poissonnier \<\> Jun Sakai \<\> Justin Patrin \<\> Justin Patrin \<\> Kalle Bronsen \<\> kamalgill \<\> Kamil Breguła \<\> Kanan Rahimov \<\> Kareem Zidane \<\> Keith Perkins \<\> Ken Fromm \<\> Ken Reese \<\> keves \<\> Kevin Gu \<\> Kevin Harvey \<\> Kevin McCarthy \<\> Kevin Richardson \<\> Kevin Richardson \<\> Kevin Tran \<\> Kieran Brownlees \<\> Kirill Pavlov \<\> Kirill Romanov \<\> komu \<\> Konstantinos Koukopoulos \<\> Konstantin Podshumok \<\> Kornelijus Survila \<\> Kouhei Maeda \<\> Kracekumar Ramaraju \<\> Krzysztof Bujniewicz \<\> kuno \<\> Kxrr \<\> Kyle Kelley \<\> Laurent Peuch \<\> lead2gold \<\> Leo Dirac \<\> Leo Singer \<\> Lewis M. Kabui \<\> llllllllll \<\> Locker537 \<\> Loic Bistuer \<\> Loisaida Sam \<\> lookfwd \<\> Loren Abrams \<\> Loren Abrams \<\> Lucas Wiman \<\> lucio \<\> Luis Clara Gomez \<\> Lukas Linhart \<\> Łukasz Kożuchowski \<\> Łukasz Langa \<\> Łukasz Oleś \<\> Luke Burden \<\> Luke Hutscal \<\> Luke Plant \<\> Luke Pomfrey \<\> Luke Zapart \<\> mabouels \<\> Maciej Obuchowski \<\> Mads Jensen \<\> Manuel Kaufmann \<\> Manuel Vázquez Acosta \<\> Marat Sharafutdinov \<\> Marcelo Da Cruz Pinto \<\> Marc Gibbons \<\> Marc Hörsken \<\> Marcin Kuźmiński \<\> marcinkuzminski \<\> Marcio Ribeiro \<\> Marco Buttu \<\> Marco Schweighauser \<\> mariia-zelenova \<<32500603+mariia-zelenova@users.noreply.github.com>\> Marin Atanasov Nikolov \<\> Marius Gedminas \<\> mark hellewell \<\> Mark Lavin \<\> Mark Lavin \<\> Mark Parncutt \<\> Mark Story \<\> Mark Stover \<\> Mark Thurman \<\> Markus Kaiserswerth \<\> Markus Ullmann \<\> martialp \<\> Martin Davidsson \<\> Martin Galpin \<\> Martin Melin \<\> Matt Davis \<\> Matthew Duggan \<\> Matthew J Morrison \<\> Matthew Miller \<\> Matthew Schinckel \<\> mattlong \<\> Matt Long \<\> Matt Robenolt \<\> Matt Robenolt \<\> Matt Williamson \<\> Matt Williamson \<\> Matt Wise \<\> Matt Woodyard \<\> Mauro Rocco \<\> Maxim Bodyansky \<.(none)\> Maxime Beauchemin \<\> Maxime Vdb \<\> Mayflower \<\> mbacho \<\> mher \<\> Mher Movsisyan \<\> Michael Aquilina \<\> Michael Duane Mooring \<\> Michael Elsdoerfer \<.(none)\> Michael Elsdorfer \<\> Michael Elsdörfer \<\> Michael Fladischer \<\> Michael Floering \<\> Michael Howitz \<\> michael \<\> Michael \<\> michael \<.(none)\> Michael Peake \<\> Michael Permana \<\> Michael Permana \<\> Michael Robellard \<\> Michael Robellard \<\> Michal Kuffa \<\> Miguel Hernandez Martos \<\> Mike Attwood \<\> Mike Chen \<\> Mike Helmick \<\> mikemccabe \<\> Mikhail Gusarov \<\> Mikhail Korobov \<\> Mikołaj \<\> Milen Pavlov \<\> Misha Wolfson \<\> Mitar \<\> Mitar \<\> Mitchel Humpherys \<\> mklauber \<\> mlissner \<\> monkut \<\> Morgan Doocy \<\> Morris Tweed \<\> Morton Fox \<\> Môshe van der Sterre \<\> Moussa Taifi \<\> mozillazg \<\> mpavlov \<\> mperice \<\> mrmmm \<\> Muneyuki Noguchi \<\> m-vdb \<\> nadad \<\> Nathaniel Varona \<\> Nathan Van Gheem \<\> Nat Williams \<\> Neil Chintomby \<\> Neil Chintomby \<\> Nicholas Pilon \<\> nicholsonjf \<\> Nick Eaket \<<4418194+neaket360pi@users.noreply.github.com>\> Nick Johnson \<\> Nicolas Mota \<\> nicolasunravel \<\> Niklas Aldergren \<\> Noah Kantrowitz \<\> Noel Remy \<\> NoKriK \<\> Norman Richards \<\> NotSqrt \<\> nott \<\> ocean1 \<\> ocean1 \<\> ocean1 \<\> OddBloke \<\> Oleg Anashkin \<\> Olivier Aubert \<\> Omar Khan \<\> Omer Katz \<\> Omer Korner \<\> orarbel \<\> orf \<\> Ori Hoch \<\> outself \<\> Pablo Marti \<\> pachewise \<\> partizan \<\> Pär Wieslander \<\> Patrick Altman \<\> Patrick Cloke \<\> Patrick \<\> Patrick Stegmann \<\> Patrick Stegmann \<\> Patrick Zhang \<\> Paul English \<\> Paul Jensen \<\> Paul Kilgo \<\> Paul McMillan \<\> Paul McMillan \<\> Paulo \<\> Paul Pearce \<\> Pavel Savchenko \<\> Pavlo Kapyshin \<\> pegler \<\> Pepijn de Vos \<\> Peter Bittner \<\> Peter Brook \<\> Philip Garnero \<\> Pierre Fersing \<\> Piotr Maślanka \<\> Piotr Sikora \<\> PMickael \<\> PMickael \<\> Polina Giralt \<\> precious \<\> Preston Moore \<\> Primož Kerin \<\> Pysaoke \<\> Rachel Johnson \<\> Rachel Willmer \<\> raducc \<\> Raf Geens \<\> Raghuram Srinivasan \<\> Raphaël Riel \<\> Raphaël Slinckx \<\> Régis B \<\> Remigiusz Modrzejewski \<\> Rémi Marenco \<\> rfkrocktk \<\> Rick van Hattem \<\> Rick Wargo \<\> Rico Moorman \<\> Rik \<\> Rinat Shigapov \<\> Riyad Parvez \<\> rlotun \<\> rnoel \<\> Robert Knight \<\> Roberto Gaiser \<\> roderick \<\> Rodolphe Quiedeville \<\> Roger Hu \<\> Roger Hu \<\> Roman Imankulov \<\> Roman Sichny \<\> Romuald Brunet \<\> Ronan Amicel \<\> Ross Deane \<\> Ross Lawley \<\> Ross Patterson \<\> Ross \<\> Rudy Attias \<\> rumyana neykova \<\> Rumyana Neykova \<\> Rune Halvorsen \<\> Rune Halvorsen \<.(none)\> runeh \<.(none)\> Russell Keith-Magee \<\> Ryan Guest \<\> Ryan Hiebert \<\> Ryan Kelly \<\> Ryan Luckie \<\> Ryan Petrello \<\> Ryan P. Kelly \<\> Ryan P Kilby \<\> Salvatore Rinchiera \<\> Sam Cooke \<\> samjy \<\> Sammie S. Taunton \<\> Samuel Dion-Girardeau \<\> Samuel Dion-Girardeau \<\> Samuel GIFFARD \<\> Scott Cooper \<\> screeley \<.(none)\> sdcooke \<\> Sean O\'Connor \<\> Sean Wang \<\> Sebastian Kalinowski \<\> Sébastien Fievet \<\> Seong Won Mun \<\> Sergey Fursov \<\> Sergey Tikhonov \<\> Sergi Almacellas Abellana \<\> Sergio Fernandez \<\> Seungha Kim \<\> shalev67 \<\> Shitikanth \<\> Silas Sewell \<\> Simon Charette \<\> Simon Engledew \<\> Simon Josi \<\> Simon Legner \<\> Simon Peeters \<\> Simon Schmidt \<\> skovorodkin \<\> Slam \<<3lnc.slam@gmail.com>\> Smirl \<\> squfrans \<\> Srinivas Garlapati \<\> Stas Rudakou \<\> Static \<\> Steeve Morin \<\> Stefan hr Berder \<\> Stefan Kjartansson \<\> Steffen Allner \<\> Stephen Weber \<\> Steven Johns \<\> Steven Parker \<\> Steven \<\> Steven Sklar \<\> Steven Skoczen \<\> Steven Skoczen \<\> Steve Peak \<\> stipa \<\> sukrit007 \<\> Sukrit Khera \<\> Sundar Raman \<\> sunfinite \<\> sww \<\> Tadej Janež \<\> Taha Jahangir \<\> Takeshi Kanemoto \<\> TakesxiSximada \<\> Tamer Sherif \<\> Tao Qingyun \<<845767657@qq.com>\> Tarun Bhardwaj \<\> Tayfun Sen \<\> Tayfun Sen \<\> Tayfun Sen \<\> tayfun \<\> Taylor C. Richberger \<\> taylornelson \<\> Theodore Dubois \<\> Theo Spears \<\> Thierry RAMORASOAVINA \<\> Thijs Triemstra \<\> Thomas French \<\> Thomas Grainger \<\> Thomas Johansson \<\> Thomas Meson \<\> Thomas Minor \<\> Thomas Wright \<\> Timo Sugliani \<\> Timo Sugliani \<.(none)\> Titusz \<\> tnir \<\> Tobias Kunze \<\> Tocho Tochev \<\> Tomas Machalek \<\> Tomasz Święcicki \<\> Tom \'Biwaa\' Riat \<\> Tomek Święcicki \<\> Tom S \<\> tothegump \<\> Travis Swicegood \<\> Travis Swicegood \<\> Travis \<\> Trevor Skaggs \<\> Ujjwal Ojha \<\> unknown \ Valentyn Klindukh \<\> Viktor Holmqvist \<\> Vincent Barbaresi \<\> Vincent Driessen \<\> Vinod Chandru \<\> Viraj \<\> Vitaly Babiy \<\> Vitaly \<\> Vivek Anand \<\> Vlad \<\> Vladimir Gorbunov \<\> Vladimir Kryachko \<\> Vladimir Rutsky \<\> Vladislav Stepanov \<<8uk.8ak@gmail.com>\> Vsevolod \<\> Wes Turner \<\> wes \<\> Wes Winham \<\> w- \<\> whendrik \<\> Wido den Hollander \<\> Wieland Hoffmann \<\> Wiliam Souza \<\> Wil Langford \<\> William King \<\> Will \<\> Will Thompson \<\> winhamwr \<\> Wojciech Żywno \<\> W. Trevor King \<\> wyc \<\> wyc \<\> xando \<\> Xavier Damman \<\> Xavier Hardy \<\> Xavier Ordoquy \<\> xin li \<\> xray7224 \<\> y0ngdi \<<36658095+y0ngdi@users.noreply.github.com>\> Yan Kalchevskiy \<\> Yohann Rebattu \<\> Yoichi NAKAYAMA \<\> Yuhannaa \<\> YuLun Shih \<\> Yury V. Zaytsev \<\> Yuval Greenfield \<\> Zach Smith \<\> Zhang Chi \<\> Zhaorong Ma \<\> Zoran Pavlovic \<\> ztlpn \<\> 何翔宇(Sean Ho) \<\> 許邱翔 \<\> ::: note ::: title Note ::: This wall was automatically generated from git history, so sadly it doesn\'t not include the people who help with more important things like answering mailing-list questions. ::: ## Important Notes {#v420-important} ### Supported Python Versions The supported Python Versions are: - CPython 2.7 - CPython 3.4 - CPython 3.5 - CPython 3.6 - PyPy 5.8 (`pypy2`) ## News {#v420-news} ### Result Backends #### New Redis Sentinel Results Backend Redis Sentinel provides high availability for Redis. A new result backend supporting it was added. #### Cassandra Results Backend A new [cassandra_options]{.title-ref} configuration option was introduced in order to configure the cassandra client. See `conf-cassandra-result-backend`{.interpreted-text role="ref"} for more information. #### DynamoDB Results Backend A new [dynamodb_endpoint_url]{.title-ref} configuration option was introduced in order to point the result backend to a local endpoint during development or testing. See `conf-dynamodb-result-backend`{.interpreted-text role="ref"} for more information. #### Python 2/3 Compatibility Fixes Both the CouchDB and the Consul result backends accepted byte strings without decoding them to Unicode first. This is now no longer the case. ### Canvas Multiple bugs were resolved resulting in a much smoother experience when using Canvas. ### Tasks #### Bound Tasks as Error Callbacks We fixed a regression that occurred when bound tasks are used as error callbacks. This used to work in Celery 3.x but raised an exception in 4.x until this release. In both 4.0 and 4.1 the following code wouldn\'t work: ``` python @app.task(name="raise_exception", bind=True) def raise_exception(self): raise Exception("Bad things happened") @app.task(name="handle_task_exception", bind=True) def handle_task_exception(self): print("Exception detected") subtask = raise_exception.subtask() subtask.apply_async(link_error=handle_task_exception.s()) ``` #### Task Representation - Shadowing task names now works as expected. The shadowed name is properly presented in flower, the logs and the traces. - [argsrepr]{.title-ref} and [kwargsrepr]{.title-ref} were previously not used even if specified. They now work as expected. See `task-hiding-sensitive-information`{.interpreted-text role="ref"} for more information. #### Custom Requests We now allow tasks to use custom `request `{.interpreted-text role="class"} classes for custom task classes. See `task-requests-and-custom-requests`{.interpreted-text role="ref"} for more information. #### Retries with Exponential Backoff Retries can now be performed with exponential backoffs to avoid overwhelming external services with requests. See `task-autoretry`{.interpreted-text role="ref"} for more information. ### Sphinx Extension Tasks were supposed to be automatically documented when using Sphinx\'s Autodoc was used. The code that would have allowed automatic documentation had a few bugs which are now fixed. Also, The extension is now documented properly. See `sphinx`{.interpreted-text role="ref"} for more information. --- # What\'s new in Celery 4.3 (rhubarb) {#whatsnew-4.3} Author : Omer Katz (`omer.drow at gmail.com`) ::: sidebar **Change history** What\'s new documents describe the changes in major versions, we also have a `changelog`{.interpreted-text role="ref"} that lists the changes in bugfix releases (0.0.x), while older series are archived under the `history`{.interpreted-text role="ref"} section. ::: Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. To read more about Celery you should go read the `introduction `{.interpreted-text role="ref"}. While this version is backward compatible with previous versions it\'s important that you read the following section. This version is officially supported on CPython 2.7, 3.4, 3.5, 3.6 & 3.7 and is also supported on PyPy2 & PyPy3. ::: topic **Table of Contents** Make sure you read the important notes before upgrading to this version. ::: ::: {.contents local="" depth="2"} ::: ## Preface The 4.3.0 release continues to improve our efforts to provide you with the best task execution platform for Python. This release has been codenamed [Rhubarb](https://www.youtube.com/watch?v=_AWIqXzvX-U) which is one of my favorite tracks from Selected Ambient Works II. This release focuses on new features like new result backends and a revamped security serializer along with bug fixes mainly for Celery Beat, Canvas, a number of critical fixes for hanging workers and fixes for several severe memory leaks. Celery 4.3 is the first release to support Python 3.7. We hope that 4.3 will be the last release to support Python 2.7 as we now begin to work on Celery 5, the next generation of our task execution platform. However, if Celery 5 will be delayed for any reason we may release another 4.x minor version which will still support Python 2.7. If another 4.x version will be released it will most likely drop support for Python 3.4 as it will reach it\'s EOL in March 2019. We have also focused on reducing contribution friction. Thanks to **Josue Balandrano Coronel**, one of our core contributors, we now have an updated `contributing`{.interpreted-text role="ref"} document. If you intend to contribute, please review it at your earliest convenience. I have also added new issue templates, which we will continue to improve, so that the issues you open will have more relevant information which will allow us to help you to resolve them more easily. *--- Omer Katz* ### Wall of Contributors Alexander Ioannidis \<\> Amir Hossein Saeid Mehr \<\> Andrea Rabbaglietti \<\> Andrey Skabelin \<\> Anthony Ruhier \<\> Antonin Delpeuch \<\> Artem Vasilyev \<\> Asif Saif Uddin (Auvi) \<\> aviadatsnyk \<\> Axel Haustant \<\> Bojan Jovanovic \<\> Brett Jackson \<\> Brett Randall \<\> Brian Schrader \<\> Bruno Alla \<\> Buddy \<<34044521+CoffeeExpress@users.noreply.github.com>\> Charles Chan \<\> Christopher Dignam \<\> Ciaran Courtney \<<6096029+ciarancourtney@users.noreply.github.com>\> Clemens Wolff \<\> Colin Watson \<\> Daniel Hahler \<\> Dash Winterson \<\> Derek Harland \<\> Dilip Vamsi Moturi \<<16288600+dilipvamsi@users.noreply.github.com>\> Dmytro Litvinov \<\> Douglas Rohde \<\> Ed Morley \<<501702+edmorley@users.noreply.github.com>\> Fabian Becker \<\> Federico Bond \<\> Fengyuan Chen \<\> Florian CHARDIN \<\> George Psarakis \<\> Guilherme Caminha \<\> ideascf \<\> Itay \<\> Jamie Alessio \<\> Jason Held \<\> Jeremy Cohen \<\> John Arnold \<\> Jon Banafato \<\> Jon Dufresne \<\> Joshua Engelman \<\> Joshua Schmid \<\> Josue Balandrano Coronel \<\> K Davis \<\> kidoz \<\> Kiyohiro Yamaguchi \<\> Korijn van Golen \<\> Lars Kruse \<\> Lars Rinn \<\> Lewis M. Kabui \<\> madprogrammer \<\> Manuel Vázquez Acosta \<\> Marcus McHale \<\> Mariatta \<\> Mario Kostelac \<\> Matt Wiens \<\> Maximilien Cuony \<\> Maximilien de Bayser \<\> Meysam \<\> Milind Shakya \<\> na387 \<\> Nicholas Pilon \<\> Nick Parsons \<\> Nik Molnar \<\> Noah Hall \<\> Noam \<\> Omer Katz \<\> Paweł Adamczak \<\> peng weikang \<\> Prathamesh Salunkhe \<\> Przemysław Suliga \<<1270737+suligap@users.noreply.github.com>\> Raf Geens \<\> (◕ᴥ◕) \<\> Robert Kopaczewski \<\> Samuel Huang \<\> Sebastian Wojciechowski \<<42519683+sebwoj@users.noreply.github.com>\> Seunghun Lee \<\> Shanavas M \<\> Simon Charette \<\> Simon Schmidt \<\> srafehi \<\> Steven Sklar \<\> Tom Booth \<\> Tom Clancy \<\> Toni Ruža \<\> tothegump \<\> Victor Mireyev \<\> Vikas Prasad \<\> walterqian \<\> Willem \<\> Xiaodong \<\> yywing \<<386542536@qq.com>\> ::: note ::: title Note ::: This wall was automatically generated from git history, so sadly it doesn\'t not include the people who help with more important things like answering mailing-list questions. ::: ## Upgrading from Celery 4.2 Please read the important notes below as there are several breaking changes. ## Important Notes {#v430-important} ### Supported Python Versions The supported Python Versions are: - CPython 2.7 - CPython 3.4 - CPython 3.5 - CPython 3.6 - CPython 3.7 - PyPy2.7 6.0 (`pypy2`) - PyPy3.5 6.0 (`pypy3`) ### Kombu Starting from this release, the minimum required version is Kombu 4.4. #### New Compression Algorithms Kombu 4.3 includes a few new optional compression methods: - LZMA (available from stdlib if using Python 3 or from a backported package) - Brotli (available if you install either the brotli or the brotlipy package) - ZStandard (available if you install the zstandard package) Unfortunately our current protocol generates huge payloads for complex canvases. Until we migrate to our 3rd revision of the Celery protocol in Celery 5 which will resolve this issue, please use one of the new compression methods as a workaround. See `calling-compression`{.interpreted-text role="ref"} for details. ### Billiard Starting from this release, the minimum required version is Billiard 3.6. ### Eventlet Workers Pool We now require [eventlet\>=0.24.1]{.title-ref}. If you are using the eventlet workers pool please install Celery using: ``` console $ pip install -U celery[eventlet] ``` ### MessagePack Serializer We\'ve been using the deprecated [msgpack-python]{.title-ref} package for a while. This is now fixed as we depend on the [msgpack]{.title-ref} instead. If you are currently using the MessagePack serializer please uninstall the previous package and reinstall the new one using: ``` console $ pip uninstall msgpack-python -y $ pip install -U celery[msgpack] ``` ### MongoDB Result Backend We now support the [DNS seedlist connection format](https://docs.mongodb.com/manual/reference/connection-string/#dns-seedlist-connection-format) for the MongoDB result backend. This requires the [dnspython]{.title-ref} package. If you are using the MongoDB result backend please install Celery using: ``` console $ pip install -U celery[mongodb] ``` ### Redis Message Broker Due to multiple bugs in earlier versions of py-redis that were causing issues for Celery, we were forced to bump the minimum required version to 3.2.0. ### Redis Result Backend Due to multiple bugs in earlier versions of py-redis that were causing issues for Celery, we were forced to bump the minimum required version to 3.2.0. ### Riak Result Backend The official Riak client does not support Python 3.7 as of yet. In case you are using the Riak result backend, either attempt to install the client from master or avoid upgrading to Python 3.7 until this matter is resolved. In case you are using the Riak result backend with Python 3.7, we now emit a warning. Please track [basho/riak-python-client#534](https://github.com/basho/riak-python-client/issues/534) for updates. ### Dropped Support for RabbitMQ 2.x Starting from this release, we officially no longer support RabbitMQ 2.x. The last release of 2.x was in 2012 and we had to make adjustments to correctly support high availability on RabbitMQ 3.x. If for some reason, you are still using RabbitMQ 2.x we encourage you to upgrade as soon as possible since security patches are no longer applied on RabbitMQ 2.x. ### Django Support Starting from this release, the minimum required Django version is 1.11. ### Revamped auth Serializer The auth serializer received a complete overhaul. It was previously horribly broken. We now depend on [cryptography]{.title-ref} instead of [pyOpenSSL]{.title-ref} for this serializer. See `message-signing`{.interpreted-text role="ref"} for details. ## News {#v430-news} ### Brokers #### Redis Broker Support for SSL URIs The Redis broker now has support for SSL connections. You can use `broker_use_ssl`{.interpreted-text role="setting"} as you normally did and use a [rediss://]{.title-ref} URI. You can also pass the SSL configuration parameters to the URI: > [rediss://localhost:3456?ssl_keyfile=keyfile.key&ssl_certfile=certificate.crt&ssl_ca_certs=ca.pem&ssl_cert_reqs=CERT_REQUIRED]{.title-ref} #### Configurable Events Exchange Name Previously, the events exchange name was hardcoded. You can use `event_exchange`{.interpreted-text role="setting"} to determine it. The default value remains the same. #### Configurable Pidbox Exchange Name Previously, the Pidbox exchange name was hardcoded. You can use `control_exchange`{.interpreted-text role="setting"} to determine it. The default value remains the same. ### Result Backends #### Redis Result Backend Support for SSL URIs The Redis result backend now has support for SSL connections. You can use `redis_backend_use_ssl`{.interpreted-text role="setting"} to configure it and use a [rediss://]{.title-ref} URI. You can also pass the SSL configuration parameters to the URI: > [rediss://localhost:3456?ssl_keyfile=keyfile.key&ssl_certfile=certificate.crt&ssl_ca_certs=ca.pem&ssl_cert_reqs=CERT_REQUIRED]{.title-ref} #### Store Extended Task Metadata in Result When `result_extended`{.interpreted-text role="setting"} is [True]{.title-ref} the backend will store the following metadata: - Task Name - Arguments - Keyword arguments - The worker the task was executed on - Number of retries - The queue\'s name or routing key In addition, `celery.app.task.update_state`{.interpreted-text role="meth"} now accepts keyword arguments which allows you to store custom data with the result. #### Encode Results Using A Different Serializer The `result_accept_content`{.interpreted-text role="setting"} setting allows to configure different accepted content for the result backend. A special serializer ([auth]{.title-ref}) is used for signed messaging, however the result_serializer remains in json, because we don\'t want encrypted content in our result backend. To accept unsigned content from the result backend, we introduced this new configuration option to specify the accepted content from the backend. #### New Result Backends This release introduces four new result backends: > - S3 result backend > - ArangoDB result backend > - Azure Block Blob Storage result backend > - CosmosDB result backend #### S3 Result Backend Amazon Simple Storage Service (Amazon S3) is an object storage service by AWS. The results are stored using the following path template: | \<`s3_bucket`{.interpreted-text role="setting"}\>/\<`s3_base_path`{.interpreted-text role="setting"}\>/\ See `conf-s3-result-backend`{.interpreted-text role="ref"} for more information. #### ArangoDB Result Backend ArangoDB is a native multi-model database with search capabilities. The backend stores the result in the following document format: |  { |    \_key: {key}, |    task: {task} |  } See `conf-arangodb-result-backend`{.interpreted-text role="ref"} for more information. #### Azure Block Blob Storage Result Backend Azure Block Blob Storage is an object storage service by Microsoft. The backend stores the result in the following path template: | \<`azureblockblob_container_name`{.interpreted-text role="setting"}\>/\ See `conf-azureblockblob-result-backend`{.interpreted-text role="ref"} for more information. #### CosmosDB Result Backend Azure Cosmos DB is Microsoft\'s globally distributed, multi-model database service. The backend stores the result in the following document format: |  { |    id: {key}, |    value: {task} |  } See `conf-cosmosdbsql-result-backend`{.interpreted-text role="ref"} for more information. ### Tasks #### Cythonized Tasks Cythonized tasks are now supported. You can generate C code from Cython that specifies a task using the [\@task]{.title-ref} decorator and everything should work exactly the same. #### Acknowledging Tasks on Failures or Timeouts When `task_acks_late`{.interpreted-text role="setting"} is set to [True]{.title-ref} tasks are acknowledged on failures or timeouts. This makes it hard to use dead letter queues and exchanges. Celery 4.3 introduces the new `task_acks_on_failure_or_timeout`{.interpreted-text role="setting"} which allows you to avoid acknowledging tasks if they failed or timed out even if `task_acks_late`{.interpreted-text role="setting"} is set to [True]{.title-ref}. `task_acks_on_failure_or_timeout`{.interpreted-text role="setting"} is set to [True]{.title-ref} by default. #### Schedules Now Support Microseconds When scheduling tasks using `celery beat`{.interpreted-text role="program"} microseconds are no longer ignored. #### Default Task Priority You can now set the default priority of a task using the `task_default_priority`{.interpreted-text role="setting"} setting. The setting\'s value will be used if no priority is provided for a specific task. #### Tasks Optionally Inherit Parent\'s Priority Setting the `task_inherit_parent_priority`{.interpreted-text role="setting"} configuration option to [True]{.title-ref} will make Celery tasks inherit the priority of the previous task linked to it. Examples: ``` python c = celery.chain( add.s(2), # priority=None add.s(3).set(priority=5), # priority=5 add.s(4), # priority=5 add.s(5).set(priority=3), # priority=3 add.s(6), # priority=3 ) ``` ``` python @app.task(bind=True) def child_task(self): pass @app.task(bind=True) def parent_task(self): child_task.delay() # child_task will also have priority=5 parent_task.apply_async(args=[], priority=5) ``` ### Canvas #### Chords can be Executed in Eager Mode When `task_always_eager`{.interpreted-text role="setting"} is set to [True]{.title-ref}, chords are executed eagerly as well. #### Configurable Chord Join Timeout Previously, `celery.result.GroupResult.join`{.interpreted-text role="meth"} had a fixed timeout of 3 seconds. The `result_chord_join_timeout`{.interpreted-text role="setting"} setting now allows you to change it. The default remains 3 seconds. --- # What\'s new in Celery 4.4 (Cliffs) {#whatsnew-4.4} Author : Asif Saif Uddin (`auvipy at gmail.com`) ::: sidebar **Change history** What\'s new documents describe the changes in major versions, we also have a `changelog`{.interpreted-text role="ref"} that lists the changes in bugfix releases (0.0.x), while older series are archived under the `history`{.interpreted-text role="ref"} section. ::: Celery is a simple, flexible, and reliable distributed programming framework to process vast amounts of messages, while providing operations with the tools required to maintain a distributed system with python. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. To read more about Celery you should go read the `introduction `{.interpreted-text role="ref"}. While this version is backward compatible with previous versions it\'s important that you read the following section. This version is officially supported on CPython 2.7, 3.5, 3.6, 3.7 & 3.8 and is also supported on PyPy2 & PyPy3. ::: topic **Table of Contents** Make sure you read the important notes before upgrading to this version. ::: ::: {.contents local="" depth="2"} ::: ## Preface The 4.4.0 release continues to improve our efforts to provide you with the best task execution platform for Python. This release has been codenamed [Cliffs](https://www.youtube.com/watch?v=i524g6JMkwI) which is one of my favorite tracks. This release focuses on mostly bug fixes and usability improvement for developers. Many long standing bugs, usability issues, documentation issues & minor enhancement issues were squashed which improve the overall developers experience. Celery 4.4 is the first release to support Python 3.8 & pypy36-7.2. As we now begin to work on Celery 5, the next generation of our task execution platform, at least another 4.x is expected before Celery 5 stable release & will get support for at least 1 years depending on community demand and support. We have also focused on reducing contribution friction and updated the contributing tools. *--- Asif Saif Uddin* ### Wall of Contributors ::: note ::: title Note ::: This wall was automatically generated from git history, so sadly it doesn\'t not include the people who help with more important things like answering mailing-list questions. ::: ## Upgrading from Celery 4.3 Please read the important notes below as there are several breaking changes. ## Important Notes {#v440-important} ### Supported Python Versions The supported Python Versions are: - CPython 2.7 - CPython 3.5 - CPython 3.6 - CPython 3.7 - CPython 3.8 - PyPy2.7 7.2 (`pypy2`) - PyPy3.5 7.1 (`pypy3`) - PyPy3.6 7.2 (`pypy3`) ### Dropped support for Python 3.4 Celery now requires either Python 2.7 or Python 3.5 and above. Python 3.4 has reached EOL in March 2019. In order to focus our efforts we have dropped support for Python 3.4 in this version. If you still require to run Celery using Python 3.4 you can still use Celery 4.3. However we encourage you to upgrade to a supported Python version since no further security patches will be applied for Python 3.4. ### Kombu Starting from this release, the minimum required version is Kombu 4.6.6. ### Billiard Starting from this release, the minimum required version is Billiard 3.6.1. ### Redis Message Broker Due to multiple bugs in earlier versions of redis-py that were causing issues for Celery, we were forced to bump the minimum required version to 3.3.0. ### Redis Result Backend Due to multiple bugs in earlier versions of redis-py that were causing issues for Celery, we were forced to bump the minimum required version to 3.3.0. ### DynamoDB Result Backend The DynamoDB result backend has gained TTL support. As a result the minimum boto3 version was bumped to 1.9.178 which is the first version to support TTL for DynamoDB. ### S3 Results Backend To keep up with the current AWS API changes the minimum boto3 version was bumped to 1.9.125. ### SQS Message Broker To keep up with the current AWS API changes the minimum boto3 version was bumped to 1.9.125. ### Configuration [CELERY_TASK_RESULT_EXPIRES]{.title-ref} has been replaced with [CELERY_RESULT_EXPIRES]{.title-ref}. ## News {#v440-news} ### Task Pools #### Threaded Tasks Pool We reintroduced a threaded task pool using [concurrent.futures.ThreadPoolExecutor]{.title-ref}. The previous threaded task pool was experimental. In addition it was based on the [threadpool](https://pypi.org/project/threadpool/) package which is obsolete. You can use the new threaded task pool by setting `worker_pool`{.interpreted-text role="setting"} to \'threads\` or by passing [\--pool threads]{.title-ref} to the [celery worker]{.title-ref} command. ### Result Backends #### ElasticSearch Results Backend ##### HTTP Basic Authentication Support You can now use HTTP Basic Authentication when using the ElasticSearch result backend by providing the username and the password in the URI. Previously, they were ignored and only unauthenticated requests were issued. #### MongoDB Results Backend ##### Support for Authentication Source and Authentication Method You can now specify the authSource and authMethod for the MongoDB using the URI options. The following URI does just that: > `mongodb://user:password@example.com/?authSource=the_database&authMechanism=SCRAM-SHA-256` Refer to the [documentation](https://api.mongodb.com/python/current/examples/authentication.html) for details about the various options. ### Tasks #### Task class definitions can now have retry attributes You can now use [autoretry_for]{.title-ref}, [retry_kwargs]{.title-ref}, [retry_backoff]{.title-ref}, [retry_backoff_max]{.title-ref} and [retry_jitter]{.title-ref} in class-based tasks: ``` python class BaseTaskWithRetry(Task): autoretry_for = (TypeError,) retry_kwargs = {'max_retries': 5} retry_backoff = True retry_backoff_max = 700 retry_jitter = False ``` ### Canvas #### Replacing Tasks Eagerly You can now call [self.replace()]{.title-ref} on tasks which are run eagerly. They will work exactly the same as tasks which are run asynchronously. #### Chaining Groups Chaining groups no longer result in a single group. The following used to join the two groups into one. Now they correctly execute one after another: >>> result = group(add.si(1, 2), add.si(1, 2)) | group(tsum.s(), tsum.s()).delay() >>> result.get() [6, 6] --- # What\'s new in Celery 5.0 (singularity) {#whatsnew-5.0} Author : Omer Katz (`omer.drow at gmail.com`) ::: sidebar **Change history** What\'s new documents describe the changes in major versions, we also have a `changelog`{.interpreted-text role="ref"} that lists the changes in bugfix releases (0.0.x), while older series are archived under the `history`{.interpreted-text role="ref"} section. ::: Celery is a simple, flexible, and reliable distributed programming framework to process vast amounts of messages, while providing operations with the tools required to maintain a distributed system with python. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. To read more about Celery you should go read the `introduction `{.interpreted-text role="ref"}. While this version is **mostly** backward compatible with previous versions it\'s important that you read the following section as this release is a new major version. This version is officially supported on CPython 3.6, 3.7 & 3.8 and is also supported on PyPy3. ::: topic **Table of Contents** Make sure you read the important notes before upgrading to this version. ::: ::: {.contents local="" depth="2"} ::: ## Preface The 5.0.0 release is a new major release for Celery. Starting from now users should expect more frequent releases of major versions as we move fast and break things to bring you even better experience. Releases in the 5.x series are codenamed after songs of [Jon Hopkins](https://en.wikipedia.org/wiki/Jon_Hopkins). This release has been codenamed [Singularity](https://www.youtube.com/watch?v=lkvnpHFajt0). This version drops support for Python 2.7.x which has reached EOL in January 1st, 2020. This allows us, the maintainers to focus on innovating without worrying for backwards compatibility. From now on we only support Python 3.6 and above. We will maintain compatibility with Python 3.6 until it\'s EOL in December, 2021. *--- Omer Katz* ### Long Term Support Policy As we\'d like to provide some time for you to transition, we\'re designating Celery 4.x an LTS release. Celery 4.x will be supported until the 1st of August, 2021. We will accept and apply patches for bug fixes and security issues. However, no new features will be merged for that version. Celery 5.x **is not** an LTS release. We will support it until the release of Celery 6.x. We\'re in the process of defining our Long Term Support policy. Watch the next \"What\'s New\" document for updates. ### Wall of Contributors Artem Vasilyev \<\> Ash Berlin-Taylor \<\> Asif Saif Uddin (Auvi) \<\> Asif Saif Uddin \<\> Christian Clauss \<\> Germain Chazot \<\> Harry Moreno \<\> kevinbai \<\> Martin Paulus \<\> Matus Valo \<\> Matus Valo \<\> maybe-sybr \<<58414429+maybe-sybr@users.noreply.github.com>\> Omer Katz \<\> Patrick Cloke \<\> qiaocc \<\> Thomas Grainger \<\> Weiliang Li \<\> ::: note ::: title Note ::: This wall was automatically generated from git history, so sadly it doesn\'t not include the people who help with more important things like answering mailing-list questions. ::: ## Upgrading from Celery 4.x ### Step 1: Adjust your command line invocation Celery 5.0 introduces a new CLI implementation which isn\'t completely backwards compatible. The global options can no longer be positioned after the sub-command. Instead, they must be positioned as an option for the [celery]{.title-ref} command like so: celery --app path.to.app worker If you were using our `daemonizing`{.interpreted-text role="ref"} guide to deploy Celery in production, you should revisit it for updates. ### Step 2: Update your configuration with the new setting names If you haven\'t already updated your configuration when you migrated to Celery 4.0, please do so now. We elected to extend the deprecation period until 6.0 since we did not loudly warn about using these deprecated settings. Please refer to the `migration guide `{.interpreted-text role="ref"} for instructions. ### Step 3: Read the important notes in this document Make sure you are not affected by any of the important upgrade notes mentioned in the `following section `{.interpreted-text role="ref"}. You should mainly verify that any of the breaking changes in the CLI do not affect you. Please refer to `New Command Line Interface `{.interpreted-text role="ref"} for details. ### Step 4: Migrate your code to Python 3 Celery 5.0 supports only Python 3. Therefore, you must ensure your code is compatible with Python 3. If you haven\'t ported your code to Python 3, you must do so before upgrading. You can use tools like [2to3](https://docs.python.org/3.8/library/2to3.html) and [pyupgrade](https://github.com/asottile/pyupgrade) to assist you with this effort. After the migration is done, run your test suite with Celery 4 to ensure nothing has been broken. ### Step 5: Upgrade to Celery 5.0 At this point you can upgrade your workers and clients with the new version. ## Important Notes {#v500-important} ### Supported Python Versions The supported Python Versions are: - CPython 3.6 - CPython 3.7 - CPython 3.8 - PyPy3.6 7.2 (`pypy3`) ### Dropped support for Python 2.7 & 3.5 Celery now requires Python 3.6 and above. Python 2.7 has reached EOL in January 2020. In order to focus our efforts we have dropped support for Python 2.7 in this version. In addition, Python 3.5 has reached EOL in September 2020. Therefore, we are also dropping support for Python 3.5. If you still require to run Celery using Python 2.7 or Python 3.5 you can still use Celery 4.x. However we encourage you to upgrade to a supported Python version since no further security patches will be applied for Python 2.7 and as mentioned Python 3.5 is not supported for practical reasons. ### Kombu Starting from this release, the minimum required version is Kombu 5.0.0. ### Billiard Starting from this release, the minimum required version is Billiard 3.6.3. ### Eventlet Workers Pool Due to [eventlet/eventlet#526](https://github.com/eventlet/eventlet/issues/526) the minimum required version is eventlet 0.26.1. ### Gevent Workers Pool Starting from this release, the minimum required version is gevent 1.0.0. ### Couchbase Result Backend The Couchbase result backend now uses the V3 Couchbase SDK. As a result, we no longer support Couchbase Server 5.x. Also, starting from this release, the minimum required version for the database client is couchbase 3.0.0. To verify that your Couchbase Server is compatible with the V3 SDK, please refer to their [documentation](https://docs.couchbase.com/python-sdk/3.0/project-docs/compatibility.html). ### Riak Result Backend The Riak result backend has been removed as the database is no longer maintained. The Python client only supports Python 3.6 and below which prevents us from supporting it and it is also unmaintained. If you are still using Riak, refrain from upgrading to Celery 5.0 while you migrate your application to a different database. We apologize for the lack of notice in advance but we feel that the chance you\'ll be affected by this breaking change is minimal which is why we did it. ### AMQP Result Backend The AMQP result backend has been removed as it was deprecated in version 4.0. ### Removed Deprecated Modules The [celery.utils.encoding]{.title-ref} and the [celery.task]{.title-ref} modules has been deprecated in version 4.0 and therefore are removed in 5.0. If you were using the [celery.utils.encoding]{.title-ref} module before, you should import [kombu.utils.encoding]{.title-ref} instead. If you were using the [celery.task]{.title-ref} module before, you should import directly from the [celery]{.title-ref} module instead. If you were using [from celery.task import Task]{.title-ref} you should use [from celery import Task]{.title-ref} instead. If you were using the [celery.task]{.title-ref} decorator you should use [celery.shared_task]{.title-ref} instead. ### New Command Line Interface {#new_command_line_interface} The command line interface has been revamped using Click. As a result a few breaking changes has been introduced: - Postfix global options like [celery worker \--app path.to.app]{.title-ref} or [celery worker \--workdir /path/to/workdir]{.title-ref} are no longer supported. You should specify them as part of the global options of the main celery command. - `celery amqp`{.interpreted-text role="program"} and `celery shell`{.interpreted-text role="program"} require the [repl]{.title-ref} sub command to start a shell. You can now also invoke specific commands without a shell. Type [celery amqp \--help]{.title-ref} or [celery shell \--help]{.title-ref} for details. - The API for adding user options has changed. Refer to the `documentation `{.interpreted-text role="ref"} for details. Click provides shell completion [out of the box](https://click.palletsprojects.com/en/7.x/bashcomplete/). This functionality replaces our previous bash completion script and adds completion support for the zsh and fish shells. The bash completion script was exported to [extras/celery.bash](https://github.com/celery/celery/blob/master/extra/bash-completion/celery.bash) for the packager\'s convenience. ### Pytest Integration Starting from Celery 5.0, the pytest plugin is no longer enabled by default. Please refer to the `documentation `{.interpreted-text role="ref"} for instructions. ### Ordered Group Results for the Redis Result Backend Previously group results were not ordered by their invocation order. Celery 4.4.7 introduced an opt-in feature to make them ordered. It is now an opt-out behavior. If you were previously using the Redis result backend, you might need to opt-out of this behavior. Please refer to the `documentation `{.interpreted-text role="ref"} for instructions on how to disable this feature. ## News {#v500-news} ### Retry Policy for the Redis Result Backend The retry policy for the Redis result backend is now exposed through the result backend transport options. Please refer to the `documentation `{.interpreted-text role="ref"} for details. --- # What\'s new in Celery 5.1 (Sun Harmonics) {#whatsnew-5.1} Author : Josue Balandrano Coronel (`jbc at rmcomplexity.com`) ::: sidebar **Change history** What\'s new documents describe the changes in major versions, we also have a `changelog`{.interpreted-text role="ref"} that lists the changes in bugfix releases (0.0.x), while older series are archived under the `history`{.interpreted-text role="ref"} section. ::: Celery is a simple, flexible, and reliable distributed programming framework to process vast amounts of messages, while providing operations with the tools required to maintain a distributed system with python. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. To read more about Celery you should go read the `introduction `{.interpreted-text role="ref"}. While this version is **mostly** backward compatible with previous versions it\'s important that you read the following section as this release is a new major version. This version is officially supported on CPython 3.6, 3.7 & 3.8 & 3.9 and is also supported on PyPy3. ::: topic **Table of Contents** Make sure you read the important notes before upgrading to this version. ::: ::: {.contents local="" depth="2"} ::: ## Preface The 5.1.0 release is a new minor release for Celery. Starting from now users should expect more frequent releases of major versions as we move fast and break things to bring you even better experience. Releases in the 5.x series are codenamed after songs of [Jon Hopkins](https://en.wikipedia.org/wiki/Jon_Hopkins). This release has been codenamed [Sun Harmonics](https://www.youtube.com/watch?v=pCwjSoBm_pI). From now on we only support Python 3.6 and above. We will maintain compatibility with Python 3.6 until it\'s EOL in December, 2021. *--- Omer Katz* ### Long Term Support Policy As we\'d like to provide some time for you to transition, we\'re designating Celery 4.x an LTS release. Celery 4.x will be supported until the 1st of August, 2021. We will accept and apply patches for bug fixes and security issues. However, no new features will be merged for that version. Celery 5.x **is not** an LTS release. We will support it until the release of Celery 6.x. We\'re in the process of defining our Long Term Support policy. Watch the next \"What\'s New\" document for updates. ### Wall of Contributors 0xflotus \<<0xflotus@gmail.com>\> AbdealiJK \<\> Anatoliy \<\> Anna Borzenko \<\> aruseni \<\> Asif Saif Uddin (Auvi) \<\> Asif Saif Uddin \<\> Awais Qureshi \<\> careljonkhout \<\> Christian Clauss \<\> danthegoodman1 \<\> Dave Johansen \<\> David Schneider \<\> Fahmi \<\> Felix Yan \<\> Gabriel Augendre \<\> galcohen \<\> gal cohen \<\> Geunsik Lim \<\> Guillaume DE SUSANNE D\'EPINAY \<\> Hilmar Hilmarsson \<\> Illia Volochii \<\> jenhaoyang \<\> Jonathan Stoppani \<\> Josue Balandrano Coronel \<\> kosarchuksn \<\> Kostya Deev \<\> Matt Hoffman \<\> Matus Valo \<\> Myeongseok Seo \<\> Noam \<\> Omer Katz \<\> pavlos kallis \<\> Pavol Plaskoň \<\> Pengjie Song (宋鹏捷) \<\> Sardorbek Imomaliev \<\> Sergey Lyapustin \<\> Sergey Tikhonov \<\> Stephen J. Fuhry \<\> Swen Kooij \<\> tned73 \<\> Tomas Hrnciar \<\> tumb1er \<\> ::: note ::: title Note ::: This wall was automatically generated from git history, so sadly it doesn\'t not include the people who help with more important things like answering mailing-list questions. ::: ## Upgrading from Celery 4.x ### Step 1: Adjust your command line invocation Celery 5.0 introduces a new CLI implementation which isn\'t completely backwards compatible. The global options can no longer be positioned after the sub-command. Instead, they must be positioned as an option for the [celery]{.title-ref} command like so: celery --app path.to.app worker If you were using our `daemonizing`{.interpreted-text role="ref"} guide to deploy Celery in production, you should revisit it for updates. ### Step 2: Update your configuration with the new setting names If you haven\'t already updated your configuration when you migrated to Celery 4.0, please do so now. We elected to extend the deprecation period until 6.0 since we did not loudly warn about using these deprecated settings. Please refer to the `migration guide `{.interpreted-text role="ref"} for instructions. ### Step 3: Read the important notes in this document Make sure you are not affected by any of the important upgrade notes mentioned in the `following section `{.interpreted-text role="ref"}. You should verify that none of the breaking changes in the CLI do not affect you. Please refer to `New Command Line Interface `{.interpreted-text role="ref"} for details. ### Step 4: Migrate your code to Python 3 Celery 5.x only supports Python 3. Therefore, you must ensure your code is compatible with Python 3. If you haven\'t ported your code to Python 3, you must do so before upgrading. You can use tools like [2to3](https://docs.python.org/3.8/library/2to3.html) and [pyupgrade](https://github.com/asottile/pyupgrade) to assist you with this effort. After the migration is done, run your test suite with Celery 4 to ensure nothing has been broken. ### Step 5: Upgrade to Celery 5.1 At this point you can upgrade your workers and clients with the new version. ## Important Notes {#v510-important} ### Supported Python Versions The supported Python Versions are: - CPython 3.6 - CPython 3.7 - CPython 3.8 - CPython 3.9 - PyPy3.6 7.2 (`pypy3`) ### Important Notes #### Kombu Starting from v5.1, the minimum required version is Kombu 5.1.0. #### Py-AMQP Starting from Celery 5.1, py-amqp will always validate certificates received from the server and it is no longer required to manually set `cert_reqs` to `ssl.CERT_REQUIRED`. The previous default, `ssl.CERT_NONE` is insecure and we its usage should be discouraged. If you\'d like to revert to the previous insecure default set `cert_reqs` to `ssl.CERT_NONE` ``` python import ssl broker_use_ssl = { 'keyfile': '/var/ssl/private/worker-key.pem', 'certfile': '/var/ssl/amqp-server-cert.pem', 'ca_certs': '/var/ssl/myca.pem', 'cert_reqs': ssl.CERT_NONE } ``` #### Billiard Starting from v5.1, the minimum required version is Billiard 3.6.4. ### Important Notes From 5.0 #### Dropped support for Python 2.7 & 3.5 Celery now requires Python 3.6 and above. Python 2.7 has reached EOL in January 2020. In order to focus our efforts we have dropped support for Python 2.7 in this version. In addition, Python 3.5 has reached EOL in September 2020. Therefore, we are also dropping support for Python 3.5. If you still require to run Celery using Python 2.7 or Python 3.5 you can still use Celery 4.x. However we encourage you to upgrade to a supported Python version since no further security patches will be applied for Python 2.7 or Python 3.5. #### Eventlet Workers Pool Due to [eventlet/eventlet#526](https://github.com/eventlet/eventlet/issues/526) the minimum required version is eventlet 0.26.1. #### Gevent Workers Pool Starting from v5.0, the minimum required version is gevent 1.0.0. #### Couchbase Result Backend The Couchbase result backend now uses the V3 Couchbase SDK. As a result, we no longer support Couchbase Server 5.x. Also, starting from v5.0, the minimum required version for the database client is couchbase 3.0.0. To verify that your Couchbase Server is compatible with the V3 SDK, please refer to their [documentation](https://docs.couchbase.com/python-sdk/3.0/project-docs/compatibility.html). #### Riak Result Backend The Riak result backend has been removed as the database is no longer maintained. The Python client only supports Python 3.6 and below which prevents us from supporting it and it is also unmaintained. If you are still using Riak, refrain from upgrading to Celery 5.0 while you migrate your application to a different database. We apologize for the lack of notice in advance but we feel that the chance you\'ll be affected by this breaking change is minimal which is why we did it. #### AMQP Result Backend The AMQP result backend has been removed as it was deprecated in version 4.0. #### Removed Deprecated Modules The [celery.utils.encoding]{.title-ref} and the [celery.task]{.title-ref} modules has been deprecated in version 4.0 and therefore are removed in 5.0. If you were using the [celery.utils.encoding]{.title-ref} module before, you should import [kombu.utils.encoding]{.title-ref} instead. If you were using the [celery.task]{.title-ref} module before, you should import directly from the [celery]{.title-ref} module instead. If you were using [from celery.task import Task]{.title-ref} you should use [from celery import Task]{.title-ref} instead. If you were using the [celery.task]{.title-ref} decorator you should use [celery.shared_task]{.title-ref} instead. ### [azure-servicebus]{.title-ref} 7.0.0 is now required Given the SDK changes between 0.50.0 and 7.0.0 Kombu deprecates support for older [azure-servicebus]{.title-ref} versions. ## News {#v510-news} ### Support for Azure Service Bus 7.0.0 With Kombu v5.1.0 we now support Azure Services Bus. Azure have completely changed the Azure ServiceBus SDK between 0.50.0 and 7.0.0. [azure-servicebus \>= 7.0.0]{.title-ref} is now required for Kombu [5.1.0]{.title-ref} ### Add support for SQLAlchemy 1.4 Following the changes in SQLAlchemy 1.4, the declarative base is no longer an extension. Importing it from sqlalchemy.ext.declarative is deprecated and will be removed in SQLAlchemy 2.0. ### Support for Redis username authentication Previously, the username was ignored from the URI. Starting from Redis\>=6.0, that shouldn\'t be the case since ACL support has landed. Please refer to the `documentation `{.interpreted-text role="ref"} for details. ### SQS transport - support back off policy SQS now supports managed visibility timeout. This lets us implement a back off policy (for instance, an exponential policy) which means that the time between task failures will dynamically change based on the number of retries. Documentation: `kombu:reference/kombu.transport.SQS`{.interpreted-text role="doc"} ### Duplicate successful tasks The trace function fetches the metadata from the backend each time it receives a task and compares its state. If the state is SUCCESS, we log and bail instead of executing the task. The task is acknowledged and everything proceeds normally. Documentation: `worker_deduplicate_successful_tasks`{.interpreted-text role="setting"} ### Terminate tasks with late acknowledgment on connection loss Tasks with late acknowledgement keep running after restart, although the connection is lost and they cannot be acknowledged anymore. These tasks will now be terminated. Documentation: `worker_cancel_long_running_tasks_on_connection_loss`{.interpreted-text role="setting"} ### [task.apply_async(ignore_result=True)]{.title-ref} now avoids persisting the result [task.apply_async]{.title-ref} now supports passing [ignore_result]{.title-ref} which will act the same as using `@app.task(ignore_result=True)`. ### Use a thread-safe implementation of [cached_property]{.title-ref} [cached_property]{.title-ref} is heavily used in celery but it is causing issues in multi-threaded code since it is not thread safe. Celery is now using a thread-safe implementation of [cached_property]{.title-ref}. ### Tasks can now have required kwargs at any order Tasks can now be defined like this: ``` python from celery import shared_task @shared_task def my_func(*, name='default', age, city='Kyiv'): pass ``` ### SQS - support STS authentication with AWS The STS token requires a refresh after a certain period of time. After [sts_token_timeout]{.title-ref} is reached, a new token will be created. Documentation: `/getting-started/backends-and-brokers/sqs`{.interpreted-text role="doc"} ### Support Redis [health_check_interval]{.title-ref} [health_check_interval]{.title-ref} can be configured and will be passed to [redis-py]{.title-ref}. Documentation: `redis_backend_health_check_interval`{.interpreted-text role="setting"} ### Update default pickle protocol version to 4 The pickle protocol version was updated to allow Celery to serialize larger strings among other benefits. See: ### Support Redis Sentinel with SSL See documentation for more info: `/getting-started/backends-and-brokers/redis`{.interpreted-text role="doc"} --- # What\'s new in Celery 5.3 (Emerald Rush) {#whatsnew-5.3} Author : Asif Saif Uddin (`auvipy at gmail.com`). ::: sidebar **Change history** What\'s new documents describe the changes in major versions, we also have a `changelog`{.interpreted-text role="ref"} that lists the changes in bugfix releases (0.0.x), while older series are archived under the `history`{.interpreted-text role="ref"} section. ::: Celery is a simple, flexible, and reliable distributed programming framework to process vast amounts of messages, while providing operations with the tools required to maintain a distributed system with python. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. ::: note ::: title Note ::: Following the problems with Freenode, we migrated our IRC channel to Libera Chat as most projects did. You can also join us using [Gitter](https://gitter.im/celery/celery). We\'re sometimes there to answer questions. We welcome you to join. ::: To read more about Celery you should go read the `introduction `{.interpreted-text role="ref"}. While this version is **mostly** backward compatible with previous versions it\'s important that you read the following section as this release is a new major version. This version is officially supported on CPython 3.8, 3.9 & 3.10 and is also supported on PyPy3.8+. ::: topic **Table of Contents** Make sure you read the important notes before upgrading to this version. ::: ::: {.contents local="" depth="2"} ::: ## Preface ::: note ::: title Note ::: **This release contains fixes for many long standing bugs & stability issues. We encourage our users to upgrade to this release as soon as possible.** ::: The 5.3.0 release is a new feature release for Celery. Releases in the 5.x series are codenamed after songs of [Jon Hopkins](https://en.wikipedia.org/wiki/Jon_Hopkins). This release has been codenamed [Emerald Rush](https://www.youtube.com/watch?v=4sk0uDbM5lc). From now on we only support Python 3.8 and above. We will maintain compatibility with Python 3.8 until it\'s EOL in 2024. *--- Asif Saif Uddin* ### Long Term Support Policy We no longer support Celery 4.x as we don\'t have the resources to do so. If you\'d like to help us, all contributions are welcome. Celery 5.x **is not** an LTS release. We will support it until the release of Celery 6.x. We\'re in the process of defining our Long Term Support policy. Watch the next \"What\'s New\" document for updates. ### Wall of Contributors ::: note ::: title Note ::: This wall was automatically generated from git history, so sadly it doesn\'t not include the people who help with more important things like answering mailing-list questions. ::: ## Upgrading from Celery 4.x ### Step 1: Adjust your command line invocation Celery 5.0 introduces a new CLI implementation which isn\'t completely backwards compatible. The global options can no longer be positioned after the sub-command. Instead, they must be positioned as an option for the [celery]{.title-ref} command like so: celery --app path.to.app worker If you were using our `daemonizing`{.interpreted-text role="ref"} guide to deploy Celery in production, you should revisit it for updates. ### Step 2: Update your configuration with the new setting names If you haven\'t already updated your configuration when you migrated to Celery 4.0, please do so now. We elected to extend the deprecation period until 6.0 since we did not loudly warn about using these deprecated settings. Please refer to the `migration guide `{.interpreted-text role="ref"} for instructions. ### Step 3: Read the important notes in this document Make sure you are not affected by any of the important upgrade notes mentioned in the `following section `{.interpreted-text role="ref"}. You should verify that none of the breaking changes in the CLI do not affect you. Please refer to `New Command Line Interface `{.interpreted-text role="ref"} for details. ### Step 4: Migrate your code to Python 3 Celery 5.x only supports Python 3. Therefore, you must ensure your code is compatible with Python 3. If you haven\'t ported your code to Python 3, you must do so before upgrading. You can use tools like [2to3](https://docs.python.org/3.8/library/2to3.html) and [pyupgrade](https://github.com/asottile/pyupgrade) to assist you with this effort. After the migration is done, run your test suite with Celery 4 to ensure nothing has been broken. ### Step 5: Upgrade to Celery 5.3 At this point you can upgrade your workers and clients with the new version. ## Important Notes {#v530-important} ### Supported Python Versions The supported Python versions are: - CPython 3.8 - CPython 3.9 - CPython 3.10 - PyPy3.8 7.3.11 (`pypy3`) #### Experimental support Celery supports these Python versions provisionally as they are not production ready yet: - CPython 3.11 ### Quality Improvements and Stability Enhancements Celery 5.3 focuses on elevating the overall quality and stability of the project. We have dedicated significant efforts to address various bugs, enhance performance, and make improvements based on valuable user feedback. ### Better Compatibility and Upgrade Confidence Our goal with Celery 5.3 is to instill confidence in users who are currently using Celery 4 or older versions. We want to assure you that upgrading to Celery 5.3 will provide a more robust and reliable experience. ### Dropped support for Python 3.7 Celery now requires Python 3.8 and above. Python 3.7 will reach EOL in June, 2023. In order to focus our efforts we have dropped support for Python 3.6 in this version. If you still require to run Celery using Python 3.7 you can still use Celery 5.2. However we encourage you to upgrade to a supported Python version since no further security patches will be applied for Python 3.7 after the 23th of June, 2023. ### Automatic re-connection on connection loss to broker Unless `broker_connection_retry_on_startup`{.interpreted-text role="setting"} is set to False, Celery will automatically retry reconnecting to the broker after the first connection loss. `broker_connection_retry`{.interpreted-text role="setting"} controls whether to automatically retry reconnecting to the broker for subsequent reconnects. Since the message broker does not track how many tasks were already fetched before the connection was lost, Celery will reduce the prefetch count by the number of tasks that are currently running multiplied by `worker_prefetch_multiplier`{.interpreted-text role="setting"}. The prefetch count will be gradually restored to the maximum allowed after each time a task that was running before the connection was lost is complete ### Kombu Starting from v5.3.0, the minimum required version is Kombu 5.3.0. ### Redis redis-py 4.5.x is the new minimum required version. ### SQLAlchemy SQLAlchemy 1.4.x & 2.0.x is now supported in celery v5.3 ### Billiard Minimum required version is now 4.1.0 ### Deprecate pytz and use zoneinfo A switch have been made to zoneinfo for handling timezone data instead of pytz. #### Support for out-of-tree worker pool implementations Prior to version 5.3, Celery had a fixed notion of the worker pool types it supports. Celery v5.3.0 introduces the the possibility of an out-of-tree worker pool implementation. This feature ensure that the current worker pool implementations consistently call into BasePool.\_get_info(), and enhance it to report the work pool class in use via the \"celery inspect stats\" command. For example: \$ celery -A \... inspect stats -\> : OK { \... \"pool\": { \... \"implementation\": \"celery_aio_pool.pool:AsyncIOPool\", It can be used as follows: > Set the environment variable CELERY_CUSTOM_WORKER_POOL to the name of > an implementation of :class:celery.concurrency.base.BasePool in the > standard Celery format of \"package:class\". > > Select this pool using \'\--pool custom\'. ### Signal::`worker_before_create_process` Dispatched in the parent process, just before new child process is created in the prefork pool. It can be used to clean up instances that don\'t behave well when forking. ``` python @signals.worker_before_create_process.connect def clean_channels(**kwargs): grpc_singleton.clean_channel() ``` ### Setting::`beat_cron_starting_deadline` When using cron, the number of seconds `~celery.bin.beat`{.interpreted-text role="mod"} can look back when deciding whether a cron schedule is due. When set to [None]{.title-ref}, cronjobs that are past due will always run immediately. ### Redis result backend Global keyprefix The global key prefix will be prepended to all keys used for the result backend, which can be useful when a redis database is shared by different users. By default, no prefix is prepended. To configure the global keyprefix for the Redis result backend, use the `global_keyprefix` key under `result_backend_transport_options`{.interpreted-text role="setting"}: ``` python app.conf.result_backend_transport_options = { 'global_keyprefix': 'my_prefix_' } ``` ### Django Minimum django version is bumped to v2.2.28. Also added \--skip-checks flag to bypass django core checks. ### Make default worker state limits configurable Previously, [REVOKES_MAX]{.title-ref}, [REVOKE_EXPIRES]{.title-ref}, [SUCCESSFUL_MAX]{.title-ref} and [SUCCESSFUL_EXPIRES]{.title-ref} were hardcoded in [celery.worker.state]{.title-ref}. This version introduces [CELERY_WORKER\_]{.title-ref} prefixed environment variables with the same names that allow you to customize these values should you need to. ### Canvas stamping The goal of the Stamping API is to give an ability to label the signature and its components for debugging information purposes. For example, when the canvas is a complex structure, it may be necessary to label some or all elements of the formed structure. The complexity increases even more when nested groups are rolled-out or chain elements are replaced. In such cases, it may be necessary to understand which group an element is a part of or on what nested level it is. This requires a mechanism that traverses the canvas elements and marks them with specific metadata. The stamping API allows doing that based on the Visitor pattern. ### Known Issues Canvas header stamping has issues in a hybrid Celery 4.x. & Celery 5.3.x environment and is not safe for production use at the moment. --- # What\'s new in Celery 5.4 (Opalescent) {#whatsnew-5.4} Author : Tomer Nosrati (`tomer.nosrati at gmail.com`). ::: sidebar **Change history** What\'s new documents describe the changes in major versions, we also have a `changelog`{.interpreted-text role="ref"} that lists the changes in bugfix releases (0.0.x), while older series are archived under the `history`{.interpreted-text role="ref"} section. ::: Celery is a simple, flexible, and reliable distributed programming framework to process vast amounts of messages, while providing operations with the tools required to maintain a distributed system with python. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. ::: note ::: title Note ::: Following the problems with Freenode, we migrated our IRC channel to Libera Chat as most projects did. You can also join us using [Gitter](https://gitter.im/celery/celery). We\'re sometimes there to answer questions. We welcome you to join. ::: To read more about Celery you should go read the `introduction `{.interpreted-text role="ref"}. While this version is **mostly** backward compatible with previous versions it\'s important that you read the following section as this release is a new major version. This version is officially supported on CPython 3.8, 3.9 & 3.10 and is also supported on PyPy3.8+. ::: topic **Table of Contents** Make sure you read the important notes before upgrading to this version. ::: ::: {.contents local="" depth="2"} ::: ## Preface ::: note ::: title Note ::: **This release contains fixes for many long standing bugs & stability issues. We encourage our users to upgrade to this release as soon as possible.** ::: The 5.4.0 release is a new feature release for Celery. Releases in the 5.x series are codenamed after songs of [Jon Hopkins](https://en.wikipedia.org/wiki/Jon_Hopkins). This release has been codenamed [Opalescent](https://www.youtube.com/watch?v=9ByfK25WsMM). From now on we only support Python 3.8 and above. We will maintain compatibility with Python 3.8 until it\'s EOL in 2024. *--- Tomer Nosrati* ### Long Term Support Policy We no longer support Celery 4.x as we don\'t have the resources to do so. If you\'d like to help us, all contributions are welcome. Celery 5.x **is not** an LTS release. We will support it until the release of Celery 6.x. We\'re in the process of defining our Long Term Support policy. Watch the next \"What\'s New\" document for updates. ### Wall of Contributors ::: note ::: title Note ::: This wall was automatically generated from git history, so sadly it doesn\'t not include the people who help with more important things like answering mailing-list questions. ::: ## Upgrading from Celery 4.x ### Step 1: Adjust your command line invocation Celery 5.0 introduces a new CLI implementation which isn\'t completely backwards compatible. The global options can no longer be positioned after the sub-command. Instead, they must be positioned as an option for the [celery]{.title-ref} command like so: celery --app path.to.app worker If you were using our `daemonizing`{.interpreted-text role="ref"} guide to deploy Celery in production, you should revisit it for updates. ### Step 2: Update your configuration with the new setting names If you haven\'t already updated your configuration when you migrated to Celery 4.0, please do so now. We elected to extend the deprecation period until 6.0 since we did not loudly warn about using these deprecated settings. Please refer to the `migration guide `{.interpreted-text role="ref"} for instructions. ### Step 3: Read the important notes in this document Make sure you are not affected by any of the important upgrade notes mentioned in the `following section `{.interpreted-text role="ref"}. You should verify that none of the breaking changes in the CLI do not affect you. Please refer to `New Command Line Interface `{.interpreted-text role="ref"} for details. ### Step 4: Migrate your code to Python 3 Celery 5.x only supports Python 3. Therefore, you must ensure your code is compatible with Python 3. If you haven\'t ported your code to Python 3, you must do so before upgrading. You can use tools like [2to3](https://docs.python.org/3.8/library/2to3.html) and [pyupgrade](https://github.com/asottile/pyupgrade) to assist you with this effort. After the migration is done, run your test suite with Celery 4 to ensure nothing has been broken. ### Step 5: Upgrade to Celery 5.4 At this point you can upgrade your workers and clients with the new version. ## Important Notes {#v540-important} ### Supported Python Versions The supported Python versions are: - CPython 3.8 - CPython 3.9 - CPython 3.10 - PyPy3.8 7.3.11 (`pypy3`) #### Experimental support Celery supports these Python versions provisionally as they are not production ready yet: - CPython 3.11 ### Quality Improvements and Stability Enhancements Celery 5.4 focuses on elevating the overall quality and stability of the project. We have dedicated significant efforts to address various bugs, enhance performance, and make improvements based on valuable user feedback. ### Better Compatibility and Upgrade Confidence Our goal with Celery 5.4 is to instill confidence in users who are currently using Celery 4 or older versions. We want to assure you that upgrading to Celery 5.4 will provide a more robust and reliable experience. ### Dropped support for Python 3.7 Celery now requires Python 3.8 and above. Python 3.7 will reach EOL in June, 2023. In order to focus our efforts we have dropped support for Python 3.6 in this version. If you still require to run Celery using Python 3.7 you can still use Celery 5.2. However we encourage you to upgrade to a supported Python version since no further security patches will be applied for Python 3.7 after the 23th of June, 2023. ### Kombu Starting from v5.4.0, the minimum required version is Kombu 5.3. ### Redis redis-py 4.5.x is the new minimum required version. ### SQLAlchemy SQLAlchemy 1.4.x & 2.0.x is now supported in celery v5.4 ### Billiard Minimum required version is now 4.1.0 ### Deprecate pytz and use zoneinfo A switch have been made to zoneinfo for handling timezone data instead of pytz. ### Django Minimum django version is bumped to v2.2.28. Also added \--skip-checks flag to bypass django core checks. --- # What\'s new in Celery 5.5 (Immunity) {#whatsnew-5.5} Author : Tomer Nosrati (`tomer.nosrati at gmail.com`). ::: sidebar **Change history** What\'s new documents describe the changes in major versions, we also have a `changelog`{.interpreted-text role="ref"} that lists the changes in bugfix releases (0.0.x), while older series are archived under the `history`{.interpreted-text role="ref"} section. ::: Celery is a simple, flexible, and reliable distributed programming framework to process vast amounts of messages, while providing operations with the tools required to maintain a distributed system with python. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. ::: note ::: title Note ::: Following the problems with Freenode, we migrated our IRC channel to Libera Chat as most projects did. You can also join us using [Gitter](https://gitter.im/celery/celery). We\'re sometimes there to answer questions. We welcome you to join. ::: To read more about Celery you should go read the `introduction `{.interpreted-text role="ref"}. While this version is **mostly** backward compatible with previous versions it\'s important that you read the following section as this release is a new major version. This version is officially supported on CPython 3.8, 3.9, 3.10, 3.11, 3.12 and 3.13. and is also supported on PyPy3.10+. ::: topic **Table of Contents** Make sure you read the important notes before upgrading to this version. ::: ::: {.contents local="" depth="3"} ::: ## Preface ::: note ::: title Note ::: **This release contains fixes for many long standing bugs & stability issues. We encourage our users to upgrade to this release as soon as possible.** ::: The 5.5.0 release is a new feature release for Celery. Releases in the 5.x series are codenamed after songs of [Jon Hopkins](https://en.wikipedia.org/wiki/Jon_Hopkins). This release has been codenamed [Immunity](https://www.youtube.com/watch?v=Y8eQR5DMous). From now on we only support Python 3.8 and above. We will maintain compatibility with Python 3.8 until it\'s EOL in 2024. *--- Tomer Nosrati* ### Long Term Support Policy We no longer support Celery 4.x as we don\'t have the resources to do so. If you\'d like to help us, all contributions are welcome. Celery 5.x **is not** an LTS release. We will support it until the release of Celery 6.x. We\'re in the process of defining our Long Term Support policy. Watch the next \"What\'s New\" document for updates. ## Upgrading from Celery 4.x ### Step 1: Adjust your command line invocation Celery 5.0 introduces a new CLI implementation which isn\'t completely backwards compatible. The global options can no longer be positioned after the sub-command. Instead, they must be positioned as an option for the [celery]{.title-ref} command like so: celery --app path.to.app worker If you were using our `daemonizing`{.interpreted-text role="ref"} guide to deploy Celery in production, you should revisit it for updates. ### Step 2: Update your configuration with the new setting names If you haven\'t already updated your configuration when you migrated to Celery 4.0, please do so now. We elected to extend the deprecation period until 6.0 since we did not loudly warn about using these deprecated settings. Please refer to the `migration guide `{.interpreted-text role="ref"} for instructions. ### Step 3: Read the important notes in this document Make sure you are not affected by any of the important upgrade notes mentioned in the `following section `{.interpreted-text role="ref"}. You should verify that none of the breaking changes in the CLI do not affect you. Please refer to `New Command Line Interface `{.interpreted-text role="ref"} for details. ### Step 4: Migrate your code to Python 3 Celery 5.x only supports Python 3. Therefore, you must ensure your code is compatible with Python 3. If you haven\'t ported your code to Python 3, you must do so before upgrading. You can use tools like [2to3](https://docs.python.org/3.8/library/2to3.html) and [pyupgrade](https://github.com/asottile/pyupgrade) to assist you with this effort. After the migration is done, run your test suite with Celery 5 to ensure nothing has been broken. ### Step 5: Upgrade to Celery 5.5 At this point you can upgrade your workers and clients with the new version. ## Important Notes {#v550-important} ### Supported Python Versions The supported Python versions are: - CPython 3.8 - CPython 3.9 - CPython 3.10 - CPython 3.11 - CPython 3.12 - CPython 3.13 - PyPy3.10 (`pypy3`) ### Python 3.8 Support Python 3.8 will reach EOL in October, 2024. ### Minimum Dependencies #### Kombu Starting from Celery v5.5, the minimum required version is Kombu 5.5. #### Redis redis-py 4.5.2 is the new minimum required version. #### SQLAlchemy SQLAlchemy 1.4.x & 2.0.x is now supported in Celery v5.5. #### Billiard Minimum required version is now 4.2.1. #### Django Minimum django version is bumped to v2.2.28. Also added \--skip-checks flag to bypass django core checks. ## News {#v550-news} ### Redis Broker Stability Improvements Long-standing disconnection issues with the Redis broker have been identified and resolved in Kombu 5.5.0. These improvements significantly enhance stability when using Redis as a broker, particularly in high-throughput environments. Additionally, the Redis backend now has better exception handling with the new `exception_safe_to_retry` feature, which improves resilience during temporary Redis connection issues. See `conf-redis-result-backend`{.interpreted-text role="ref"} for complete documentation. ### `pycurl` replaced with `urllib3` Replaced the `pycurl`{.interpreted-text role="pypi"} dependency with `urllib3`{.interpreted-text role="pypi"}. We\'re monitoring the performance impact of this change and welcome feedback from users who notice any significant differences in their environments. ### RabbitMQ Quorum Queues Support Added support for RabbitMQ\'s new [Quorum Queues](https://www.rabbitmq.com/docs/quorum-queues) feature, including compatibility with ETA tasks. This implementation has some limitations compared to classic queues, so please refer to the documentation for details. [Native Delayed Delivery](https://docs.particular.net/transports/rabbitmq/delayed-delivery) is automatically enabled when quorum queues are detected to implement the ETA mechanism. See `using-quorum-queues`{.interpreted-text role="ref"} for complete documentation. Configuration options: - `broker_native_delayed_delivery_queue_type`{.interpreted-text role="setting"}: Specifies the queue type for delayed delivery (default: `quorum`) - `task_default_queue_type`{.interpreted-text role="setting"}: Sets the default queue type for tasks (default: `classic`) - `worker_detect_quorum_queues`{.interpreted-text role="setting"}: Controls automatic detection of quorum queues (default: `True`) ### Soft Shutdown Mechanism Soft shutdown is a time limited warm shutdown, initiated just before the cold shutdown. The worker will allow `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} seconds for all currently executing tasks to finish before it terminates. If the time limit is reached, the worker will initiate a cold shutdown and cancel all currently executing tasks. This feature is particularly valuable when using brokers with visibility timeout mechanisms, such as Redis or SQS. It allows the worker enough time to re-queue tasks that were not completed before exiting, preventing task loss during worker shutdown. See `worker-stopping`{.interpreted-text role="ref"} for complete documentation on worker shutdown types. Configuration options: - `worker_soft_shutdown_timeout`{.interpreted-text role="setting"}: Sets the duration in seconds for the soft shutdown period (default: `0.0`, disabled) - `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"}: Controls whether soft shutdown should be enabled even when the worker is idle (default: `False`) ### Pydantic Support New native support for Pydantic models in tasks. This integration allows you to leverage Pydantic\'s powerful data validation and serialization capabilities directly in your Celery tasks. Example usage: ``` python from pydantic import BaseModel from celery import Celery app = Celery('tasks') class ArgModel(BaseModel): value: int class ReturnModel(BaseModel): value: str @app.task(pydantic=True) def x(arg: ArgModel) -> ReturnModel: # args/kwargs type hinted as Pydantic model will be converted assert isinstance(arg, ArgModel) # The returned model will be converted to a dict automatically return ReturnModel(value=f"example: {arg.value}") ``` See `task-pydantic`{.interpreted-text role="ref"} for complete documentation. Configuration options: - `pydantic=True`: Enables Pydantic integration for the task - `pydantic_strict=True/False`: Controls whether strict validation is enabled (default: `False`) - `pydantic_context={...}`: Provides additional context for validation - `pydantic_dump_kwargs={...}`: Customizes serialization behavior ### Google Pub/Sub Transport New support for Google Cloud Pub/Sub as a message transport, expanding Celery\'s cloud integration options. See `broker-gcpubsub`{.interpreted-text role="ref"} for complete documentation. For the Google Pub/Sub support you have to install additional dependencies: ``` console $ pip install "celery[gcpubsub]" ``` Then configure your Celery application to use the Google Pub/Sub transport: ``` python broker_url = 'gcpubsub://projects/project-id' ``` ### Python 3.13 Support Official support for Python 3.13. All core dependencies have been updated to ensure compatibility, including Kombu and py-amqp. This release maintains compatibility with Python 3.8 through 3.13, as well as PyPy 3.10+. ### REMAP_SIGTERM Support The \"REMAP_SIGTERM\" feature, previously undocumented, has been tested, documented, and is now officially supported. This feature allows you to remap the SIGTERM signal to SIGQUIT, enabling you to initiate a soft or cold shutdown using TERM instead of QUIT. This is particularly useful in containerized environments where SIGTERM is the standard signal for graceful termination. See `Cold Shutdown documentation `{.interpreted-text role="ref"} for more info. To enable this feature, set the environment variable: ``` bash export REMAP_SIGTERM="SIGQUIT" ``` Database Backend Improvements \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-- New `create_tables_at_setup` option for the database backend. This option controls when database tables are created, allowing for non-lazy table creation. By default (`create_tables_at_setup=True`), tables are created during backend initialization. Setting this to `False` defers table creation until they are actually needed, which can be useful in certain deployment scenarios where you want more control over database schema management. See `conf-database-result-backend`{.interpreted-text role="ref"} for complete documentation. --- # What\'s new in Celery 5.6 (Recovery) {#whatsnew-5.6} Author : Tomer Nosrati (`tomer.nosrati at gmail.com`). ::: sidebar **Change history** What\'s new documents describe the changes in major versions, we also have a `changelog`{.interpreted-text role="ref"} that lists the changes in bugfix releases (0.0.x), while older series are archived under the `history`{.interpreted-text role="ref"} section. ::: Celery is a simple, flexible, and reliable distributed programming framework to process vast amounts of messages, while providing operations with the tools required to maintain a distributed system with python. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, you should come join us `on IRC `{.interpreted-text role="ref"} or `our mailing-list `{.interpreted-text role="ref"}. ::: note ::: title Note ::: Following the problems with Freenode, we migrated our IRC channel to Libera Chat as most projects did. You can also join us using [Gitter](https://gitter.im/celery/celery). We\'re sometimes there to answer questions. We welcome you to join. ::: To read more about Celery you should go read the `introduction `{.interpreted-text role="ref"}. While this version is **mostly** backward compatible with previous versions it\'s important that you read the following section as this release is a new major version. This version is officially supported on CPython 3.9, 3.10, 3.11, 3.12 and 3.13, and is also supported on PyPy3.11+. ::: topic **Table of Contents** Make sure you read the important notes before upgrading to this version. ::: ::: {.contents local="" depth="3"} ::: ## Preface ::: note ::: title Note ::: **This release contains fixes for many long standing bugs & stability issues. We encourage our users to upgrade to this release as soon as possible.** ::: The 5.6.0 release is a new feature release for Celery. Releases in the 5.x series are codenamed after songs of [Jon Hopkins](https://en.wikipedia.org/wiki/Jon_Hopkins). This release has been codenamed [Recovery](https://www.youtube.com/watch?v=MaqlsAmlbzo). This is the last version to support Python 3.9. Support for Python 3.8 was removed after v5.6.0b1. *--- Tomer Nosrati* ### Long Term Support Policy We no longer support Celery 4.x as we don\'t have the resources to do so. If you\'d like to help us, all contributions are welcome. Celery 5.x **is not** an LTS release. We will support it until the release of Celery 6.x. We\'re in the process of defining our Long Term Support policy. Watch the next \"What\'s New\" document for updates. ## Upgrading from Celery 4.x ### Step 1: Adjust your command line invocation Celery 5.0 introduces a new CLI implementation which isn\'t completely backwards compatible. The global options can no longer be positioned after the sub-command. Instead, they must be positioned as an option for the [celery]{.title-ref} command like so: celery --app path.to.app worker If you were using our `daemonizing`{.interpreted-text role="ref"} guide to deploy Celery in production, you should revisit it for updates. ### Step 2: Update your configuration with the new setting names If you haven\'t already updated your configuration when you migrated to Celery 4.0, please do so now. We elected to extend the deprecation period until 6.0 since we did not loudly warn about using these deprecated settings. Please refer to the `migration guide `{.interpreted-text role="ref"} for instructions. ### Step 3: Read the important notes in this document Make sure you are not affected by any of the important upgrade notes mentioned in the `following section `{.interpreted-text role="ref"}. You should verify that none of the breaking changes in the CLI do not affect you. Please refer to `New Command Line Interface `{.interpreted-text role="ref"} for details. ### Step 4: Migrate your code to Python 3 Celery 5.x only supports Python 3. Therefore, you must ensure your code is compatible with Python 3. If you haven\'t ported your code to Python 3, you must do so before upgrading. You can use tools like [2to3](https://docs.python.org/3.8/library/2to3.html) and [pyupgrade](https://github.com/asottile/pyupgrade) to assist you with this effort. After the migration is done, run your test suite with Celery 5 to ensure nothing has been broken. ### Step 5: Upgrade to Celery 5.6 At this point you can upgrade your workers and clients with the new version. ## Important Notes {#v560-important} ### Supported Python Versions The supported Python versions are: - CPython 3.9 - CPython 3.10 - CPython 3.11 - CPython 3.12 - CPython 3.13 - PyPy3.11 (`pypy3`) ### Python 3.9 Support Python 3.9 will reach EOL in October, 2025. ### Minimum Dependencies #### Kombu Starting from Celery v5.6, the minimum required version is Kombu 5.6. #### Redis redis-py 4.5.2 is the new minimum required version. #### SQLAlchemy SQLAlchemy 1.4.x & 2.0.x is now supported in Celery v5.6. #### Billiard Minimum required version is now 4.2.4. #### Django Minimum django version is bumped to v2.2.28. Also added \--skip-checks flag to bypass django core checks. ## News {#v560-news} ### SQS: Reverted to `pycurl` from `urllib3` The switch from `pycurl` to `urllib3` for the SQS transport (introduced in Celery 5.5.0 via Kombu) has been reverted due to critical issues affecting SQS users. ### Security Fix: Broker Credential Leak Prevention Fixed a security issue where broker URLs containing passwords were being logged in plaintext by the delayed delivery mechanism. Broker credentials are now properly sanitized in all log output. ### Memory Leak Fixes Two significant memory leaks have been fixed in this release: **Exception Handling Memory Leak**: Fixed a critical memory leak in task exception handling that was particularly severe on Python 3.11+ due to enhanced traceback data. The fix properly breaks reference cycles in tracebacks to allow garbage collection. This resolves a long-standing issue that caused worker memory to grow unbounded over time. **Pending Result Memory Leak**: Fixed a memory leak where `AsyncResult` subscriptions were not being cleaned up when results were forgotten. This affected users who frequently use `AsyncResult.forget()` in their workflows. ### ETA Task Memory Limit New configuration option to prevent out-of-memory crashes when workers fetch large numbers of ETA or countdown tasks. Previously, workers could exhaust available memory when the broker contained many scheduled tasks. Configuration option: - `worker_eta_task_limit`{.interpreted-text role="setting"}: Sets the maximum number of ETA tasks to hold in worker memory at once (default: `None`, unlimited) Example usage: ``` python app.conf.worker_eta_task_limit = 1000 ``` ### Queue Type Selection for Auto-created Queues New configuration options allow specifying the queue type and exchange type when Celery auto-creates missing queues. This is particularly useful for RabbitMQ users who want to use quorum queues with auto-created queues. Configuration options: - `task_create_missing_queue_type`{.interpreted-text role="setting"}: Sets the queue type for auto-created queues (e.g., `quorum`, `classic`) - `task_create_missing_queue_exchange_type`{.interpreted-text role="setting"}: Sets the exchange type for auto-created queues Example usage: ``` python app.conf.task_create_missing_queue_type = 'quorum' ``` ### Django Connection Pool Support Fixed an issue where Django applications using psycopg3 connection pooling would experience `psycopg_pool.PoolTimeout` errors after worker forks. Celery now properly closes Django\'s connection pools before forking, similar to how Django itself handles this in its autoreload mechanism. ### Redis Backend Improvements **Credential Provider Support**: Added the `redis_backend_credential_provider`{.interpreted-text role="setting"} setting to the Redis backend. This enables integration with AWS ElastiCache using IAM authentication and other credential provider mechanisms. **Client Name Support**: Added the `redis_client_name`{.interpreted-text role="setting"} setting to the Redis backend, making it easier to identify Celery connections when monitoring Redis servers. ### Cold Shutdown Improvements Fixed an issue where tasks would incorrectly fail with a timeout error during cold shutdown. The worker now properly skips timeout failure handling during the cold shutdown phase, allowing tasks to complete or be properly requeued. --- # Celery - Distributed Task Queue Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It\'s a task queue with focus on real-time processing, while also supporting task scheduling. Celery has a large and diverse community of users and contributors, don\'t hesitate to ask questions or `get involved `{.interpreted-text role="ref"}. Celery is Open Source and licensed under the [BSD License](http://www.opensource.org/licenses/BSD-3-Clause). [![Open Collective logo](https://opencollective.com/static/images/opencollectivelogo-footer-n.svg){width="240px"}](https://opencollective.com/celery) [Open Collective](https://opencollective.com/celery) is our community-powered funding platform that fuels Celery\'s ongoing development. Your sponsorship directly supports improvements, maintenance, and innovative features that keep Celery robust and reliable. ## Getting Started - If you\'re new to Celery you can get started by following the `first-steps`{.interpreted-text role="ref"} tutorial. - You can also check out the `FAQ `{.interpreted-text role="ref"}. ## Contents ::: {.toctree maxdepth="1"} copyright ::: ::: {.toctree maxdepth="2"} getting-started/index userguide/index ::: ::: {.toctree maxdepth="1"} django/index contributing community tutorials/index faq changelog reference/index internals/index history/index glossary ::: ## Indices and tables - `genindex`{.interpreted-text role="ref"} - `modindex`{.interpreted-text role="ref"} - `search`{.interpreted-text role="ref"} --- # \"The Big Instance\" Refactor The [app]{.title-ref} branch is a work-in-progress to remove the use of a global configuration in Celery. Celery can now be instantiated and several instances of Celery may exist in the same process space. Also, large parts can be customized without resorting to monkey patching. ## Examples Creating a Celery instance: >>> from celery import Celery >>> app = Celery() >>> app.config_from_object('celeryconfig') >>> #app.config_from_envvar('CELERY_CONFIG_MODULE') Creating tasks: ``` python @app.task def add(x, y): return x + y ``` Creating custom Task subclasses: ``` python Task = celery.create_task_cls() class DebugTask(Task): def on_failure(self, *args, **kwargs): import pdb pdb.set_trace() @app.task(base=DebugTask) def add(x, y): return x + y ``` Starting a worker: ``` python worker = celery.Worker(loglevel='INFO') ``` Getting access to the configuration: ``` python celery.conf.task_always_eager = True celery.conf['task_always_eager'] = True ``` Controlling workers: >>> celery.control.inspect().active() >>> celery.control.rate_limit(add.name, '100/m') >>> celery.control.broadcast('shutdown') >>> celery.control.discard_all() Other interesting attributes: # Establish broker connection. >>> celery.broker_connection() # AMQP Specific features. >>> celery.amqp >>> celery.amqp.Router >>> celery.amqp.get_queues() >>> celery.amqp.get_task_consumer() # Loader >>> celery.loader # Default backend >>> celery.backend As you can probably see, this really opens up another dimension of customization abilities. ## Deprecated - `celery.task.ping` `celery.task.PingTask` Inferior to the ping remote control command. Will be removed in Celery 2.3. ## Aliases (Pending deprecation) - `celery.execute` : - `.send_task` -\> {`app.send_task`} - `.delay_task` -\> *no alternative* - `celery.log` : - `.get_default_logger` -\> {`app.log.get_default_logger`} - `.setup_logger` -\> {`app.log.setup_logger`} - `.get_task_logger` -\> {`app.log.get_task_logger`} - `.setup_task_logger` -\> {`app.log.setup_task_logger`} - `.setup_logging_subsystem` -\> {`app.log.setup_logging_subsystem`} - `.redirect_stdouts_to_logger` -\> {`app.log.redirect_stdouts_to_logger`} - `celery.messaging` : - `.establish_connection` -\> {`app.broker_connection`} - `.with_connection` -\> {`app.with_connection`} - `.get_consumer_set` -\> {`app.amqp.get_task_consumer`} - `.TaskPublisher` -\> {`app.amqp.TaskPublisher`} - `.TaskConsumer` -\> {`app.amqp.TaskConsumer`} - `.ConsumerSet` -\> {`app.amqp.ConsumerSet`} - `celery.conf.*` -\> {`app.conf`} > **NOTE**: All configuration keys are now named the same as in the > configuration. So the key `task_always_eager` is accessed as: > > >>> app.conf.task_always_eager > > instead of: > > >>> from celery import conf > >>> conf.always_eager > > - `.get_queues` -\> {`app.amqp.get_queues`} - `celery.utils.info` : - `.humanize_seconds` -\> `celery.utils.time.humanize_seconds` - `.textindent` -\> `celery.utils.textindent` - `.get_broker_info` -\> {`app.amqp.get_broker_info`} - `.format_broker_info` -\> {`app.amqp.format_broker_info`} - `.format_queues` -\> {`app.amqp.format_queues`} ## Default App Usage To be backward compatible, it must be possible to use all the classes/functions without passing an explicit app instance. This is achieved by having all app-dependent objects use `~celery.app.default_app`{.interpreted-text role="data"} if the app instance is missing. ``` python from celery.app import app_or_default class SomeClass: def __init__(self, app=None): self.app = app_or_default(app) ``` The problem with this approach is that there\'s a chance that the app instance is lost along the way, and everything seems to be working normally. Testing app instance leaks is hard. The environment variable `CELERY_TRACE_APP`{.interpreted-text role="envvar"} can be used, when this is enabled `celery.app.app_or_default`{.interpreted-text role="func"} will raise an exception whenever it has to go back to the default app instance. ### App Dependency Tree - {`app`} : - `celery.loaders.base.BaseLoader` - `celery.backends.base.BaseBackend` - {`app.TaskSet`} : - `celery.task.sets.TaskSet` (`app.TaskSet`) - \[`app.TaskSetResult`\] : - `celery.result.TaskSetResult` (`app.TaskSetResult`) - {`app.AsyncResult`} : - `celery.result.BaseAsyncResult` / `celery.result.AsyncResult` - `celery.bin.worker.WorkerCommand` : - `celery.apps.worker.Worker` : - `celery.worker.WorkerController` : - `celery.worker.consumer.Consumer` : - `celery.worker.request.Request` - `celery.events.EventDispatcher` - `celery.worker.control.ControlDispatch` : - `celery.worker.control.registry.Panel` - `celery.pidbox.BroadcastPublisher` - `celery.pidbox.BroadcastConsumer` - `celery.beat.EmbeddedService` - `celery.bin.events.EvCommand` : - `celery.events.snapshot.evcam` : - `celery.events.snapshot.Polaroid` - `celery.events.EventReceiver` - `celery.events.cursesmon.evtop` : - `celery.events.EventReceiver` - `celery.events.cursesmon.CursesMonitor` - `celery.events.dumper` : - `celery.events.EventReceiver` - `celery.bin.amqp.AMQPAdmin` - `celery.bin.beat.BeatCommand` : - `celery.apps.beat.Beat` : - `celery.beat.Service` : - `celery.beat.Scheduler` --- # Celery Deprecation Time-line {#deprecation-timeline} ::: {.contents local=""} ::: ## Removals for version 5.0 {#deprecations-v5.0} ### Old Task API #### Compat Task Modules {#deprecate-compat-task-modules} - Module `celery.decorators` will be removed: > This means you need to change: > > ``` python > from celery.decorators import task > ``` > > Into: > > ``` python > from celery import task > ``` - Module `celery.task` will be removed > This means you should change: > > ``` python > from celery.task import task > ``` > > into: > > ``` python > from celery import shared_task > ``` > > \-- and: > > ``` python > from celery import task > ``` > > into: > > ``` python > from celery import shared_task > ``` > > \-- and: > > ``` python > from celery.task import Task > ``` > > into: > > ``` python > from celery import Task > ``` Note that the new `~celery.Task`{.interpreted-text role="class"} class no longer uses `classmethod`{.interpreted-text role="func"} for these methods: > - delay > - apply_async > - retry > - apply > - AsyncResult > - subtask This also means that you can\'t call these methods directly on the class, but have to instantiate the task first: ``` pycon >>> MyTask.delay() # NO LONGER WORKS >>> MyTask().delay() # WORKS! ``` ### Task attributes The task attributes: - `queue` - `exchange` - `exchange_type` - `routing_key` - `delivery_mode` - `priority` is deprecated and must be set by `task_routes`{.interpreted-text role="setting"} instead. ### Modules to Remove - `celery.execute` This module only contains `send_task`: this must be replaced with `@send_task`{.interpreted-text role="attr"} instead. - `celery.decorators` > See `deprecate-compat-task-modules`{.interpreted-text role="ref"} - `celery.log` > Use `@log`{.interpreted-text role="attr"} instead. - `celery.messaging` > Use `@amqp`{.interpreted-text role="attr"} instead. - `celery.registry` > Use `celery.app.registry`{.interpreted-text role="mod"} instead. - `celery.task.control` > Use `@control`{.interpreted-text role="attr"} instead. - `celery.task.schedules` > Use `celery.schedules`{.interpreted-text role="mod"} instead. - `celery.task.chords` > Use `celery.chord`{.interpreted-text role="func"} instead. ### Settings #### `BROKER` Settings **Setting name** **Replace with** ------------------- ------------------------------------------------ `BROKER_HOST` `broker_url`{.interpreted-text role="setting"} `BROKER_PORT` `broker_url`{.interpreted-text role="setting"} `BROKER_USER` `broker_url`{.interpreted-text role="setting"} `BROKER_PASSWORD` `broker_url`{.interpreted-text role="setting"} `BROKER_VHOST` `broker_url`{.interpreted-text role="setting"} #### `REDIS` Result Backend Settings **Setting name** **Replace with** ------------------------- ---------------------------------------------------- `CELERY_REDIS_HOST` `result_backend`{.interpreted-text role="setting"} `CELERY_REDIS_PORT` `result_backend`{.interpreted-text role="setting"} `CELERY_REDIS_DB` `result_backend`{.interpreted-text role="setting"} `CELERY_REDIS_PASSWORD` `result_backend`{.interpreted-text role="setting"} `REDIS_HOST` `result_backend`{.interpreted-text role="setting"} `REDIS_PORT` `result_backend`{.interpreted-text role="setting"} `REDIS_DB` `result_backend`{.interpreted-text role="setting"} `REDIS_PASSWORD` `result_backend`{.interpreted-text role="setting"} ### Task_sent signal The `task_sent`{.interpreted-text role="signal"} signal will be removed in version 4.0. Please use the `before_task_publish`{.interpreted-text role="signal"} and `after_task_publish`{.interpreted-text role="signal"} signals instead. ### Result Apply to: `~celery.result.AsyncResult`{.interpreted-text role="class"}, `~celery.result.EagerResult`{.interpreted-text role="class"}: - `Result.wait()` -\> `Result.get()` - `Result.task_id()` -\> `Result.id` - `Result.status` -\> `Result.state`. #### Settings {#deprecations-v3.1} **Setting name** **Replace with** ----------------------------------- ---------------------------------------------------- `CELERY_AMQP_TASK_RESULT_EXPIRES` `result_expires`{.interpreted-text role="setting"} ## Removals for version 2.0 {#deprecations-v2.0} - The following settings will be removed: **Setting name** **Replace with** ------------------------------------------------- ------------------------------------------ [CELERY_AMQP_CONSUMER_QUEUES]{.title-ref} [task_queues]{.title-ref} [CELERY_AMQP_CONSUMER_QUEUES]{.title-ref} [task_queues]{.title-ref} [CELERY_AMQP_EXCHANGE]{.title-ref} [task_default_exchange]{.title-ref} [CELERY_AMQP_EXCHANGE_TYPE]{.title-ref} [task_default_exchange_type]{.title-ref} [CELERY_AMQP_CONSUMER_ROUTING_KEY]{.title-ref} [task_queues]{.title-ref} [CELERY_AMQP_PUBLISHER_ROUTING_KEY]{.title-ref} [task_default_routing_key]{.title-ref} - `CELERY_LOADER`{.interpreted-text role="envvar"} definitions without class name. > For example,, [celery.loaders.default]{.title-ref}, needs to > include the class name: > [celery.loaders.default.Loader]{.title-ref}. - `TaskSet.run`{.interpreted-text role="meth"}. Use `celery.task.base.TaskSet.apply_async`{.interpreted-text role="meth"} : instead. --- # Contributors Guide to the Code {#internals-guide} ::: {.contents local=""} ::: ## Philosophy ### The API\>RCP Precedence Rule - The API is more important than Readability - Readability is more important than Convention - Convention is more important than Performance : - ...unless the code is a proven hot-spot. More important than anything else is the end-user API. Conventions must step aside, and any suffering is always alleviated if the end result is a better API. ## Conventions and Idioms Used ### Classes #### Naming - Follows `8`{.interpreted-text role="pep"}. - Class names must be [CamelCase]{.title-ref}. - but not if they\'re verbs, verbs shall be \`lower_case\`: > ``` python > # - test case for a class > class TestMyClass(Case): # BAD > pass > > class test_MyClass(Case): # GOOD > pass > > # - test case for a function > class TestMyFunction(Case): # BAD > pass > > class test_my_function(Case): # GOOD > pass > > # - "action" class (verb) > class UpdateTwitterStatus: # BAD > pass > > class update_twitter_status: # GOOD > pass > ``` > > ::: note > ::: title > Note > ::: > > Sometimes it makes sense to have a class mask as a function, and > there\'s precedence for this in the Python standard library (e.g., > `~contextlib.contextmanager`{.interpreted-text role="class"}). > Celery examples include `~celery.signature`{.interpreted-text > role="class"}, `~celery.chord`{.interpreted-text role="class"}, > `inspect`, `~kombu.utils.functional.promise`{.interpreted-text > role="class"} and more.. > ::: - Factory functions and methods must be [CamelCase]{.title-ref} (excluding verbs): > ``` python > class Celery: > > def consumer_factory(self): # BAD > ... > > def Consumer(self): # GOOD > ... > ``` #### Default values Class attributes serve as default values for the instance, as this means that they can be set by either instantiation or inheritance. **Example:** ``` python class Producer: active = True serializer = 'json' def __init__(self, serializer=None, active=None): self.serializer = serializer or self.serializer # must check for None when value can be false-y self.active = active if active is not None else self.active ``` A subclass can change the default value: ``` python TaskProducer(Producer): serializer = 'pickle' ``` and the value can be set at instantiation: ``` pycon >>> producer = TaskProducer(serializer='msgpack') ``` #### Exceptions Custom exceptions raised by an objects methods and properties should be available as an attribute and documented in the method/property that throw. This way a user doesn\'t have to find out where to import the exception from, but rather use `help(obj)` and access the exception class from the instance directly. **Example**: ``` python class Empty(Exception): pass class Queue: Empty = Empty def get(self): """Get the next item from the queue. :raises Queue.Empty: if there are no more items left. """ try: return self.queue.popleft() except IndexError: raise self.Empty() ``` #### Composites Similarly to exceptions, composite classes should be override-able by inheritance and/or instantiation. Common sense can be used when selecting what classes to include, but often it\'s better to add one too many: predicting what users need to override is hard (this has saved us from many a monkey patch). **Example**: ``` python class Worker: Consumer = Consumer def __init__(self, connection, consumer_cls=None): self.Consumer = consumer_cls or self.Consumer def do_work(self): with self.Consumer(self.connection) as consumer: self.connection.drain_events() ``` ## Applications vs. \"single mode\" In the beginning Celery was developed for Django, simply because this enabled us get the project started quickly, while also having a large potential user base. In Django there\'s a global settings object, so multiple Django projects can\'t co-exist in the same process space, this later posed a problem for using Celery with frameworks that don\'t have this limitation. Therefore the app concept was introduced. When using apps you use \'celery\' objects instead of importing things from Celery sub-modules, this (unfortunately) also means that Celery essentially has two API\'s. Here\'s an example using Celery in single-mode: ``` python from celery import task from celery.task.control import inspect from .models import CeleryStats @task def write_stats_to_db(): stats = inspect().stats(timeout=1) for node_name, reply in stats: CeleryStats.objects.update_stat(node_name, stats) ``` and here\'s the same using Celery app objects: ``` python from .celery import celery from .models import CeleryStats @app.task def write_stats_to_db(): stats = celery.control.inspect().stats(timeout=1) for node_name, reply in stats: CeleryStats.objects.update_stat(node_name, stats) ``` In the example above the actual application instance is imported from a module in the project, this module could look something like this: ``` python from celery import Celery app = Celery(broker='amqp://') ``` ## Module Overview - celery.app > This is the core of Celery: the entry-point for all functionality. - celery.loaders > Every app must have a loader. The loader decides how configuration > is read; what happens when the worker starts; when a task starts > and ends; and so on. > > The loaders included are: > > > - app > > > > > Custom Celery app instances uses this loader by default. > > > > - default > > > > > \"single-mode\" uses this loader by default. > > Extension loaders also exist, for example > `celery-pylons`{.interpreted-text role="pypi"}. - celery.worker > This is the worker implementation. - celery.backends > Task result backends live here. - celery.apps > Major user applications: worker and beat. The command-line > wrappers for these are in celery.bin (see below) - celery.bin > Command-line applications. `setup.py`{.interpreted-text > role="file"} creates setuptools entry-points for these. - celery.concurrency > Execution pool implementations (prefork, eventlet, gevent, solo, > thread). - celery.db > Database models for the SQLAlchemy database result backend. > (should be moved into `celery.backends.database`{.interpreted-text > role="mod"}) - celery.events > Sending and consuming monitoring events, also includes curses > monitor, event dumper and utilities to work with in-memory cluster > state. - celery.execute.trace > How tasks are executed and traced by the worker, and in eager > mode. - celery.security > Security related functionality, currently a serializer using > cryptographic digests. - celery.task > single-mode interface to creating tasks, and controlling workers. - t.unit (int distribution) > The unit test suite. - celery.utils > Utility functions used by the Celery code base. Much of it is > there to be compatible across Python versions. - celery.contrib > Additional public code that doesn\'t fit into any other > name-space. ## Worker overview - [celery.bin.worker:Worker]{.title-ref} > This is the command-line interface to the worker. > > Responsibilities: > > : - Daemonization when > `--detach `{.interpreted-text > role="option"} set, > - dropping privileges when using > `--uid `{.interpreted-text > role="option"}/ > `--gid `{.interpreted-text > role="option"} arguments > - Installs \"concurrency patches\" (eventlet/gevent monkey > patches). `app.worker_main(argv)` calls `instantiate('celery.bin.worker:Worker')(app).execute_from_commandline(argv)` - [app.Worker]{.title-ref} -\> [celery.apps.worker:Worker]{.title-ref} > Responsibilities: > - sets up logging and redirects standard outs > - installs signal handlers > ([TERM]{.title-ref}/[HUP]{.title-ref}/[STOP]{.title-ref}/[USR1]{.title-ref} > (cry)/[USR2]{.title-ref} (rdb)) > - prints banner and warnings (e.g., pickle warning) > - handles the `celery worker --purge`{.interpreted-text > role="option"} argument - [app.WorkController]{.title-ref} -\> [celery.worker.WorkController]{.title-ref} > This is the real worker, built up around bootsteps. --- # Internals Release : Date : ::: {.toctree maxdepth="2"} guide deprecation worker protocol app-overview reference/index ::: --- # Message Protocol ::: {.contents local=""} ::: ## Task messages[]{#message-protocol-task} {#internals-task-message-protocol} ### Version 2 {#message-protocol-task-v2} #### Definition ``` python properties = { 'correlation_id': uuid task_id, 'content_type': string mimetype, 'content_encoding': string encoding, # optional 'reply_to': string queue_or_url, } headers = { 'lang': string 'py' 'task': string task, 'id': uuid task_id, 'root_id': uuid root_id, 'parent_id': uuid parent_id, 'group': uuid group_id, # optional 'meth': string method_name, 'shadow': string alias_name, 'eta': iso8601 ETA, 'expires': iso8601 expires, 'retries': int retries, 'timelimit': (soft, hard), 'argsrepr': str repr(args), 'kwargsrepr': str repr(kwargs), 'origin': str nodename, 'replaced_task_nesting': int } body = ( object[] args, Mapping kwargs, Mapping embed { 'callbacks': Signature[] callbacks, 'errbacks': Signature[] errbacks, 'chain': Signature[] chain, 'chord': Signature chord_callback, } ) ``` #### Example This example sends a task message using version 2 of the protocol: ``` python # chain: add(add(add(2, 2), 4), 8) == 2 + 2 + 4 + 8 import json import os import socket task_id = uuid() args = (2, 2) kwargs = {} basic_publish( message=json.dumps((args, kwargs, None)), application_headers={ 'lang': 'py', 'task': 'proj.tasks.add', 'argsrepr': repr(args), 'kwargsrepr': repr(kwargs), 'origin': '@'.join([os.getpid(), socket.gethostname()]) } properties={ 'correlation_id': task_id, 'content_type': 'application/json', 'content_encoding': 'utf-8', } ) ``` #### Changes from version 1 - Protocol version detected by the presence of a `task` message header. - Support for multiple languages via the `lang` header. > Worker may redirect the message to a worker that supports the > language. - Meta-data moved to headers. > This means that workers/intermediates can inspect the message and > make decisions based on the headers without decoding the payload > (that may be language specific, for example serialized by the > Python specific pickle serializer). - Always UTC > There\'s no `utc` flag anymore, so any time information missing > timezone will be expected to be in UTC time. - Body is only for language specific data. > - Python stores args/kwargs and embedded signatures in body. > - If a message uses raw encoding then the raw data will be > passed as a single argument to the function. > - Java/C, etc. can use a Thrift/protobuf document as the body - `origin` is the name of the node sending the task. - Dispatches to actor based on `task`, `meth` headers > `meth` is unused by Python, but may be used in the future to > specify class+method pairs. - Chain gains a dedicated field. > Reducing the chain into a recursive `callbacks` argument causes > problems when the recursion limit is exceeded. > > This is fixed in the new message protocol by specifying a list of > signatures, each task will then pop a task off the list when > sending the next message: > > ``` python > execute_task(message) > chain = embed['chain'] > if chain: > sig = maybe_signature(chain.pop()) > sig.apply_async(chain=chain) > ``` - `correlation_id` replaces `task_id` field. - `root_id` and `parent_id` fields helps keep track of work-flows. - `shadow` lets you specify a different name for logs, monitors can be used for concepts like tasks that calls a function specified as argument: > ``` python > from celery.utils.imports import qualname > > class PickleTask(Task): > > def unpack_args(self, fun, args=()): > return fun, args > > def apply_async(self, args, kwargs, **options): > fun, real_args = self.unpack_args(*args) > return super().apply_async( > (fun, real_args, kwargs), shadow=qualname(fun), **options > ) > > @app.task(base=PickleTask) > def call(fun, args, kwargs): > return fun(*args, **kwargs) > ``` ### Version 1[]{#message-protocol-task-v1} {#task-message-protocol-v1} In version 1 of the protocol all fields are stored in the message body: meaning workers and intermediate consumers must deserialize the payload to read the fields. #### Message body - `task` : [string]{.title-ref} : Name of the task. **required** - `id` : [string]{.title-ref} : Unique id of the task (UUID). **required** - `args` : [list]{.title-ref} : List of arguments. Will be an empty list if not provided. - `kwargs` : [dictionary]{.title-ref} : Dictionary of keyword arguments. Will be an empty dictionary if not provided. - `retries` : [int]{.title-ref} : Current number of times this task has been retried. Defaults to [0]{.title-ref} if not specified. - `eta` : [string]{.title-ref} (ISO 8601) : Estimated time of arrival. This is the date and time in ISO 8601 format. If not provided the message isn\'t scheduled, but will be executed asap. - `expires` : [string]{.title-ref} (ISO 8601) : ::: versionadded 2.0.2 ::: Expiration date. This is the date and time in ISO 8601 format. If not provided the message will never expire. The message will be expired when the message is received and the expiration date has been exceeded. - `taskset` : [string]{.title-ref} : The group this task is part of (if any). - `chord` : [Signature]{.title-ref} : ::: versionadded 2.3 ::: Signifies that this task is one of the header parts of a chord. The value of this key is the body of the cord that should be executed when all of the tasks in the header has returned. - `utc` : [bool]{.title-ref} : ::: versionadded 2.5 ::: If true time uses the UTC timezone, if not the current local timezone should be used. - `callbacks` : [\Signature]{.title-ref} : ::: versionadded 3.0 ::: A list of signatures to call if the task exited successfully. - `errbacks` : [\Signature]{.title-ref} : ::: versionadded 3.0 ::: A list of signatures to call if an error occurs while executing the task. - `timelimit` : [\(float, float)]{.title-ref} : ::: versionadded 3.1 ::: Task execution time limit settings. This is a tuple of hard and soft time limit value ([int]{.title-ref}/[float]{.title-ref} or `None`{.interpreted-text role="const"} for no limit). Example value specifying a soft time limit of 3 seconds, and a hard time limit of 10 seconds: {'timelimit': (3.0, 10.0)} #### Example message This is an example invocation of a [celery.task.ping]{.title-ref} task in json format: ``` javascript {"id": "4cc7438e-afd4-4f8f-a2f3-f46567e7ca77", "task": "celery.task.PingTask", "args": [], "kwargs": {}, "retries": 0, "eta": "2009-11-17T12:30:56.527191"} ``` ### Task Serialization Several types of serialization formats are supported using the [content_type]{.title-ref} message header. The MIME-types supported by default are shown in the following table. > Scheme MIME Type > --------- -------------------------------- > json application/json > yaml application/x-yaml > pickle application/x-python-serialize > msgpack application/x-msgpack ## Event Messages {#message-protocol-event} Event messages are always JSON serialized and can contain arbitrary message body fields. Since version 4.0. the body can consist of either a single mapping (one event), or a list of mappings (multiple events). There are also standard fields that must always be present in an event message: ### Standard body fields - *string* `type` > The type of event. This is a string containing the *category* and > *action* separated by a dash delimiter (e.g., `task-succeeded`). - *string* `hostname` > The fully qualified hostname of where the event occurred at. - *unsigned long long* `clock` > The logical clock value for this event (Lamport time-stamp). - *float* `timestamp` > The UNIX time-stamp corresponding to the time of when the event > occurred. - *signed short* `utcoffset` > This field describes the timezone of the originating host, and is > specified as the number of hours ahead of/behind UTC (e.g., -2 or > +1). - *unsigned long long* `pid` > The process id of the process the event originated in. ### Standard event types For a list of standard event types and their fields see the `event-reference`{.interpreted-text role="ref"}. ### Example message This is the message fields for a `task-succeeded` event: ``` python properties = { 'routing_key': 'task.succeeded', 'exchange': 'celeryev', 'content_type': 'application/json', 'content_encoding': 'utf-8', 'delivery_mode': 1, } headers = { 'hostname': 'worker1@george.vandelay.com', } body = { 'type': 'task-succeeded', 'hostname': 'worker1@george.vandelay.com', 'pid': 6335, 'clock': 393912923921, 'timestamp': 1401717709.101747, 'utcoffset': -1, 'uuid': '9011d855-fdd1-4f8f-adb3-a413b499eafb', 'retval': '4', 'runtime': 0.0003212, ) ``` --- # `celery._state` ::: {.contents local=""} ::: ::: currentmodule celery.\_state ::: ::: {.automodule members="" undoc-members=""} celery.\_state ::: --- # `celery.app.annotations` ::: {.contents local=""} ::: ::: currentmodule celery.app.annotations ::: ::: {.automodule members="" undoc-members=""} celery.app.annotations ::: --- # `celery.app.routes` ::: {.contents local=""} ::: ::: currentmodule celery.app.routes ::: ::: {.automodule members="" undoc-members=""} celery.app.routes ::: --- # `celery.app.trace` ::: {.contents local=""} ::: ::: currentmodule celery.app.trace ::: ::: {.automodule members="" undoc-members=""} celery.app.trace ::: --- # `celery.backends.arangodb` ::: {.contents local=""} ::: ::: currentmodule celery.backends.arangodb ::: ::: {.automodule members="" undoc-members=""} celery.backends.arangodb ::: --- # `celery.backends.asynchronous` ::: {.contents local=""} ::: ::: currentmodule celery.backends.asynchronous ::: ::: {.automodule members="" undoc-members=""} celery.backends.asynchronous ::: --- # `celery.backends.azureblockblob` ::: {.contents local=""} ::: ::: currentmodule celery.backends.azureblockblob ::: ::: {.automodule members="" undoc-members=""} celery.backends.azureblockblob ::: --- # `celery.backends.base` ::: {.contents local=""} ::: ::: currentmodule celery.backends.base ::: ::: {.automodule members="" undoc-members=""} celery.backends.base ::: --- # `celery.backends.cache` ::: {.contents local=""} ::: ::: currentmodule celery.backends.cache ::: ::: {.automodule members="" undoc-members=""} celery.backends.cache ::: --- # `celery.backends.cassandra` ::: {.contents local=""} ::: ::: currentmodule celery.backends.cassandra ::: ::: {.automodule members="" undoc-members=""} celery.backends.cassandra ::: --- # celery.backends.consul ::: {.contents local=""} ::: ::: currentmodule celery.backends.consul ::: ::: {.automodule members="" undoc-members=""} celery.backends.consul ::: --- # `celery.backends.cosmosdbsql` ::: {.contents local=""} ::: ::: currentmodule celery.backends.cosmosdbsql ::: ::: {.automodule members="" undoc-members=""} celery.backends.cosmosdbsql ::: --- # `celery.backends.couchbase` ::: {.contents local=""} ::: ::: currentmodule celery.backends.couchbase ::: ::: {.automodule members="" undoc-members=""} celery.backends.couchbase ::: --- # `celery.backends.couchdb` ::: {.contents local=""} ::: ::: currentmodule celery.backends.couchdb ::: ::: {.automodule members="" undoc-members=""} celery.backends.couchdb ::: --- # `celery.backends.database` ::: {.contents local=""} ::: ::: currentmodule celery.backends.database ::: ::: {.automodule members="" undoc-members=""} celery.backends.database ::: --- # `celery.backends.database.models` ::: {.contents local=""} ::: ::: currentmodule celery.backends.database.models ::: ::: {.automodule members="" undoc-members=""} celery.backends.database.models ::: --- # `celery.backends.database.session` ::: {.contents local=""} ::: ::: currentmodule celery.backends.database.session ::: ::: {.automodule members="" undoc-members=""} celery.backends.database.session ::: --- # `celery.backends.dynamodb` ::: {.contents local=""} ::: ::: currentmodule celery.backends.dynamodb ::: ::: {.automodule members="" undoc-members=""} celery.backends.dynamodb ::: --- # `celery.backends.elasticsearch` ::: {.contents local=""} ::: ::: currentmodule celery.backends.elasticsearch ::: ::: {.automodule members="" undoc-members=""} celery.backends.elasticsearch ::: --- # `celery.backends.filesystem` ::: {.contents local=""} ::: ::: currentmodule celery.backends.filesystem ::: ::: {.automodule members="" undoc-members=""} celery.backends.filesystem ::: --- # `celery.backends.gcs` ::: {.contents local=""} ::: ::: currentmodule celery.backends.gcs ::: ::: {.automodule members="" undoc-members=""} celery.backends.gcs ::: --- # `celery.backends` ::: {.contents local=""} ::: ::: currentmodule celery.backends ::: ::: {.automodule members="" undoc-members=""} celery.backends ::: --- # `celery.backends.mongodb` ::: {.contents local=""} ::: ::: currentmodule celery.backends.mongodb ::: ::: {.automodule members="" undoc-members=""} celery.backends.mongodb ::: --- # `celery.backends.redis` ::: {.contents local=""} ::: ::: currentmodule celery.backends.redis ::: ::: {.automodule members="" undoc-members=""} celery.backends.redis ::: --- # `celery.backends.rpc` ::: {.contents local=""} ::: ::: currentmodule celery.backends.rpc ::: ::: {.automodule members="" undoc-members=""} celery.backends.rpc ::: --- # `celery.backends.s3` ::: {.contents local=""} ::: ::: currentmodule celery.backends.s3 ::: ::: {.automodule members="" undoc-members=""} celery.backends.s3 ::: --- # `celery.concurrency.base` ::: {.contents local=""} ::: ::: currentmodule celery.concurrency.base ::: ::: {.automodule members="" undoc-members=""} celery.concurrency.base ::: --- # `celery.concurrency.eventlet` ::: {.contents local=""} ::: ::: currentmodule celery.concurrency.eventlet ::: ::: {.automodule members="" undoc-members=""} celery.concurrency.eventlet ::: --- # `celery.concurrency.gevent` ::: {.contents local=""} ::: ::: currentmodule celery.concurrency.gevent ::: ::: {.automodule members="" undoc-members=""} celery.concurrency.gevent ::: --- # `celery.concurrency` ::: {.contents local=""} ::: ::: currentmodule celery.concurrency ::: ::: {.automodule members="" undoc-members=""} celery.concurrency ::: --- # `celery.concurrency.prefork` ::: {.contents local=""} ::: ::: currentmodule celery.concurrency.prefork ::: ::: {.automodule members="" undoc-members=""} celery.concurrency.prefork ::: --- # `celery.concurrency.solo` ::: {.contents local=""} ::: ::: currentmodule celery.concurrency.solo ::: ::: {.automodule members="" undoc-members=""} celery.concurrency.solo ::: --- # `celery.concurrency.thread` ::: {.contents local=""} ::: ::: currentmodule celery.concurrency.thread ::: ::: {.automodule members="" undoc-members=""} celery.concurrency.thread ::: --- # `celery.events.cursesmon` ::: {.contents local=""} ::: ::: currentmodule celery.events.cursesmon ::: ::: {.automodule members="" undoc-members=""} celery.events.cursesmon ::: --- # `celery.events.dumper` ::: {.contents local=""} ::: ::: currentmodule celery.events.dumper ::: ::: {.automodule members="" undoc-members=""} celery.events.dumper ::: --- # `celery.events.snapshot` ::: {.contents local=""} ::: ::: currentmodule celery.events.snapshot ::: ::: {.automodule members="" undoc-members=""} celery.events.snapshot ::: --- # `celery.platforms` ::: {.contents local=""} ::: ::: currentmodule celery.platforms ::: ::: {.automodule members="" undoc-members=""} celery.platforms ::: --- # `celery.security.certificate` ::: {.contents local=""} ::: ::: currentmodule celery.security.certificate ::: ::: {.automodule members="" undoc-members=""} celery.security.certificate ::: --- # `celery.security.key` ::: {.contents local=""} ::: ::: currentmodule celery.security.key ::: ::: {.automodule members="" undoc-members=""} celery.security.key ::: --- # `celery.security.serialization` ::: {.contents local=""} ::: ::: currentmodule celery.security.serialization ::: ::: {.automodule members="" undoc-members=""} celery.security.serialization ::: --- # `celery.security.utils` ::: {.contents local=""} ::: ::: currentmodule celery.security.utils ::: ::: {.automodule members="" undoc-members=""} celery.security.utils ::: --- # `celery.utils.abstract` ::: {.contents local=""} ::: ::: currentmodule celery.utils.abstract ::: ::: {.automodule members="" undoc-members=""} celery.utils.abstract ::: --- # `celery.utils.collections` ::: currentmodule celery.utils.collections ::: ::: {.contents local=""} ::: ::: {.automodule members="" undoc-members=""} celery.utils.collections ::: --- # `celery.utils.deprecated` ::: {.contents local=""} ::: ::: currentmodule celery.utils.deprecated ::: ::: {.automodule members="" undoc-members=""} celery.utils.deprecated ::: --- # `celery.utils.dispatch` ::: {.contents local=""} ::: ::: currentmodule celery.utils.dispatch ::: ::: {.automodule members="" undoc-members=""} celery.utils.dispatch ::: --- # `celery.utils.dispatch.signal` ::: {.contents local=""} ::: ::: currentmodule celery.utils.dispatch.signal ::: ::: {.automodule members="" undoc-members=""} celery.utils.dispatch.signal ::: --- # `celery.utils.functional` ::: {.contents local=""} ::: ::: currentmodule celery.utils.functional ::: ::: {.automodule members="" undoc-members=""} celery.utils.functional ::: --- # `celery.utils.graph` ::: {.contents local=""} ::: ::: currentmodule celery.utils.graph ::: ::: {.automodule members="" undoc-members=""} celery.utils.graph ::: --- # `celery.utils.imports` ::: {.contents local=""} ::: ::: currentmodule celery.utils.imports ::: ::: {.automodule members="" undoc-members=""} celery.utils.imports ::: --- # `celery.utils.iso8601` ::: {.contents local=""} ::: ::: currentmodule celery.utils.iso8601 ::: ::: {.automodule members="" undoc-members=""} celery.utils.iso8601 ::: --- # `celery.utils.log` ::: {.contents local=""} ::: ::: currentmodule celery.utils.log ::: ::: {.automodule members="" undoc-members=""} celery.utils.log ::: --- # `celery.utils` ::: {.contents local=""} ::: ::: currentmodule celery.utils ::: ::: {.automodule members="" undoc-members=""} celery.utils ::: --- # `celery.utils.nodenames` ::: {.contents local=""} ::: ::: currentmodule celery.utils.nodenames ::: ::: {.automodule members="" undoc-members=""} celery.utils.nodenames ::: --- # `celery.utils.objects` ::: {.contents local=""} ::: ::: currentmodule celery.utils.objects ::: ::: {.automodule members="" undoc-members=""} celery.utils.objects ::: --- # `celery.utils.saferepr` ::: {.contents local=""} ::: ::: currentmodule celery.utils.saferepr ::: ::: {.automodule members="" undoc-members=""} celery.utils.saferepr ::: --- # `celery.utils.serialization` ::: {.contents local=""} ::: ::: currentmodule celery.utils.serialization ::: ::: {.automodule members="" undoc-members=""} celery.utils.serialization ::: --- # `celery.utils.sysinfo` ::: {.contents local=""} ::: ::: currentmodule celery.utils.sysinfo ::: ::: {.automodule members="" undoc-members=""} celery.utils.sysinfo ::: --- # `celery.utils.term` ::: {.contents local=""} ::: ::: currentmodule celery.utils.term ::: ::: {.automodule members="" undoc-members=""} celery.utils.term ::: --- # `celery.utils.text` ::: {.contents local=""} ::: ::: currentmodule celery.utils.text ::: ::: {.automodule members="" undoc-members=""} celery.utils.text ::: --- # `celery.utils.threads` ::: {.contents local=""} ::: ::: currentmodule celery.utils.threads ::: ::: {.automodule members="" undoc-members=""} celery.utils.threads ::: --- # `celery.utils.time` ::: {.contents local=""} ::: ::: currentmodule celery.utils.time ::: ::: {.automodule members="" undoc-members=""} celery.utils.time ::: --- # `celery.utils.timer2` ::: {.contents local=""} ::: ::: currentmodule celery.utils.timer2 ::: ::: {.automodule members="" undoc-members=""} celery.utils.timer2 ::: --- # `celery.worker.autoscale` ::: {.contents local=""} ::: ::: currentmodule celery.worker.autoscale ::: ::: {.automodule members="" undoc-members=""} celery.worker.autoscale ::: --- # `celery.worker.components` ::: {.contents local=""} ::: ::: currentmodule celery.worker.components ::: ::: {.automodule members="" undoc-members=""} celery.worker.components ::: --- # `celery.worker.control` ::: {.contents local=""} ::: ::: currentmodule celery.worker.control ::: ::: {.automodule members="" undoc-members=""} celery.worker.control ::: --- # `celery.worker.heartbeat` ::: {.contents local=""} ::: ::: currentmodule celery.worker.heartbeat ::: ::: {.automodule members="" undoc-members=""} celery.worker.heartbeat ::: --- # `celery.worker.loops` ::: {.contents local=""} ::: ::: currentmodule celery.worker.loops ::: ::: {.automodule members="" undoc-members=""} celery.worker.loops ::: --- # `celery.worker.pidbox` ::: {.contents local=""} ::: ::: currentmodule celery.worker.pidbox ::: ::: {.automodule members="" undoc-members=""} celery.worker.pidbox ::: --- # Internal Module Reference Release : Date : ::: {.toctree maxdepth="1"} celery.worker.components celery.worker.loops celery.worker.heartbeat celery.worker.control celery.worker.pidbox celery.worker.autoscale celery.concurrency celery.concurrency.solo celery.concurrency.prefork celery.concurrency.eventlet celery.concurrency.gevent celery.concurrency.thread celery.concurrency.base celery.backends celery.backends.base celery.backends.asynchronous celery.backends.azureblockblob celery.backends.rpc celery.backends.database celery.backends.cache celery.backends.consul celery.backends.couchdb celery.backends.mongodb celery.backends.elasticsearch celery.backends.redis celery.backends.cassandra celery.backends.couchbase celery.backends.arangodb celery.backends.dynamodb celery.backends.filesystem celery.backends.cosmosdbsql celery.backends.s3 celery.backends.gcs celery.app.trace celery.app.annotations celery.app.routes celery.security.certificate celery.security.key celery.security.serialization celery.security.utils celery.events.snapshot celery.events.cursesmon celery.events.dumper celery.backends.database.models celery.backends.database.session celery.utils celery.utils.abstract celery.utils.collections celery.utils.nodenames celery.utils.deprecated celery.utils.functional celery.utils.graph celery.utils.objects celery.utils.term celery.utils.time celery.utils.iso8601 celery.utils.saferepr celery.utils.serialization celery.utils.sysinfo celery.utils.threads celery.utils.timer2 celery.utils.imports celery.utils.log celery.utils.text celery.utils.dispatch celery.utils.dispatch.signal celery.platforms celery.\_state ::: --- # Internals: The worker {#internals-worker} ::: {.contents local=""} ::: ## Introduction The worker consists of 4 main components: the consumer, the scheduler, the mediator and the task pool. All these components runs in parallel working with two data structures: the ready queue and the ETA schedule. ## Data structures ### timer The timer uses `heapq`{.interpreted-text role="mod"} to schedule internal functions. It\'s very efficient and can handle hundred of thousands of entries. ## Components ### Consumer Receives messages from the broker using `Kombu`{.interpreted-text role="pypi"}. When a message is received it\'s converted into a `celery.worker.request.Request`{.interpreted-text role="class"} object. Tasks with an ETA, or rate-limit are entered into the [timer]{.title-ref}, messages that can be immediately processed are sent to the execution pool. ETA and rate-limit when used together will result in the rate limit being observed with the task being scheduled after the ETA. ### Timer The timer schedules internal functions, like cleanup and internal monitoring, but also it schedules ETA tasks and rate limited tasks. If the scheduled tasks ETA has passed it is moved to the execution pool. ### TaskPool This is a slightly modified `multiprocessing.Pool`{.interpreted-text role="class"}. It mostly works the same way, except it makes sure all of the workers are running at all times. If a worker is missing, it replaces it with a new one. --- ::: currentmodule celery.app.amqp ::: ::: automodule celery.app.amqp ::: {.contents local=""} ::: # AMQP ::: autoclass AMQP ::: attribute Connection Broker connection class used. Default is `kombu.Connection`{.interpreted-text role="class"}. ::: ::: attribute Consumer Base Consumer class used. Default is `kombu.Consumer`{.interpreted-text role="class"}. ::: ::: attribute Producer Base Producer class used. Default is `kombu.Producer`{.interpreted-text role="class"}. ::: ::: attribute queues All currently defined task queues (a `Queues`{.interpreted-text role="class"} instance). ::: ::: attribute argsrepr_maxsize Max size of positional argument representation used for logging purposes. Default is 1024. ::: ::: attribute kwargsrepr_maxsize Max size of keyword argument representation used for logging purposes. Default is 1024. ::: ::: automethod Queues ::: ::: automethod Router ::: ::: automethod flush_routes ::: ::: autoattribute create_task_message ::: ::: autoattribute send_task_message ::: ::: autoattribute default_queue ::: ::: autoattribute default_exchange ::: ::: autoattribute producer_pool ::: ::: autoattribute router ::: ::: autoattribute routes ::: ::: # Queues ::: {.autoclass members="" undoc-members=""} Queues ::: ::: --- # `celery.app.autoretry` ::: {.contents local=""} ::: ::: currentmodule celery.app.autoretry ::: ::: {.automodule members="" undoc-members=""} celery.app.autoretry ::: --- # `celery.app.backends` ::: {.contents local=""} ::: ::: currentmodule celery.app.backends ::: ::: {.automodule members="" undoc-members=""} celery.app.backends ::: --- # `celery.app.builtins` ::: {.contents local=""} ::: ::: currentmodule celery.app.builtins ::: ::: {.automodule members="" undoc-members=""} celery.app.builtins ::: --- # `celery.app.control` ::: {.contents local=""} ::: ::: currentmodule celery.app.control ::: ::: {.automodule members="" undoc-members=""} celery.app.control ::: --- # `celery.app.defaults` ::: {.contents local=""} ::: ::: currentmodule celery.app.defaults ::: ::: {.automodule members="" undoc-members=""} celery.app.defaults ::: --- # `celery.app.events` ::: {.contents local=""} ::: ::: currentmodule celery.app.events ::: ::: {.automodule members="" undoc-members=""} celery.app.events ::: --- # `celery.app.log` ::: {.contents local=""} ::: ::: currentmodule celery.app.log ::: ::: {.automodule members="" undoc-members=""} celery.app.log ::: --- ::: currentmodule celery.app ::: ::: automodule celery.app ::: {.contents local=""} ::: # Proxies ::: autodata default_app ::: # Functions ::: autofunction app_or_default ::: ::: autofunction enable_trace ::: ::: autofunction disable_trace ::: ::: --- # `celery.app.registry` ::: {.contents local=""} ::: ::: currentmodule celery.app.registry ::: ::: {.automodule members="" undoc-members=""} celery.app.registry ::: --- # `celery.app.task` ::: {.contents local=""} ::: ::: currentmodule celery.app.task ::: ::: {.automodule members="Task, Context, TaskType"} celery.app.task ::: --- # `celery.app.utils` ::: {.contents local=""} ::: ::: currentmodule celery.app.utils ::: ::: {.automodule members="" undoc-members=""} celery.app.utils ::: --- # `celery.apps.beat` ::: {.contents local=""} ::: ::: currentmodule celery.apps.beat ::: ::: {.automodule members="" undoc-members=""} celery.apps.beat ::: --- # `celery.apps.multi` ::: {.contents local=""} ::: ::: currentmodule celery.apps.multi ::: ::: {.automodule members="" undoc-members=""} celery.apps.multi ::: --- # `celery.apps.worker` ::: {.contents local=""} ::: ::: currentmodule celery.apps.worker ::: ::: {.automodule members="" undoc-members=""} celery.apps.worker ::: --- # `celery.beat` ::: {.contents local=""} ::: ::: currentmodule celery.beat ::: ::: {.automodule members="" undoc-members=""} celery.beat ::: --- # `celery.bin.amqp` ::: {.contents local=""} ::: ::: currentmodule celery.bin.amqp ::: ::: {.automodule members="" undoc-members=""} celery.bin.amqp ::: --- # `celery.bin.base` ::: {.contents local=""} ::: ::: currentmodule celery.bin.base ::: ::: {.automodule members="" undoc-members=""} celery.bin.base ::: --- # `celery.bin.beat` ::: {.contents local=""} ::: ::: currentmodule celery.bin.beat ::: ::: {.automodule members="" undoc-members=""} celery.bin.beat ::: --- # `celery.bin.call` ::: {.contents local=""} ::: ::: currentmodule celery.bin.call ::: ::: {.automodule members="" undoc-members=""} celery.bin.call ::: --- # `celery.bin.celery` ::: {.contents local=""} ::: ::: currentmodule celery.bin.celery ::: ::: {.automodule members="" undoc-members=""} celery.bin.celery ::: --- # `celery.bin.control` ::: {.contents local=""} ::: ::: currentmodule celery.bin.control ::: ::: {.automodule members="" undoc-members=""} celery.bin.control ::: --- # `celery.bin.events` ::: {.contents local=""} ::: ::: currentmodule celery.bin.events ::: ::: {.automodule members="" undoc-members=""} celery.bin.events ::: --- # `celery.bin.graph` ::: {.contents local=""} ::: ::: currentmodule celery.bin.graph ::: ::: {.automodule members="" undoc-members=""} celery.bin.graph ::: --- # `celery.bin.list` ::: {.contents local=""} ::: ::: currentmodule celery.bin.list ::: ::: {.automodule members="" undoc-members=""} celery.bin.list ::: --- # `celery.bin.logtool` ::: {.contents local=""} ::: ::: currentmodule celery.bin.logtool ::: ::: {.automodule members="" undoc-members=""} celery.bin.logtool ::: --- # `celery.bin.migrate` ::: {.contents local=""} ::: ::: currentmodule celery.bin.migrate ::: ::: {.automodule members="" undoc-members=""} celery.bin.migrate ::: --- # `celery.bin.multi` ::: {.contents local=""} ::: ::: currentmodule celery.bin.multi ::: ::: {.automodule members="" undoc-members=""} celery.bin.multi ::: --- # `celery.bin.purge` ::: {.contents local=""} ::: ::: currentmodule celery.bin.purge ::: ::: {.automodule members="" undoc-members=""} celery.bin.purge ::: --- # `celery.bin.result` ::: {.contents local=""} ::: ::: currentmodule celery.bin.result ::: ::: {.automodule members="" undoc-members=""} celery.bin.result ::: --- # `celery.bin.shell` ::: {.contents local=""} ::: ::: currentmodule celery.bin.shell ::: ::: {.automodule members="" undoc-members=""} celery.bin.shell ::: --- # `celery.bin.upgrade` ::: {.contents local=""} ::: ::: currentmodule celery.bin.upgrade ::: ::: {.automodule members="" undoc-members=""} celery.bin.upgrade ::: --- # `celery.bin.worker` ::: {.contents local=""} ::: ::: currentmodule celery.bin.worker ::: ::: {.automodule members="" undoc-members=""} celery.bin.worker ::: --- # `celery.bootsteps` ::: {.contents local=""} ::: ::: currentmodule celery.bootsteps ::: ::: {.automodule members="" undoc-members=""} celery.bootsteps ::: --- # `celery.contrib.abortable` ::: {.contents local=""} ::: ::: currentmodule celery.contrib.abortable ::: ::: {.automodule members="" undoc-members=""} celery.contrib.abortable ::: --- # `celery.contrib.django.task` ::: versionadded 5.4 ::: ::: {.contents local=""} ::: ## API Reference ::: currentmodule celery.contrib.django.task ::: ::: {.automodule members="" undoc-members=""} celery.contrib.django.task ::: --- # `celery.contrib.migrate` ::: {.contents local=""} ::: ::: currentmodule celery.contrib.migrate ::: ::: {.automodule members="" undoc-members=""} celery.contrib.migrate ::: --- # `celery.contrib.pytest` ::: {.contents local=""} ::: ## API Reference ::: currentmodule celery.contrib.pytest ::: ::: {.automodule members="" undoc-members=""} celery.contrib.pytest ::: --- # `celery.contrib.rdb` ::: currentmodule celery.contrib.rdb ::: ::: automodule celery.contrib.rdb ::: autofunction set_trace ::: ::: autofunction debugger ::: ::: autoclass Rdb ::: ::: --- # celery.contrib.sphinx ::: currentmodule celery.contrib.sphinx ::: ::: {.automodule members=""} celery.contrib.sphinx ::: --- # `celery.contrib.testing.app` ::: {.contents local=""} ::: ## API Reference ::: currentmodule celery.contrib.testing.app ::: ::: {.automodule members="" undoc-members=""} celery.contrib.testing.app ::: --- # `celery.contrib.testing.manager` ::: {.contents local=""} ::: ## API Reference ::: currentmodule celery.contrib.testing.manager ::: ::: {.automodule members="" undoc-members=""} celery.contrib.testing.manager ::: --- # `celery.contrib.testing.mocks` ::: {.contents local=""} ::: ## API Reference ::: currentmodule celery.contrib.testing.mocks ::: ::: {.automodule members="" undoc-members=""} celery.contrib.testing.mocks ::: --- # `celery.contrib.testing.worker` ::: {.contents local=""} ::: ## API Reference ::: currentmodule celery.contrib.testing.worker ::: ::: {.automodule members="" undoc-members=""} celery.contrib.testing.worker ::: --- # `celery.events.state` ::: {.contents local=""} ::: ::: currentmodule celery.events.dispatcher ::: ::: {.automodule members="" undoc-members=""} celery.events.dispatcher ::: --- # `celery.events.event` ::: {.contents local=""} ::: ::: currentmodule celery.events.event ::: ::: {.automodule members="" undoc-members=""} celery.events.event ::: --- # `celery.events` ::: {.contents local=""} ::: ::: currentmodule celery.events ::: ::: {.automodule members="" undoc-members=""} celery.events ::: --- # `celery.events.receiver` ::: {.contents local=""} ::: ::: currentmodule celery.events.receiver ::: ::: {.automodule members="" undoc-members=""} celery.events.receiver ::: --- # `celery.events.state` ::: {.contents local=""} ::: ::: currentmodule celery.events.state ::: ::: {.automodule members="" undoc-members=""} celery.events.state ::: --- # `celery.exceptions` ::: {.contents local=""} ::: ::: currentmodule celery.exceptions ::: ::: {.automodule members="" undoc-members=""} celery.exceptions ::: --- # `celery.loaders.app` ::: {.contents local=""} ::: ::: currentmodule celery.loaders.app ::: ::: {.automodule members="" undoc-members=""} celery.loaders.app ::: --- # `celery.loaders.base` ::: {.contents local=""} ::: ::: currentmodule celery.loaders.base ::: ::: {.automodule members="" undoc-members=""} celery.loaders.base ::: --- # `celery.loaders.default` ::: {.contents local=""} ::: ::: currentmodule celery.loaders.default ::: ::: {.automodule members="" undoc-members=""} celery.loaders.default ::: --- # `celery.loaders` ::: {.contents local=""} ::: ::: currentmodule celery.loaders ::: ::: {.automodule members="" undoc-members=""} celery.loaders ::: --- # `celery`{.interpreted-text role="mod"} \-\-- Distributed processing ::: currentmodule celery ::: ::: {.module synopsis="Distributed processing"} celery ::: ::: moduleauthor Ask Solem \<\> ::: ::: sectionauthor Ask Solem \<\> ::: ------------------------------------------------------------------------ This module is the main entry-point for the Celery API. It includes commonly needed things for calling tasks, and creating Celery applications. ----------------------------------------------- ------------------------------------------- `Celery`{.interpreted-text role="class"} Celery application instance `group`{.interpreted-text role="class"} group tasks together `chain`{.interpreted-text role="class"} chain tasks together `chord`{.interpreted-text role="class"} chords enable callbacks for groups `signature`{.interpreted-text role="func"} create a new task signature `Signature`{.interpreted-text role="class"} object describing a task invocation `current_app`{.interpreted-text role="data"} proxy to the current application instance `current_task`{.interpreted-text role="data"} proxy to the currently executing task ----------------------------------------------- ------------------------------------------- ## `Celery`{.interpreted-text role="class"} application objects ::: versionadded 2.5 ::: ::: autoclass Celery ::: autoattribute user_options ::: ::: autoattribute steps ::: ::: autoattribute current_task ::: ::: autoattribute current_worker_task ::: ::: autoattribute amqp ::: ::: autoattribute backend ::: ::: autoattribute loader ::: ::: autoattribute control ::: ::: autoattribute events ::: ::: autoattribute log ::: ::: autoattribute tasks ::: ::: autoattribute pool ::: ::: autoattribute producer_pool ::: ::: autoattribute Task ::: ::: autoattribute timezone ::: ::: autoattribute builtin_fixups ::: ::: autoattribute oid ::: ::: automethod close ::: ::: automethod signature ::: ::: automethod bugreport ::: ::: automethod config_from_object ::: ::: automethod config_from_envvar ::: ::: automethod autodiscover_tasks ::: ::: automethod add_defaults ::: ::: automethod add_periodic_task ::: ::: automethod setup_security ::: ::: automethod task ::: ::: automethod send_task ::: ::: automethod gen_task_name ::: ::: autoattribute AsyncResult ::: ::: autoattribute GroupResult ::: ::: autoattribute Worker ::: ::: autoattribute WorkController ::: ::: autoattribute Beat ::: ::: automethod connection_for_read ::: ::: automethod connection_for_write ::: ::: automethod connection ::: ::: automethod connection_or_acquire ::: ::: automethod producer_or_acquire ::: ::: automethod select_queues ::: ::: automethod now ::: ::: automethod set_current ::: ::: automethod set_default ::: ::: automethod finalize ::: ::: automethod on_init ::: ::: automethod prepare_config ::: ::: data on_configure Signal sent when app is loading configuration. ::: ::: data on_after_configure Signal sent after app has prepared the configuration. ::: ::: data on_after_finalize Signal sent after app has been finalized. ::: ::: data on_after_fork Signal sent in child process after fork. ::: ::: ## Canvas primitives See `guide-canvas`{.interpreted-text role="ref"} for more about creating task work-flows. ::: autoclass group ::: ::: autoclass chain ::: ::: autoclass chord ::: ::: autofunction signature ::: ::: autoclass Signature ::: ## Proxies ::: data current_app The currently set app for this thread. ::: ::: data current_task The task currently being executed (only set in the worker, or when eager/apply is used). ::: --- # `celery.result` ::: {.contents local=""} ::: ::: currentmodule celery.result ::: ::: {.automodule members="" undoc-members=""} celery.result ::: --- # `celery.schedules` ::: {.contents local=""} ::: ::: currentmodule celery.schedules ::: ::: {.automodule members="" undoc-members=""} celery.schedules ::: --- # `celery.security` ::: {.contents local=""} ::: ::: currentmodule celery.security ::: ::: {.automodule members="" undoc-members=""} celery.security ::: --- # `celery.signals` ::: {.contents local=""} ::: ::: currentmodule celery.signals ::: ::: {.automodule members="" undoc-members=""} celery.signals ::: --- ::: currentmodule celery.states ::: ::: {.contents local=""} ::: ::: {.automodule members=""} celery.states ::: --- # `celery.utils.debug` ::: {.contents local=""} ::: ## Sampling Memory Usage This module can be used to diagnose and sample the memory usage used by parts of your application. For example, to sample the memory usage of calling tasks you can do this: ``` python from celery.utils.debug import sample_mem, memdump from tasks import add try: for i in range(100): for j in range(100): add.delay(i, j) sample_mem() finally: memdump() ``` ## API Reference ::: currentmodule celery.utils.debug ::: ::: automodule celery.utils.debug ::: autofunction sample_mem ::: ::: autofunction memdump ::: ::: autofunction sample ::: ::: autofunction mem_rss ::: ::: autofunction ps ::: ::: --- # `celery.worker.consumer.agent` ::: {.contents local=""} ::: ::: currentmodule celery.worker.consumer.agent ::: ::: {.automodule members="" undoc-members=""} celery.worker.consumer.agent ::: --- # `celery.worker.consumer.connection` ::: {.contents local=""} ::: ::: currentmodule celery.worker.consumer.connection ::: ::: {.automodule members="" undoc-members=""} celery.worker.consumer.connection ::: --- # `celery.worker.consumer.consumer` ::: {.contents local=""} ::: ::: currentmodule celery.worker.consumer.consumer ::: ::: {.automodule members="" undoc-members=""} celery.worker.consumer.consumer ::: --- # `celery.worker.consumer.control` ::: {.contents local=""} ::: ::: currentmodule celery.worker.consumer.control ::: ::: {.automodule members="" undoc-members=""} celery.worker.consumer.control ::: --- # `celery.worker.consumer.events` ::: {.contents local=""} ::: ::: currentmodule celery.worker.consumer.events ::: ::: {.automodule members="" undoc-members=""} celery.worker.consumer.events ::: --- # `celery.worker.consumer.gossip` ::: {.contents local=""} ::: ::: currentmodule celery.worker.consumer.gossip ::: ::: {.automodule members="" undoc-members=""} celery.worker.consumer.gossip ::: --- # `celery.worker.consumer.heart` ::: {.contents local=""} ::: ::: currentmodule celery.worker.consumer.heart ::: ::: {.automodule members="" undoc-members=""} celery.worker.consumer.heart ::: --- # `celery.worker.consumer` ::: {.contents local=""} ::: ::: currentmodule celery.worker.consumer ::: ::: {.automodule members="" undoc-members=""} celery.worker.consumer ::: --- # `celery.worker.consumer.mingle` ::: {.contents local=""} ::: ::: currentmodule celery.worker.consumer.mingle ::: ::: {.automodule members="" undoc-members=""} celery.worker.consumer.mingle ::: --- # `celery.worker.consumer.tasks` ::: {.contents local=""} ::: ::: currentmodule celery.worker.consumer.tasks ::: ::: {.automodule members="" undoc-members=""} celery.worker.consumer.tasks ::: --- # `celery.worker` ::: {.contents local=""} ::: ::: currentmodule celery.worker ::: ::: {.automodule members="" undoc-members=""} celery.worker ::: --- # `celery.worker.request` ::: {.contents local=""} ::: ::: currentmodule celery.worker.request ::: ::: {.automodule members="" undoc-members=""} celery.worker.request ::: --- # `celery.worker.state` ::: {.contents local=""} ::: ::: currentmodule celery.worker.state ::: ::: {.automodule members="" undoc-members=""} celery.worker.state ::: --- # `celery.worker.strategy` ::: {.contents local=""} ::: ::: currentmodule celery.worker.strategy ::: ::: {.automodule members="" undoc-members=""} celery.worker.strategy ::: --- # `celery.worker.worker` ::: {.contents local=""} ::: ::: currentmodule celery.worker.worker ::: ::: {.automodule members="" undoc-members=""} celery.worker.worker ::: --- # Command Line Interface ::: note ::: title Note ::: The prefix [CELERY\_]{.title-ref} must be added to the names of the environment variables described below. E.g., [APP]{.title-ref} becomes [CELERY_APP]{.title-ref}. ::: ::: {.click prog="celery" nested="full"} celery.bin.celery:celery ::: --- # API Reference {#apiref} Release : Date : ::: {.toctree maxdepth="1"} cli celery celery.app celery.app.task celery.app.amqp celery.app.defaults celery.app.control celery.app.registry celery.app.backends celery.app.builtins celery.app.events celery.app.log celery.app.utils celery.app.autoretry celery.bootsteps celery.result celery.schedules celery.signals celery.security celery.utils.debug celery.exceptions celery.loaders celery.loaders.app celery.loaders.default celery.loaders.base celery.states celery.contrib.abortable celery.contrib.django.task celery.contrib.migrate celery.contrib.pytest celery.contrib.sphinx celery.contrib.testing.worker celery.contrib.testing.app celery.contrib.testing.manager celery.contrib.testing.mocks celery.contrib.rdb celery.events celery.events.receiver celery.events.dispatcher celery.events.event celery.events.state celery.beat celery.apps.worker celery.apps.beat celery.apps.multi celery.worker celery.worker.request celery.worker.state celery.worker.strategy celery.worker.consumer celery.worker.consumer.agent celery.worker.consumer.connection celery.worker.consumer.consumer celery.worker.consumer.control celery.worker.consumer.events celery.worker.consumer.gossip celery.worker.consumer.heart celery.worker.consumer.mingle celery.worker.consumer.tasks celery.worker.worker celery.bin.base celery.bin.celery celery.bin.worker celery.bin.beat celery.bin.events celery.bin.logtool celery.bin.amqp celery.bin.graph celery.bin.multi celery.bin.call celery.bin.control celery.bin.list celery.bin.migrate celery.bin.purge celery.bin.result celery.bin.shell celery.bin.upgrade ::: --- # Tutorials Release : Date : ::: {.toctree maxdepth="2"} task-cookbook ::: --- # Task Cookbook {#cookbook-tasks} ::: {.contents local=""} ::: ## Ensuring a task is only executed one at a time {#cookbook-task-serial} You can accomplish this by using a lock. In this example we\'ll be using the cache framework to set a lock that\'s accessible for all workers. It\'s part of an imaginary RSS feed importer called [djangofeeds]{.title-ref}. The task takes a feed URL as a single argument, and imports that feed into a Django model called [Feed]{.title-ref}. We ensure that it\'s not possible for two or more workers to import the same feed at the same time by setting a cache key consisting of the MD5 check-sum of the feed URL. The cache key expires after some time in case something unexpected happens, and something always will\... For this reason your tasks run-time shouldn\'t exceed the timeout. ::: note ::: title Note ::: In order for this to work correctly you need to be using a cache backend where the `.add` operation is atomic. `memcached` is known to work well for this purpose. ::: ``` python import time from celery import task from celery.utils.log import get_task_logger from contextlib import contextmanager from django.core.cache import cache from hashlib import md5 from djangofeeds.models import Feed logger = get_task_logger(__name__) LOCK_EXPIRE = 60 * 10 # Lock expires in 10 minutes @contextmanager def memcache_lock(lock_id, oid): timeout_at = time.monotonic() + LOCK_EXPIRE - 3 # cache.add fails if the key already exists status = cache.add(lock_id, oid, LOCK_EXPIRE) try: yield status finally: # memcache delete is very slow, but we have to use it to take # advantage of using add() for atomic locking if time.monotonic() < timeout_at and status: # don't release the lock if we exceeded the timeout # to lessen the chance of releasing an expired lock # owned by someone else # also don't release the lock if we didn't acquire it cache.delete(lock_id) @task(bind=True) def import_feed(self, feed_url): # The cache key consists of the task name and the MD5 digest # of the feed URL. feed_url_hexdigest = md5(feed_url).hexdigest() lock_id = '{0}-lock-{1}'.format(self.name, feed_url_hexdigest) logger.debug('Importing feed: %s', feed_url) with memcache_lock(lock_id, self.app.oid) as acquired: if acquired: return Feed.objects.import_feed(feed_url).url logger.debug( 'Feed %s is already being imported by another worker', feed_url) ``` --- # Application {#guide-app} ::: {.contents local="" depth="1"} ::: The Celery library must be instantiated before use, this instance is called an application (or *app* for short). The application is thread-safe so that multiple Celery applications with different configurations, components, and tasks can co-exist in the same process space. Let\'s create one now: ``` pycon >>> from celery import Celery >>> app = Celery() >>> app ``` The last line shows the textual representation of the application: including the name of the app class (`Celery`), the name of the current main module (`__main__`), and the memory address of the object (`0x100469fd0`). ## Main Name Only one of these is important, and that\'s the main module name. Let\'s look at why that is. When you send a task message in Celery, that message won\'t contain any source code, but only the name of the task you want to execute. This works similarly to how host names work on the internet: every worker maintains a mapping of task names to their actual functions, called the *task registry*. Whenever you define a task, that task will also be added to the local registry: ``` pycon >>> @app.task ... def add(x, y): ... return x + y >>> add <@task: __main__.add> >>> add.name __main__.add >>> app.tasks['__main__.add'] <@task: __main__.add> ``` and there you see that `__main__` again; whenever Celery isn\'t able to detect what module the function belongs to, it uses the main module name to generate the beginning of the task name. This is only a problem in a limited set of use cases: > 1. If the module that the task is defined in is run as a program. > 2. If the application is created in the Python shell (REPL). For example here, where the tasks module is also used to start a worker with `@worker_main`{.interpreted-text role="meth"}: `tasks.py`{.interpreted-text role="file"}: ``` python from celery import Celery app = Celery() @app.task def add(x, y): return x + y if __name__ == '__main__': args = ['worker', '--loglevel=INFO'] app.worker_main(argv=args) ``` When this module is executed the tasks will be named starting with \"`__main__`\", but when the module is imported by another process, say to call a task, the tasks will be named starting with \"`tasks`\" (the real name of the module): ``` pycon >>> from tasks import add >>> add.name tasks.add ``` You can specify another name for the main module: ``` pycon >>> app = Celery('tasks') >>> app.main 'tasks' >>> @app.task ... def add(x, y): ... return x + y >>> add.name tasks.add ``` ::: seealso `task-names`{.interpreted-text role="ref"} ::: ## Configuration There are several options you can set that\'ll change how Celery works. These options can be set directly on the app instance, or you can use a dedicated configuration module. The configuration is available as `@conf`{.interpreted-text role="attr"}: ``` pycon >>> app.conf.timezone 'Europe/London' ``` where you can also set configuration values directly: ``` pycon >>> app.conf.enable_utc = True ``` or update several keys at once by using the `update` method: ``` python >>> app.conf.update( ... enable_utc=True, ... timezone='Europe/London', ...) ``` The configuration object consists of multiple dictionaries that are consulted in order: > 1. Changes made at run-time. > 2. The configuration module (if any) > 3. The default configuration (`celery.app.defaults`{.interpreted-text > role="mod"}). You can even add new default sources by using the `@add_defaults`{.interpreted-text role="meth"} method. ::: seealso Go to the `Configuration reference `{.interpreted-text role="ref"} for a complete listing of all the available settings, and their default values. ::: ### `config_from_object` The `@config_from_object`{.interpreted-text role="meth"} method loads configuration from a configuration object. This can be a configuration module, or any object with configuration attributes. Note that any configuration that was previously set will be reset when `~@config_from_object`{.interpreted-text role="meth"} is called. If you want to set additional configuration you should do so after. #### Example 1: Using the name of a module The `@config_from_object`{.interpreted-text role="meth"} method can take the fully qualified name of a Python module, or even the name of a Python attribute, for example: `"celeryconfig"`, `"myproj.config.celery"`, or `"myproj.config:CeleryConfig"`: ``` python from celery import Celery app = Celery() app.config_from_object('celeryconfig') ``` The `celeryconfig` module may then look like this: `celeryconfig.py`{.interpreted-text role="file"}: ``` python enable_utc = True timezone = 'Europe/London' ``` and the app will be able to use it as long as `import celeryconfig` is possible. #### Example 2: Passing an actual module object You can also pass an already imported module object, but this isn\'t always recommended. ::: tip ::: title Tip ::: Using the name of a module is recommended as this means the module does not need to be serialized when the prefork pool is used. If you\'re experiencing configuration problems or pickle errors then please try using the name of a module instead. ::: ``` python import celeryconfig from celery import Celery app = Celery() app.config_from_object(celeryconfig) ``` #### Example 3: Using a configuration class/object ``` python from celery import Celery app = Celery() class Config: enable_utc = True timezone = 'Europe/London' app.config_from_object(Config) # or using the fully qualified name of the object: # app.config_from_object('module:Config') ``` ### `config_from_envvar` The `@config_from_envvar`{.interpreted-text role="meth"} takes the configuration module name from an environment variable For example \-- to load configuration from a module specified in the environment variable named `CELERY_CONFIG_MODULE`{.interpreted-text role="envvar"}: ``` python import os from celery import Celery #: Set default configuration module name os.environ.setdefault('CELERY_CONFIG_MODULE', 'celeryconfig') app = Celery() app.config_from_envvar('CELERY_CONFIG_MODULE') ``` You can then specify the configuration module to use via the environment: ``` console $ CELERY_CONFIG_MODULE="celeryconfig.prod" celery worker -l INFO ``` ### Censored configuration {#app-censored-config} If you ever want to print out the configuration, as debugging information or similar, you may also want to filter out sensitive information like passwords and API keys. Celery comes with several utilities useful for presenting the configuration, one is `~celery.app.utils.Settings.humanize`{.interpreted-text role="meth"}: ``` pycon >>> app.conf.humanize(with_defaults=False, censored=True) ``` This method returns the configuration as a tabulated string. This will only contain changes to the configuration by default, but you can include the built-in default keys and values by enabling the `with_defaults` argument. If you instead want to work with the configuration as a dictionary, you can use the `~celery.app.utils.Settings.table`{.interpreted-text role="meth"} method: ``` pycon >>> app.conf.table(with_defaults=False, censored=True) ``` Please note that Celery won\'t be able to remove all sensitive information, as it merely uses a regular expression to search for commonly named keys. If you add custom settings containing sensitive information you should name the keys using a name that Celery identifies as secret. A configuration setting will be censored if the name contains any of these sub-strings: `API`, `TOKEN`, `KEY`, `SECRET`, `PASS`, `SIGNATURE`, `DATABASE` ## Laziness The application instance is lazy, meaning it won\'t be evaluated until it\'s actually needed. Creating a `@Celery`{.interpreted-text role="class"} instance will only do the following: > 1. Create a logical clock instance, used for events. > 2. Create the task registry. > 3. Set itself as the current app (but not if the `set_as_current` > argument was disabled) > 4. Call the `@on_init`{.interpreted-text role="meth"} callback (does > nothing by default). The `@task`{.interpreted-text role="meth"} decorators don\'t create the tasks at the point when the task is defined, instead it\'ll defer the creation of the task to happen either when the task is used, or after the application has been *finalized*, This example shows how the task isn\'t created until you use the task, or access an attribute (in this case `repr`{.interpreted-text role="meth"}): ``` pycon >>> @app.task >>> def add(x, y): ... return x + y >>> type(add) >>> add.__evaluated__() False >>> add # <-- causes repr(add) to happen <@task: __main__.add> >>> add.__evaluated__() True ``` *Finalization* of the app happens either explicitly by calling `@finalize`{.interpreted-text role="meth"} \-- or implicitly by accessing the `@tasks`{.interpreted-text role="attr"} attribute. Finalizing the object will: > 1. Copy tasks that must be shared between apps > > > Tasks are shared by default, but if the `shared` argument to the > > task decorator is disabled, then the task will be private to the > > app it\'s bound to. > > 2. Evaluate all pending task decorators. > > 3. Make sure all tasks are bound to the current app. > > > Tasks are bound to an app so that they can read default values > > from the configuration. ::: {#default-app} ::: topic **The \"default app\"** Celery didn\'t always have applications, it used to be that there was only a module-based API. A compatibility API was available at the old location until the release of Celery 5.0, but has been removed. Celery always creates a special app - the \"default app\", and this is used if no custom application has been instantiated. The `celery.task`{.interpreted-text role="mod"} module is no longer available. Use the methods on the app instance, not the module based API: ``` python from celery.task import Task # << OLD Task base class. from celery import Task # << NEW base class. ``` ::: ::: ## Breaking the chain While it\'s possible to depend on the current app being set, the best practice is to always pass the app instance around to anything that needs it. I call this the \"app chain\", since it creates a chain of instances depending on the app being passed. The following example is considered bad practice: ``` python from celery import current_app class Scheduler: def run(self): app = current_app ``` Instead it should take the `app` as an argument: ``` python class Scheduler: def __init__(self, app): self.app = app ``` Internally Celery uses the `celery.app.app_or_default`{.interpreted-text role="func"} function so that everything also works in the module-based compatibility API ``` python from celery.app import app_or_default class Scheduler: def __init__(self, app=None): self.app = app_or_default(app) ``` In development you can set the `CELERY_TRACE_APP`{.interpreted-text role="envvar"} environment variable to raise an exception if the app chain breaks: ``` console $ CELERY_TRACE_APP=1 celery worker -l INFO ``` ::: topic **Evolving the API** Celery has changed a lot from 2009 since it was initially created. For example, in the beginning it was possible to use any callable as a task: ``` pycon def hello(to): return 'hello {0}'.format(to) >>> from celery.execute import apply_async >>> apply_async(hello, ('world!',)) ``` or you could also create a `Task` class to set certain options, or override other behavior ``` python from celery import Task from celery.registry import tasks class Hello(Task): queue = 'hipri' def run(self, to): return 'hello {0}'.format(to) tasks.register(Hello) >>> Hello.delay('world!') ``` Later, it was decided that passing arbitrary call-able\'s was an anti-pattern, since it makes it very hard to use serializers other than pickle, and the feature was removed in 2.0, replaced by task decorators: ``` python from celery import app @app.task(queue='hipri') def hello(to): return 'hello {0}'.format(to) ``` ::: ## Abstract Tasks All tasks created using the `@task`{.interpreted-text role="meth"} decorator will inherit from the application\'s base `~@Task`{.interpreted-text role="attr"} class. You can specify a different base class using the `base` argument: ``` python @app.task(base=OtherTask): def add(x, y): return x + y ``` To create a custom task class you should inherit from the neutral base class: `celery.Task`{.interpreted-text role="class"}. ``` python from celery import Task class DebugTask(Task): def __call__(self, *args, **kwargs): print('TASK STARTING: {0.name}[{0.request.id}]'.format(self)) return self.run(*args, **kwargs) ``` ::: tip ::: title Tip ::: If you override the task\'s `__call__` method, then it\'s very important that you also call `self.run` to execute the body of the task. Do not call `super().__call__`. The `__call__` method of the neutral base class `celery.Task`{.interpreted-text role="class"} is only present for reference. For optimization, this has been unrolled into `celery.app.trace.build_tracer.trace_task` which calls `run` directly on the custom task class if no `__call__` method is defined. ::: The neutral base class is special because it\'s not bound to any specific app yet. Once a task is bound to an app it\'ll read configuration to set default values, and so on. To realize a base class you need to create a task using the `@task`{.interpreted-text role="meth"} decorator: ``` python @app.task(base=DebugTask) def add(x, y): return x + y ``` It\'s even possible to change the default base class for an application by changing its `@Task`{.interpreted-text role="meth"} attribute: ``` pycon >>> from celery import Celery, Task >>> app = Celery() >>> class MyBaseTask(Task): ... queue = 'hipri' >>> app.Task = MyBaseTask >>> app.Task >>> @app.task ... def add(x, y): ... return x + y >>> add <@task: __main__.add> >>> add.__class__.mro() [>, , , ] ``` --- # Calling Tasks {#guide-calling} ::: {.contents local="" depth="1"} ::: ## Basics {#calling-basics} This document describes Celery\'s uniform \"Calling API\" used by task instances and the `canvas `{.interpreted-text role="ref"}. The API defines a standard set of execution options, as well as three methods: > - `apply_async(args[, kwargs[, …]])` > > > Sends a task message. > > - `delay(*args, **kwargs)` > > > Shortcut to send a task message, but doesn\'t support execution > > options. > > - *calling* (`__call__`) > > > Applying an object supporting the calling API (e.g., > > `add(2, 2)`) means that the task will not be executed by a > > worker, but in the current process instead (a message won\'t be > > sent). ::: {#calling-cheat} ::: topic **Quick Cheat Sheet** - `T.delay(arg, kwarg=value)` : Star arguments shortcut to `.apply_async`. (`.delay(*args, **kwargs)` calls `.apply_async(args, kwargs)`). - `T.apply_async((arg,), {'kwarg': value})` - `T.apply_async(countdown=10)` : executes in 10 seconds from now. - `T.apply_async(eta=now + timedelta(seconds=10))` : executes in 10 seconds from now, specified using `eta` - `T.apply_async(countdown=60, expires=120)` : executes in one minute from now, but expires after 2 minutes. - `T.apply_async(expires=now + timedelta(days=2))` : expires in 2 days, set using `~datetime.datetime`{.interpreted-text role="class"}. - `T.apply_async(task_id=f'my_own_task_id')` : sets the id of the task to my_own_task_id instead of a uuid that is normally generated ::: ::: ### Example The `~@Task.delay`{.interpreted-text role="meth"} method is convenient as it looks like calling a regular function: ``` python task.delay(arg1, arg2, kwarg1='x', kwarg2='y') ``` Using `~@Task.apply_async`{.interpreted-text role="meth"} instead you have to write: ``` python task.apply_async(args=[arg1, arg2], kwargs={'kwarg1': 'x', 'kwarg2': 'y'}) ``` ::: sidebar **Tip** If the task isn\'t registered in the current process you can use `~@send_task`{.interpreted-text role="meth"} to call the task by name instead. ::: So [delay]{.title-ref} is clearly convenient, but if you want to set additional execution options you have to use `apply_async`. The rest of this document will go into the task execution options in detail. All examples use a task called [add]{.title-ref}, returning the sum of two arguments: ``` python @app.task def add(x, y): return x + y ``` ::: topic **There\'s another way...** You\'ll learn more about this later while reading about the `Canvas `{.interpreted-text role="ref"}, but `~celery.signature`{.interpreted-text role="class"}\'s are objects used to pass around the signature of a task invocation, (for example to send it over the network), and they also support the Calling API: ``` python task.s(arg1, arg2, kwarg1='x', kwargs2='y').apply_async() ``` ::: ## Linking (callbacks/errbacks) {#calling-links} Celery supports linking tasks together so that one task follows another. The callback task will be applied with the result of the parent task as a partial argument: ``` python add.apply_async((2, 2), link=add.s(16)) ``` ::: sidebar **What\'s `s`?** The `add.s` call used here is called a signature. If you don\'t know what they are you should read about them in the `canvas guide `{.interpreted-text role="ref"}. There you can also learn about `~celery.chain`{.interpreted-text role="class"}: a simpler way to chain tasks together. In practice the `link` execution option is considered an internal primitive, and you\'ll probably not use it directly, but use chains instead. ::: Here the result of the first task (4) will be sent to a new task that adds 16 to the previous result, forming the expression $(2 + 2) + 16 = 20$ You can also cause a callback to be applied if task raises an exception (*errback*). The worker won\'t actually call the errback as a task, but will instead call the errback function directly so that the raw request, exception and traceback objects can be passed to it. This is an example error callback: ``` python @app.task def error_handler(request, exc, traceback): print('Task {0} raised exception: {1!r}\n{2!r}'.format( request.id, exc, traceback)) ``` it can be added to the task using the `link_error` execution option: ``` python add.apply_async((2, 2), link_error=error_handler.s()) ``` In addition, both the `link` and `link_error` options can be expressed as a list: ``` python add.apply_async((2, 2), link=[add.s(16), other_task.s()]) ``` The callbacks/errbacks will then be called in order, and all callbacks will be called with the return value of the parent task as a partial argument. In the case of a chord, we can handle errors using multiple handling strategies. See `chord error handling `{.interpreted-text role="ref"} for more information. ## On message {#calling-on-message} Celery supports catching all states changes by setting on_message callback. For example for long-running tasks to send task progress you can do something like this: ``` python @app.task(bind=True) def hello(self, a, b): time.sleep(1) self.update_state(state="PROGRESS", meta={'progress': 50}) time.sleep(1) self.update_state(state="PROGRESS", meta={'progress': 90}) time.sleep(1) return 'hello world: %i' % (a+b) ``` ``` python def on_raw_message(body): print(body) a, b = 1, 1 r = hello.apply_async(args=(a, b)) print(r.get(on_message=on_raw_message, propagate=False)) ``` Will generate output like this: ``` text {'task_id': '5660d3a3-92b8-40df-8ccc-33a5d1d680d7', 'result': {'progress': 50}, 'children': [], 'status': 'PROGRESS', 'traceback': None} {'task_id': '5660d3a3-92b8-40df-8ccc-33a5d1d680d7', 'result': {'progress': 90}, 'children': [], 'status': 'PROGRESS', 'traceback': None} {'task_id': '5660d3a3-92b8-40df-8ccc-33a5d1d680d7', 'result': 'hello world: 10', 'children': [], 'status': 'SUCCESS', 'traceback': None} hello world: 10 ``` ## ETA and Countdown {#calling-eta} The ETA (estimated time of arrival) lets you set a specific date and time that is the earliest time at which your task will be executed. [countdown]{.title-ref} is a shortcut to set ETA by seconds into the future. ``` pycon >>> result = add.apply_async((2, 2), countdown=3) >>> result.get() # this takes at least 3 seconds to return 4 ``` The task is guaranteed to be executed at some time *after* the specified date and time, but not necessarily at that exact time. Possible reasons for broken deadlines may include many items waiting in the queue, or heavy network latency. To make sure your tasks are executed in a timely manner you should monitor the queue for congestion. Use Munin, or similar tools, to receive alerts, so appropriate action can be taken to ease the workload. See `monitoring-munin`{.interpreted-text role="ref"}. While [countdown]{.title-ref} is an integer, [eta]{.title-ref} must be a `~datetime.datetime`{.interpreted-text role="class"} object, specifying an exact date and time (including millisecond precision, and timezone information): ``` pycon >>> from datetime import datetime, timedelta, timezone >>> tomorrow = datetime.now(timezone.utc) + timedelta(days=1) >>> add.apply_async((2, 2), eta=tomorrow) ``` ::: warning ::: title Warning ::: Tasks with [eta]{.title-ref} or [countdown]{.title-ref} are immediately fetched by the worker and until the scheduled time passes, they reside in the worker\'s memory. When using those options to schedule lots of tasks for a distant future, those tasks may accumulate in the worker and make a significant impact on the RAM usage. Moreover, tasks are not acknowledged until the worker starts executing them. If using Redis as a broker, task will get redelivered when [countdown]{.title-ref} exceeds [visibility_timeout]{.title-ref} (see `redis-caveats`{.interpreted-text role="ref"}). Therefore, using [eta]{.title-ref} and [countdown]{.title-ref} **is not recommended** for scheduling tasks for a distant future. Ideally, use values no longer than several minutes. For longer durations, consider using database-backed periodic tasks, e.g. with `django-celery-beat`{.interpreted-text role="pypi"} if using Django (see `beat-custom-schedulers`{.interpreted-text role="ref"}). ::: ::: warning ::: title Warning ::: When using RabbitMQ as a message broker when specifying a `countdown` over 15 minutes, you may encounter the problem that the worker terminates with an `~amqp.exceptions.PreconditionFailed`{.interpreted-text role="exc"} error will be raised: ``` pycon amqp.exceptions.PreconditionFailed: (0, 0): (406) PRECONDITION_FAILED - consumer ack timed out on channel ``` In RabbitMQ since version 3.8.15 the default value for `consumer_timeout` is 15 minutes. Since version 3.8.17 it was increased to 30 minutes. If a consumer does not ack its delivery for more than the timeout value, its channel will be closed with a `PRECONDITION_FAILED` channel exception. See [Delivery Acknowledgement Timeout](https://www.rabbitmq.com/consumers.html#acknowledgement-timeout) for more information. To solve the problem, in RabbitMQ configuration file `rabbitmq.conf` you should specify the `consumer_timeout` parameter greater than or equal to your countdown value. For example, you can specify a very large value of `consumer_timeout = 31622400000`, which is equal to 1 year in milliseconds, to avoid problems in the future. ::: ## Expiration {#calling-expiration} The [expires]{.title-ref} argument defines an optional expiry time, either as seconds after task publish, or a specific date and time using `~datetime.datetime`{.interpreted-text role="class"}: ``` pycon >>> # Task expires after one minute from now. >>> add.apply_async((10, 10), expires=60) >>> # Also supports datetime >>> from datetime import datetime, timedelta, timezone >>> add.apply_async((10, 10), kwargs, ... expires=datetime.now(timezone.utc) + timedelta(days=1)) ``` When a worker receives an expired task it will mark the task as `REVOKED`{.interpreted-text role="state"} (`~@TaskRevokedError`{.interpreted-text role="exc"}). ## Message Sending Retry {#calling-retry} Celery will automatically retry sending messages in the event of connection failure, and retry behavior can be configured \-- like how often to retry, or a maximum number of retries \-- or disabled all together. To disable retry you can set the `retry` execution option to `False`{.interpreted-text role="const"}: ``` python add.apply_async((2, 2), retry=False) ``` ::: topic **Related Settings** ::: {.hlist columns="2"} - `task_publish_retry`{.interpreted-text role="setting"} - `task_publish_retry_policy`{.interpreted-text role="setting"} ::: ::: ### Retry Policy A retry policy is a mapping that controls how retries behave, and can contain the following keys: - [max_retries]{.title-ref} > Maximum number of retries before giving up, in this case the > exception that caused the retry to fail will be raised. > > A value of `None`{.interpreted-text role="const"} means it will > retry forever. > > The default is to retry 3 times. - [interval_start]{.title-ref} > Defines the number of seconds (float or integer) to wait between > retries. Default is 0 (the first retry will be instantaneous). - [interval_step]{.title-ref} > On each consecutive retry this number will be added to the retry > delay (float or integer). Default is 0.2. - [interval_max]{.title-ref} > Maximum number of seconds (float or integer) to wait between > retries. Default is 0.2. - [retry_errors]{.title-ref} > [retry_errors]{.title-ref} is a tuple of exception classes that > should be retried. It will be ignored if not specified. Default is > None (ignored). > > For example, if you want to retry only tasks that were timed out, > you can use `~kombu.exceptions.TimeoutError`{.interpreted-text > role="exc"}: > > ``` python > from kombu.exceptions import TimeoutError > > add.apply_async((2, 2), retry=True, retry_policy={ > 'max_retries': 3, > 'retry_errors': (TimeoutError, ), > }) > ``` > > ::: versionadded > 5.3 > ::: For example, the default policy correlates to: ``` python add.apply_async((2, 2), retry=True, retry_policy={ 'max_retries': 3, 'interval_start': 0, 'interval_step': 0.2, 'interval_max': 0.2, 'retry_errors': None, }) ``` the maximum time spent retrying will be 0.4 seconds. It\'s set relatively short by default because a connection failure could lead to a retry pile effect if the broker connection is down \-- For example, many web server processes waiting to retry, blocking other incoming requests. ## Connection Error Handling {#calling-connection-errors} When you send a task and the message transport connection is lost, or the connection cannot be initiated, an `~kombu.exceptions.OperationalError`{.interpreted-text role="exc"} error will be raised: ``` pycon >>> from proj.tasks import add >>> add.delay(2, 2) Traceback (most recent call last): File "", line 1, in File "celery/app/task.py", line 388, in delay return self.apply_async(args, kwargs) File "celery/app/task.py", line 503, in apply_async **options File "celery/app/base.py", line 662, in send_task amqp.send_task_message(P, name, message, **options) File "celery/backends/rpc.py", line 275, in on_task_call maybe_declare(self.binding(producer.channel), retry=True) File "/opt/celery/kombu/kombu/messaging.py", line 204, in _get_channel channel = self._channel = channel() File "/opt/celery/py-amqp/amqp/connection.py", line 272, in connect self.transport.connect() File "/opt/celery/py-amqp/amqp/transport.py", line 100, in connect self._connect(self.host, self.port, self.connect_timeout) File "/opt/celery/py-amqp/amqp/transport.py", line 141, in _connect self.sock.connect(sa) kombu.exceptions.OperationalError: [Errno 61] Connection refused ``` If you have `retries `{.interpreted-text role="ref"} enabled this will only happen after retries are exhausted, or when disabled immediately. You can handle this error too: ``` pycon >>> from celery.utils.log import get_logger >>> logger = get_logger(__name__) >>> try: ... add.delay(2, 2) ... except add.OperationalError as exc: ... logger.exception('Sending task raised: %r', exc) ``` ::: note ::: title Note ::: With RabbitMQ, these errors only indicate the broker is unreachable. Messages can still be silently dropped when the broker hits resource limits. Enable `confirm_publish` in `broker_transport_options`{.interpreted-text role="setting"} to detect this. ::: ## Serializers {#calling-serializers} ::: sidebar **Security** The pickle module allows for execution of arbitrary functions, please see the `security guide `{.interpreted-text role="ref"}. Celery also comes with a special serializer that uses cryptography to sign your messages. ::: Data transferred between clients and workers needs to be serialized, so every message in Celery has a `content_type` header that describes the serialization method used to encode it. The default serializer is [JSON]{.title-ref}, but you can change this using the `task_serializer`{.interpreted-text role="setting"} setting, or for each individual task, or even per message. There\'s built-in support for [JSON]{.title-ref}, `pickle`{.interpreted-text role="mod"}, [YAML]{.title-ref} and `msgpack`, and you can also add your own custom serializers by registering them into the Kombu serializer registry ::: seealso `Message Serialization `{.interpreted-text role="ref"} in the Kombu user guide. ::: Each option has its advantages and disadvantages. json \-- JSON is supported in many programming languages, is now : a standard part of Python (since 2.6), and is fairly fast to decode. The primary disadvantage to JSON is that it limits you to the following data types: strings, Unicode, floats, Boolean, dictionaries, and lists. Decimals and dates are notably missing. Binary data will be transferred using Base64 encoding, increasing the size of the transferred data by 34% compared to an encoding format where native binary types are supported. However, if your data fits inside the above constraints and you need cross-language support, the default setting of JSON is probably your best choice. See for more information. ::: note ::: title Note ::: (From Python official docs ) Keys in key/value pairs of JSON are always of the type `str`{.interpreted-text role="class"}. When a dictionary is converted into JSON, all the keys of the dictionary are coerced to strings. As a result of this, if a dictionary is converted into JSON and then back into a dictionary, the dictionary may not equal the original one. That is, `loads(dumps(x)) != x` if x has non-string keys. ::: ::: warning ::: title Warning ::: With more complex workflows created using `guide-canvas`{.interpreted-text role="ref"}, the JSON serializer has been observed to drastically inflate message sizes due to recursive references, leading to resource issues. The *pickle* serializer is not vulnerable to this and may therefore be preferable in such cases. ::: pickle \-- If you have no desire to support any language other than : Python, then using the pickle encoding will gain you the support of all built-in Python data types (except class instances), smaller messages when sending binary files, and a slight speedup over JSON processing. See `pickle`{.interpreted-text role="mod"} for more information. yaml \-- YAML has many of the same characteristics as json, : except that it natively supports more data types (including dates, recursive references, etc.). However, the Python libraries for YAML are a good bit slower than the libraries for JSON. If you need a more expressive set of data types and need to maintain cross-language compatibility, then YAML may be a better fit than the above. To use it, install Celery with: ``` console $ pip install celery[yaml] ``` See for more information. msgpack \-- msgpack is a binary serialization format that\'s closer to JSON : in features. The format compresses better, so is a faster to parse and encode compared to JSON. To use it, install Celery with: ``` console $ pip install celery[msgpack] ``` See for more information. To use a custom serializer you need to add the content type to `accept_content`{.interpreted-text role="setting"}. By default, only JSON is accepted, and tasks containing other content headers are rejected. The following order is used to decide the serializer used when sending a task: > 1. The [serializer]{.title-ref} execution option. > 2. The `@-Task.serializer`{.interpreted-text role="attr"} attribute > 3. The `task_serializer`{.interpreted-text role="setting"} setting. Example setting a custom serializer for a single task invocation: ``` pycon >>> add.apply_async((10, 10), serializer='json') ``` ## Compression {#calling-compression} Celery can compress messages using the following builtin schemes: - [brotli]{.title-ref} > brotli is optimized for the web, in particular small text > documents. It is most effective for serving static content such as > fonts and html pages. > > To use it, install Celery with: > > ``` console > $ pip install celery[brotli] > ``` - [bzip2]{.title-ref} > bzip2 creates smaller files than gzip, but compression and > decompression speeds are noticeably slower than those of gzip. > > To use it, please ensure your Python executable was compiled with > bzip2 support. > > If you get the following `ImportError`{.interpreted-text > role="class"}: > > ``` pycon > >>> import bz2 > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named 'bz2' > ``` > > it means that you should recompile your Python version with bzip2 > support. - [gzip]{.title-ref} > gzip is suitable for systems that require a small memory > footprint, making it ideal for systems with limited memory. It is > often used to generate files with the \".tar.gz\" extension. > > To use it, please ensure your Python executable was compiled with > gzip support. > > If you get the following `ImportError`{.interpreted-text > role="class"}: > > ``` pycon > >>> import gzip > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named 'gzip' > ``` > > it means that you should recompile your Python version with gzip > support. - [lzma]{.title-ref} > lzma provides a good compression ratio and executes with fast > compression and decompression speeds at the expense of higher > memory usage. > > To use it, please ensure your Python executable was compiled with > lzma support and that your Python version is 3.3 and above. > > If you get the following `ImportError`{.interpreted-text > role="class"}: > > ``` pycon > >>> import lzma > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named 'lzma' > ``` > > it means that you should recompile your Python version with lzma > support. > > Alternatively, you can also install a backport using: > > ``` console > $ pip install celery[lzma] > ``` - [zlib]{.title-ref} > zlib is an abstraction of the Deflate algorithm in library form > which includes support both for the gzip file format and a > lightweight stream format in its API. It is a crucial component of > many software systems - Linux kernel and Git VCS just to name a > few. > > To use it, please ensure your Python executable was compiled with > zlib support. > > If you get the following `ImportError`{.interpreted-text > role="class"}: > > ``` pycon > >>> import zlib > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named 'zlib' > ``` > > it means that you should recompile your Python version with zlib > support. - [zstd]{.title-ref} > zstd targets real-time compression scenarios at zlib-level and > better compression ratios. It\'s backed by a very fast entropy > stage, provided by Huff0 and FSE library. > > To use it, install Celery with: > > ``` console > $ pip install celery[zstd] > ``` You can also create your own compression schemes and register them in the `kombu compression registry `{.interpreted-text role="func"}. The following order is used to decide the compression scheme used when sending a task: > 1. The [compression]{.title-ref} execution option. > 2. The `@-Task.compression`{.interpreted-text role="attr"} attribute. > 3. The `task_compression`{.interpreted-text role="setting"} > attribute. Example specifying the compression used when calling a task: >>> add.apply_async((2, 2), compression='zlib') ## Connections {#calling-connections} ::: sidebar **Automatic Pool Support** Since version 2.3 there\'s support for automatic connection pools, so you don\'t have to manually handle connections and publishers to reuse connections. The connection pool is enabled by default since version 2.5. See the `broker_pool_limit`{.interpreted-text role="setting"} setting for more information. ::: You can handle the connection manually by creating a publisher: ``` python numbers = [(2, 2), (4, 4), (8, 8), (16, 16)] results = [] with add.app.pool.acquire(block=True) as connection: with add.get_publisher(connection) as publisher: try: for i, j in numbers: res = add.apply_async((i, j), publisher=publisher) results.append(res) print([res.get() for res in results]) ``` Though this particular example is much better expressed as a group: ``` pycon >>> from celery import group >>> numbers = [(2, 2), (4, 4), (8, 8), (16, 16)] >>> res = group(add.s(i, j) for i, j in numbers).apply_async() >>> res.get() [4, 8, 16, 32] ``` ## Routing options {#calling-routing} Celery can route tasks to different queues. Simple routing (name \<-\> name) is accomplished using the `queue` option: add.apply_async(queue='priority.high') You can then assign workers to the `priority.high` queue by using the workers `-Q `{.interpreted-text role="option"} argument: ``` console $ celery -A proj worker -l INFO -Q celery,priority.high ``` ::: seealso Hard-coding queue names in code isn\'t recommended, the best practice is to use configuration routers (`task_routes`{.interpreted-text role="setting"}). To find out more about routing, please see `guide-routing`{.interpreted-text role="ref"}. ::: ## Results options {#calling-results} You can enable or disable result storage using the `task_ignore_result`{.interpreted-text role="setting"} setting or by using the `ignore_result` option: ``` pycon >>> result = add.apply_async((1, 2), ignore_result=True) >>> result.get() None >>> # Do not ignore result (default) ... >>> result = add.apply_async((1, 2), ignore_result=False) >>> result.get() 3 ``` If you\'d like to store additional metadata about the task in the result backend set the `result_extended`{.interpreted-text role="setting"} setting to `True`. ::: note ::: title Note ::: `result_extended` controls what *Celery* includes as extended task metadata, but it does not automatically add scheduler-specific metadata. For example, some integrations (e.g. `django-celery-beat`{.interpreted-text role="pypi"} together with `django-celery-results`{.interpreted-text role="pypi"}) may record the *periodic task name* in the result backend only when the scheduler provides it as part of the published message. When you call tasks manually using `apply_async`/`delay`, that periodic task context is usually not present unless you add it explicitly (e.g. via message headers/properties in `apply_async` options). For example: ``` python result = task.apply_async( headers={"periodic_task_name": "task_name"}, ) ``` ::: ::: seealso For more information on tasks, please see `guide-tasks`{.interpreted-text role="ref"}. ::: ### Advanced Options These options are for advanced users who want to take use of AMQP\'s full routing capabilities. Interested parties may read the `routing guide `{.interpreted-text role="ref"}. - exchange > Name of exchange (or a `kombu.entity.Exchange`{.interpreted-text > role="class"}) to send the message to. - routing_key > Routing key used to determine. - priority > A number between [0]{.title-ref} and [255]{.title-ref}, where > [255]{.title-ref} is the highest priority. > > Supported by: RabbitMQ, Redis (priority reversed, 0 is highest). --- # Canvas: Designing Work-flows {#guide-canvas} ::: {.contents local="" depth="2"} ::: ## Signatures[]{#canvas-subtasks} {#canvas-signatures} ::: versionadded 2.0 ::: You just learned how to call a task using the tasks `delay` method in the `calling `{.interpreted-text role="ref"} guide, and this is often all you need, but sometimes you may want to pass the signature of a task invocation to another process or as an argument to another function. A `~celery.signature`{.interpreted-text role="func"} wraps the arguments, keyword arguments, and execution options of a single task invocation in a way such that it can be passed to functions or even serialized and sent across the wire. - You can create a signature for the `add` task using its name like this: > ``` pycon > >>> from celery import signature > >>> signature('tasks.add', args=(2, 2), countdown=10) > tasks.add(2, 2) > ``` This task has a signature of arity 2 (two arguments): `(2, 2)`, and sets the countdown execution option to 10. - or you can create one using the task\'s `signature` method: > ``` pycon > >>> add.signature((2, 2), countdown=10) > tasks.add(2, 2) > ``` - There\'s also a shortcut using star arguments: > ``` pycon > >>> add.s(2, 2) > tasks.add(2, 2) > ``` - Keyword arguments are also supported: > ``` pycon > >>> add.s(2, 2, debug=True) > tasks.add(2, 2, debug=True) > ``` - From any signature instance you can inspect the different fields: > ``` pycon > >>> s = add.signature((2, 2), {'debug': True}, countdown=10) > >>> s.args > (2, 2) > >>> s.kwargs > {'debug': True} > >>> s.options > {'countdown': 10} > ``` - It supports the \"Calling API\" of `delay`, `apply_async`, etc., including being called directly (`__call__`). > Calling the signature will execute the task inline in the current > process: > > ``` pycon > >>> add(2, 2) > 4 > >>> add.s(2, 2)() > 4 > ``` > > `delay` is our beloved shortcut to `apply_async` taking > star-arguments: > > ``` pycon > >>> result = add.delay(2, 2) > >>> result.get() > 4 > ``` > > `apply_async` takes the same arguments as the > `Task.apply_async <@Task.apply_async>`{.interpreted-text > role="meth"} method: > > ``` pycon > >>> add.apply_async(args, kwargs, **options) > >>> add.signature(args, kwargs, **options).apply_async() > > >>> add.apply_async((2, 2), countdown=1) > >>> add.signature((2, 2), countdown=1).apply_async() > ``` - You can\'t define options with `~@Task.s`{.interpreted-text role="meth"}, but a chaining `set` call takes care of that: > ``` pycon > >>> add.s(2, 2).set(countdown=1) > proj.tasks.add(2, 2) > ``` ### Partials With a signature, you can execute the task in a worker: ``` pycon >>> add.s(2, 2).delay() >>> add.s(2, 2).apply_async(countdown=1) ``` Or you can call it directly in the current process: ``` pycon >>> add.s(2, 2)() 4 ``` Specifying additional args, kwargs, or options to `apply_async`/`delay` creates partials: - Any arguments added will be prepended to the args in the signature: > ``` pycon > >>> partial = add.s(2) # incomplete signature > >>> partial.delay(4) # 4 + 2 > >>> partial.apply_async((4,)) # same > ``` - Any keyword arguments added will be merged with the kwargs in the signature, with the new keyword arguments taking precedence: > ``` pycon > >>> s = add.s(2, 2) > >>> s.delay(debug=True) # -> add(2, 2, debug=True) > >>> s.apply_async(kwargs={'debug': True}) # same > ``` - Any options added will be merged with the options in the signature, with the new options taking precedence: > ``` pycon > >>> s = add.signature((2, 2), countdown=10) > >>> s.apply_async(countdown=1) # countdown is now 1 > ``` You can also clone signatures to create derivatives: ``` pycon >>> s = add.s(2) proj.tasks.add(2) >>> s.clone(args=(4,), kwargs={'debug': True}) proj.tasks.add(4, 2, debug=True) ``` ### Immutability ::: versionadded 3.0 ::: Partials are meant to be used with callbacks, any tasks linked, or chord callbacks will be applied with the result of the parent task. Sometimes you want to specify a callback that doesn\'t take additional arguments, and in that case you can set the signature to be immutable: ``` pycon >>> add.apply_async((2, 2), link=reset_buffers.signature(immutable=True)) ``` The `.si()` shortcut can also be used to create immutable signatures: ``` pycon >>> add.apply_async((2, 2), link=reset_buffers.si()) ``` Only the execution options can be set when a signature is immutable, so it\'s not possible to call the signature with partial args/kwargs. ::: note ::: title Note ::: In this tutorial I sometimes use the prefix operator [\~]{.title-ref} to signatures. You probably shouldn\'t use it in your production code, but it\'s a handy shortcut when experimenting in the Python shell: ``` pycon >>> ~sig >>> # is the same as >>> sig.delay().get() ``` ::: ### Callbacks {#canvas-callbacks} ::: versionadded 3.0 ::: Callbacks can be added to any task using the `link` argument to `apply_async`: ``` pycon add.apply_async((2, 2), link=other_task.s()) ``` The callback will only be applied if the task exited successfully, and it will be applied with the return value of the parent task as argument. As I mentioned earlier, any arguments you add to a signature, will be prepended to the arguments specified by the signature itself! If you have the signature: ``` pycon >>> sig = add.s(10) ``` then [sig.delay(result)]{.title-ref} becomes: ``` pycon >>> add.apply_async(args=(result, 10)) ``` \... Now let\'s call our `add` task with a callback using partial arguments: ``` pycon >>> add.apply_async((2, 2), link=add.s(8)) ``` As expected this will first launch one task calculating $2 + 2$, then another task calculating $4 + 8$. ## The Primitives ::: versionadded 3.0 ::: ::: topic **Overview** - `group` > The group primitive is a signature that takes a list of tasks that > should be applied in parallel. - `chain` > The chain primitive lets us link together signatures so that one > is called after the other, essentially forming a *chain* of > callbacks. - `chord` > A chord is just like a group but with a callback. A chord consists > of a header group and a body, where the body is a task that should > execute after all of the tasks in the header are complete. - `map` > The map primitive works like the built-in `map` function, but > creates a temporary task where a list of arguments is applied to > the task. For example, `task.map([1, 2])` \-- results in a single > task being called, applying the arguments in order to the task > function so that the result is: > > ``` python > res = [task(1), task(2)] > ``` - `starmap` > Works exactly like map except the arguments are applied as > `*args`. For example `add.starmap([(2, 2), (4, 4)])` results in a > single task calling: > > ``` python > res = [add(2, 2), add(4, 4)] > ``` - `chunks` > Chunking splits a long list of arguments into parts, for example > the operation: > > ``` pycon > >>> items = zip(range(1000), range(1000)) # 1000 items > >>> add.chunks(items, 10) > ``` > > will split the list of items into chunks of 10, resulting in 100 > tasks (each processing 10 items in sequence). ::: The primitives are also signature objects themselves, so that they can be combined in any number of ways to compose complex work-flows. Here\'re some examples: - Simple chain > Here\'s a simple chain, the first task executes passing its return > value to the next task in the chain, and so on. > > ``` pycon > >>> from celery import chain > > >>> # 2 + 2 + 4 + 8 > >>> res = chain(add.s(2, 2), add.s(4), add.s(8))() > >>> res.get() > 16 > ``` > > This can also be written using pipes: > > ``` pycon > >>> (add.s(2, 2) | add.s(4) | add.s(8))().get() > 16 > ``` - Immutable signatures > Signatures can be partial so arguments can be added to the > existing arguments, but you may not always want that, for example > if you don\'t want the result of the previous task in a chain. > > In that case you can mark the signature as immutable, so that the > arguments cannot be changed: > > ``` pycon > >>> add.signature((2, 2), immutable=True) > ``` > > There\'s also a `.si()` shortcut for this, and this is the > preferred way of creating signatures: > > ``` pycon > >>> add.si(2, 2) > ``` > > Now you can create a chain of independent tasks instead: > > ``` pycon > >>> res = (add.si(2, 2) | add.si(4, 4) | add.si(8, 8))() > >>> res.get() > 16 > > >>> res.parent.get() > 8 > > >>> res.parent.parent.get() > 4 > ``` - Simple group > You can easily create a group of tasks to execute in parallel: > > ``` pycon > >>> from celery import group > >>> res = group(add.s(i, i) for i in range(10))() > >>> res.get(timeout=1) > [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] > ``` - Simple chord > The chord primitive enables us to add a callback to be called when > all of the tasks in a group have finished executing. This is often > required for algorithms that aren\'t *embarrassingly parallel*: > > ``` pycon > >>> from celery import chord > >>> res = chord((add.s(i, i) for i in range(10)), tsum.s())() > >>> res.get() > 90 > ``` > > The above example creates 10 tasks that all start in parallel, and > when all of them are complete the return values are combined into > a list and sent to the `tsum` task. > > The body of a chord can also be immutable, so that the return > value of the group isn\'t passed on to the callback: > > ``` pycon > >>> chord((import_contact.s(c) for c in contacts), > ... notify_complete.si(import_id)).apply_async() > ``` > > Note the use of `.si` above; this creates an immutable signature, > meaning any new arguments passed (including to return value of the > previous task) will be ignored. - Blow your mind by combining > Chains can be partial too: > > ``` pycon > >>> c1 = (add.s(4) | mul.s(8)) > > # (16 + 4) * 8 > >>> res = c1(16) > >>> res.get() > 160 > ``` > > this means that you can combine chains: > > ``` pycon > # ((4 + 16) * 2 + 4) * 8 > >>> c2 = (add.s(4, 16) | mul.s(2) | (add.s(4) | mul.s(8))) > > >>> res = c2() > >>> res.get() > 352 > ``` > > Chaining a group together with another task will automatically > upgrade it to be a chord: > > ``` pycon > >>> c3 = (group(add.s(i, i) for i in range(10)) | tsum.s()) > >>> res = c3() > >>> res.get() > 90 > ``` > > Groups and chords accepts partial arguments too, so in a chain the > return value of the previous task is forwarded to all tasks in the > group: > > ``` pycon > >>> new_user_workflow = (create_user.s() | group( > ... import_contacts.s(), > ... send_welcome_email.s())) > ... new_user_workflow.delay(username='artv', > ... first='Art', > ... last='Vandelay', > ... email='art@vandelay.com') > ``` > > If you don\'t want to forward arguments to the group then you can > make the signatures in the group immutable: > > ``` pycon > >>> res = (add.s(4, 4) | group(add.si(i, i) for i in range(10)))() > >>> res.get() > [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] > > >>> res.parent.get() > 8 > ``` ::: warning ::: title Warning ::: With more complex workflows, the default JSON serializer has been observed to drastically inflate message sizes due to recursive references, leading to resource issues. The *pickle* serializer is not vulnerable to this and may therefore be preferable in such cases. ::: ### Chains {#canvas-chain} ::: versionadded 3.0 ::: Tasks can be linked together: the linked task is called when the task returns successfully: ``` pycon >>> res = add.apply_async((2, 2), link=mul.s(16)) >>> res.get() 4 ``` The linked task will be applied with the result of its parent task as the first argument. In the above case where the result was 4, this will result in `mul(4, 16)`. The results will keep track of any subtasks called by the original task, and this can be accessed from the result instance: ``` pycon >>> res.children [] >>> res.children[0].get() 64 ``` The result instance also has a `~@AsyncResult.collect`{.interpreted-text role="meth"} method that treats the result as a graph, enabling you to iterate over the results: ``` pycon >>> list(res.collect()) [(, 4), (, 64)] ``` By default `~@AsyncResult.collect`{.interpreted-text role="meth"} will raise an `~@IncompleteStream`{.interpreted-text role="exc"} exception if the graph isn\'t fully formed (one of the tasks hasn\'t completed yet), but you can get an intermediate representation of the graph too: ``` pycon >>> for result, value in res.collect(intermediate=True): .... ``` You can link together as many tasks as you like, and signatures can be linked too: ``` pycon >>> s = add.s(2, 2) >>> s.link(mul.s(4)) >>> s.link(log_result.s()) ``` You can also add *error callbacks* using the [on_error]{.title-ref} method: ``` pycon >>> add.s(2, 2).on_error(log_error.s()).delay() ``` This will result in the following `.apply_async` call when the signature is applied: ``` pycon >>> add.apply_async((2, 2), link_error=log_error.s()) ``` The worker won\'t actually call the errback as a task, but will instead call the errback function directly so that the raw request, exception and traceback objects can be passed to it. Here\'s an example errback: ``` python import os from proj.celery import app @app.task def log_error(request, exc, traceback): with open(os.path.join('/var/errors', request.id), 'a') as fh: print('--\n\n{0} {1} {2}'.format( request.id, exc, traceback), file=fh) ``` To make it even easier to link tasks together there\'s a special signature called `~celery.chain`{.interpreted-text role="class"} that lets you chain tasks together: ``` pycon >>> from celery import chain >>> from proj.tasks import add, mul >>> # (4 + 4) * 8 * 10 >>> res = chain(add.s(4, 4), mul.s(8), mul.s(10)) proj.tasks.add(4, 4) | proj.tasks.mul(8) | proj.tasks.mul(10) ``` Calling the chain will call the tasks in the current process and return the result of the last task in the chain: ``` pycon >>> res = chain(add.s(4, 4), mul.s(8), mul.s(10))() >>> res.get() 640 ``` It also sets `parent` attributes so that you can work your way up the chain to get intermediate results: ``` pycon >>> res.parent.get() 64 >>> res.parent.parent.get() 8 >>> res.parent.parent ``` Chains can also be made using the `|` (pipe) operator: ``` pycon >>> (add.s(2, 2) | mul.s(8) | mul.s(10)).apply_async() ``` #### Task ID ::: versionadded 5.4 ::: A chain will inherit the task id of the last task in the chain. #### Graphs In addition you can work with the result graph as a `~celery.utils.graph.DependencyGraph`{.interpreted-text role="class"}: ``` pycon >>> res = chain(add.s(4, 4), mul.s(8), mul.s(10))() >>> res.parent.parent.graph 285fa253-fcf8-42ef-8b95-0078897e83e6(1) 463afec2-5ed4-4036-b22d-ba067ec64f52(0) 872c3995-6fa0-46ca-98c2-5a19155afcf0(2) 285fa253-fcf8-42ef-8b95-0078897e83e6(1) 463afec2-5ed4-4036-b22d-ba067ec64f52(0) ``` You can even convert these graphs to *dot* format: ``` pycon >>> with open('graph.dot', 'w') as fh: ... res.parent.parent.graph.to_dot(fh) ``` and create images: ``` console $ dot -Tpng graph.dot -o graph.png ``` ![image](../images/result_graph.png) ### Groups {#canvas-group} ::: versionadded 3.0 ::: ::: note ::: title Note ::: Similarly to chords, tasks used in a group must *not* ignore their results. See \"`chord-important-notes`{.interpreted-text role="ref"}\" for more information. ::: A group can be used to execute several tasks in parallel. The `~celery.group`{.interpreted-text role="class"} function takes a list of signatures: ``` pycon >>> from celery import group >>> from proj.tasks import add >>> group(add.s(2, 2), add.s(4, 4)) (proj.tasks.add(2, 2), proj.tasks.add(4, 4)) ``` If you **call** the group, the tasks will be applied one after another in the current process, and a `~celery.result.GroupResult`{.interpreted-text role="class"} instance is returned that can be used to keep track of the results, or tell how many tasks are ready and so on: ``` pycon >>> g = group(add.s(2, 2), add.s(4, 4)) >>> res = g() >>> res.get() [4, 8] ``` Group also supports iterators: ``` pycon >>> group(add.s(i, i) for i in range(100))() ``` A group is a signature object, so it can be used in combination with other signatures. #### Group Callbacks and Error Handling {#group-callbacks} Groups can have callback and errback signatures linked to them as well, however the behaviour can be somewhat surprising due to the fact that groups are not real tasks and simply pass linked tasks down to their encapsulated signatures. This means that the return values of a group are not collected to be passed to a linked callback signature. Additionally, linking the task will *not* guarantee that it will activate only when all group tasks have finished. As an example, the following snippet using a simple [add(a, b)]{.title-ref} task is faulty since the linked [add.s()]{.title-ref} signature will not receive the finalised group result as one might expect. ``` pycon >>> g = group(add.s(2, 2), add.s(4, 4)) >>> g.link(add.s()) >>> res = g() [4, 8] ``` Note that the finalised results of the first two tasks are returned, but the callback signature will have run in the background and raised an exception since it did not receive the two arguments it expects. Group errbacks are passed down to encapsulated signatures as well which opens the possibility for an errback linked only once to be called more than once if multiple tasks in a group were to fail. As an example, the following snippet using a [fail()]{.title-ref} task which raises an exception can be expected to invoke the [log_error()]{.title-ref} signature once for each failing task which gets run in the group. ``` pycon >>> g = group(fail.s(), fail.s()) >>> g.link_error(log_error.s()) >>> res = g() ``` With this in mind, it\'s generally advisable to create idempotent or counting tasks which are tolerant to being called repeatedly for use as errbacks. These use cases are better addressed by the `~celery.chord`{.interpreted-text role="class"} class which is supported on certain backend implementations. #### Group Results The group task returns a special result too, this result works just like normal task results, except that it works on the group as a whole: ``` pycon >>> from celery import group >>> from tasks import add >>> job = group([ ... add.s(2, 2), ... add.s(4, 4), ... add.s(8, 8), ... add.s(16, 16), ... add.s(32, 32), ... ]) >>> result = job.apply_async() >>> result.ready() # have all subtasks completed? True >>> result.successful() # were all subtasks successful? True >>> result.get() [4, 8, 16, 32, 64] ``` The `~celery.result.GroupResult`{.interpreted-text role="class"} takes a list of `~celery.result.AsyncResult`{.interpreted-text role="class"} instances and operates on them as if it was a single task. It supports the following operations: - `~celery.result.GroupResult.successful`{.interpreted-text role="meth"} > Return `True`{.interpreted-text role="const"} if all of the > subtasks finished successfully (e.g., didn\'t raise an exception). - `~celery.result.GroupResult.failed`{.interpreted-text role="meth"} > Return `True`{.interpreted-text role="const"} if any of the > subtasks failed. - `~celery.result.GroupResult.waiting`{.interpreted-text role="meth"} > Return `True`{.interpreted-text role="const"} if any of the > subtasks isn\'t ready yet. - `~celery.result.GroupResult.ready`{.interpreted-text role="meth"} > Return `True`{.interpreted-text role="const"} if all of the > subtasks are ready. - `~celery.result.GroupResult.completed_count`{.interpreted-text role="meth"} > Return the number of completed subtasks. Note that > [complete]{.title-ref} means [successful]{.title-ref} in this > context. In other words, the return value of this method is the > number of `successful` tasks. - `~celery.result.GroupResult.revoke`{.interpreted-text role="meth"} > Revoke all of the subtasks. - `~celery.result.GroupResult.join`{.interpreted-text role="meth"} > Gather the results of all subtasks and return them in the same > order as they were called (as a list). #### Group Unrolling A group with a single signature will be unrolled to a single signature when chained. This means that the following group may pass either a list of results or a single result to the chain depending on the number of items in the group. ``` pycon >>> from celery import chain, group >>> from tasks import add >>> chain(add.s(2, 2), group(add.s(1)), add.s(1)) add(2, 2) | add(1) | add(1) >>> chain(add.s(2, 2), group(add.s(1), add.s(2)), add.s(1)) add(2, 2) | %add((add(1), add(2)), 1) ``` This means that you should be careful and make sure the `add` task can accept either a list or a single item as input if you plan to use it as part of a larger canvas. ::: warning ::: title Warning ::: In Celery 4.x the following group below would not unroll into a chain due to a bug but instead the canvas would be upgraded into a chord. ``` pycon >>> from celery import chain, group >>> from tasks import add >>> chain(group(add.s(1, 1)), add.s(2)) %add([add(1, 1)], 2) ``` In Celery 5.x this bug was fixed and the group is correctly unrolled into a single signature. ``` pycon >>> from celery import chain, group >>> from tasks import add >>> chain(group(add.s(1, 1)), add.s(2)) add(1, 1) | add(2) ``` ::: ### Chords {#canvas-chord} ::: versionadded 2.3 ::: ::: note ::: title Note ::: Tasks used within a chord must *not* ignore their results. If the result backend is disabled for *any* task (header or body) in your chord you should read \"`chord-important-notes`{.interpreted-text role="ref"}\". Chords are not currently supported with the RPC result backend. ::: A chord is a task that only executes after all of the tasks in a group have finished executing. Let\'s calculate the sum of the expression $1 + 1 + 2 + 2 + 3 + 3 ... n + n$ up to a hundred digits. First you need two tasks, `add`{.interpreted-text role="func"} and `tsum`{.interpreted-text role="func"} (`sum`{.interpreted-text role="func"} is already a standard function): ``` python @app.task def add(x, y): return x + y @app.task def tsum(numbers): return sum(numbers) ``` Now you can use a chord to calculate each addition step in parallel, and then get the sum of the resulting numbers: ``` pycon >>> from celery import chord >>> from tasks import add, tsum >>> chord(add.s(i, i) ... for i in range(100))(tsum.s()).get() 9900 ``` This is obviously a very contrived example, the overhead of messaging and synchronization makes this a lot slower than its Python counterpart: ``` pycon >>> sum(i + i for i in range(100)) ``` The synchronization step is costly, so you should avoid using chords as much as possible. Still, the chord is a powerful primitive to have in your toolbox as synchronization is a required step for many parallel algorithms. Let\'s break the chord expression down: ``` pycon >>> callback = tsum.s() >>> header = [add.s(i, i) for i in range(100)] >>> result = chord(header)(callback) >>> result.get() 9900 ``` Remember, the callback can only be executed after all of the tasks in the header have returned. Each step in the header is executed as a task, in parallel, possibly on different nodes. The callback is then applied with the return value of each task in the header. The task id returned by `chord`{.interpreted-text role="meth"} is the id of the callback, so you can wait for it to complete and get the final return value (but remember to `never have a task wait for other tasks `{.interpreted-text role="ref"}) #### Error handling {#chord-errors} So what happens if one of the tasks raises an exception? The chord callback result will transition to the failure state, and the error is set to the `~@ChordError`{.interpreted-text role="exc"} exception: ``` pycon >>> c = chord([add.s(4, 4), raising_task.s(), add.s(8, 8)]) >>> result = c() >>> result.get() ``` ``` pytb Traceback (most recent call last): File "", line 1, in File "*/celery/result.py", line 120, in get interval=interval) File "*/celery/backends/amqp.py", line 150, in wait_for raise meta['result'] celery.exceptions.ChordError: Dependency 97de6f3f-ea67-4517-a21c-d867c61fcb47 raised ValueError('something something',) ``` While the traceback may be different depending on the result backend used, you can see that the error description includes the id of the task that failed and a string representation of the original exception. You can also find the original traceback in `result.traceback`. Note that the rest of the tasks will still execute, so the third task (`add.s(8, 8)`) is still executed even though the middle task failed. Also the `~@ChordError`{.interpreted-text role="exc"} only shows the task that failed first (in time): it doesn\'t respect the ordering of the header group. To perform an action when a chord fails you can therefore attach an errback to the chord callback: ``` python @app.task def on_chord_error(request, exc, traceback): print('Task {0!r} raised error: {1!r}'.format(request.id, exc)) ``` ``` pycon >>> c = (group(add.s(i, i) for i in range(10)) | ... tsum.s().on_error(on_chord_error.s())).delay() ``` Chords may have callback and errback signatures linked to them, which addresses some of the issues with linking signatures to groups. Doing so will link the provided signature to the chord\'s body which can be expected to gracefully invoke callbacks just once upon completion of the body, or errbacks just once if any task in the chord header or body fails. This behavior can be manipulated to allow error handling of the chord header using the `task_allow_error_cb_on_chord_header `{.interpreted-text role="ref"} flag. Enabling this flag will cause the chord header to invoke the errback for the body (default behavior) *and* any task in the chord\'s header that fails. #### Important Notes {#chord-important-notes} Tasks used within a chord must *not* ignore their results. In practice this means that you must enable a `result_backend`{.interpreted-text role="const"} in order to use chords. Additionally, if `task_ignore_result`{.interpreted-text role="const"} is set to `True`{.interpreted-text role="const"} in your configuration, be sure that the individual tasks to be used within the chord are defined with `ignore_result=False`{.interpreted-text role="const"}. This applies to both Task subclasses and decorated tasks. Example Task subclass: ``` python class MyTask(Task): ignore_result = False ``` Example decorated task: ``` python @app.task(ignore_result=False) def another_task(project): do_something() ``` By default the synchronization step is implemented by having a recurring task poll the completion of the group every second, calling the signature when ready. Example implementation: ``` python from celery import maybe_signature @app.task(bind=True) def unlock_chord(self, group, callback, interval=1, max_retries=None): if group.ready(): return maybe_signature(callback).delay(group.join()) raise self.retry(countdown=interval, max_retries=max_retries) ``` This is used by all result backends except Redis, Memcached and DynamoDB: they increment a counter after each task in the header, then applies the callback when the counter exceeds the number of tasks in the set. The Redis, Memcached and DynamoDB approach is a much better solution, but not easily implemented in other backends (suggestions welcome!). ::: note ::: title Note ::: Chords don\'t properly work with Redis before version 2.2; you\'ll need to upgrade to at least redis-server 2.2 to use them. ::: ::: note ::: title Note ::: If you\'re using chords with the Redis result backend and also overriding the `Task.after_return`{.interpreted-text role="meth"} method, you need to make sure to call the super method or else the chord callback won\'t be applied. ``` python def after_return(self, *args, **kwargs): do_something() super().after_return(*args, **kwargs) ``` ::: ### Map & Starmap {#canvas-map} `~celery.map`{.interpreted-text role="class"} and `~celery.starmap`{.interpreted-text role="class"} are built-in tasks that call the provided calling task for every element in a sequence. They differ from `~celery.group`{.interpreted-text role="class"} in that: - only one task message is sent. - the operation is sequential. For example using `map`: ``` pycon >>> from proj.tasks import add >>> ~tsum.map([list(range(10)), list(range(100))]) [45, 4950] ``` is the same as having a task doing: ``` python @app.task def temp(): return [tsum(range(10)), tsum(range(100))] ``` and using `starmap`: ``` pycon >>> ~add.starmap(zip(range(10), range(10))) [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] ``` is the same as having a task doing: ``` python @app.task def temp(): return [add(i, i) for i in range(10)] ``` Both `map` and `starmap` are signature objects, so they can be used as other signatures and combined in groups etc., for example to call the starmap after 10 seconds: ``` pycon >>> add.starmap(zip(range(10), range(10))).apply_async(countdown=10) ``` ### Chunks {#canvas-chunks} Chunking lets you divide an iterable of work into pieces, so that if you have one million objects, you can create 10 tasks with a hundred thousand objects each. Some may worry that chunking your tasks results in a degradation of parallelism, but this is rarely true for a busy cluster and in practice since you\'re avoiding the overhead of messaging it may considerably increase performance. To create a chunks\' signature you can use `@Task.chunks`{.interpreted-text role="meth"}: ``` pycon >>> add.chunks(zip(range(100), range(100)), 10) ``` As with `~celery.group`{.interpreted-text role="class"} the act of sending the messages for the chunks will happen in the current process when called: ``` pycon >>> from proj.tasks import add >>> res = add.chunks(zip(range(100), range(100)), 10)() >>> res.get() [[0, 2, 4, 6, 8, 10, 12, 14, 16, 18], [20, 22, 24, 26, 28, 30, 32, 34, 36, 38], [40, 42, 44, 46, 48, 50, 52, 54, 56, 58], [60, 62, 64, 66, 68, 70, 72, 74, 76, 78], [80, 82, 84, 86, 88, 90, 92, 94, 96, 98], [100, 102, 104, 106, 108, 110, 112, 114, 116, 118], [120, 122, 124, 126, 128, 130, 132, 134, 136, 138], [140, 142, 144, 146, 148, 150, 152, 154, 156, 158], [160, 162, 164, 166, 168, 170, 172, 174, 176, 178], [180, 182, 184, 186, 188, 190, 192, 194, 196, 198]] ``` while calling `.apply_async` will create a dedicated task so that the individual tasks are applied in a worker instead: ``` pycon >>> add.chunks(zip(range(100), range(100)), 10).apply_async() ``` You can also convert chunks to a group: ``` pycon >>> group = add.chunks(zip(range(100), range(100)), 10).group() ``` and with the group skew the countdown of each task by increments of one: ``` pycon >>> group.skew(start=1, stop=10)() ``` This means that the first task will have a countdown of one second, the second task a countdown of two seconds, and so on. ## Stamping {#canvas-stamping} ::: versionadded 5.3 ::: The goal of the Stamping API is to give an ability to label the signature and its components for debugging information purposes. For example, when the canvas is a complex structure, it may be necessary to label some or all elements of the formed structure. The complexity increases even more when nested groups are rolled-out or chain elements are replaced. In such cases, it may be necessary to understand which group an element is a part of or on what nested level it is. This requires a mechanism that traverses the canvas elements and marks them with specific metadata. The stamping API allows doing that based on the Visitor pattern. For example, ``` pycon >>> sig1 = add.si(2, 2) >>> sig1_res = sig1.freeze() >>> g = group(sig1, add.si(3, 3)) >>> g.stamp(stamp='your_custom_stamp') >>> res = g.apply_async() >>> res.get(timeout=TIMEOUT) [4, 6] >>> sig1_res._get_task_meta()['stamp'] ['your_custom_stamp'] ``` will initialize a group `g` and mark its components with stamp `your_custom_stamp`. For this feature to be useful, you need to set the `result_extended`{.interpreted-text role="setting"} configuration option to `True` or directive `result_extended = True`. ### Canvas stamping We can also stamp the canvas with custom stamping logic, using the visitor class `StampingVisitor` as the base class for the custom stamping visitor. ### Custom stamping If more complex stamping logic is required, it is possible to implement custom stamping behavior based on the Visitor pattern. The class that implements this custom logic must inherit `StampingVisitor` and implement appropriate methods. For example, the following example `InGroupVisitor` will label tasks that are in side of some group by label `in_group`. ``` python class InGroupVisitor(StampingVisitor): def __init__(self): self.in_group = False def on_group_start(self, group, **headers) -> dict: self.in_group = True return {"in_group": [self.in_group], "stamped_headers": ["in_group"]} def on_group_end(self, group, **headers) -> None: self.in_group = False def on_chain_start(self, chain, **headers) -> dict: return {"in_group": [self.in_group], "stamped_headers": ["in_group"]} def on_signature(self, sig, **headers) -> dict: return {"in_group": [self.in_group], "stamped_headers": ["in_group"]} ``` The following example shows another custom stamping visitor, which labels all tasks with a custom `monitoring_id` which can represent a UUID value of an external monitoring system, that can be used to track the task execution by including the id with such a visitor implementation. This `monitoring_id` can be a randomly generated UUID, or a unique identifier of the span id used by the external monitoring system, etc. ``` python class MonitoringIdStampingVisitor(StampingVisitor): def on_signature(self, sig, **headers) -> dict: return {'monitoring_id': uuid4().hex} ``` ::: important ::: title Important ::: The `stamped_headers` key in the dictionary returned by `on_signature()` (or any other visitor method) is **optional**: ``` python # Approach 1: Without stamped_headers - ALL keys are treated as stamps def on_signature(self, sig, **headers) -> dict: return {'monitoring_id': uuid4().hex} # monitoring_id becomes a stamp # Approach 2: With stamped_headers - ONLY listed keys are stamps def on_signature(self, sig, **headers) -> dict: return { 'monitoring_id': uuid4().hex, # This will be a stamp 'other_data': 'value', # This will NOT be a stamp 'stamped_headers': ['monitoring_id'] # Only monitoring_id is stamped } ``` If the `stamped_headers` key is not specified, the stamping visitor will assume all keys in the returned dictionary are stamped headers. ::: Next, let\'s see how to use the `MonitoringIdStampingVisitor` example stamping visitor. ``` python sig_example = signature('t1') sig_example.stamp(visitor=MonitoringIdStampingVisitor()) group_example = group([signature('t1'), signature('t2')]) group_example.stamp(visitor=MonitoringIdStampingVisitor()) chord_example = chord([signature('t1'), signature('t2')], signature('t3')) chord_example.stamp(visitor=MonitoringIdStampingVisitor()) chain_example = chain(signature('t1'), group(signature('t2'), signature('t3')), signature('t4')) chain_example.stamp(visitor=MonitoringIdStampingVisitor()) ``` Lastly, it\'s important to mention that each monitoring id stamp in the example above would be different from each other between tasks. ### Callbacks stamping The stamping API also supports stamping callbacks implicitly. This means that when a callback is added to a task, the stamping visitor will be applied to the callback as well. ::: warning ::: title Warning ::: The callback must be linked to the signature before stamping. ::: For example, let\'s examine the following custom stamping visitor that uses the implicit approach where all returned dictionary keys are automatically treated as stamped headers without explicitly specifying [stamped_headers]{.title-ref}. ``` python class CustomStampingVisitor(StampingVisitor): def on_signature(self, sig, **headers) -> dict: # 'header' will automatically be treated as a stamped header # without needing to specify 'stamped_headers': ['header'] return {'header': 'value'} def on_callback(self, callback, **header) -> dict: # 'on_callback' will automatically be treated as a stamped header return {'on_callback': True} def on_errback(self, errback, **header) -> dict: # 'on_errback' will automatically be treated as a stamped header return {'on_errback': True} ``` This custom stamping visitor will stamp the signature, callbacks, and errbacks with `{'header': 'value'}` and stamp the callbacks and errbacks with `{'on_callback': True}` and `{'on_errback': True}` respectively as shown below. ``` python c = chord([add.s(1, 1), add.s(2, 2)], xsum.s()) callback = signature('sig_link') errback = signature('sig_link_error') c.link(callback) c.link_error(errback) c.stamp(visitor=CustomStampingVisitor()) ``` This example will result in the following stamps: ``` python >>> c.options {'header': 'value', 'stamped_headers': ['header']} >>> c.tasks.tasks[0].options {'header': 'value', 'stamped_headers': ['header']} >>> c.tasks.tasks[1].options {'header': 'value', 'stamped_headers': ['header']} >>> c.body.options {'header': 'value', 'stamped_headers': ['header']} >>> c.body.options['link'][0].options {'header': 'value', 'on_callback': True, 'stamped_headers': ['header', 'on_callback']} >>> c.body.options['link_error'][0].options {'header': 'value', 'on_errback': True, 'stamped_headers': ['header', 'on_errback']} ``` --- # Concurrency with Eventlet {#concurrency-eventlet} ## Introduction {#eventlet-introduction} The [Eventlet](http://eventlet.net) homepage describes it as a concurrent networking library for Python that allows you to change how you run your code, not how you write it. > - It uses [epoll(4)](http://linux.die.net/man/4/epoll) or > [libevent](http://monkey.org/~provos/libevent/) for [highly > scalable non-blocking > I/O](https://en.wikipedia.org/wiki/Asynchronous_I/O#Select.28.2Fpoll.29_loops). > - [Coroutines](https://en.wikipedia.org/wiki/Coroutine) ensure that > the developer uses a blocking style of programming that\'s similar > to threading, but provide the benefits of non-blocking I/O. > - The event dispatch is implicit: meaning you can easily use > Eventlet from the Python interpreter, or as a small part of a > larger application. Celery supports Eventlet as an alternative execution pool implementation and in some cases superior to prefork. However, you need to ensure one task doesn\'t block the event loop too long. Generally, CPU-bound operations don\'t go well with Eventlet. Also note that some libraries, usually with C extensions, cannot be monkeypatched and therefore cannot benefit from using Eventlet. Please refer to their documentation if you are not sure. For example, pylibmc does not allow cooperation with Eventlet but psycopg2 does when both of them are libraries with C extensions. The prefork pool can take use of multiple processes, but how many is often limited to a few processes per CPU. With Eventlet you can efficiently spawn hundreds, or thousands of green threads. In an informal test with a feed hub system the Eventlet pool could fetch and process hundreds of feeds every second, while the prefork pool spent 14 seconds processing 100 feeds. Note that this is one of the applications async I/O is especially good at (asynchronous HTTP requests). You may want a mix of both Eventlet and prefork workers, and route tasks according to compatibility or what works best. ## Enabling Eventlet You can enable the Eventlet pool by using the `celery worker -P`{.interpreted-text role="option"} worker option. ``` console $ celery -A proj worker -P eventlet -c 1000 ``` ## Examples {#eventlet-examples} See the [Eventlet examples](https://github.com/celery/celery/tree/main/examples/eventlet) directory in the Celery distribution for some examples taking use of Eventlet support. --- # Concurrency with gevent {#concurrency-eventlet} ## Introduction {#gevent-introduction} The [gevent](http://www.gevent.org/) homepage describes it a [coroutine](https://en.wikipedia.org/wiki/Coroutine) -based [Python](http://python.org) networking library that uses [greenlet](https://greenlet.readthedocs.io) to provide a high-level synchronous API on top of the [libev](http://software.schmorp.de/pkg/libev.html) or [libuv](http://libuv.org) event loop. Features include: - Fast event loop based on [libev](http://software.schmorp.de/pkg/libev.html) or [libuv](http://libuv.org). - Lightweight execution units based on greenlets. - API that reuses concepts from the Python standard library (for examples there are [events](http://www.gevent.org/api/gevent.event.html#gevent.event.Event) and [queues](http://www.gevent.org/api/gevent.queue.html#gevent.queue.Queue)). - [Cooperative sockets with SSL support](http://www.gevent.org/api/index.html#networking) - [Cooperative DNS queries](http://www.gevent.org/dns.html) performed through a threadpool, dnspython, or c-ares. - [Monkey patching utility](http://www.gevent.org/intro.html#monkey-patching) to get 3rd party modules to become cooperative - TCP/UDP/HTTP servers - Subprocess support (through [gevent.subprocess](http://www.gevent.org/api/gevent.subprocess.html#module-gevent.subprocess)) - Thread pools gevent is [inspired by eventlet](http://blog.gevent.org/2010/02/27/why-gevent/) but features a more consistent API, simpler implementation and better performance. Read why others [use gevent](http://groups.google.com/group/gevent/browse_thread/thread/4de9703e5dca8271) and check out the list of the [open source projects based on gevent](https://github.com/gevent/gevent/wiki/Projects). ## Enabling gevent You can enable the gevent pool by using the `celery worker -P gevent`{.interpreted-text role="option"} or `celery worker --pool=gevent`{.interpreted-text role="option"} worker option. ``` console $ celery -A proj worker -P gevent -c 1000 ``` ## Examples {#eventlet-examples} See the [gevent examples](https://github.com/celery/celery/tree/main/examples/gevent) directory in the Celery distribution for some examples taking use of Eventlet support. ## Known issues There is a known issue using python 3.11 and gevent. The issue is documented [here](https://github.com/celery/celery/issues/8425) and addressed in a [gevent issue](https://github.com/gevent/gevent/issues/1985). Upgrading to greenlet 3.0 solves it. --- # Concurrency Release : Date : Concurrency in Celery enables the parallel execution of tasks. The default model, [prefork]{.title-ref}, is well-suited for many scenarios and generally recommended for most users. In fact, switching to another mode will silently disable certain features like [soft_timeout]{.title-ref} and [max_tasks_per_child]{.title-ref}. This page gives a quick overview of the available options which you can pick between using the [\--pool]{.title-ref} option when starting the worker. ## Overview of Concurrency Options - \`prefork\`: The default option, ideal for CPU-bound tasks and most use cases. It is robust and recommended unless there\'s a specific need for another model. - [eventlet]{.title-ref} and \`gevent\`: Designed for IO-bound tasks, these models use greenlets for high concurrency. Note that certain features, like [soft_timeout]{.title-ref}, are not available in these modes. These have detailed documentation pages linked below. - \`solo\`: Executes tasks sequentially in the main thread. - \`threads\`: Utilizes threading for concurrency, available if the [concurrent.futures]{.title-ref} module is present. - \`custom\`: Enables specifying a custom worker pool implementation through environment variables. ::: {.toctree maxdepth="2"} eventlet gevent ::: ::: note ::: title Note ::: While alternative models like [eventlet]{.title-ref} and [gevent]{.title-ref} are available, they may lack certain features compared to [prefork]{.title-ref}. We recommend [prefork]{.title-ref} as the starting point unless specific requirements dictate otherwise. ::: --- # Configuration and defaults {#configuration} This document describes the configuration options available. If you\'re using the default loader, you must create the `celeryconfig.py`{.interpreted-text role="file"} module and make sure it\'s available on the Python path. ::: {.contents local="" depth="2"} ::: ## Example configuration file {#conf-example} This is an example configuration file to get you started. It should contain all you need to run a basic Celery set-up. ``` python ## Broker settings. broker_url = 'amqp://guest:guest@localhost:5672//' # List of modules to import when the Celery worker starts. imports = ('myapp.tasks',) ## Using the database to store task state and results. result_backend = 'db+sqlite:///results.db' task_annotations = {'tasks.add': {'rate_limit': '10/s'}} ``` ## New lowercase settings {#conf-old-settings-map} Version 4.0 introduced new lower case settings and setting organization. The major difference between previous versions, apart from the lower case names, are the renaming of some prefixes, like `celery_beat_` to `beat_`, `celeryd_` to `worker_`, and most of the top level `celery_` settings have been moved into a new `task_` prefix. ::: warning ::: title Warning ::: Celery will still be able to read old configuration files until Celery 6.0. Afterwards, support for the old configuration files will be removed. We provide the `celery upgrade` command that should handle plenty of cases (including `Django `{.interpreted-text role="ref"}). Please migrate to the new configuration scheme as soon as possible. ::: **Setting name** **Replace with** ---------------------------------------------------------- ---------------------------------------------------------------------------- `CELERY_ACCEPT_CONTENT` `accept_content`{.interpreted-text role="setting"} `CELERY_ENABLE_UTC` `enable_utc`{.interpreted-text role="setting"} `CELERY_IMPORTS` `imports`{.interpreted-text role="setting"} `CELERY_INCLUDE` `include`{.interpreted-text role="setting"} `CELERY_TIMEZONE` `timezone`{.interpreted-text role="setting"} `CELERYBEAT_MAX_LOOP_INTERVAL` `beat_max_loop_interval`{.interpreted-text role="setting"} `CELERYBEAT_SCHEDULE` `beat_schedule`{.interpreted-text role="setting"} `CELERYBEAT_SCHEDULER` `beat_scheduler`{.interpreted-text role="setting"} `CELERYBEAT_SCHEDULE_FILENAME` `beat_schedule_filename`{.interpreted-text role="setting"} `CELERYBEAT_SYNC_EVERY` `beat_sync_every`{.interpreted-text role="setting"} `BROKER_URL` `broker_url`{.interpreted-text role="setting"} `BROKER_TRANSPORT` `broker_transport`{.interpreted-text role="setting"} `BROKER_TRANSPORT_OPTIONS` `broker_transport_options`{.interpreted-text role="setting"} `BROKER_CONNECTION_TIMEOUT` `broker_connection_timeout`{.interpreted-text role="setting"} `BROKER_CONNECTION_RETRY` `broker_connection_retry`{.interpreted-text role="setting"} `BROKER_CONNECTION_MAX_RETRIES` `broker_connection_max_retries`{.interpreted-text role="setting"} `BROKER_FAILOVER_STRATEGY` `broker_failover_strategy`{.interpreted-text role="setting"} `BROKER_HEARTBEAT` `broker_heartbeat`{.interpreted-text role="setting"} `BROKER_LOGIN_METHOD` `broker_login_method`{.interpreted-text role="setting"} \`\`BROKER_NATIVE_DELAYED_DELIVERY_QUEUE_TYPE \`[ :setting:\`broker_native_delayed_delivery_queue_type]{.title-ref} `BROKER_POOL_LIMIT` `broker_pool_limit`{.interpreted-text role="setting"} `BROKER_USE_SSL` `broker_use_ssl`{.interpreted-text role="setting"} `CELERY_CACHE_BACKEND` `cache_backend`{.interpreted-text role="setting"} `CELERY_CACHE_BACKEND_OPTIONS` `cache_backend_options`{.interpreted-text role="setting"} `CASSANDRA_COLUMN_FAMILY` `cassandra_table`{.interpreted-text role="setting"} `CASSANDRA_ENTRY_TTL` `cassandra_entry_ttl`{.interpreted-text role="setting"} `CASSANDRA_KEYSPACE` `cassandra_keyspace`{.interpreted-text role="setting"} `CASSANDRA_PORT` `cassandra_port`{.interpreted-text role="setting"} `CASSANDRA_READ_CONSISTENCY` `cassandra_read_consistency`{.interpreted-text role="setting"} `CASSANDRA_SERVERS` `cassandra_servers`{.interpreted-text role="setting"} `CASSANDRA_WRITE_CONSISTENCY` `cassandra_write_consistency`{.interpreted-text role="setting"} `CASSANDRA_OPTIONS` `cassandra_options`{.interpreted-text role="setting"} `S3_ACCESS_KEY_ID` `s3_access_key_id`{.interpreted-text role="setting"} `S3_SECRET_ACCESS_KEY` `s3_secret_access_key`{.interpreted-text role="setting"} `S3_BUCKET` `s3_bucket`{.interpreted-text role="setting"} `S3_BASE_PATH` `s3_base_path`{.interpreted-text role="setting"} `S3_ENDPOINT_URL` `s3_endpoint_url`{.interpreted-text role="setting"} `S3_REGION` `s3_region`{.interpreted-text role="setting"} `CELERY_COUCHBASE_BACKEND_SETTINGS` `couchbase_backend_settings`{.interpreted-text role="setting"} `CELERY_ARANGODB_BACKEND_SETTINGS` `arangodb_backend_settings`{.interpreted-text role="setting"} `CELERY_MONGODB_BACKEND_SETTINGS` `mongodb_backend_settings`{.interpreted-text role="setting"} `CELERY_EVENT_QUEUE_EXPIRES` `event_queue_expires`{.interpreted-text role="setting"} `CELERY_EVENT_QUEUE_TTL` `event_queue_ttl`{.interpreted-text role="setting"} `CELERY_EVENT_QUEUE_DURABLE` `event_queue_durable`{.interpreted-text role="setting"} `CELERY_EVENT_QUEUE_EXCLUSIVE` `event_queue_exclusive`{.interpreted-text role="setting"} `CELERY_EVENT_QUEUE_PREFIX` `event_queue_prefix`{.interpreted-text role="setting"} `CELERY_EVENT_SERIALIZER` `event_serializer`{.interpreted-text role="setting"} `CELERY_REDIS_DB` `redis_db`{.interpreted-text role="setting"} `CELERY_REDIS_HOST` `redis_host`{.interpreted-text role="setting"} `CELERY_REDIS_MAX_CONNECTIONS` `redis_max_connections`{.interpreted-text role="setting"} `CELERY_REDIS_USERNAME` `redis_username`{.interpreted-text role="setting"} `CELERY_REDIS_PASSWORD` `redis_password`{.interpreted-text role="setting"} `CELERY_REDIS_PORT` `redis_port`{.interpreted-text role="setting"} `CELERY_REDIS_BACKEND_USE_SSL` `redis_backend_use_ssl`{.interpreted-text role="setting"} [\`CELERY_REDIS_BACKEND_CREDENTIAL_PROVIDER]{.title-ref} [ :setting:\`redis_backend_credential_provider]{.title-ref} `CELERY_RESULT_BACKEND` `result_backend`{.interpreted-text role="setting"} `CELERY_MAX_CACHED_RESULTS` `result_cache_max`{.interpreted-text role="setting"} `CELERY_MESSAGE_COMPRESSION` `result_compression`{.interpreted-text role="setting"} `CELERY_RESULT_EXCHANGE` `result_exchange`{.interpreted-text role="setting"} `CELERY_RESULT_EXCHANGE_TYPE` `result_exchange_type`{.interpreted-text role="setting"} `CELERY_RESULT_EXPIRES` `result_expires`{.interpreted-text role="setting"} `CELERY_RESULT_PERSISTENT` `result_persistent`{.interpreted-text role="setting"} `CELERY_RESULT_SERIALIZER` `result_serializer`{.interpreted-text role="setting"} `CELERY_RESULT_DBURI` Use `result_backend`{.interpreted-text role="setting"} instead. `CELERY_RESULT_ENGINE_OPTIONS` `database_engine_options`{.interpreted-text role="setting"} `[...]_DB_SHORT_LIVED_SESSIONS` `database_short_lived_sessions`{.interpreted-text role="setting"} `CELERY_RESULT_DB_TABLE_NAMES` `database_db_names`{.interpreted-text role="setting"} `CELERY_SECURITY_CERTIFICATE` `security_certificate`{.interpreted-text role="setting"} `CELERY_SECURITY_CERT_STORE` `security_cert_store`{.interpreted-text role="setting"} `CELERY_SECURITY_KEY` `security_key`{.interpreted-text role="setting"} `CELERY_SECURITY_KEY_PASSWORD` `security_key_password`{.interpreted-text role="setting"} `CELERY_ACKS_LATE` `task_acks_late`{.interpreted-text role="setting"} `CELERY_ACKS_ON_FAILURE_OR_TIMEOUT` `task_acks_on_failure_or_timeout`{.interpreted-text role="setting"} `CELERY_TASK_ALWAYS_EAGER` `task_always_eager`{.interpreted-text role="setting"} `CELERY_ANNOTATIONS` `task_annotations`{.interpreted-text role="setting"} `CELERY_COMPRESSION` `task_compression`{.interpreted-text role="setting"} `CELERY_CREATE_MISSING_QUEUES` `task_create_missing_queues`{.interpreted-text role="setting"} `CELERY_CREATE_MISSING_QUEUE_TYPE` `task_create_missing_queue_type`{.interpreted-text role="setting"} \`\`CELERY_CREATE_MISSING_QUEUE_EXCHANGE_TYPE \`[ :setting:\`task_create_missing_queue_exchange_type]{.title-ref} `CELERY_DEFAULT_DELIVERY_MODE` `task_default_delivery_mode`{.interpreted-text role="setting"} `CELERY_DEFAULT_EXCHANGE` `task_default_exchange`{.interpreted-text role="setting"} `CELERY_DEFAULT_EXCHANGE_TYPE` `task_default_exchange_type`{.interpreted-text role="setting"} `CELERY_DEFAULT_QUEUE` `task_default_queue`{.interpreted-text role="setting"} `CELERY_DEFAULT_QUEUE_TYPE` `task_default_queue_type`{.interpreted-text role="setting"} `CELERY_DEFAULT_RATE_LIMIT` `task_default_rate_limit`{.interpreted-text role="setting"} `CELERY_DEFAULT_ROUTING_KEY` `task_default_routing_key`{.interpreted-text role="setting"} `CELERY_EAGER_PROPAGATES` `task_eager_propagates`{.interpreted-text role="setting"} `CELERY_IGNORE_RESULT` `task_ignore_result`{.interpreted-text role="setting"} `CELERY_PUBLISH_RETRY` `task_publish_retry`{.interpreted-text role="setting"} `CELERY_PUBLISH_RETRY_POLICY` `task_publish_retry_policy`{.interpreted-text role="setting"} `CELERY_QUEUES` `task_queues`{.interpreted-text role="setting"} `CELERY_ROUTES` `task_routes`{.interpreted-text role="setting"} `CELERY_SEND_SENT_EVENT` `task_send_sent_event`{.interpreted-text role="setting"} `CELERY_TASK_SERIALIZER` `task_serializer`{.interpreted-text role="setting"} `CELERYD_SOFT_TIME_LIMIT` `task_soft_time_limit`{.interpreted-text role="setting"} `CELERY_TASK_TRACK_STARTED` `task_track_started`{.interpreted-text role="setting"} `CELERY_TASK_REJECT_ON_WORKER_LOST` `task_reject_on_worker_lost`{.interpreted-text role="setting"} `CELERYD_TIME_LIMIT` `task_time_limit`{.interpreted-text role="setting"} `CELERY_ALLOW_ERROR_CB_ON_CHORD_HEADER` `task_allow_error_cb_on_chord_header`{.interpreted-text role="setting"} `CELERYD_AGENT` `worker_agent`{.interpreted-text role="setting"} `CELERYD_AUTOSCALER` `worker_autoscaler`{.interpreted-text role="setting"} `CELERYD_CONCURRENCY` `worker_concurrency`{.interpreted-text role="setting"} `CELERYD_CONSUMER` `worker_consumer`{.interpreted-text role="setting"} `CELERY_WORKER_DIRECT` `worker_direct`{.interpreted-text role="setting"} `CELERY_DISABLE_RATE_LIMITS` `worker_disable_rate_limits`{.interpreted-text role="setting"} `CELERY_ENABLE_REMOTE_CONTROL` `worker_enable_remote_control`{.interpreted-text role="setting"} `CELERYD_HIJACK_ROOT_LOGGER` `worker_hijack_root_logger`{.interpreted-text role="setting"} `CELERYD_LOG_COLOR` `worker_log_color`{.interpreted-text role="setting"} `CELERY_WORKER_LOG_FORMAT` `worker_log_format`{.interpreted-text role="setting"} `CELERYD_WORKER_LOST_WAIT` `worker_lost_wait`{.interpreted-text role="setting"} `CELERYD_MAX_TASKS_PER_CHILD` `worker_max_tasks_per_child`{.interpreted-text role="setting"} `CELERYD_POOL` `worker_pool`{.interpreted-text role="setting"} `CELERYD_POOL_PUTLOCKS` `worker_pool_putlocks`{.interpreted-text role="setting"} `CELERYD_POOL_RESTARTS` `worker_pool_restarts`{.interpreted-text role="setting"} `CELERYD_PREFETCH_MULTIPLIER` `worker_prefetch_multiplier`{.interpreted-text role="setting"} `CELERYD_ETA_TASK_LIMIT` `worker_eta_task_limit`{.interpreted-text role="setting"} `CELERYD_ENABLE_PREFETCH_COUNT_REDUCTION` `worker_enable_prefetch_count_reduction`{.interpreted-text role="setting"} `CELERYD_REDIRECT_STDOUTS` `worker_redirect_stdouts`{.interpreted-text role="setting"} `CELERYD_REDIRECT_STDOUTS_LEVEL` `worker_redirect_stdouts_level`{.interpreted-text role="setting"} `CELERY_SEND_EVENTS` `worker_send_task_events`{.interpreted-text role="setting"} `CELERYD_STATE_DB` `worker_state_db`{.interpreted-text role="setting"} `CELERY_WORKER_TASK_LOG_FORMAT` `worker_task_log_format`{.interpreted-text role="setting"} `CELERYD_TIMER` `worker_timer`{.interpreted-text role="setting"} `CELERYD_TIMER_PRECISION` `worker_timer_precision`{.interpreted-text role="setting"} `CELERYD_DETECT_QUORUM_QUEUES` `worker_detect_quorum_queues`{.interpreted-text role="setting"} ## Configuration Directives ### General settings {#conf-datetime} ::: setting accept_content ::: #### `accept_content` Default: `{'json'}` (set, list, or tuple). A white-list of content-types/serializers to allow. If a message is received that\'s not in this list then the message will be discarded with an error. By default only json is enabled but any content type can be added, including pickle and yaml; when this is the case make sure untrusted parties don\'t have access to your broker. See `guide-security`{.interpreted-text role="ref"} for more. Example: # using serializer name accept_content = ['json'] # or the actual content-type (MIME) accept_content = ['application/json'] ::: setting result_accept_content ::: #### `result_accept_content` Default: `None` (can be set, list or tuple). ::: versionadded 4.3 ::: A white-list of content-types/serializers to allow for the result backend. If a message is received that\'s not in this list then the message will be discarded with an error. By default it is the same serializer as `accept_content`. However, a different serializer for accepted content of the result backend can be specified. Usually this is needed if signed messaging is used and the result is stored unsigned in the result backend. See `guide-security`{.interpreted-text role="ref"} for more. Example: # using serializer name result_accept_content = ['json'] # or the actual content-type (MIME) result_accept_content = ['application/json'] ### Time and date settings ::: setting enable_utc ::: #### `enable_utc` ::: versionadded 2.5 ::: Default: Enabled by default since version 3.0. If enabled dates and times in messages will be converted to use the UTC timezone. Note that workers running Celery versions below 2.5 will assume a local timezone for all messages, so only enable if all workers have been upgraded. ::: setting timezone ::: #### `timezone` ::: versionadded 2.5 ::: Default: `"UTC"`. Configure Celery to use a custom time zone. The timezone value can be any time zone supported by the [ZoneInfo](https://docs.python.org/3/library/zoneinfo.html) library. If not set the UTC timezone is used. For backwards compatibility there\'s also a `enable_utc`{.interpreted-text role="setting"} setting, and when this is set to false the system local timezone is used instead. ### Task settings {#conf-tasks} ::: setting task_annotations ::: #### `task_annotations` ::: versionadded 2.5 ::: Default: `None`{.interpreted-text role="const"}. This setting can be used to rewrite any task attribute from the configuration. The setting can be a dict, or a list of annotation objects that filter for tasks and return a map of attributes to change. This will change the `rate_limit` attribute for the `tasks.add` task: ``` python task_annotations = {'tasks.add': {'rate_limit': '10/s'}} ``` or change the same for all tasks: ``` python task_annotations = {'*': {'rate_limit': '10/s'}} ``` You can change methods too, for example the `on_failure` handler: ``` python def my_on_failure(self, exc, task_id, args, kwargs, einfo): print('Oh no! Task failed: {0!r}'.format(exc)) task_annotations = {'*': {'on_failure': my_on_failure}} ``` If you need more flexibility then you can use objects instead of a dict to choose the tasks to annotate: ``` python class MyAnnotate: def annotate(self, task): if task.name.startswith('tasks.'): return {'rate_limit': '10/s'} task_annotations = (MyAnnotate(), {other,}) ``` ::: setting task_compression ::: #### `task_compression` Default: `None`{.interpreted-text role="const"} Default compression used for task messages. Can be `gzip`, `bzip2` (if available), or any custom compression schemes registered in the Kombu compression registry. The default is to send uncompressed messages. ::: setting task_protocol ::: #### `task_protocol` Default: 2 (since 4.0). Set the default task message protocol version used to send tasks. Supports protocols: 1 and 2. Protocol 2 is supported by 3.1.24 and 4.x+. ::: setting task_serializer ::: #### `task_serializer` Default: `"json"` (since 4.0, earlier: pickle). A string identifying the default serialization method to use. Can be [json]{.title-ref} (default), [pickle]{.title-ref}, [yaml]{.title-ref}, [msgpack]{.title-ref}, or any custom serialization methods that have been registered with `kombu.serialization.registry`{.interpreted-text role="mod"}. ::: seealso `calling-serializers`{.interpreted-text role="ref"}. ::: ::: setting task_publish_retry ::: #### `task_publish_retry` ::: versionadded 2.2 ::: Default: Enabled. Decides if publishing task messages will be retried in the case of connection loss or other connection errors. See also `task_publish_retry_policy`{.interpreted-text role="setting"}. ::: setting task_publish_retry_policy ::: #### `task_publish_retry_policy` ::: versionadded 2.2 ::: Default: See `calling-retry`{.interpreted-text role="ref"}. Defines the default policy when retrying publishing a task message in the case of connection loss or other connection errors. ### Task execution settings {#conf-task-execution} ::: setting task_always_eager ::: #### `task_always_eager` Default: Disabled. If this is `True`{.interpreted-text role="const"}, all tasks will be executed locally by blocking until the task returns. `apply_async()` and `Task.delay()` will return an `~celery.result.EagerResult`{.interpreted-text role="class"} instance, that emulates the API and behavior of `~celery.result.AsyncResult`{.interpreted-text role="class"}, except the result is already evaluated. That is, tasks will be executed locally instead of being sent to the queue. ::: setting task_eager_propagates ::: #### `task_eager_propagates` Default: Disabled. If this is `True`{.interpreted-text role="const"}, eagerly executed tasks (applied by [task.apply()]{.title-ref}, or when the `task_always_eager`{.interpreted-text role="setting"} setting is enabled), will propagate exceptions. It\'s the same as always running `apply()` with `throw=True`. ::: setting task_store_eager_result ::: #### `task_store_eager_result` ::: versionadded 5.1 ::: Default: Disabled. If this is `True`{.interpreted-text role="const"} and `task_always_eager`{.interpreted-text role="setting"} is `True`{.interpreted-text role="const"} and `task_ignore_result`{.interpreted-text role="setting"} is `False`{.interpreted-text role="const"}, the results of eagerly executed tasks will be saved to the backend. By default, even with `task_always_eager`{.interpreted-text role="setting"} set to `True`{.interpreted-text role="const"} and `task_ignore_result`{.interpreted-text role="setting"} set to `False`{.interpreted-text role="const"}, the result will not be saved. ::: setting task_remote_tracebacks ::: #### `task_remote_tracebacks` Default: Disabled. If enabled task results will include the workers stack when re-raising task errors. This requires the `tblib`{.interpreted-text role="pypi"} library, that can be installed using `pip`{.interpreted-text role="command"}: ``` console $ pip install celery[tblib] ``` See `bundles`{.interpreted-text role="ref"} for information on combining multiple extension requirements. ::: setting task_ignore_result ::: #### `task_ignore_result` Default: Disabled. Whether to store the task return values or not (tombstones). If you still want to store errors, just not successful return values, you can set `task_store_errors_even_if_ignored`{.interpreted-text role="setting"}. ::: setting task_store_errors_even_if_ignored ::: #### `task_store_errors_even_if_ignored` Default: Disabled. If set, the worker stores all task errors in the result store even if `Task.ignore_result `{.interpreted-text role="attr"} is on. ::: setting task_track_started ::: #### `task_track_started` Default: Disabled. If `True`{.interpreted-text role="const"} the task will report its status as \'started\' when the task is executed by a worker. The default value is `False`{.interpreted-text role="const"} as the normal behavior is to not report that level of granularity. Tasks are either pending, finished, or waiting to be retried. Having a \'started\' state can be useful for when there are long running tasks and there\'s a need to report what task is currently running. ::: setting task_time_limit ::: #### `task_time_limit` Default: No time limit. Task hard time limit in seconds. The worker processing the task will be killed and replaced with a new one when this is exceeded. ::: setting task_allow_error_cb_on_chord_header ::: #### `task_allow_error_cb_on_chord_header` ::: versionadded 5.3 ::: Default: Disabled. Enabling this flag will allow linking an error callback to a chord header, which by default will not link when using `link_error()`, and preventing from the chord\'s body to execute if any of the tasks in the header fails. Consider the following canvas with the flag disabled (default behavior): ``` python header = group([t1, t2]) body = t3 c = chord(header, body) c.link_error(error_callback_sig) ``` If *any* of the header tasks failed (`t1` or `t2`), by default, the chord body (`t3`) would **not execute**, and `error_callback_sig` will be called **once** (for the body). Enabling this flag will change the above behavior by: 1. `error_callback_sig` will be linked to `t1` and `t2` (as well as `t3`). 2. If *any* of the header tasks failed, `error_callback_sig` will be called **for each** failed header task **and** the `body` (even if the body didn\'t run). Consider now the following canvas with the flag enabled: ``` python header = group([failingT1, failingT2]) body = t3 c = chord(header, body) c.link_error(error_callback_sig) ``` If *all* of the header tasks failed (`failingT1` and `failingT2`), then the chord body (`t3`) would **not execute**, and `error_callback_sig` will be called **3 times** (two times for the header and one time for the body). Lastly, consider the following canvas with the flag enabled: ``` python header = group([failingT1, failingT2]) body = t3 upgraded_chord = chain(header, body) upgraded_chord.link_error(error_callback_sig) ``` This canvas will behave exactly the same as the previous one, since the `chain` will be upgraded to a `chord` internally. ::: setting task_soft_time_limit ::: #### `task_soft_time_limit` Default: No soft time limit. Task soft time limit in seconds. The `~@SoftTimeLimitExceeded`{.interpreted-text role="exc"} exception will be raised when this is exceeded. For example, the task can catch this to clean up before the hard time limit comes: ``` python from celery.exceptions import SoftTimeLimitExceeded @app.task def mytask(): try: return do_work() except SoftTimeLimitExceeded: cleanup_in_a_hurry() ``` ::: setting task_acks_late ::: #### `task_acks_late` Default: Disabled. Late ack means the task messages will be acknowledged **after** the task has been executed, not *right before* (the default behavior). ::: seealso FAQ: `faq-acks_late-vs-retry`{.interpreted-text role="ref"}. ::: ::: setting task_acks_on_failure_or_timeout ::: #### `task_acks_on_failure_or_timeout` Default: Enabled When enabled messages for all tasks will be acknowledged even if they fail or time out. Configuring this setting only applies to tasks that are acknowledged **after** they have been executed and only if `task_acks_late`{.interpreted-text role="setting"} is enabled. ::: setting task_reject_on_worker_lost ::: #### `task_reject_on_worker_lost` Default: Disabled. Even if `task_acks_late`{.interpreted-text role="setting"} is enabled, the worker will acknowledge tasks when the worker process executing them abruptly exits or is signaled (e.g., `KILL`{.interpreted-text role="sig"}/`INT`{.interpreted-text role="sig"}, etc). Setting this to true allows the message to be re-queued instead, so that the task will execute again by the same worker, or another worker. ::: warning ::: title Warning ::: Enabling this can cause message loops; make sure you know what you\'re doing. ::: ::: setting task_default_rate_limit ::: #### `task_default_rate_limit` Default: No rate limit. The global default rate limit for tasks. This value is used for tasks that doesn\'t have a custom rate limit ::: seealso The `worker_disable_rate_limits`{.interpreted-text role="setting"} setting can disable all rate limits. ::: ### Task result backend settings {#conf-result-backend} ::: setting result_backend ::: #### `result_backend` Default: No result backend enabled by default. The backend used to store task results (tombstones). Can be one of the following: - `rpc` : Send results back as AMQP messages See `conf-rpc-result-backend`{.interpreted-text role="ref"}. - `database` : Use a relational database supported by [SQLAlchemy](http://sqlalchemy.org). See `conf-database-result-backend`{.interpreted-text role="ref"}. - `redis` : Use [Redis](#redis) to store the results. See `conf-redis-result-backend`{.interpreted-text role="ref"}. - `cache` : Use [Memcached](http://memcached.org) to store the results. See `conf-cache-result-backend`{.interpreted-text role="ref"}. - `mongodb` : Use [MongoDB](http://mongodb.org) to store the results. See `conf-mongodb-result-backend`{.interpreted-text role="ref"}. - `cassandra` : Use [Cassandra](http://cassandra.apache.org/) to store the results. See `conf-cassandra-result-backend`{.interpreted-text role="ref"}. - `elasticsearch` : Use [Elasticsearch](https://aws.amazon.com/elasticsearch-service/) to store the results. See `conf-elasticsearch-result-backend`{.interpreted-text role="ref"}. - `ironcache` : Use [IronCache](http://www.iron.io/cache) to store the results. See `conf-ironcache-result-backend`{.interpreted-text role="ref"}. - `couchbase` : Use [Couchbase](https://www.couchbase.com/) to store the results. See `conf-couchbase-result-backend`{.interpreted-text role="ref"}. - `arangodb` : Use [ArangoDB](https://www.arangodb.com/) to store the results. See `conf-arangodb-result-backend`{.interpreted-text role="ref"}. - `couchdb` : Use [CouchDB](http://www.couchdb.com/) to store the results. See `conf-couchdb-result-backend`{.interpreted-text role="ref"}. - `cosmosdbsql (experimental)` : Use the [CosmosDB](https://azure.microsoft.com/en-us/services/cosmos-db/) PaaS to store the results. See `conf-cosmosdbsql-result-backend`{.interpreted-text role="ref"}. - `filesystem` : Use a shared directory to store the results. See `conf-filesystem-result-backend`{.interpreted-text role="ref"}. - `consul` : Use the [Consul](https://consul.io/) K/V store to store the results See `conf-consul-result-backend`{.interpreted-text role="ref"}. - `azureblockblob` : Use the [AzureBlockBlob](https://azure.microsoft.com/en-us/services/storage/blobs/) PaaS store to store the results See `conf-azureblockblob-result-backend`{.interpreted-text role="ref"}. - `s3` : Use the [S3](https://aws.amazon.com/s3/) to store the results See `conf-s3-result-backend`{.interpreted-text role="ref"}. - `gcs` : Use the [GCS](https://cloud.google.com/storage/) to store the results See `conf-gcs-result-backend`{.interpreted-text role="ref"}. > While the AMQP result backend is very efficient, you must make sure > you only receive the same result once. See > `userguide/calling`{.interpreted-text role="doc"}). ::: setting result_backend_always_retry ::: #### `result_backend_always_retry` Default: `False`{.interpreted-text role="const"} If enable, backend will try to retry on the event of recoverable exceptions instead of propagating the exception. It will use an exponential backoff sleep time between 2 retries. ::: setting result_backend_max_sleep_between_retries_ms ::: #### `result_backend_max_sleep_between_retries_ms` Default: 10000 This specifies the maximum sleep time between two backend operation retry. ::: setting result_backend_base_sleep_between_retries_ms ::: #### `result_backend_base_sleep_between_retries_ms` Default: 10 This specifies the base amount of sleep time between two backend operation retry. ::: setting result_backend_max_retries ::: #### `result_backend_max_retries` Default: Inf This is the maximum of retries in case of recoverable exceptions. ::: setting result_backend_thread_safe ::: #### `result_backend_thread_safe` Default: False If True, then the backend object is shared across threads. This may be useful for using a shared connection pool instead of creating a connection for every thread. ::: setting result_backend_transport_options ::: #### `result_backend_transport_options` Default: `{}` (empty mapping). A dict of additional options passed to the underlying transport. See your transport user manual for supported options (if any). Example setting the visibility timeout (supported by Redis and SQS transports): ``` python result_backend_transport_options = {'visibility_timeout': 18000} # 5 hours ``` ::: setting result_serializer ::: #### `result_serializer` Default: `json` since 4.0 (earlier: pickle). Result serialization format. See `calling-serializers`{.interpreted-text role="ref"} for information about supported serialization formats. ::: setting result_compression ::: #### `result_compression` Default: No compression. Optional compression method used for task results. Supports the same options as the `task_compression`{.interpreted-text role="setting"} setting. ::: setting result_extended ::: #### `result_extended` Default: `False` Enables extended task result attributes (name, args, kwargs, worker, retries, queue, delivery_info) to be written to backend. ::: setting result_expires ::: #### `result_expires` Default: Expire after 1 day. Time (in seconds, or a `~datetime.timedelta`{.interpreted-text role="class"} object) for when after stored task tombstones will be deleted. A built-in periodic task will delete the results after this time (`celery.backend_cleanup`), assuming that `celery beat` is enabled. The task runs daily at 4am. A value of `None`{.interpreted-text role="const"} or 0 means results will never expire (depending on backend specifications). ::: note ::: title Note ::: For the moment this only works with the AMQP, database, cache, Couchbase, filesystem and Redis backends. When using the database or filesystem backend, `celery beat` must be running for the results to be expired. ::: ::: setting result_cache_max ::: #### `result_cache_max` Default: Disabled by default. Enables client caching of results. This can be useful for the old deprecated \'amqp\' backend where the result is unavailable as soon as one result instance consumes it. This is the total number of results to cache before older results are evicted. A value of 0 or None means no limit, and a value of `-1`{.interpreted-text role="const"} will disable the cache. Disabled by default. ::: setting result_chord_join_timeout ::: #### `result_chord_join_timeout` Default: 3.0. The timeout in seconds (int/float) when joining a group\'s results within a chord. ::: setting result_chord_retry_interval ::: #### `result_chord_retry_interval` Default: 1.0. Default interval for retrying chord tasks. ::: setting override_backends ::: #### `override_backends` Default: Disabled by default. Path to class that implements backend. Allows to override backend implementation. This can be useful if you need to store additional metadata about executed tasks, override retry policies, etc. Example: ``` python override_backends = {"db": "custom_module.backend.class"} ``` ### Database backend settings {#conf-database-result-backend} #### Database URL Examples To use the database backend you have to configure the `result_backend`{.interpreted-text role="setting"} setting with a connection URL and the `db+` prefix: ``` python result_backend = 'db+scheme://user:password@host:port/dbname' ``` Examples: # sqlite (filename) result_backend = 'db+sqlite:///results.sqlite' # mysql result_backend = 'db+mysql://scott:tiger@localhost/foo' # postgresql result_backend = 'db+postgresql://scott:tiger@localhost/mydatabase' # oracle result_backend = 'db+oracle://scott:tiger@127.0.0.1:1521/sidname' ``` python ``` Please see [Supported Databases](http://www.sqlalchemy.org/docs/core/engines.html#supported-databases) for a table of supported databases, and [Connection String](http://www.sqlalchemy.org/docs/core/engines.html#database-urls) for more information about connection strings (this is the part of the URI that comes after the `db+` prefix). ::: setting database_create_tables_at_setup ::: #### `database_create_tables_at_setup` ::: versionadded 5.5.0 ::: Default: True by default. - If [True]{.title-ref}, Celery will create the tables in the database during setup. - If [False]{.title-ref}, Celery will create the tables lazily, i.e. wait for the first task to be executed before creating the tables. ::: note ::: title Note ::: Before celery 5.5, the tables were created lazily i.e. it was equivalent to [database_create_tables_at_setup]{.title-ref} set to False. ::: ::: setting database_engine_options ::: #### `database_engine_options` Default: `{}` (empty mapping). To specify additional SQLAlchemy database engine options you can use the `database_engine_options`{.interpreted-text role="setting"} setting: # echo enables verbose logging from SQLAlchemy. app.conf.database_engine_options = {'echo': True} ::: setting database_short_lived_sessions ::: #### `database_short_lived_sessions` Default: Disabled by default. Short lived sessions are disabled by default. If enabled they can drastically reduce performance, especially on systems processing lots of tasks. This option is useful on low-traffic workers that experience errors as a result of cached database connections going stale through inactivity. For example, intermittent errors like [(OperationalError) (2006, \'MySQL server has gone away\')]{.title-ref} can be fixed by enabling short lived sessions. This option only affects the database backend. ::: setting database_table_schemas ::: #### `database_table_schemas` Default: `{}` (empty mapping). When SQLAlchemy is configured as the result backend, Celery automatically creates two tables to store result meta-data for tasks. This setting allows you to customize the schema of the tables: ``` python # use custom schema for the database result backend. database_table_schemas = { 'task': 'celery', 'group': 'celery', } ``` ::: setting database_table_names ::: #### `database_table_names` Default: `{}` (empty mapping). When SQLAlchemy is configured as the result backend, Celery automatically creates two tables to store result meta-data for tasks. This setting allows you to customize the table names: ``` python # use custom table names for the database result backend. database_table_names = { 'task': 'myapp_taskmeta', 'group': 'myapp_groupmeta', } ``` ### RPC backend settings {#conf-rpc-result-backend} ::: setting result_persistent ::: #### `result_persistent` Default: Disabled by default (transient messages). If set to `True`{.interpreted-text role="const"}, result messages will be persistent. This means the messages won\'t be lost after a broker restart. #### Example configuration ``` python result_backend = 'rpc://' result_persistent = False ``` **Please note**: using this backend could trigger the raise of `celery.backends.rpc.BacklogLimitExceeded` if the task tombstone is too *old*. E.g. ``` python for i in range(10000): r = debug_task.delay() print(r.state) # this would raise celery.backends.rpc.BacklogLimitExceeded ``` ### Cache backend settings {#conf-cache-result-backend} ::: note ::: title Note ::: The cache backend supports the `pylibmc`{.interpreted-text role="pypi"} and `python-memcached`{.interpreted-text role="pypi"} libraries. The latter is used only if `pylibmc`{.interpreted-text role="pypi"} isn\'t installed. ::: Using a single Memcached server: ``` python result_backend = 'cache+memcached://127.0.0.1:11211/' ``` Using multiple Memcached servers: ``` python result_backend = """ cache+memcached://172.19.26.240:11211;172.19.26.242:11211/ """.strip() ``` The \"memory\" backend stores the cache in memory only: ``` python result_backend = 'cache' cache_backend = 'memory' ``` ::: setting cache_backend_options ::: #### `cache_backend_options` Default: `{}` (empty mapping). You can set `pylibmc`{.interpreted-text role="pypi"} options using the `cache_backend_options`{.interpreted-text role="setting"} setting: ``` python cache_backend_options = { 'binary': True, 'behaviors': {'tcp_nodelay': True}, } ``` ::: setting cache_backend ::: #### `cache_backend` This setting is no longer used in celery\'s builtin backends as it\'s now possible to specify the cache backend directly in the `result_backend`{.interpreted-text role="setting"} setting. ::: note ::: title Note ::: The `django-celery-results`{.interpreted-text role="ref"} library uses `cache_backend` for choosing django caches. ::: ### MongoDB backend settings {#conf-mongodb-result-backend} ::: note ::: title Note ::: The MongoDB backend requires the `pymongo`{.interpreted-text role="mod"} library: ::: ::: setting mongodb_backend_settings ::: #### mongodb_backend_settings This is a dict supporting the following keys: - database : The database name to connect to. Defaults to `celery`. - taskmeta_collection : The collection name to store task meta data. Defaults to `celery_taskmeta`. - max_pool_size : Passed as max_pool_size to PyMongo\'s Connection or MongoClient constructor. It is the maximum number of TCP connections to keep open to MongoDB at a given time. If there are more open connections than max_pool_size, sockets will be closed when they are released. Defaults to 10. - options > Additional keyword arguments to pass to the mongodb connection > constructor. See the `pymongo`{.interpreted-text role="mod"} docs > to see a list of arguments supported. ::: note ::: title Note ::: With pymongo\>=4.14, options are case-sensitive when they were previously case-insensitive. See `~pymongo.mongo_client.MongoClient`{.interpreted-text role="class"} to determine the correct case. ::: #### Example configuration {#example-mongodb-result-config} ``` python result_backend = 'mongodb://localhost:27017/' mongodb_backend_settings = { 'database': 'mydb', 'taskmeta_collection': 'my_taskmeta_collection', } ``` ### Redis backend settings {#conf-redis-result-backend} #### Configuring the backend URL ::: note ::: title Note ::: The Redis backend requires the `redis`{.interpreted-text role="pypi"} library. To install this package use `pip`{.interpreted-text role="command"}: ``` console $ pip install celery[redis] ``` See `bundles`{.interpreted-text role="ref"} for information on combining multiple extension requirements. ::: This backend requires the `result_backend`{.interpreted-text role="setting"} setting to be set to a Redis or [Redis over TLS](https://www.iana.org/assignments/uri-schemes/prov/rediss) URL: result_backend = 'redis://username:password@host:port/db' For example: result_backend = 'redis://localhost/0' is the same as: result_backend = 'redis://' Use the `rediss://` protocol to connect to redis over TLS: result_backend = 'rediss://username:password@host:port/db?ssl_cert_reqs=required' Note that the `ssl_cert_reqs` string should be one of `required`, `optional`, or `none` (though, for backwards compatibility with older Celery versions, the string may also be one of `CERT_REQUIRED`, `CERT_OPTIONAL`, `CERT_NONE`, but those values only work for Celery, not for Redis directly). If a Unix socket connection should be used, the URL needs to be in the format:: result_backend = 'socket:///path/to/redis.sock' The fields of the URL are defined as follows: 1. `username` > ::: versionadded > 5.1.0 > ::: > > Username used to connect to the database. > > Note that this is only supported in Redis\>=6.0 and with > py-redis\>=3.4.0 installed. > > If you use an older database version or an older client version > you can omit the username: > > result_backend = 'redis://:password@host:port/db' 2. `password` > Password used to connect to the database. 3. `host` > Host name or IP address of the Redis server (e.g., > [localhost]{.title-ref}). 4. `port` > Port to the Redis server. Default is 6379. 5. `db` > Database number to use. Default is 0. The db can include an > optional leading slash. When using a TLS connection (protocol is `rediss://`), you may pass in all values in `broker_use_ssl`{.interpreted-text role="setting"} as query parameters. Paths to certificates must be URL encoded, and `ssl_cert_reqs` is required. Example: ``` python result_backend = 'rediss://:password@host:port/db?\ ssl_cert_reqs=required\ &ssl_ca_certs=%2Fvar%2Fssl%2Fmyca.pem\ # /var/ssl/myca.pem &ssl_certfile=%2Fvar%2Fssl%2Fredis-server-cert.pem\ # /var/ssl/redis-server-cert.pem &ssl_keyfile=%2Fvar%2Fssl%2Fprivate%2Fworker-key.pem' # /var/ssl/private/worker-key.pem ``` Note that the `ssl_cert_reqs` string should be one of `required`, `optional`, or `none` (though, for backwards compatibility, the string may also be one of `CERT_REQUIRED`, `CERT_OPTIONAL`, `CERT_NONE`). ::: setting redis_backend_health_check_interval ::: ::: versionadded 5.1.0 ::: #### `redis_backend_health_check_interval` Default: Not configured The Redis backend supports health checks. This value must be set as an integer whose value is the number of seconds between health checks. If a ConnectionError or a TimeoutError is encountered during the health check, the connection will be re-established and the command retried exactly once. ::: setting redis_backend_use_ssl ::: #### `redis_backend_use_ssl` Default: Disabled. The Redis backend supports SSL. This value must be set in the form of a dictionary. The valid key-value pairs are the same as the ones mentioned in the `redis` sub-section under `broker_use_ssl`{.interpreted-text role="setting"}. ::: setting redis_backend_credential_provider ::: ::: versionadded 5.6 ::: #### `redis_backend_credential_provider` Default: Disabled. The Redis backend supports credential provider. This value must be set in the form of a class path string or a class instance. e.g. `mymodule.myfile.myclass` check more details in [RedisCredentialProvider](https://redis.readthedocs.io/en/stable/examples/connection_examples.html#Connecting-to-a-redis-instance-with-standard-credential-provider) doc. ::: setting redis_max_connections ::: #### `redis_max_connections` Default: No limit. Maximum number of connections available in the Redis connection pool used for sending and retrieving results. ::: warning ::: title Warning ::: Redis will raise a [ConnectionError]{.title-ref} if the number of concurrent connections exceeds the maximum. ::: ::: setting redis_socket_connect_timeout ::: #### `redis_socket_connect_timeout` ::: versionadded 4.0.1 ::: Default: `None`{.interpreted-text role="const"} Socket timeout for connections to Redis from the result backend in seconds (int/float) ::: setting redis_socket_timeout ::: #### `redis_socket_timeout` Default: 120.0 seconds. Socket timeout for reading/writing operations to the Redis server in seconds (int/float), used by the redis result backend. ::: setting redis_retry_on_timeout ::: #### `redis_retry_on_timeout` ::: versionadded 4.4.1 ::: Default: `False`{.interpreted-text role="const"} To retry reading/writing operations on TimeoutError to the Redis server, used by the redis result backend. Shouldn\'t set this variable if using Redis connection by unix socket. ::: setting redis_socket_keepalive ::: #### `redis_socket_keepalive` ::: versionadded 4.4.1 ::: Default: `False`{.interpreted-text role="const"} Socket TCP keepalive to keep connections healthy to the Redis server, used by the redis result backend. ::: setting redis_client_name ::: #### `redis_client_name` ::: versionadded 5.6 ::: Default: `None`{.interpreted-text role="const"} Sets the client name for Redis connections used by the result backend. This can help identify connections in Redis monitoring tools. ### Cassandra/AstraDB backend settings {#conf-cassandra-result-backend} ::: note ::: title Note ::: This Cassandra backend driver requires `cassandra-driver`{.interpreted-text role="pypi"}. This backend can refer to either a regular Cassandra installation or a managed Astra DB instance. Depending on which one, exactly one between the `cassandra_servers`{.interpreted-text role="setting"} and `cassandra_secure_bundle_path`{.interpreted-text role="setting"} settings must be provided (but not both). To install, use `pip`{.interpreted-text role="command"}: ``` console $ pip install celery[cassandra] ``` See `bundles`{.interpreted-text role="ref"} for information on combining multiple extension requirements. ::: This backend requires the following configuration directives to be set. ::: setting cassandra_servers ::: #### `cassandra_servers` Default: `[]` (empty list). List of `host` Cassandra servers. This must be provided when connecting to a Cassandra cluster. Passing this setting is strictly exclusive to `cassandra_secure_bundle_path`{.interpreted-text role="setting"}. Example: cassandra_servers = ['localhost'] ::: setting cassandra_secure_bundle_path ::: #### `cassandra_secure_bundle_path` Default: None. Absolute path to the secure-connect-bundle zip file to connect to an Astra DB instance. Passing this setting is strictly exclusive to `cassandra_servers`{.interpreted-text role="setting"}. Example: cassandra_secure_bundle_path = '/home/user/bundles/secure-connect.zip' When connecting to Astra DB, it is necessary to specify the plain-text auth provider and the associated username and password, which take the value of the Client ID and the Client Secret, respectively, of a valid token generated for the Astra DB instance. See below for an Astra DB configuration example. ::: setting cassandra_port ::: #### `cassandra_port` Default: 9042. Port to contact the Cassandra servers on. ::: setting cassandra_keyspace ::: #### `cassandra_keyspace` Default: None. The keyspace in which to store the results. For example: cassandra_keyspace = 'tasks_keyspace' ::: setting cassandra_table ::: #### `cassandra_table` Default: None. The table (column family) in which to store the results. For example: cassandra_table = 'tasks' ::: setting cassandra_read_consistency ::: #### `cassandra_read_consistency` Default: None. The read consistency used. Values can be `ONE`, `TWO`, `THREE`, `QUORUM`, `ALL`, `LOCAL_QUORUM`, `EACH_QUORUM`, `LOCAL_ONE`. ::: setting cassandra_write_consistency ::: #### `cassandra_write_consistency` Default: None. The write consistency used. Values can be `ONE`, `TWO`, `THREE`, `QUORUM`, `ALL`, `LOCAL_QUORUM`, `EACH_QUORUM`, `LOCAL_ONE`. ::: setting cassandra_entry_ttl ::: #### `cassandra_entry_ttl` Default: None. Time-to-live for status entries. They will expire and be removed after that many seconds after adding. A value of `None`{.interpreted-text role="const"} (default) means they will never expire. ::: setting cassandra_auth_provider ::: #### `cassandra_auth_provider` Default: `None`{.interpreted-text role="const"}. AuthProvider class within `cassandra.auth` module to use. Values can be `PlainTextAuthProvider` or `SaslAuthProvider`. ::: setting cassandra_auth_kwargs ::: #### `cassandra_auth_kwargs` Default: `{}` (empty mapping). Named arguments to pass into the authentication provider. For example: ``` python cassandra_auth_kwargs = { username: 'cassandra', password: 'cassandra' } ``` ::: setting cassandra_options ::: #### `cassandra_options` Default: `{}` (empty mapping). Named arguments to pass into the `cassandra.cluster` class. ``` python cassandra_options = { 'cql_version': '3.2.1' 'protocol_version': 3 } ``` #### Example configuration (Cassandra) ``` python result_backend = 'cassandra://' cassandra_servers = ['localhost'] cassandra_keyspace = 'celery' cassandra_table = 'tasks' cassandra_read_consistency = 'QUORUM' cassandra_write_consistency = 'QUORUM' cassandra_entry_ttl = 86400 ``` #### Example configuration (Astra DB) ``` python result_backend = 'cassandra://' cassandra_keyspace = 'celery' cassandra_table = 'tasks' cassandra_read_consistency = 'QUORUM' cassandra_write_consistency = 'QUORUM' cassandra_auth_provider = 'PlainTextAuthProvider' cassandra_auth_kwargs = { 'username': '<>', 'password': '<>' } cassandra_secure_bundle_path = '/path/to/secure-connect-bundle.zip' cassandra_entry_ttl = 86400 ``` #### Additional configuration The Cassandra driver, when establishing the connection, undergoes a stage of negotiating the protocol version with the server(s). Similarly, a load-balancing policy is automatically supplied (by default `DCAwareRoundRobinPolicy`, which in turn has a `local_dc` setting, also determined by the driver upon connection). When possible, one should explicitly provide these in the configuration: moreover, future versions of the Cassandra driver will require at least the load-balancing policy to be specified (using [execution profiles](https://docs.datastax.com/en/developer/python-driver/3.25/execution_profiles/), as shown below). A full configuration for the Cassandra backend would thus have the following additional lines: ``` python from cassandra.policies import DCAwareRoundRobinPolicy from cassandra.cluster import ExecutionProfile from cassandra.cluster import EXEC_PROFILE_DEFAULT myEProfile = ExecutionProfile( load_balancing_policy=DCAwareRoundRobinPolicy( local_dc='datacenter1', # replace with your DC name ) ) cassandra_options = { 'protocol_version': 5, # for Cassandra 4, change if needed 'execution_profiles': {EXEC_PROFILE_DEFAULT: myEProfile}, } ``` And similarly for Astra DB: ``` python from cassandra.policies import DCAwareRoundRobinPolicy from cassandra.cluster import ExecutionProfile from cassandra.cluster import EXEC_PROFILE_DEFAULT myEProfile = ExecutionProfile( load_balancing_policy=DCAwareRoundRobinPolicy( local_dc='europe-west1', # for Astra DB, region name = dc name ) ) cassandra_options = { 'protocol_version': 4, # for Astra DB 'execution_profiles': {EXEC_PROFILE_DEFAULT: myEProfile}, } ``` ### S3 backend settings {#conf-s3-result-backend} ::: note ::: title Note ::: This s3 backend driver requires `s3`{.interpreted-text role="pypi"}. To install, use `s3`{.interpreted-text role="command"}: ``` console $ pip install celery[s3] ``` See `bundles`{.interpreted-text role="ref"} for information on combining multiple extension requirements. ::: This backend requires the following configuration directives to be set. ::: setting s3_access_key_id ::: #### `s3_access_key_id` Default: None. The s3 access key id. For example: s3_access_key_id = 'access_key_id' ::: setting s3_secret_access_key ::: #### `s3_secret_access_key` Default: None. The s3 secret access key. For example: s3_secret_access_key = 'access_secret_access_key' ::: setting s3_bucket ::: #### `s3_bucket` Default: None. The s3 bucket name. For example: s3_bucket = 'bucket_name' ::: setting s3_base_path ::: #### `s3_base_path` Default: None. A base path in the s3 bucket to use to store result keys. For example: s3_base_path = '/prefix' ::: setting s3_endpoint_url ::: #### `s3_endpoint_url` Default: None. A custom s3 endpoint url. Use it to connect to a custom self-hosted s3 compatible backend (Ceph, Scality\...). For example: s3_endpoint_url = 'https://.s3.custom.url' ::: setting s3_region ::: #### `s3_region` Default: None. The s3 aws region. For example: s3_region = 'us-east-1' #### Example configuration ``` python s3_access_key_id = 's3-access-key-id' s3_secret_access_key = 's3-secret-access-key' s3_bucket = 'mybucket' s3_base_path = '/celery_result_backend' s3_endpoint_url = 'https://endpoint_url' ``` ### Azure Block Blob backend settings {#conf-azureblockblob-result-backend} To use [AzureBlockBlob](https://azure.microsoft.com/en-us/services/storage/blobs/) as the result backend you simply need to configure the `result_backend`{.interpreted-text role="setting"} setting with the correct URL. The required URL format is `azureblockblob://` followed by the storage connection string. You can find the storage connection string in the `Access Keys` pane of your storage account resource in the Azure Portal. #### Example configuration ``` python result_backend = 'azureblockblob://DefaultEndpointsProtocol=https;AccountName=somename;AccountKey=Lou...bzg==;EndpointSuffix=core.windows.net' ``` ::: setting azureblockblob_container_name ::: #### `azureblockblob_container_name` Default: celery. The name for the storage container in which to store the results. ::: setting azureblockblob_base_path ::: #### `azureblockblob_base_path` ::: versionadded 5.1 ::: Default: None. A base path in the storage container to use to store result keys. For example: azureblockblob_base_path = 'prefix/' ::: setting azureblockblob_retry_initial_backoff_sec ::: #### `azureblockblob_retry_initial_backoff_sec` Default: 2. The initial backoff interval, in seconds, for the first retry. Subsequent retries are attempted with an exponential strategy. ::: setting azureblockblob_retry_increment_base ::: #### `azureblockblob_retry_increment_base` Default: 2. ::: setting azureblockblob_retry_max_attempts ::: #### `azureblockblob_retry_max_attempts` Default: 3. The maximum number of retry attempts. ::: setting azureblockblob_connection_timeout ::: #### `azureblockblob_connection_timeout` Default: 20. Timeout in seconds for establishing the azure block blob connection. ::: setting azureblockblob_read_timeout ::: #### `azureblockblob_read_timeout` Default: 120. Timeout in seconds for reading of an azure block blob. ### GCS backend settings {#conf-gcs-result-backend} ::: note ::: title Note ::: This gcs backend driver requires `google-cloud-storage`{.interpreted-text role="pypi"} and `google-cloud-firestore`{.interpreted-text role="pypi"}. To install, use `gcs`{.interpreted-text role="command"}: ``` console $ pip install celery[gcs] ``` See `bundles`{.interpreted-text role="ref"} for information on combining multiple extension requirements. ::: GCS could be configured via the URL provided in `result_backend`{.interpreted-text role="setting"}, for example: result_backend = 'gs://mybucket/some-prefix?gcs_project=myproject&ttl=600' result_backend = 'gs://mybucket/some-prefix?gcs_project=myproject?firestore_project=myproject2&ttl=600' This backend requires the following configuration directives to be set: ::: setting gcs_bucket ::: #### `gcs_bucket` Default: None. The gcs bucket name. For example: gcs_bucket = 'bucket_name' ::: setting gcs_project ::: #### `gcs_project` Default: None. The gcs project name. For example: gcs_project = 'test-project' ::: setting gcs_base_path ::: #### `gcs_base_path` Default: None. A base path in the gcs bucket to use to store all result keys. For example: gcs_base_path = '/prefix' #### `gcs_ttl` Default: 0. The time to live in seconds for the results blobs. Requires a GCS bucket with \"Delete\" Object Lifecycle Management action enabled. Use it to automatically delete results from Cloud Storage Buckets. For example to auto remove results after 24 hours: gcs_ttl = 86400 #### `gcs_threadpool_maxsize` Default: 10. Threadpool size for GCS operations. Same value defines the connection pool size. Allows to control the number of concurrent operations. For example: gcs_threadpool_maxsize = 20 #### `firestore_project` Default: gcs_project. The Firestore project for Chord reference counting. Allows native chord ref counts. If not specified defaults to `gcs_project`{.interpreted-text role="setting"}. For example: firestore_project = 'test-project2' #### Example configuration ``` python gcs_bucket = 'mybucket' gcs_project = 'myproject' gcs_base_path = '/celery_result_backend' gcs_ttl = 86400 ``` ### Elasticsearch backend settings {#conf-elasticsearch-result-backend} To use [Elasticsearch](https://aws.amazon.com/elasticsearch-service/) as the result backend you simply need to configure the `result_backend`{.interpreted-text role="setting"} setting with the correct URL. #### Example configuration ``` python result_backend = 'elasticsearch://example.com:9200/index_name/doc_type' ``` ::: setting elasticsearch_retry_on_timeout ::: #### `elasticsearch_retry_on_timeout` Default: `False`{.interpreted-text role="const"} Should timeout trigger a retry on different node? ::: setting elasticsearch_max_retries ::: #### `elasticsearch_max_retries` Default: 3. Maximum number of retries before an exception is propagated. ::: setting elasticsearch_timeout ::: #### `elasticsearch_timeout` Default: 10.0 seconds. Global timeout,used by the elasticsearch result backend. ::: setting elasticsearch_save_meta_as_text ::: #### `elasticsearch_save_meta_as_text` Default: `True`{.interpreted-text role="const"} Should meta saved as text or as native json. Result is always serialized as text. ### AWS DynamoDB backend settings {#conf-dynamodb-result-backend} ::: note ::: title Note ::: The Dynamodb backend requires the `boto3`{.interpreted-text role="pypi"} library. To install this package use `pip`{.interpreted-text role="command"}: ``` console $ pip install celery[dynamodb] ``` See `bundles`{.interpreted-text role="ref"} for information on combining multiple extension requirements. ::: ::: warning ::: title Warning ::: The Dynamodb backend is not compatible with tables that have a sort key defined. If you want to query the results table based on something other than the partition key, please define a global secondary index (GSI) instead. ::: This backend requires the `result_backend`{.interpreted-text role="setting"} setting to be set to a DynamoDB URL: result_backend = 'dynamodb://aws_access_key_id:aws_secret_access_key@region:port/table?read=n&write=m' For example, specifying the AWS region and the table name: result_backend = 'dynamodb://@us-east-1/celery_results' or retrieving AWS configuration parameters from the environment, using the default table name (`celery`) and specifying read and write provisioned throughput: result_backend = 'dynamodb://@/?read=5&write=5' or using the [downloadable version](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html) of DynamoDB [locally](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.Endpoint.html): result_backend = 'dynamodb://@localhost:8000' or using downloadable version or other service with conforming API deployed on any host: result_backend = 'dynamodb://@us-east-1' dynamodb_endpoint_url = 'http://192.168.0.40:8000' The fields of the DynamoDB URL in `result_backend` are defined as follows: 1. `aws_access_key_id & aws_secret_access_key` > The credentials for accessing AWS API resources. These can also be > resolved by the `boto3`{.interpreted-text role="pypi"} library > from various sources, as described > [here](http://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials). 2. `region` > The AWS region, e.g. `us-east-1` or `localhost` for the > [Downloadable > Version](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html). > See the `boto3`{.interpreted-text role="pypi"} library > [documentation](http://boto3.readthedocs.io/en/latest/guide/configuration.html#environment-variable-configuration) > for definition options. 3. `port` The listening port of the local DynamoDB instance, if you are using the downloadable version. If you have not specified the `region` parameter as `localhost`, setting this parameter has **no effect**. 4. `table` > Table name to use. Default is `celery`. See the [DynamoDB Naming > Rules](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-naming-rules) > for information on the allowed characters and length. 5. `read & write` > The Read & Write Capacity Units for the created DynamoDB table. > Default is `1` for both read and write. More details can be found > in the [Provisioned Throughput > documentation](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ProvisionedThroughput.html). 6. `ttl_seconds` > Time-to-live (in seconds) for results before they expire. The > default is to not expire results, while also leaving the DynamoDB > table\'s Time to Live settings untouched. If `ttl_seconds` is set > to a positive value, results will expire after the specified > number of seconds. Setting `ttl_seconds` to a negative value means > to not expire results, and also to actively disable the DynamoDB > table\'s Time to Live setting. Note that trying to change a > table\'s Time to Live setting multiple times in quick succession > will cause a throttling error. More details can be found in the > [DynamoDB TTL > documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html) ### IronCache backend settings {#conf-ironcache-result-backend} ::: note ::: title Note ::: The IronCache backend requires the `iron_celery`{.interpreted-text role="pypi"} library: To install this package use `pip`{.interpreted-text role="command"}: ``` console $ pip install iron_celery ``` ::: IronCache is configured via the URL provided in `result_backend`{.interpreted-text role="setting"}, for example: result_backend = 'ironcache://project_id:token@' Or to change the cache name: ironcache:://project_id:token@/awesomecache For more information, see: ### Couchbase backend settings {#conf-couchbase-result-backend} ::: note ::: title Note ::: The Couchbase backend requires the `couchbase`{.interpreted-text role="pypi"} library. To install this package use `pip`{.interpreted-text role="command"}: ``` console $ pip install celery[couchbase] ``` See `bundles`{.interpreted-text role="ref"} for instructions how to combine multiple extension requirements. ::: This backend can be configured via the `result_backend`{.interpreted-text role="setting"} set to a Couchbase URL: ``` python result_backend = 'couchbase://username:password@host:port/bucket' ``` ::: setting couchbase_backend_settings ::: #### `couchbase_backend_settings` Default: `{}` (empty mapping). This is a dict supporting the following keys: - `host` > Host name of the Couchbase server. Defaults to `localhost`. - `port` > The port the Couchbase server is listening to. Defaults to `8091`. - `bucket` > The default bucket the Couchbase server is writing to. Defaults to > `default`. - `username` > User name to authenticate to the Couchbase server as (optional). - `password` > Password to authenticate to the Couchbase server (optional). ### ArangoDB backend settings {#conf-arangodb-result-backend} ::: note ::: title Note ::: The ArangoDB backend requires the `pyArango`{.interpreted-text role="pypi"} library. To install this package use `pip`{.interpreted-text role="command"}: ``` console $ pip install celery[arangodb] ``` See `bundles`{.interpreted-text role="ref"} for instructions how to combine multiple extension requirements. ::: This backend can be configured via the `result_backend`{.interpreted-text role="setting"} set to a ArangoDB URL: ``` python result_backend = 'arangodb://username:password@host:port/database/collection' ``` ::: setting arangodb_backend_settings ::: #### `arangodb_backend_settings` Default: `{}` (empty mapping). This is a dict supporting the following keys: - `host` > Host name of the ArangoDB server. Defaults to `localhost`. - `port` > The port the ArangoDB server is listening to. Defaults to `8529`. - `database` > The default database in the ArangoDB server is writing to. > Defaults to `celery`. - `collection` > The default collection in the ArangoDB servers database is writing > to. Defaults to `celery`. - `username` > User name to authenticate to the ArangoDB server as (optional). - `password` > Password to authenticate to the ArangoDB server (optional). - `http_protocol` > HTTP Protocol in ArangoDB server connection. Defaults to `http`. - `verify` > HTTPS Verification check while creating the ArangoDB connection. > Defaults to `False`. ### CosmosDB backend settings (experimental) {#conf-cosmosdbsql-result-backend} To use [CosmosDB](https://azure.microsoft.com/en-us/services/cosmos-db/) as the result backend, you simply need to configure the `result_backend`{.interpreted-text role="setting"} setting with the correct URL. #### Example configuration ``` python result_backend = 'cosmosdbsql://:{InsertAccountPrimaryKeyHere}@{InsertAccountNameHere}.documents.azure.com' ``` ::: setting cosmosdbsql_database_name ::: #### `cosmosdbsql_database_name` Default: celerydb. The name for the database in which to store the results. ::: setting cosmosdbsql_collection_name ::: #### `cosmosdbsql_collection_name` Default: celerycol. The name of the collection in which to store the results. ::: setting cosmosdbsql_consistency_level ::: #### `cosmosdbsql_consistency_level` Default: Session. Represents the consistency levels supported for Azure Cosmos DB client operations. Consistency levels by order of strength are: Strong, BoundedStaleness, Session, ConsistentPrefix and Eventual. ::: setting cosmosdbsql_max_retry_attempts ::: #### `cosmosdbsql_max_retry_attempts` Default: 9. Maximum number of retries to be performed for a request. ::: setting cosmosdbsql_max_retry_wait_time ::: #### `cosmosdbsql_max_retry_wait_time` Default: 30. Maximum wait time in seconds to wait for a request while the retries are happening. ### CouchDB backend settings {#conf-couchdb-result-backend} ::: note ::: title Note ::: The CouchDB backend requires the `pycouchdb`{.interpreted-text role="pypi"} library: To install this Couchbase package use `pip`{.interpreted-text role="command"}: ``` console $ pip install celery[couchdb] ``` See `bundles`{.interpreted-text role="ref"} for information on combining multiple extension requirements. ::: This backend can be configured via the `result_backend`{.interpreted-text role="setting"} set to a CouchDB URL: result_backend = 'couchdb://username:password@host:port/container' The URL is formed out of the following parts: - `username` > User name to authenticate to the CouchDB server as (optional). - `password` > Password to authenticate to the CouchDB server (optional). - `host` > Host name of the CouchDB server. Defaults to `localhost`. - `port` > The port the CouchDB server is listening to. Defaults to `8091`. - `container` > The default container the CouchDB server is writing to. Defaults > to `default`. ### File-system backend settings {#conf-filesystem-result-backend} This backend can be configured using a file URL, for example: CELERY_RESULT_BACKEND = 'file:///var/celery/results' The configured directory needs to be shared and writable by all servers using the backend. If you\'re trying Celery on a single system you can simply use the backend without any further configuration. For larger clusters you could use NFS, [GlusterFS](http://www.gluster.org/), CIFS, [HDFS](http://hadoop.apache.org/) (using FUSE), or any other file-system. ### Consul K/V store backend settings {#conf-consul-result-backend} ::: note ::: title Note ::: The Consul backend requires the `python-consul2`{.interpreted-text role="pypi"} library: To install this package use `pip`{.interpreted-text role="command"}: ``` console $ pip install python-consul2 ``` ::: The Consul backend can be configured using a URL, for example: CELERY_RESULT_BACKEND = 'consul://localhost:8500/' or: result_backend = 'consul://localhost:8500/' The backend will store results in the K/V store of Consul as individual keys. The backend supports auto expire of results using TTLs in Consul. The full syntax of the URL is: ``` text consul://host:port[?one_client=1] ``` The URL is formed out of the following parts: - `host` > Host name of the Consul server. - `port` > The port the Consul server is listening to. - `one_client` > By default, for correctness, the backend uses a separate client > connection per operation. In cases of extreme load, the rate of > creation of new connections can cause HTTP 429 \"too many > connections\" error responses from the Consul server when under > load. The recommended way to handle this is to enable retries in > `python-consul2` using the patch at > . > > Alternatively, if `one_client` is set, a single client connection > will be used for all operations instead. This should eliminate the > HTTP 429 errors, but the storage of results in the backend can > become unreliable. ### Message Routing {#conf-messaging} ::: {#conf-messaging-routing} ::: setting task_queues ::: ::: #### `task_queues` Default: `None`{.interpreted-text role="const"} (queue taken from default queue settings). Most users will not want to specify this setting and should rather use the `automatic routing facilities `{.interpreted-text role="ref"}. If you really want to configure advanced routing, this setting should be a list of `kombu.Queue`{.interpreted-text role="class"} objects the worker will consume from. Note that workers can be overridden this setting via the `-Q `{.interpreted-text role="option"} option, or individual queues from this list (by name) can be excluded using the `-X `{.interpreted-text role="option"} option. Also see `routing-basics`{.interpreted-text role="ref"} for more information. The default is a queue/exchange/binding key of `celery`, with exchange type `direct`. See also `task_routes`{.interpreted-text role="setting"} ::: setting task_routes ::: #### `task_routes` Default: `None`{.interpreted-text role="const"}. A list of routers, or a single router used to route tasks to queues. When deciding the final destination of a task the routers are consulted in order. A router can be specified as either: - A function with the signature `(name, args, kwargs, options, task=None, **kwargs)` - A string providing the path to a router function. - A dict containing router specification: : Will be converted to a `celery.routes.MapRoute`{.interpreted-text role="class"} instance. - A list of `(pattern, route)` tuples: : Will be converted to a `celery.routes.MapRoute`{.interpreted-text role="class"} instance. Examples: ``` python task_routes = { 'celery.ping': 'default', 'mytasks.add': 'cpu-bound', 'feed.tasks.*': 'feeds', # <-- glob pattern re.compile(r'(image|video)\.tasks\..*'): 'media', # <-- regex 'video.encode': { 'queue': 'video', 'exchange': 'media', 'routing_key': 'media.video.encode', }, } task_routes = ('myapp.tasks.route_task', {'celery.ping': 'default'}) ``` Where `myapp.tasks.route_task` could be: ``` python def route_task(self, name, args, kwargs, options, task=None, **kw): if task == 'celery.ping': return {'queue': 'default'} ``` `route_task` may return a string or a dict. A string then means it\'s a queue name in `task_queues`{.interpreted-text role="setting"}, a dict means it\'s a custom route. When sending tasks, the routers are consulted in order. The first router that doesn\'t return `None` is the route to use. The message options is then merged with the found route settings, where the task\'s settings have priority. Example if `~celery.execute.apply_async`{.interpreted-text role="func"} has these arguments: ``` python Task.apply_async(immediate=False, exchange='video', routing_key='video.compress') ``` and a router returns: ``` python {'immediate': True, 'exchange': 'urgent'} ``` the final message options will be: ``` python immediate=False, exchange='video', routing_key='video.compress' ``` (and any default message options defined in the `~celery.app.task.Task`{.interpreted-text role="class"} class) Values defined in `task_routes`{.interpreted-text role="setting"} have precedence over values defined in `task_queues`{.interpreted-text role="setting"} when merging the two. With the follow settings: ``` python task_queues = { 'cpubound': { 'exchange': 'cpubound', 'routing_key': 'cpubound', }, } task_routes = { 'tasks.add': { 'queue': 'cpubound', 'routing_key': 'tasks.add', 'serializer': 'json', }, } ``` The final routing options for `tasks.add` will become: ``` javascript {'exchange': 'cpubound', 'routing_key': 'tasks.add', 'serializer': 'json'} ``` See `routers`{.interpreted-text role="ref"} for more examples. ::: setting task_queue_max_priority ::: #### `task_queue_max_priority` brokers : RabbitMQ Default: `None`{.interpreted-text role="const"}. See `routing-options-rabbitmq-priorities`{.interpreted-text role="ref"}. ::: setting task_default_priority ::: #### `task_default_priority` brokers : RabbitMQ, Redis Default: `None`{.interpreted-text role="const"}. See `routing-options-rabbitmq-priorities`{.interpreted-text role="ref"}. ::: setting task_inherit_parent_priority ::: #### `task_inherit_parent_priority` brokers : RabbitMQ Default: `False`{.interpreted-text role="const"}. If enabled, child tasks will inherit priority of the parent task. ``` python # The last task in chain will also have priority set to 5. chain = celery.chain(add.s(2) | add.s(2).set(priority=5) | add.s(3)) ``` Priority inheritance also works when calling child tasks from a parent task with [delay]{.title-ref} or [apply_async]{.title-ref}. See `routing-options-rabbitmq-priorities`{.interpreted-text role="ref"}. ::: setting worker_direct ::: #### `worker_direct` Default: Disabled. This option enables so that every worker has a dedicated queue, so that tasks can be routed to specific workers. The queue name for each worker is automatically generated based on the worker hostname and a `.dq` suffix, using the `C.dq2` exchange. For example the queue name for the worker with node name `w1@example.com` becomes: w1@example.com.dq Then you can route the task to the worker by specifying the hostname as the routing key and the `C.dq2` exchange: task_routes = { 'tasks.add': {'exchange': 'C.dq2', 'routing_key': 'w1@example.com'} } ::: setting task_create_missing_queues ::: #### `task_create_missing_queues` Default: Enabled. If enabled (default), any queues specified that aren\'t defined in `task_queues`{.interpreted-text role="setting"} will be automatically created. See `routing-automatic`{.interpreted-text role="ref"}. ::: setting task_create_missing_queue_type ::: `task_create_missing_queue_type` \~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~ .. versionadded:: 5.6 Default: `"classic"` When Celery needs to declare a queue that doesn't exist (i.e., when `task_create_missing_queues` is enabled), this setting defines what type of RabbitMQ queue to create. - `"classic"` (default): declares a standard classic queue. - `"quorum"`: declares a RabbitMQ quorum queue (adds `x-queue-type: quorum`). ::: setting task_create_missing_queue_exchange_type ::: `task_create_missing_queue_exchange_type` \~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~ .. versionadded:: 5.6 Default: `None` If this option is None or the empty string (the default), Celery leaves the exchange exactly as returned by your `app.amqp.Queues.autoexchange`{.interpreted-text role="attr"} hook. You can set this to a specific exchange type, such as `"direct"`, `"topic"`, or `"fanout"`, to create the missing queue with that exchange type. ::: tip ::: title Tip ::: ::: Combine this setting with task_create_missing_queue_type = \"quorum\" to create quorum queues bound to a topic exchange, for example: app.conf.task_create_missing_queues=True app.conf.task_create_missing_queue_type="quorum" app.conf.task_create_missing_queue_exchange_type="topic" ::: note ::: title Note ::: ::: Like the queue-type setting above, this option does not affect queues that you define explicitly in `task_queues`{.interpreted-text role="setting"}; it applies only to queues created implicitly at runtime. ::: setting task_default_queue ::: #### `task_default_queue` Default: `"celery"`. The name of the default queue used by [.apply_async]{.title-ref} if the message has no route or no custom queue has been specified. This queue must be listed in `task_queues`{.interpreted-text role="setting"}. If `task_queues`{.interpreted-text role="setting"} isn\'t specified then it\'s automatically created containing one queue entry, where this name is used as the name of that queue. ::: seealso `routing-changing-default-queue`{.interpreted-text role="ref"} ::: ::: setting task_default_queue_type ::: #### `task_default_queue_type` ::: versionadded 5.5 ::: Default: `"classic"`. This setting is used to allow changing the default queue type for the `task_default_queue`{.interpreted-text role="setting"} queue. The other viable option is `"quorum"` which is only supported by RabbitMQ and sets the queue type to `quorum` using the `x-queue-type` queue argument. If the `worker_detect_quorum_queues`{.interpreted-text role="setting"} setting is enabled, the worker will automatically detect the queue type and disable the global QoS accordingly. ::: warning ::: title Warning ::: Quorum queues require confirm publish to be enabled. Use `broker_transport_options`{.interpreted-text role="setting"} to enable confirm publish by setting: ``` python broker_transport_options = {"confirm_publish": True} ``` For more information, see [RabbitMQ documentation](https://www.rabbitmq.com/docs/quorum-queues#use-cases). ::: ::: setting task_default_exchange ::: #### `task_default_exchange` Default: Uses the value set for `task_default_queue`{.interpreted-text role="setting"}. Name of the default exchange to use when no custom exchange is specified for a key in the `task_queues`{.interpreted-text role="setting"} setting. ::: setting task_default_exchange_type ::: #### `task_default_exchange_type` Default: `"direct"`. Default exchange type used when no custom exchange type is specified for a key in the `task_queues`{.interpreted-text role="setting"} setting. ::: setting task_default_routing_key ::: #### `task_default_routing_key` Default: Uses the value set for `task_default_queue`{.interpreted-text role="setting"}. The default routing key used when no custom routing key is specified for a key in the `task_queues`{.interpreted-text role="setting"} setting. ::: setting task_default_delivery_mode ::: #### `task_default_delivery_mode` Default: `"persistent"`. Can be [transient]{.title-ref} (messages not written to disk) or [persistent]{.title-ref} (written to disk). ### Broker Settings {#conf-broker-settings} ::: setting broker_url ::: #### `broker_url` Default: `"amqp://"` Default broker URL. This must be a URL in the form of: transport://userid:password@hostname:port/virtual_host Only the scheme part (`transport://`) is required, the rest is optional, and defaults to the specific transports default values. The transport part is the broker implementation to use, and the default is `amqp`, (uses `librabbitmq` if installed or falls back to `pyamqp`). There are also other choices available, including; `redis://`, `sqs://`, and `qpid://`. The scheme can also be a fully qualified path to your own transport implementation: broker_url = 'proj.transports.MyTransport://localhost' More than one broker URL, of the same transport, can also be specified. The broker URLs can be passed in as a single string that\'s semicolon delimited: broker_url = 'transport://userid:password@hostname:port//;transport://userid:password@hostname:port//' Or as a list: broker_url = [ 'transport://userid:password@localhost:port//', 'transport://userid:password@hostname:port//' ] The brokers will then be used in the `broker_failover_strategy`{.interpreted-text role="setting"}. See `kombu:connection-urls`{.interpreted-text role="ref"} in the Kombu documentation for more information. ::: setting broker_read_url ::: ::: setting broker_write_url ::: #### `broker_read_url` / `broker_write_url` Default: Taken from `broker_url`{.interpreted-text role="setting"}. These settings can be configured, instead of `broker_url`{.interpreted-text role="setting"} to specify different connection parameters for broker connections used for consuming and producing. Example: broker_read_url = 'amqp://user:pass@broker.example.com:56721' broker_write_url = 'amqp://user:pass@broker.example.com:56722' Both options can also be specified as a list for failover alternates, see `broker_url`{.interpreted-text role="setting"} for more information. ::: setting broker_failover_strategy ::: #### `broker_failover_strategy` Default: `"round-robin"`. Default failover strategy for the broker Connection object. If supplied, may map to a key in \'kombu.connection.failover_strategies\', or be a reference to any method that yields a single item from a supplied list. Example: # Random failover strategy def random_failover_strategy(servers): it = list(servers) # don't modify callers list shuffle = random.shuffle for _ in repeat(None): shuffle(it) yield it[0] broker_failover_strategy = random_failover_strategy ::: setting broker_heartbeat ::: #### `broker_heartbeat` transports supported : `pyamqp` Default: `120.0` (negotiated by server). Note: This value is only used by the worker, clients do not use a heartbeat at the moment. It\'s not always possible to detect connection loss in a timely manner using TCP/IP alone, so AMQP defines something called heartbeats that\'s is used both by the client and the broker to detect if a connection was closed. If the heartbeat value is 10 seconds, then the heartbeat will be monitored at the interval specified by the `broker_heartbeat_checkrate`{.interpreted-text role="setting"} setting (by default this is set to double the rate of the heartbeat value, so for the 10 seconds, the heartbeat is checked every 5 seconds). ::: setting broker_heartbeat_checkrate ::: #### `broker_heartbeat_checkrate` transports supported : `pyamqp` Default: 2.0. At intervals the worker will monitor that the broker hasn\'t missed too many heartbeats. The rate at which this is checked is calculated by dividing the `broker_heartbeat`{.interpreted-text role="setting"} value with this value, so if the heartbeat is 10.0 and the rate is the default 2.0, the check will be performed every 5 seconds (twice the heartbeat sending rate). ::: setting broker_use_ssl ::: #### `broker_use_ssl` transports supported : `pyamqp`, `redis` Default: Disabled. Toggles SSL usage on broker connection and SSL settings. The valid values for this option vary by transport. ##### `pyamqp` If `True` the connection will use SSL with default SSL settings. If set to a dict, will configure SSL connection according to the specified policy. The format used is Python\'s `ssl.wrap_socket`{.interpreted-text role="func"} options. Note that SSL socket is generally served on a separate port by the broker. Example providing a client cert and validating the server cert against a custom certificate authority: ``` python import ssl broker_use_ssl = { 'keyfile': '/var/ssl/private/worker-key.pem', 'certfile': '/var/ssl/amqp-server-cert.pem', 'ca_certs': '/var/ssl/myca.pem', 'cert_reqs': ssl.CERT_REQUIRED } ``` ::: versionadded 5.1 Starting from Celery 5.1, py-amqp will always validate certificates received from the server and it is no longer required to manually set `cert_reqs` to `ssl.CERT_REQUIRED`. The previous default, `ssl.CERT_NONE` is insecure and we its usage should be discouraged. If you\'d like to revert to the previous insecure default set `cert_reqs` to `ssl.CERT_NONE` ::: ##### `redis` The setting must be a dict with the following keys: - `ssl_cert_reqs` (required): one of the `SSLContext.verify_mode` values: : - `ssl.CERT_NONE` - `ssl.CERT_OPTIONAL` - `ssl.CERT_REQUIRED` - `ssl_ca_certs` (optional): path to the CA certificate - `ssl_certfile` (optional): path to the client certificate - `ssl_keyfile` (optional): path to the client key ::: setting broker_pool_limit ::: #### `broker_pool_limit` ::: versionadded 2.3 ::: Default: 10. The maximum number of connections that can be open in the connection pool. The pool is enabled by default since version 2.5, with a default limit of ten connections. This number can be tweaked depending on the number of threads/green-threads (eventlet/gevent) using a connection. For example running eventlet with 1000 greenlets that use a connection to the broker, contention can arise and you should consider increasing the limit. If set to `None`{.interpreted-text role="const"} or 0 the connection pool will be disabled and connections will be established and closed for every use. ::: setting broker_connection_timeout ::: #### `broker_connection_timeout` Default: 4.0. The default timeout in seconds before we give up establishing a connection to the AMQP server. This setting is disabled when using gevent. ::: note ::: title Note ::: The broker connection timeout only applies to a worker attempting to connect to the broker. It does not apply to producer sending a task, see `broker_transport_options`{.interpreted-text role="setting"} for how to provide a timeout for that situation. ::: ::: setting broker_connection_retry ::: #### `broker_connection_retry` Default: Enabled. Automatically try to re-establish the connection to the AMQP broker if lost after the initial connection is made. The time between retries is increased for each retry, and is not exhausted before `broker_connection_max_retries`{.interpreted-text role="setting"} is exceeded. ::: warning ::: title Warning ::: The broker_connection_retry configuration setting will no longer determine whether broker connection retries are made during startup in Celery 6.0 and above. If you wish to refrain from retrying connections on startup, you should set broker_connection_retry_on_startup to False instead. ::: ::: setting broker_connection_retry_on_startup ::: #### `broker_connection_retry_on_startup` Default: Enabled. Automatically try to establish the connection to the AMQP broker on Celery startup if it is unavailable. The time between retries is increased for each retry, and is not exhausted before `broker_connection_max_retries`{.interpreted-text role="setting"} is exceeded. ::: setting broker_connection_max_retries ::: #### `broker_connection_max_retries` Default: 100. Maximum number of retries before we give up re-establishing a connection to the AMQP broker. If this is set to `None`{.interpreted-text role="const"}, we\'ll retry forever. #### `broker_channel_error_retry` ::: versionadded 5.3 ::: Default: Disabled. Automatically try to re-establish the connection to the AMQP broker if any invalid response has been returned. The retry count and interval is the same as that of [broker_connection_retry]{.title-ref}. Also, this option doesn\'t work when [broker_connection_retry]{.title-ref} is [False]{.title-ref}. ::: setting broker_login_method ::: #### `broker_login_method` Default: `"AMQPLAIN"`. Set custom amqp login method. ::: setting broker_native_delayed_delivery_queue_type ::: #### `broker_native_delayed_delivery_queue_type` ::: versionadded 5.5 ::: transports supported : `pyamqp` Default: `"quorum"`. This setting is used to allow changing the default queue type for the native delayed delivery queues. The other viable option is `"classic"` which is only supported by RabbitMQ and sets the queue type to `classic` using the `x-queue-type` queue argument. ::: setting broker_transport_options ::: #### `broker_transport_options` ::: versionadded 2.2 ::: Default: `{}` (empty mapping). A dict of additional options passed to the underlying transport. See your transport user manual for supported options (if any). Example setting the visibility timeout (supported by Redis and SQS transports): ``` python broker_transport_options = {'visibility_timeout': 18000} # 5 hours ``` Example setting the producer connection maximum number of retries (so producers won\'t retry forever if the broker isn\'t available at the first task execution): ``` python broker_transport_options = {'max_retries': 5} ``` Example enabling publisher confirms (supported by the `pyamqp` transport). Without this, messages can be silently dropped when the broker hits resource limits: ``` python broker_transport_options = {'confirm_publish': True} ``` ### Worker {#conf-worker} ::: setting imports ::: #### `imports` Default: `[]` (empty list). A sequence of modules to import when the worker starts. This is used to specify the task modules to import, but also to import signal handlers and additional remote control commands, etc. The modules will be imported in the original order. ::: setting include ::: #### `include` Default: `[]` (empty list). Exact same semantics as `imports`{.interpreted-text role="setting"}, but can be used as a means to have different import categories. The modules in this setting are imported after the modules in `imports`{.interpreted-text role="setting"}. ::: setting worker_deduplicate_successful_tasks ::: #### `worker_deduplicate_successful_tasks` ::: versionadded 5.1 ::: Default: False Before each task execution, instruct the worker to check if this task is a duplicate message. Deduplication occurs only with tasks that have the same identifier, enabled late acknowledgment, were redelivered by the message broker and their state is `SUCCESS` in the result backend. To avoid overflowing the result backend with queries, a local cache of successfully executed tasks is checked before querying the result backend in case the task was already successfully executed by the same worker that received the task. This cache can be made persistent by setting the `worker_state_db`{.interpreted-text role="setting"} setting. If the result backend is not [persistent](https://github.com/celery/celery/blob/main/celery/backends/base.py#L102) (the RPC backend, for example), this setting is ignored. ::: {#conf-concurrency} ::: setting worker_concurrency ::: ::: #### `worker_concurrency` Default: Number of CPU cores. The number of concurrent worker processes/threads/green threads executing tasks. If you\'re doing mostly I/O you can have more processes, but if mostly CPU-bound, try to keep it close to the number of CPUs on your machine. If not set, the number of CPUs/cores on the host will be used. ::: setting worker_prefetch_multiplier ::: #### `worker_prefetch_multiplier` Default: 4. How many messages to prefetch at a time multiplied by the number of concurrent processes. The default is 4 (four messages for each process). The default setting is usually a good choice, however \-- if you have very long running tasks waiting in the queue and you have to start the workers, note that the first worker to start will receive four times the number of messages initially. Thus the tasks may not be fairly distributed to the workers. To limit the broker to only deliver one message per process at a time, set `worker_prefetch_multiplier`{.interpreted-text role="setting"} to 1. Changing that setting to 0 will allow the worker to keep consuming as many messages as it wants. If you need to completely disable broker prefetching while still using early acknowledgments, enable `worker_disable_prefetch`{.interpreted-text role="setting"}. When this option is enabled the worker only fetches a task from the broker when one of its processes is available. ::: note ::: title Note ::: This feature is currently only supported when using Redis as the broker. ::: You can also enable this via the `--disable-prefetch `{.interpreted-text role="option"} command line flag. For more on prefetching, read `optimizing-prefetch-limit`{.interpreted-text role="ref"} ::: setting worker_eta_task_limit ::: #### `worker_eta_task_limit` ::: versionadded 5.6 ::: Default: No limit (None). The maximum number of ETA/countdown tasks that a worker can hold in memory at once. When this limit is reached, the worker will not receive new tasks from the broker until some of the existing ETA tasks are executed. This setting helps prevent memory exhaustion when a queue contains a large number of tasks with ETA/countdown values, as these tasks are held in memory until their execution time. Without this limit, workers may fetch thousands of ETA tasks into memory, potentially causing out-of-memory issues. ::: note ::: title Note ::: Tasks with ETA/countdown aren\'t affected by prefetch limits. ::: ::: setting worker_disable_prefetch ::: #### `worker_disable_prefetch` ::: versionadded 5.6 ::: Default: `False`. When enabled, a worker will only consume messages from the broker when it has an available process to execute them. This disables prefetching while still using early acknowledgments, ensuring that tasks are fairly distributed between workers. ::: note ::: title Note ::: This feature is currently only supported when using Redis as the broker. Using this setting with other brokers will result in a warning and the setting will be ignored. ::: ::: setting worker_enable_prefetch_count_reduction ::: #### `worker_enable_prefetch_count_reduction` ::: versionadded 5.4 ::: Default: Enabled. The `worker_enable_prefetch_count_reduction` setting governs the restoration behavior of the prefetch count to its maximum allowable value following a connection loss to the message broker. By default, this setting is enabled. Upon a connection loss, Celery will attempt to reconnect to the broker automatically, provided the `broker_connection_retry_on_startup`{.interpreted-text role="setting"} or `broker_connection_retry`{.interpreted-text role="setting"} is not set to False. During the period of lost connection, the message broker does not keep track of the number of tasks already fetched. Therefore, to manage the task load effectively and prevent overloading, Celery reduces the prefetch count based on the number of tasks that are currently running. The prefetch count is the number of messages that a worker will fetch from the broker at a time. The reduced prefetch count helps ensure that tasks are not fetched excessively during periods of reconnection. With `worker_enable_prefetch_count_reduction` set to its default value (Enabled), the prefetch count will be gradually restored to its maximum allowed value each time a task that was running before the connection was lost is completed. This behavior helps maintain a balanced distribution of tasks among the workers while managing the load effectively. To disable the reduction and restoration of the prefetch count to its maximum allowed value on reconnection, set `worker_enable_prefetch_count_reduction` to False. Disabling this setting might be useful in scenarios where a fixed prefetch count is desired to control the rate of task processing or manage the worker load, especially in environments with fluctuating connectivity. The `worker_enable_prefetch_count_reduction` setting provides a way to control the restoration behavior of the prefetch count following a connection loss, aiding in maintaining a balanced task distribution and effective load management across the workers. ::: setting worker_lost_wait ::: #### `worker_lost_wait` Default: 10.0 seconds. In some cases a worker may be killed without proper cleanup, and the worker may have published a result before terminating. This value specifies how long we wait for any missing results before raising a `@WorkerLostError`{.interpreted-text role="exc"} exception. ::: setting worker_max_tasks_per_child ::: #### `worker_max_tasks_per_child` Maximum number of tasks a pool worker process can execute before it\'s replaced with a new one. Default is no limit. ::: setting worker_max_memory_per_child ::: #### `worker_max_memory_per_child` Default: No limit. Type: int (kilobytes) Maximum amount of resident memory, in kilobytes (1024 bytes), that may be consumed by a worker before it will be replaced by a new worker. If a single task causes a worker to exceed this limit, the task will be completed, and the worker will be replaced afterwards. Example: ``` python worker_max_memory_per_child = 12288 # 12 * 1024 = 12 MB ``` ::: setting worker_disable_rate_limits ::: #### `worker_disable_rate_limits` Default: Disabled (rate limits enabled). Disable all rate limits, even if tasks has explicit rate limits set. ::: setting worker_state_db ::: #### `worker_state_db` Default: `None`{.interpreted-text role="const"}. Name of the file used to stores persistent worker state (like revoked tasks). Can be a relative or absolute path, but be aware that the suffix [.db]{.title-ref} may be appended to the file name (depending on Python version). Can also be set via the `celery worker --statedb`{.interpreted-text role="option"} argument. ::: setting worker_timer_precision ::: #### `worker_timer_precision` Default: 1.0 seconds. Set the maximum time in seconds that the ETA scheduler can sleep between rechecking the schedule. Setting this value to 1 second means the schedulers precision will be 1 second. If you need near millisecond precision you can set this to 0.1. ::: setting worker_enable_remote_control ::: #### `worker_enable_remote_control` Default: Enabled by default. Specify if remote control of the workers is enabled. ::: setting worker_proc_alive_timeout ::: #### `worker_proc_alive_timeout` Default: 4.0. The timeout in seconds (int/float) when waiting for a new worker process to start up. ::: setting worker_cancel_long_running_tasks_on_connection_loss ::: #### `worker_cancel_long_running_tasks_on_connection_loss` ::: versionadded 5.1 ::: Default: Disabled by default. Kill all long-running tasks with late acknowledgment enabled on connection loss. Tasks which have not been acknowledged before the connection loss cannot do so anymore since their channel is gone and the task is redelivered back to the queue. This is why tasks with late acknowledged enabled must be idempotent as they may be executed more than once. In this case, the task is being executed twice per connection loss (and sometimes in parallel in other workers). When turning this option on, those tasks which have not been completed are cancelled and their execution is terminated. Tasks which have completed in any way before the connection loss are recorded as such in the result backend as long as `task_ignore_result`{.interpreted-text role="setting"} is not enabled. ::: warning ::: title Warning ::: This feature was introduced as a future breaking change. If it is turned off, Celery will emit a warning message. In Celery 6.0, the `worker_cancel_long_running_tasks_on_connection_loss`{.interpreted-text role="setting"} will be set to `True` by default as the current behavior leads to more problems than it solves. ::: ::: setting worker_detect_quorum_queues ::: #### `worker_detect_quorum_queues` ::: versionadded 5.5 ::: Default: Enabled. Automatically detect if any of the queues in `task_queues`{.interpreted-text role="setting"} are quorum queues (including the `task_default_queue`{.interpreted-text role="setting"}) and disable the global QoS if any quorum queue is detected. ::: setting worker_soft_shutdown_timeout ::: #### `worker_soft_shutdown_timeout` ::: versionadded 5.5 ::: Default: 0.0. The standard `warm shutdown `{.interpreted-text role="ref"} will wait for all tasks to finish before shutting down unless the cold shutdown is triggered. The `soft shutdown `{.interpreted-text role="ref"} will add a waiting time before the cold shutdown is initiated. This setting specifies how long the worker will wait before the cold shutdown is initiated and the worker is terminated. This will apply also when the worker initiate `cold shutdown `{.interpreted-text role="ref"} without doing a warm shutdown first. If the value is set to 0.0, the soft shutdown will be practically disabled. Regardless of the value, the soft shutdown will be disabled if there are no tasks running (unless `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} is enabled). Experiment with this value to find the optimal time for your tasks to finish gracefully before the worker is terminated. Recommended values can be 10, 30, 60 seconds. Too high value can lead to a long waiting time before the worker is terminated and trigger a `KILL`{.interpreted-text role="sig"} signal to forcefully terminate the worker by the host system. ::: setting worker_enable_soft_shutdown_on_idle ::: #### `worker_enable_soft_shutdown_on_idle` ::: versionadded 5.5 ::: Default: False. If the `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} is set to a value greater than 0.0, the worker will skip the `soft shutdown `{.interpreted-text role="ref"} anyways if there are no tasks running. This setting will enable the soft shutdown even if there are no tasks running. ::: tip ::: title Tip ::: When the worker received ETA tasks, but the ETA has not been reached yet, and a shutdown is initiated, the worker will **skip** the soft shutdown and initiate the cold shutdown immediately if there are no tasks running. This may lead to failure in re-queueing the ETA tasks during worker teardown. To mitigate this, enable this configuration to ensure the worker waits regadless, which gives enough time for a graceful shutdown and successful re-queueing of the ETA tasks. ::: ### Events {#conf-events} ::: setting worker_send_task_events ::: #### `worker_send_task_events` Default: Disabled by default. Send task-related events so that tasks can be monitored using tools like [flower]{.title-ref}. Sets the default value for the workers `-E `{.interpreted-text role="option"} argument. ::: setting task_send_sent_event ::: #### `task_send_sent_event` ::: versionadded 2.2 ::: Default: Disabled by default. If enabled, a `task-sent`{.interpreted-text role="event"} event will be sent for every task so tasks can be tracked before they\'re consumed by a worker. ::: setting event_queue_ttl ::: #### `event_queue_ttl` transports supported : `amqp` Default: 5.0 seconds. Message expiry time in seconds (int/float) for when messages sent to a monitor clients event queue is deleted (`x-message-ttl`) For example, if this value is set to 10 then a message delivered to this queue will be deleted after 10 seconds. ::: setting event_queue_expires ::: #### `event_queue_expires` transports supported : `amqp` Default: 60.0 seconds. Expiry time in seconds (int/float) for when after a monitor clients event queue will be deleted (`x-expires`). ::: setting event_queue_durable ::: #### `event_queue_durable` transports supported : `amqp` ::: versionadded 5.6 ::: Default: `False` If enabled, the event receiver\'s queue will be marked as *durable*, meaning it will survive broker restarts. ::: setting event_queue_exclusive ::: #### `event_queue_exclusive` transports supported : `amqp` ::: versionadded 5.6 ::: Default: `False` If enabled, the event queue will be *exclusive* to the current connection and automatically deleted when the connection closes. ::: warning ::: title Warning ::: You **cannot** set both `event_queue_durable` and `event_queue_exclusive` to `True` at the same time. Celery will raise an `ImproperlyConfigured`{.interpreted-text role="exc"} error if both are set. ::: ::: setting event_queue_prefix ::: #### `event_queue_prefix` Default: `"celeryev"`. The prefix to use for event receiver queue names. ::: setting event_exchange ::: #### `event_exchange` Default: `"celeryev"`. Name of the event exchange. ::: warning ::: title Warning ::: This option is in experimental stage, please use it with caution. ::: ::: setting event_serializer ::: #### `event_serializer` Default: `"json"`. Message serialization format used when sending event messages. ::: seealso `calling-serializers`{.interpreted-text role="ref"}. ::: ::: setting events_logfile ::: #### `events_logfile` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional file path for `celery events`{.interpreted-text role="program"} to log into (defaults to [stdout]{.title-ref}). ::: setting events_pidfile ::: #### `events_pidfile` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional file path for `celery events`{.interpreted-text role="program"} to create/store its PID file (default to no PID file created). ::: setting events_uid ::: #### `events_uid` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional user ID to use when events `celery events`{.interpreted-text role="program"} drops its privileges (defaults to no UID change). ::: setting events_gid ::: #### `events_gid` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional group ID to use when `celery events`{.interpreted-text role="program"} daemon drops its privileges (defaults to no GID change). ::: setting events_umask ::: #### `events_umask` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional [umask]{.title-ref} to use when `celery events`{.interpreted-text role="program"} creates files (log, pid\...) when daemonizing. ::: setting events_executable ::: #### `events_executable` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional [python]{.title-ref} executable path for `celery events`{.interpreted-text role="program"} to use when deaemonizing (defaults to `sys.executable`{.interpreted-text role="data"}). ### Remote Control Commands {#conf-control} ::: note ::: title Note ::: To disable remote control commands see the `worker_enable_remote_control`{.interpreted-text role="setting"} setting. ::: ::: setting control_queue_ttl ::: #### `control_queue_ttl` Default: 300.0 Time in seconds, before a message in a remote control command queue will expire. If using the default of 300 seconds, this means that if a remote control command is sent and no worker picks it up within 300 seconds, the command is discarded. This setting also applies to remote control reply queues. ::: setting control_queue_expires ::: #### `control_queue_expires` Default: 10.0 Time in seconds, before an unused remote control command queue is deleted from the broker. This setting also applies to remote control reply queues. ::: setting control_exchange ::: #### `control_exchange` Default: `"celery"`. Name of the control command exchange. ::: warning ::: title Warning ::: This option is in experimental stage, please use it with caution. ::: ::: {#conf-logging} ::: setting control_queue_durable ::: ::: ### `control_queue_durable` - **Default:** `False` - **Type:** `bool` If set to `True`, the control exchange and queue will be durable --- they will survive broker restarts. ::: setting control_queue_exclusive ::: ### `control_queue_exclusive` - **Default:** `False` - **Type:** `bool` If set to `True`, the control queue will be exclusive to a single connection. This is generally not recommended in distributed environments. ::: warning ::: title Warning ::: Setting both `control_queue_durable` and `control_queue_exclusive` to `True` is not supported and will raise an error. ::: ### Logging ::: setting worker_hijack_root_logger ::: #### `worker_hijack_root_logger` ::: versionadded 2.2 ::: Default: Enabled by default (hijack root logger). By default any previously configured handlers on the root logger will be removed. If you want to customize your own logging handlers, then you can disable this behavior by setting [worker_hijack_root_logger = False]{.title-ref}. ::: note ::: title Note ::: Logging can also be customized by connecting to the `celery.signals.setup_logging`{.interpreted-text role="signal"} signal. ::: ::: setting worker_log_color ::: #### `worker_log_color` Default: Enabled if app is logging to a terminal. Enables/disables colors in logging output by the Celery apps. ::: setting worker_log_format ::: #### `worker_log_format` Default: ``` text "[%(asctime)s: %(levelname)s/%(processName)s] %(message)s" ``` The format to use for log messages. See the Python `logging`{.interpreted-text role="mod"} module for more information about log formats. ::: setting worker_task_log_format ::: #### `worker_task_log_format` Default: ``` text "[%(asctime)s: %(levelname)s/%(processName)s] %(task_name)s[%(task_id)s]: %(message)s" ``` The format to use for log messages logged in tasks. See the Python `logging`{.interpreted-text role="mod"} module for more information about log formats. ::: setting worker_redirect_stdouts ::: #### `worker_redirect_stdouts` Default: Enabled by default. If enabled [stdout]{.title-ref} and [stderr]{.title-ref} will be redirected to the current logger. Used by `celery worker`{.interpreted-text role="program"} and `celery beat`{.interpreted-text role="program"}. ::: setting worker_redirect_stdouts_level ::: #### `worker_redirect_stdouts_level` Default: `WARNING`{.interpreted-text role="const"}. The log level output to [stdout]{.title-ref} and [stderr]{.title-ref} is logged as. Can be one of `DEBUG`{.interpreted-text role="const"}, `INFO`{.interpreted-text role="const"}, `WARNING`{.interpreted-text role="const"}, `ERROR`{.interpreted-text role="const"}, or `CRITICAL`{.interpreted-text role="const"}. ### Security {#conf-security} ::: setting security_key ::: #### `security_key` Default: `None`{.interpreted-text role="const"}. ::: versionadded 2.5 ::: The relative or absolute path to a file containing the private key used to sign messages when `message-signing`{.interpreted-text role="ref"} is used. ::: setting security_key_password ::: #### `security_key_password` Default: `None`{.interpreted-text role="const"}. ::: versionadded 5.3.0 ::: The password used to decrypt the private key when `message-signing`{.interpreted-text role="ref"} is used. ::: setting security_certificate ::: #### `security_certificate` Default: `None`{.interpreted-text role="const"}. ::: versionadded 2.5 ::: The relative or absolute path to an X.509 certificate file used to sign messages when `message-signing`{.interpreted-text role="ref"} is used. ::: setting security_cert_store ::: #### `security_cert_store` Default: `None`{.interpreted-text role="const"}. ::: versionadded 2.5 ::: The directory containing X.509 certificates used for `message-signing`{.interpreted-text role="ref"}. Can be a glob with wild-cards, (for example `/etc/certs/*.pem`{.interpreted-text role="file"}). ::: setting security_digest ::: #### `security_digest` Default: `sha256`{.interpreted-text role="const"}. ::: versionadded 4.3 ::: A cryptography digest used to sign messages when `message-signing`{.interpreted-text role="ref"} is used. ### Custom Component Classes (advanced) {#conf-custom-components} ::: setting worker_pool ::: #### `worker_pool` Default: `"prefork"` (`celery.concurrency.prefork:TaskPool`). Name of the pool class used by the worker. ::: admonition Eventlet/Gevent Never use this option to select the eventlet or gevent pool. You must use the `-P `{.interpreted-text role="option"} option to `celery worker`{.interpreted-text role="program"} instead, to ensure the monkey patches aren\'t applied too late, causing things to break in strange ways. ::: ::: setting worker_pool_restarts ::: #### `worker_pool_restarts` Default: Disabled by default. If enabled the worker pool can be restarted using the `pool_restart`{.interpreted-text role="control"} remote control command. ::: setting worker_autoscaler ::: #### `worker_autoscaler` ::: versionadded 2.2 ::: Default: `"celery.worker.autoscale:Autoscaler"`. Name of the autoscaler class to use. ::: setting worker_consumer ::: #### `worker_consumer` Default: `"celery.worker.consumer:Consumer"`. Name of the consumer class used by the worker. ::: setting worker_timer ::: #### `worker_timer` Default: `"kombu.asynchronous.hub.timer:Timer"`. Name of the ETA scheduler class used by the worker. Default is or set by the pool implementation. ::: setting worker_logfile ::: #### `worker_logfile` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional file path for `celery worker`{.interpreted-text role="program"} to log into (defaults to [stdout]{.title-ref}). ::: setting worker_pidfile ::: #### `worker_pidfile` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional file path for `celery worker`{.interpreted-text role="program"} to create/store its PID file (defaults to no PID file created). ::: setting worker_uid ::: #### `worker_uid` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional user ID to use when `celery worker`{.interpreted-text role="program"} daemon drops its privileges (defaults to no UID change). ::: setting worker_gid ::: #### `worker_gid` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional group ID to use when `celery worker`{.interpreted-text role="program"} daemon drops its privileges (defaults to no GID change). ::: setting worker_umask ::: #### `worker_umask` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional [umask]{.title-ref} to use when `celery worker`{.interpreted-text role="program"} creates files (log, pid\...) when daemonizing. ::: setting worker_executable ::: #### `worker_executable` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional [python]{.title-ref} executable path for `celery worker`{.interpreted-text role="program"} to use when deaemonizing (defaults to `sys.executable`{.interpreted-text role="data"}). ### Beat Settings (`celery beat`{.interpreted-text role="program"}) {#conf-celerybeat} ::: setting beat_schedule ::: #### `beat_schedule` Default: `{}` (empty mapping). The periodic task schedule used by `~celery.bin.beat`{.interpreted-text role="mod"}. See `beat-entries`{.interpreted-text role="ref"}. ::: setting beat_scheduler ::: #### `beat_scheduler` Default: `"celery.beat:PersistentScheduler"`. The default scheduler class. May be set to `"django_celery_beat.schedulers:DatabaseScheduler"` for instance, if used alongside `django-celery-beat`{.interpreted-text role="pypi"} extension. Can also be set via the `celery beat -S`{.interpreted-text role="option"} argument. ::: setting beat_schedule_filename ::: #### `beat_schedule_filename` Default: `"celerybeat-schedule"`. Name of the file used by [PersistentScheduler]{.title-ref} to store the last run times of periodic tasks. Can be a relative or absolute path, but be aware that the suffix [.db]{.title-ref} may be appended to the file name (depending on Python version). Can also be set via the `celery beat --schedule`{.interpreted-text role="option"} argument. ::: setting beat_sync_every ::: #### `beat_sync_every` Default: 0. The number of periodic tasks that can be called before another database sync is issued. A value of 0 (default) means sync based on timing - default of 3 minutes as determined by scheduler.sync_every. If set to 1, beat will call sync after every task message sent. ::: setting beat_max_loop_interval ::: #### `beat_max_loop_interval` Default: 0. The maximum number of seconds `~celery.bin.beat`{.interpreted-text role="mod"} can sleep between checking the schedule. The default for this value is scheduler specific. For the default Celery beat scheduler the value is 300 (5 minutes), but for the `django-celery-beat`{.interpreted-text role="pypi"} database scheduler it\'s 5 seconds because the schedule may be changed externally, and so it must take changes to the schedule into account. Also when running Celery beat embedded (`-B `{.interpreted-text role="option"}) on Jython as a thread the max interval is overridden and set to 1 so that it\'s possible to shut down in a timely manner. ::: setting beat_cron_starting_deadline ::: #### `beat_cron_starting_deadline` ::: versionadded 5.3 ::: Default: None. When using cron, the number of seconds `~celery.bin.beat`{.interpreted-text role="mod"} can look back when deciding whether a cron schedule is due. When set to [None]{.title-ref}, cronjobs that are past due will always run immediately. ::: warning ::: title Warning ::: Setting this higher than 3600 (1 hour) is highly discouraged. ::: ::: setting beat_logfile ::: #### `beat_logfile` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional file path for `celery beat`{.interpreted-text role="program"} to log into (defaults to [stdout]{.title-ref}). ::: setting beat_pidfile ::: #### `beat_pidfile` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional file path for `celery beat`{.interpreted-text role="program"} to create/store it PID file (defaults to no PID file created). ::: setting beat_uid ::: #### `beat_uid` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional user ID to use when beat `celery beat`{.interpreted-text role="program"} drops its privileges (defaults to no UID change). ::: setting beat_gid ::: #### `beat_gid` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional group ID to use when `celery beat`{.interpreted-text role="program"} daemon drops its privileges (defaults to no GID change). ::: setting beat_umask ::: #### `beat_umask` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional [umask]{.title-ref} to use when `celery beat`{.interpreted-text role="program"} creates files (log, pid\...) when daemonizing. ::: setting beat_executable ::: #### `beat_executable` ::: versionadded 5.4 ::: Default: `None`{.interpreted-text role="const"} An optional [python]{.title-ref} executable path for `celery beat`{.interpreted-text role="program"} to use when deaemonizing (defaults to `sys.executable`{.interpreted-text role="data"}). --- # Daemonization {#daemonizing} ::: {.contents local=""} ::: Most Linux distributions these days use systemd for managing the lifecycle of system and user services. You can check if your Linux distribution uses systemd by typing: ``` console $ systemctl --version systemd 249 (v249.9-1.fc35) +PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified ``` If you have output similar to the above, please refer to `our systemd documentation `{.interpreted-text role="ref"} for guidance. However, the init.d script should still work in those Linux distributions as well since systemd provides the systemd-sysv compatibility layer which generates services automatically from the init.d scripts we provide. If you package Celery for multiple Linux distributions and some do not support systemd or to other Unix systems as well, you may want to refer to `our init.d documentation `{.interpreted-text role="ref"}. ## Generic init-scripts {#daemon-generic} See the [extra/generic-init.d/](https://github.com/celery/celery/tree/main/extra/generic-init.d/) directory Celery distribution. This directory contains generic bash init-scripts for the `celery worker`{.interpreted-text role="program"} program, these should run on Linux, FreeBSD, OpenBSD, and other Unix-like platforms. ### Init-script: `celeryd` {#generic-initd-celeryd} Usage : [/etc/init.d/celeryd {start\|stop\|restart\|status}]{.title-ref} Configuration file : `/etc/default/celeryd`{.interpreted-text role="file"} To configure this script to run the worker properly you probably need to at least tell it where to change directory to when it starts (to find the module containing your app, or your configuration module). The daemonization script is configured by the file `/etc/default/celeryd`{.interpreted-text role="file"}. This is a shell (`sh`{.interpreted-text role="command"}) script where you can add environment variables like the configuration options below. To add real environment variables affecting the worker you must also export them (e.g., `export DISPLAY=":0"`{.interpreted-text role="command"}) ::: admonition Superuser privileges required The init-scripts can only be used by root, and the shell configuration file must also be owned by root. Unprivileged users don\'t need to use the init-script, instead they can use the `celery multi`{.interpreted-text role="program"} utility (or `celery worker --detach`{.interpreted-text role="program"}): ``` console $ celery -A proj multi start worker1 \ --pidfile="$HOME/run/celery/%n.pid" \ --logfile="$HOME/log/celery/%n%I.log" $ celery -A proj multi restart worker1 \ --logfile="$HOME/log/celery/%n%I.log" \ --pidfile="$HOME/run/celery/%n.pid $ celery multi stopwait worker1 --pidfile="$HOME/run/celery/%n.pid" ``` ::: #### Example configuration {#generic-initd-celeryd-example} This is an example configuration for a Python project. `/etc/default/celeryd`{.interpreted-text role="file"}: ``` bash # Names of nodes to start # most people will only start one node: CELERYD_NODES="worker1" # but you can also start multiple and configure settings # for each in CELERYD_OPTS #CELERYD_NODES="worker1 worker2 worker3" # alternatively, you can specify the number of nodes to start: #CELERYD_NODES=10 # Absolute or relative path to the 'celery' command: CELERY_BIN="/usr/local/bin/celery" #CELERY_BIN="/virtualenvs/def/bin/celery" # App instance to use # comment out this line if you don't use an app CELERY_APP="proj" # or fully qualified: #CELERY_APP="proj.tasks:app" # Where to chdir at start. CELERYD_CHDIR="/opt/Myproject/" # Extra command-line arguments to the worker CELERYD_OPTS="--time-limit=300 --concurrency=8" # Configure node-specific settings by appending node name to arguments: #CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1" # Set logging level to DEBUG #CELERYD_LOG_LEVEL="DEBUG" # %n will be replaced with the first part of the nodename. CELERYD_LOG_FILE="/var/log/celery/%n%I.log" CELERYD_PID_FILE="/var/run/celery/%n.pid" # Workers should run as an unprivileged user. # You need to create this user manually (or you can choose # a user/group combination that already exists (e.g., nobody). CELERYD_USER="celery" CELERYD_GROUP="celery" # If enabled pid and log directories will be created if missing, # and owned by the userid/group configured. CELERY_CREATE_DIRS=1 ``` #### Using a login shell You can inherit the environment of the `CELERYD_USER` by using a login shell: ``` bash CELERYD_SU_ARGS="-l" ``` Note that this isn\'t recommended, and that you should only use this option when absolutely necessary. #### Example Django configuration {#generic-initd-celeryd-django-example} Django users now uses the exact same template as above, but make sure that the module that defines your Celery app instance also sets a default value for `DJANGO_SETTINGS_MODULE`{.interpreted-text role="envvar"} as shown in the example Django project in `django-first-steps`{.interpreted-text role="ref"}. #### Available options {#generic-initd-celeryd-options} - `CELERY_APP` > App instance to use (value for > `--app `{.interpreted-text role="option"} argument). - `CELERY_BIN` > Absolute or relative path to the `celery`{.interpreted-text > role="program"} program. Examples: > > > - `celery`{.interpreted-text role="file"} > > - `/usr/local/bin/celery`{.interpreted-text role="file"} > > - `/virtualenvs/proj/bin/celery`{.interpreted-text > > role="file"} > > - `/virtualenvs/proj/bin/python -m celery`{.interpreted-text > > role="file"} - `CELERYD_NODES` > List of node names to start (separated by space). - `CELERYD_OPTS` > Additional command-line arguments for the worker, see [celery > worker \--help]{.title-ref} for a list. This also supports the > extended syntax used by [multi]{.title-ref} to configure settings > for individual nodes. See [celery multi \--help]{.title-ref} for > some multi-node configuration examples. - `CELERYD_CHDIR` > Path to change directory to at start. Default is to stay in the > current directory. - `CELERYD_PID_FILE` > Full path to the PID file. Default is /var/run/celery/%n.pid - `CELERYD_LOG_FILE` > Full path to the worker log file. Default is > /var/log/celery/%n%I.log **Note**: Using [%I]{.title-ref} is > important when using the prefork pool as having multiple processes > share the same log file will lead to race conditions. - `CELERYD_LOG_LEVEL` > Worker log level. Default is INFO. - `CELERYD_USER` > User to run the worker as. Default is current user. - `CELERYD_GROUP` > Group to run worker as. Default is current user. - `CELERY_CREATE_DIRS` > Always create directories (log directory and pid file directory). > Default is to only create directories when no custom > logfile/pidfile set. - `CELERY_CREATE_RUNDIR` > Always create pidfile directory. By default only enabled when no > custom pidfile location set. - `CELERY_CREATE_LOGDIR` > Always create logfile directory. By default only enable when no > custom logfile location set. ### Init-script: `celerybeat` {#generic-initd-celerybeat} Usage : [/etc/init.d/celerybeat {start\|stop\|restart}]{.title-ref} Configuration file : `/etc/default/celerybeat`{.interpreted-text role="file"} or `/etc/default/celeryd`{.interpreted-text role="file"}. #### Example configuration {#generic-initd-celerybeat-example} This is an example configuration for a Python project: \`/etc/default/celerybeat\`: ``` bash # Absolute or relative path to the 'celery' command: CELERY_BIN="/usr/local/bin/celery" #CELERY_BIN="/virtualenvs/def/bin/celery" # App instance to use # comment out this line if you don't use an app CELERY_APP="proj" # or fully qualified: #CELERY_APP="proj.tasks:app" # Where to chdir at start. CELERYBEAT_CHDIR="/opt/Myproject/" # Extra arguments to celerybeat CELERYBEAT_OPTS="--schedule=/var/run/celery/celerybeat-schedule" ``` #### Example Django configuration {#generic-initd-celerybeat-django-example} You should use the same template as above, but make sure the `DJANGO_SETTINGS_MODULE` variable is set (and exported), and that `CELERYD_CHDIR` is set to the projects directory: ``` bash export DJANGO_SETTINGS_MODULE="settings" CELERYD_CHDIR="/opt/MyProject" ``` #### Available options {#generic-initd-celerybeat-options} - `CELERY_APP` > App instance to use (value for > `--app `{.interpreted-text role="option"} argument). - `CELERYBEAT_OPTS` > Additional arguments to `celery beat`{.interpreted-text > role="program"}, see `celery beat --help`{.interpreted-text > role="command"} for a list of available options. - `CELERYBEAT_PID_FILE` > Full path to the PID file. Default is > `/var/run/celeryd.pid`{.interpreted-text role="file"}. - `CELERYBEAT_LOG_FILE` > Full path to the log file. Default is > `/var/log/celeryd.log`{.interpreted-text role="file"}. - `CELERYBEAT_LOG_LEVEL` > Log level to use. Default is `INFO`. - `CELERYBEAT_USER` > User to run beat as. Default is the current user. - `CELERYBEAT_GROUP` > Group to run beat as. Default is the current user. - `CELERY_CREATE_DIRS` > Always create directories (log directory and pid file directory). > Default is to only create directories when no custom > logfile/pidfile set. - `CELERY_CREATE_RUNDIR` > Always create pidfile directory. By default only enabled when no > custom pidfile location set. - `CELERY_CREATE_LOGDIR` > Always create logfile directory. By default only enable when no > custom logfile location set. ### Troubleshooting {#generic-initd-troubleshooting} If you can\'t get the init-scripts to work, you should try running them in *verbose mode*: ``` console # sh -x /etc/init.d/celeryd start ``` This can reveal hints as to why the service won\'t start. If the worker starts with *\"OK\"* but exits almost immediately afterwards and there\'s no evidence in the log file, then there\'s probably an error but as the daemons standard outputs are already closed you\'ll not be able to see them anywhere. For this situation you can use the `C_FAKEFORK`{.interpreted-text role="envvar"} environment variable to skip the daemonization step: ``` console # C_FAKEFORK=1 sh -x /etc/init.d/celeryd start ``` and now you should be able to see the errors. Commonly such errors are caused by insufficient permissions to read from, or write to a file, and also by syntax errors in configuration modules, user modules, third-party libraries, or even from Celery itself (if you\'ve found a bug you should `report it `{.interpreted-text role="ref"}). ## Usage `systemd` {#daemon-systemd-generic} - [extra/systemd/](https://github.com/celery/celery/tree/main/extra/systemd/) ::: {#generic-systemd-celery} Usage : [systemctl {start\|stop\|restart\|status} celery.service]{.title-ref} Configuration file : /etc/conf.d/celery ::: ### Service file: celery.service This is an example systemd file: `/etc/systemd/system/celery.service`{.interpreted-text role="file"}: ``` bash [Unit] Description=Celery Service After=network.target [Service] Type=forking User=celery Group=celery EnvironmentFile=/etc/conf.d/celery WorkingDirectory=/opt/celery ExecStart=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi start $CELERYD_NODES \ --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \ --loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS' ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait $CELERYD_NODES \ --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \ --loglevel="${CELERYD_LOG_LEVEL}"' ExecReload=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi restart $CELERYD_NODES \ --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \ --loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS' Restart=always [Install] WantedBy=multi-user.target ``` Once you\'ve put that file in `/etc/systemd/system`{.interpreted-text role="file"}, you should run `systemctl daemon-reload`{.interpreted-text role="command"} in order that Systemd acknowledges that file. You should also run that command each time you modify it. Use `systemctl enable celery.service`{.interpreted-text role="command"} if you want the celery service to automatically start when (re)booting the system. Optionally you can specify extra dependencies for the celery service: e.g. if you use RabbitMQ as a broker, you could specify `rabbitmq-server.service` in both `After=` and `Requires=` in the `[Unit]` [systemd section](https://www.freedesktop.org/software/systemd/man/systemd.unit.html#%5BUnit%5D%20Section%20Options). To configure user, group, `chdir`{.interpreted-text role="command"} change settings: `User`, `Group`, and `WorkingDirectory` defined in `/etc/systemd/system/celery.service`{.interpreted-text role="file"}. You can also use systemd-tmpfiles in order to create working directories (for logs and pid). file : [/etc/tmpfiles.d/celery.conf]{.title-ref} ``` bash d /run/celery 0755 celery celery - d /var/log/celery 0755 celery celery - ``` #### Example configuration {#generic-systemd-celery-example} This is an example configuration for a Python project: `/etc/conf.d/celery`{.interpreted-text role="file"}: ``` bash # Name of nodes to start # here we have a single node CELERYD_NODES="w1" # or we could have three nodes: #CELERYD_NODES="w1 w2 w3" # Absolute or relative path to the 'celery' command: CELERY_BIN="/usr/local/bin/celery" #CELERY_BIN="/virtualenvs/def/bin/celery" # App instance to use # comment out this line if you don't use an app CELERY_APP="proj" # or fully qualified: #CELERY_APP="proj.tasks:app" # How to call manage.py CELERYD_MULTI="multi" # Extra command-line arguments to the worker CELERYD_OPTS="--time-limit=300 --concurrency=8" # - %n will be replaced with the first part of the nodename. # - %I will be replaced with the current child process index # and is important when using the prefork pool to avoid race conditions. CELERYD_PID_FILE="/var/run/celery/%n.pid" CELERYD_LOG_FILE="/var/log/celery/%n%I.log" CELERYD_LOG_LEVEL="INFO" # you may wish to add these options for Celery Beat CELERYBEAT_PID_FILE="/var/run/celery/beat.pid" CELERYBEAT_LOG_FILE="/var/log/celery/beat.log" ``` ### Service file: celerybeat.service This is an example systemd file for Celery Beat: `/etc/systemd/system/celerybeat.service`{.interpreted-text role="file"}: ``` bash [Unit] Description=Celery Beat Service After=network.target [Service] Type=simple User=celery Group=celery EnvironmentFile=/etc/conf.d/celery WorkingDirectory=/opt/celery ExecStart=/bin/sh -c '${CELERY_BIN} -A ${CELERY_APP} beat \ --pidfile=${CELERYBEAT_PID_FILE} \ --logfile=${CELERYBEAT_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL}' Restart=always [Install] WantedBy=multi-user.target ``` Once you\'ve put that file in `/etc/systemd/system`{.interpreted-text role="file"}, you should run `systemctl daemon-reload`{.interpreted-text role="command"} in order that Systemd acknowledges that file. You should also run that command each time you modify it. Use `systemctl enable celerybeat.service`{.interpreted-text role="command"} if you want the celery beat service to automatically start when (re)booting the system. ## Running the worker with superuser privileges (root) Running the worker with superuser privileges is a very dangerous practice. There should always be a workaround to avoid running as root. Celery may run arbitrary code in messages serialized with pickle - this is dangerous, especially when run as root. By default Celery won\'t run workers as root. The associated error message may not be visible in the logs but may be seen if `C_FAKEFORK`{.interpreted-text role="envvar"} is used. To force Celery to run workers as root use `C_FORCE_ROOT`{.interpreted-text role="envvar"}. When running as root without `C_FORCE_ROOT`{.interpreted-text role="envvar"} the worker will appear to start with *\"OK\"* but exit immediately after with no apparent errors. This problem may appear when running the project in a new development or production environment (inadvertently) as root. ## `supervisor`{.interpreted-text role="pypi"} {#daemon-supervisord} - [extra/supervisord/](https://github.com/celery/celery/tree/main/extra/supervisord/) ## `launchd` (macOS) {#daemon-launchd} - [extra/macOS](https://github.com/celery/celery/tree/main/extra/macOS/) --- # Debugging {#guide-debugging} ## Debugging Tasks Remotely (using pdb) {#tut-remote_debug} ### Basics `celery.contrib.rdb`{.interpreted-text role="mod"} is an extended version of `pdb`{.interpreted-text role="mod"} that enables remote debugging of processes that doesn\'t have terminal access. Example usage: ``` python from celery import task from celery.contrib import rdb @task() def add(x, y): result = x + y rdb.set_trace() # <- set break-point return result ``` `~celery.contrib.rdb.set_trace`{.interpreted-text role="func"} sets a break-point at the current location and creates a socket you can telnet into to remotely debug your task. The debugger may be started by multiple processes at the same time, so rather than using a fixed port the debugger will search for an available port, starting from the base port (6900 by default). The base port can be changed using the environment variable `CELERY_RDB_PORT`{.interpreted-text role="envvar"}. By default the debugger will only be available from the local host, to enable access from the outside you have to set the environment variable `CELERY_RDB_HOST`{.interpreted-text role="envvar"}. When the worker encounters your break-point it\'ll log the following information: ``` text [INFO/MainProcess] Received task: tasks.add[d7261c71-4962-47e5-b342-2448bedd20e8] [WARNING/PoolWorker-1] Remote Debugger:6900: Please telnet 127.0.0.1 6900. Type `exit` in session to continue. [2011-01-18 14:25:44,119: WARNING/PoolWorker-1] Remote Debugger:6900: Waiting for client... ``` If you telnet the port specified you\'ll be presented with a [pdb]{.title-ref} shell: ``` console $ telnet localhost 6900 Connected to localhost. Escape character is '^]'. > /opt/devel/demoapp/tasks.py(128)add() -> return result (Pdb) ``` Enter `help` to get a list of available commands, It may be a good idea to read the [Python Debugger Manual](http://docs.python.org/library/pdb.html) if you have never used [pdb]{.title-ref} before. To demonstrate, we\'ll read the value of the `result` variable, change it and continue execution of the task: ``` text (Pdb) result 4 (Pdb) result = 'hello from rdb' (Pdb) continue Connection closed by foreign host. ``` The result of our vandalism can be seen in the worker logs: ``` text [2011-01-18 14:35:36,599: INFO/MainProcess] Task tasks.add[d7261c71-4962-47e5-b342-2448bedd20e8] succeeded in 61.481s: 'hello from rdb' ``` ### Tips #### Enabling the break-point signal {#breakpoint_signal} If the environment variable `CELERY_RDBSIG`{.interpreted-text role="envvar"} is set, the worker will open up an rdb instance whenever the [SIGUSR2]{.title-ref} signal is sent. This is the case for both main and worker processes. For example starting the worker with: ``` console $ CELERY_RDBSIG=1 celery worker -l INFO ``` You can start an rdb session for any of the worker processes by executing: ``` console $ kill -USR2 ``` --- # Extensions and Bootsteps {#guide-extending} ::: {.contents local="" depth="2"} ::: ## Custom Message Consumers {#extending-custom-consumers} You may want to embed custom Kombu consumers to manually process your messages. For that purpose a special `~celery.bootstep.ConsumerStep`{.interpreted-text role="class"} bootstep class exists, where you only need to define the `get_consumers` method, that must return a list of `kombu.Consumer`{.interpreted-text role="class"} objects to start whenever the connection is established: ``` python from celery import Celery from celery import bootsteps from kombu import Consumer, Exchange, Queue my_queue = Queue('custom', Exchange('custom'), 'routing_key') app = Celery(broker='amqp://') class MyConsumerStep(bootsteps.ConsumerStep): def get_consumers(self, channel): return [Consumer(channel, queues=[my_queue], callbacks=[self.handle_message], accept=['json'])] def handle_message(self, body, message): print('Received message: {0!r}'.format(body)) message.ack() app.steps['consumer'].add(MyConsumerStep) def send_me_a_message(who, producer=None): with app.producer_or_acquire(producer) as producer: producer.publish( {'hello': who}, serializer='json', exchange=my_queue.exchange, routing_key='routing_key', declare=[my_queue], retry=True, ) if __name__ == '__main__': send_me_a_message('world!') ``` ::: note ::: title Note ::: Kombu Consumers can take use of two different message callback dispatching mechanisms. The first one is the `callbacks` argument that accepts a list of callbacks with a `(body, message)` signature, the second one is the `on_message` argument that takes a single callback with a `(message,)` signature. The latter won\'t automatically decode and deserialize the payload. ``` python def get_consumers(self, channel): return [Consumer(channel, queues=[my_queue], on_message=self.on_message)] def on_message(self, message): payload = message.decode() print( 'Received message: {0!r} {props!r} rawlen={s}'.format( payload, props=message.properties, s=len(message.body), )) message.ack() ``` ::: ## Blueprints {#extending-blueprints} Bootsteps is a technique to add functionality to the workers. A bootstep is a custom class that defines hooks to do custom actions at different stages in the worker. Every bootstep belongs to a blueprint, and the worker currently defines two blueprints: **Worker**, and **Consumer** ------------------------------------------------------------------------ **Figure A:** Bootsteps in the Worker and Consumer blueprints. Starting : from the bottom up the first step in the worker blueprint is the Timer, and the last step is to start the Consumer blueprint, that then establishes the broker connection and starts consuming messages. ![](../images/worker_graph_full.png) ------------------------------------------------------------------------ ## Worker {#extending-worker_blueprint} The Worker is the first blueprint to start, and with it starts major components like the event loop, processing pool, and the timer used for ETA tasks and other timed events. When the worker is fully started it continues with the Consumer blueprint, that sets up how tasks are executed, connects to the broker and starts the message consumers. The `~celery.worker.WorkController`{.interpreted-text role="class"} is the core worker implementation, and contains several methods and attributes that you can use in your bootstep. ### Attributes {#extending-worker_blueprint-attributes} ::: {#extending-worker-app} ::: attribute app The current app instance. ::: ::: ::: {#extending-worker-hostname} ::: attribute hostname The workers node name (e.g., [worker1@example.com]{.title-ref}) ::: ::: ::: {#extending-worker-blueprint} ::: attribute blueprint This is the worker `~celery.bootsteps.Blueprint`{.interpreted-text role="class"}. ::: ::: ::: {#extending-worker-hub} ::: attribute hub Event loop object (`~kombu.asynchronous.Hub`{.interpreted-text role="class"}). You can use this to register callbacks in the event loop. This is only supported by async I/O enabled transports (amqp, redis), in which case the [worker.use_eventloop]{.title-ref} attribute should be set. Your worker bootstep must require the Hub bootstep to use this: ``` python class WorkerStep(bootsteps.StartStopStep): requires = {'celery.worker.components:Hub'} ``` ::: ::: ::: {#extending-worker-pool} ::: attribute pool The current process/eventlet/gevent/thread pool. See `celery.concurrency.base.BasePool`{.interpreted-text role="class"}. Your worker bootstep must require the Pool bootstep to use this: ``` python class WorkerStep(bootsteps.StartStopStep): requires = {'celery.worker.components:Pool'} ``` ::: ::: ::: {#extending-worker-timer} ::: attribute timer `~kombu.asynchronous.timer.Timer`{.interpreted-text role="class"} used to schedule functions. Your worker bootstep must require the Timer bootstep to use this: ``` python class WorkerStep(bootsteps.StartStopStep): requires = {'celery.worker.components:Timer'} ``` ::: ::: ::: {#extending-worker-statedb} ::: attribute statedb `Database `{.interpreted-text role="class"}\` to persist state between worker restarts. This is only defined if the `statedb` argument is enabled. Your worker bootstep must require the `Statedb` bootstep to use this: ``` python class WorkerStep(bootsteps.StartStopStep): requires = {'celery.worker.components:Statedb'} ``` ::: ::: ::: {#extending-worker-autoscaler} ::: attribute autoscaler `~celery.worker.autoscaler.Autoscaler`{.interpreted-text role="class"} used to automatically grow and shrink the number of processes in the pool. This is only defined if the `autoscale` argument is enabled. Your worker bootstep must require the [Autoscaler]{.title-ref} bootstep to use this: ``` python class WorkerStep(bootsteps.StartStopStep): requires = ('celery.worker.autoscaler:Autoscaler',) ``` ::: ::: ::: {#extending-worker-autoreloader} ::: attribute autoreloader `~celery.worker.autoreloder.Autoreloader`{.interpreted-text role="class"} used to automatically reload use code when the file-system changes. This is only defined if the `autoreload` argument is enabled. Your worker bootstep must require the [Autoreloader]{.title-ref} bootstep to use this; ``` python class WorkerStep(bootsteps.StartStopStep): requires = ('celery.worker.autoreloader:Autoreloader',) ``` ::: ::: ### Example worker bootstep An example Worker bootstep could be: ``` python from celery import bootsteps class ExampleWorkerStep(bootsteps.StartStopStep): requires = {'celery.worker.components:Pool'} def __init__(self, worker, **kwargs): print('Called when the WorkController instance is constructed') print('Arguments to WorkController: {0!r}'.format(kwargs)) def create(self, worker): # this method can be used to delegate the action methods # to another object that implements ``start`` and ``stop``. return self def start(self, worker): print('Called when the worker is started.') def stop(self, worker): print('Called when the worker shuts down.') def terminate(self, worker): print('Called when the worker terminates') ``` Every method is passed the current `WorkController` instance as the first argument. Another example could use the timer to wake up at regular intervals: ``` python from celery import bootsteps class DeadlockDetection(bootsteps.StartStopStep): requires = {'celery.worker.components:Timer'} def __init__(self, worker, deadlock_timeout=3600): self.timeout = deadlock_timeout self.requests = [] self.tref = None def start(self, worker): # run every 30 seconds. self.tref = worker.timer.call_repeatedly( 30.0, self.detect, (worker,), priority=10, ) def stop(self, worker): if self.tref: self.tref.cancel() self.tref = None def detect(self, worker): # update active requests for req in worker.active_requests: if req.time_start and time() - req.time_start > self.timeout: raise SystemExit() ``` ### Customizing Task Handling Logs The Celery worker emits messages to the Python logging subsystem for various events throughout the lifecycle of a task. These messages can be customized by overriding the `LOG_` format strings which are defined in `celery/app/trace.py`{.interpreted-text role="file"}. For example: ``` python import celery.app.trace celery.app.trace.LOG_SUCCESS = "This is a custom message" ``` The various format strings are all provided with the task name and ID for `%` formatting, and some of them receive extra fields like the return value or the exception which caused a task to fail. These fields can be used in custom format strings like so: ``` python import celery.app.trace celery.app.trace.LOG_REJECTED = "%(name)r is cursed and I won't run it: %(exc)s" ``` ## Consumer {#extending-consumer_blueprint} The Consumer blueprint establishes a connection to the broker, and is restarted every time this connection is lost. Consumer bootsteps include the worker heartbeat, the remote control command consumer, and importantly, the task consumer. When you create consumer bootsteps you must take into account that it must be possible to restart your blueprint. An additional \'shutdown\' method is defined for consumer bootsteps, this method is called when the worker is shutdown. ### Attributes {#extending-consumer-attributes} ::: {#extending-consumer-app} ::: attribute app The current app instance. ::: ::: ::: {#extending-consumer-controller} ::: attribute controller The parent `~@WorkController`{.interpreted-text role="class"} object that created this consumer. ::: ::: ::: {#extending-consumer-hostname} ::: attribute hostname The workers node name (e.g., [worker1@example.com]{.title-ref}) ::: ::: ::: {#extending-consumer-blueprint} ::: attribute blueprint This is the worker `~celery.bootsteps.Blueprint`{.interpreted-text role="class"}. ::: ::: ::: {#extending-consumer-hub} ::: attribute hub Event loop object (`~kombu.asynchronous.Hub`{.interpreted-text role="class"}). You can use this to register callbacks in the event loop. This is only supported by async I/O enabled transports (amqp, redis), in which case the [worker.use_eventloop]{.title-ref} attribute should be set. Your worker bootstep must require the Hub bootstep to use this: ``` python class WorkerStep(bootsteps.StartStopStep): requires = {'celery.worker.components:Hub'} ``` ::: ::: ::: {#extending-consumer-connection} ::: attribute connection The current broker connection (`kombu.Connection`{.interpreted-text role="class"}). A consumer bootstep must require the \'Connection\' bootstep to use this: ``` python class Step(bootsteps.StartStopStep): requires = {'celery.worker.consumer.connection:Connection'} ``` ::: ::: ::: {#extending-consumer-event_dispatcher} ::: attribute event_dispatcher A `@events.Dispatcher`{.interpreted-text role="class"} object that can be used to send events. A consumer bootstep must require the [Events]{.title-ref} bootstep to use this. ``` python class Step(bootsteps.StartStopStep): requires = {'celery.worker.consumer.events:Events'} ``` ::: ::: ::: {#extending-consumer-gossip} ::: attribute gossip Worker to worker broadcast communication (`~celery.worker.consumer.gossip.Gossip`{.interpreted-text role="class"}). A consumer bootstep must require the [Gossip]{.title-ref} bootstep to use this. ``` python class RatelimitStep(bootsteps.StartStopStep): """Rate limit tasks based on the number of workers in the cluster.""" requires = {'celery.worker.consumer.gossip:Gossip'} def start(self, c): self.c = c self.c.gossip.on.node_join.add(self.on_cluster_size_change) self.c.gossip.on.node_leave.add(self.on_cluster_size_change) self.c.gossip.on.node_lost.add(self.on_node_lost) self.tasks = [ self.app.tasks['proj.tasks.add'] self.app.tasks['proj.tasks.mul'] ] self.last_size = None def on_cluster_size_change(self, worker): cluster_size = len(list(self.c.gossip.state.alive_workers())) if cluster_size != self.last_size: for task in self.tasks: task.rate_limit = 1.0 / cluster_size self.c.reset_rate_limits() self.last_size = cluster_size def on_node_lost(self, worker): # may have processed heartbeat too late, so wake up soon # in order to see if the worker recovered. self.c.timer.call_after(10.0, self.on_cluster_size_change) ``` **Callbacks** - ` gossip.on.node_join` > Called whenever a new node joins the cluster, providing a > `~celery.events.state.Worker`{.interpreted-text role="class"} > instance. - ` gossip.on.node_leave` > Called whenever a new node leaves the cluster (shuts down), > providing a `~celery.events.state.Worker`{.interpreted-text > role="class"} instance. - ` gossip.on.node_lost` > Called whenever heartbeat was missed for a worker instance in the > cluster (heartbeat not received or processed in time), providing a > `~celery.events.state.Worker`{.interpreted-text role="class"} > instance. > > This doesn\'t necessarily mean the worker is actually offline, so > use a time out mechanism if the default heartbeat timeout isn\'t > sufficient. ::: ::: ::: {#extending-consumer-pool} ::: attribute pool The current process/eventlet/gevent/thread pool. See `celery.concurrency.base.BasePool`{.interpreted-text role="class"}. ::: ::: ::: {#extending-consumer-timer} ::: attribute timer `Timer >> app = Celery() >>> app.steps['worker'].add(MyWorkerStep) # < add class, don't instantiate >>> app.steps['consumer'].add(MyConsumerStep) >>> app.steps['consumer'].update([StepA, StepB]) >>> app.steps['consumer'] {step:proj.StepB{()}, step:proj.MyConsumerStep{()}, step:proj.StepA{()} ``` The order of steps isn\'t important here as the order is decided by the resulting dependency graph (`Step.requires`). To illustrate how you can install bootsteps and how they work, this is an example step that prints some useless debugging information. It can be added both as a worker and consumer bootstep: ``` python from celery import Celery from celery import bootsteps class InfoStep(bootsteps.Step): def __init__(self, parent, **kwargs): # here we can prepare the Worker/Consumer object # in any way we want, set attribute defaults, and so on. print('{0!r} is in init'.format(parent)) def start(self, parent): # our step is started together with all other Worker/Consumer # bootsteps. print('{0!r} is starting'.format(parent)) def stop(self, parent): # the Consumer calls stop every time the consumer is # restarted (i.e., connection is lost) and also at shutdown. # The Worker will call stop at shutdown only. print('{0!r} is stopping'.format(parent)) def shutdown(self, parent): # shutdown is called by the Consumer at shutdown, it's not # called by Worker. print('{0!r} is shutting down'.format(parent)) app = Celery(broker='amqp://') app.steps['worker'].add(InfoStep) app.steps['consumer'].add(InfoStep) ``` Starting the worker with this step installed will give us the following logs: ``` text is in init is in init [2013-05-29 16:18:20,544: WARNING/MainProcess] is starting [2013-05-29 16:18:21,577: WARNING/MainProcess] is starting is stopping is stopping is shutting down ``` The `print` statements will be redirected to the logging subsystem after the worker has been initialized, so the \"is starting\" lines are time-stamped. You may notice that this does no longer happen at shutdown, this is because the `stop` and `shutdown` methods are called inside a *signal handler*, and it\'s not safe to use logging inside such a handler. Logging with the Python logging module isn\'t `reentrant`{.interpreted-text role="term"}: meaning you cannot interrupt the function then call it again later. It\'s important that the `stop` and `shutdown` methods you write is also `reentrant`{.interpreted-text role="term"}. Starting the worker with `--loglevel=debug `{.interpreted-text role="option"} will show us more information about the boot process: ``` text [2013-05-29 16:18:20,509: DEBUG/MainProcess] | Worker: Preparing bootsteps. [2013-05-29 16:18:20,511: DEBUG/MainProcess] | Worker: Building graph... is in init [2013-05-29 16:18:20,511: DEBUG/MainProcess] | Worker: New boot order: {Hub, Pool, Timer, StateDB, Autoscaler, InfoStep, Beat, Consumer} [2013-05-29 16:18:20,514: DEBUG/MainProcess] | Consumer: Preparing bootsteps. [2013-05-29 16:18:20,514: DEBUG/MainProcess] | Consumer: Building graph... is in init [2013-05-29 16:18:20,515: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Mingle, Events, Gossip, InfoStep, Agent, Heart, Control, Tasks, event loop} [2013-05-29 16:18:20,522: DEBUG/MainProcess] | Worker: Starting Hub [2013-05-29 16:18:20,522: DEBUG/MainProcess] ^-- substep ok [2013-05-29 16:18:20,522: DEBUG/MainProcess] | Worker: Starting Pool [2013-05-29 16:18:20,542: DEBUG/MainProcess] ^-- substep ok [2013-05-29 16:18:20,543: DEBUG/MainProcess] | Worker: Starting InfoStep [2013-05-29 16:18:20,544: WARNING/MainProcess] is starting [2013-05-29 16:18:20,544: DEBUG/MainProcess] ^-- substep ok [2013-05-29 16:18:20,544: DEBUG/MainProcess] | Worker: Starting Consumer [2013-05-29 16:18:20,544: DEBUG/MainProcess] | Consumer: Starting Connection [2013-05-29 16:18:20,559: INFO/MainProcess] Connected to amqp://guest@127.0.0.1:5672// [2013-05-29 16:18:20,560: DEBUG/MainProcess] ^-- substep ok [2013-05-29 16:18:20,560: DEBUG/MainProcess] | Consumer: Starting Mingle [2013-05-29 16:18:20,560: INFO/MainProcess] mingle: searching for neighbors [2013-05-29 16:18:21,570: INFO/MainProcess] mingle: no one here [2013-05-29 16:18:21,570: DEBUG/MainProcess] ^-- substep ok [2013-05-29 16:18:21,571: DEBUG/MainProcess] | Consumer: Starting Events [2013-05-29 16:18:21,572: DEBUG/MainProcess] ^-- substep ok [2013-05-29 16:18:21,572: DEBUG/MainProcess] | Consumer: Starting Gossip [2013-05-29 16:18:21,577: DEBUG/MainProcess] ^-- substep ok [2013-05-29 16:18:21,577: DEBUG/MainProcess] | Consumer: Starting InfoStep [2013-05-29 16:18:21,577: WARNING/MainProcess] is starting [2013-05-29 16:18:21,578: DEBUG/MainProcess] ^-- substep ok [2013-05-29 16:18:21,578: DEBUG/MainProcess] | Consumer: Starting Heart [2013-05-29 16:18:21,579: DEBUG/MainProcess] ^-- substep ok [2013-05-29 16:18:21,579: DEBUG/MainProcess] | Consumer: Starting Control [2013-05-29 16:18:21,583: DEBUG/MainProcess] ^-- substep ok [2013-05-29 16:18:21,583: DEBUG/MainProcess] | Consumer: Starting Tasks [2013-05-29 16:18:21,606: DEBUG/MainProcess] basic.qos: prefetch_count->80 [2013-05-29 16:18:21,606: DEBUG/MainProcess] ^-- substep ok [2013-05-29 16:18:21,606: DEBUG/MainProcess] | Consumer: Starting event loop [2013-05-29 16:18:21,608: WARNING/MainProcess] celery@example.com ready. ``` ## Command-line programs {#extending-programs} ### Adding new command-line options {#extending-commandoptions} #### Command-specific options {#extending-command-options} You can add additional command-line options to the `worker`, `beat`, and `events` commands by modifying the `~@user_options`{.interpreted-text role="attr"} attribute of the application instance. Celery commands uses the `click`{.interpreted-text role="mod"} module to parse command-line arguments, and so to add custom arguments you need to add `click.Option`{.interpreted-text role="class"} instances to the relevant set. Example adding a custom option to the `celery worker`{.interpreted-text role="program"} command: ``` python from celery import Celery from click import Option app = Celery(broker='amqp://') app.user_options['worker'].add(Option(('--enable-my-option',), is_flag=True, help='Enable custom option.')) ``` All bootsteps will now receive this argument as a keyword argument to `Bootstep.__init__`: ``` python from celery import bootsteps class MyBootstep(bootsteps.Step): def __init__(self, parent, enable_my_option=False, **options): super().__init__(parent, **options) if enable_my_option: party() app.steps['worker'].add(MyBootstep) ``` #### Preload options {#extending-preload_options} The `celery`{.interpreted-text role="program"} umbrella command supports the concept of \'preload options\'. These are special options passed to all sub-commands. You can add new preload options, for example to specify a configuration template: ``` python from celery import Celery from celery import signals from click import Option app = Celery() app.user_options['preload'].add(Option(('-Z', '--template'), default='default', help='Configuration template to use.')) @signals.user_preload_options.connect def on_preload_parsed(options, **kwargs): use_template(options['template']) ``` ### Adding new `celery`{.interpreted-text role="program"} sub-commands {#extending-subcommands} New commands can be added to the `celery`{.interpreted-text role="program"} umbrella command by using [setuptools entry-points](http://reinout.vanrees.org/weblog/2010/01/06/zest-releaser-entry-points.html). Entry-points is special meta-data that can be added to your packages `setup.py` program, and then after installation, read from the system using the `importlib`{.interpreted-text role="mod"} module. Celery recognizes `celery.commands` entry-points to install additional sub-commands, where the value of the entry-point must point to a valid click command. This is how the `Flower`{.interpreted-text role="pypi"} monitoring extension may add the `celery flower`{.interpreted-text role="program"} command, by adding an entry-point in `setup.py`{.interpreted-text role="file"}: ``` python setup( name='flower', entry_points={ 'celery.commands': [ 'flower = flower.command:flower', ], } ) ``` The command definition is in two parts separated by the equal sign, where the first part is the name of the sub-command (flower), then the second part is the fully qualified symbol path to the function that implements the command: ``` text flower.command:flower ``` The module path and the name of the attribute should be separated by colon as above. In the module `flower/command.py`{.interpreted-text role="file"}, the command function may be defined as the following: ``` python import click @click.command() @click.option('--port', default=8888, type=int, help='Webserver port') @click.option('--debug', is_flag=True) def flower(port, debug): print('Running our command') ``` ## Worker API ### `~kombu.asynchronous.Hub`{.interpreted-text role="class"} - The workers async event loop supported transports : amqp, redis ::: versionadded 3.0 ::: The worker uses asynchronous I/O when the amqp or redis broker transports are used. The eventual goal is for all transports to use the event-loop, but that will take some time so other transports still use a threading-based solution. ::: method hub.add(fd, callback, flags) ::: ::: method hub.add_reader(fd, callback, \*args) Add callback to be called when `fd` is readable. The callback will stay registered until explicitly removed using `hub.remove(fd) `{.interpreted-text role="meth"}, or the file descriptor is automatically discarded because it\'s no longer valid. Note that only one callback can be registered for any given file descriptor at a time, so calling `add` a second time will remove any callback that was previously registered for that file descriptor. A file descriptor is any file-like object that supports the `fileno` method, or it can be the file descriptor number (int). ::: ::: method hub.add_writer(fd, callback, \*args) Add callback to be called when `fd` is writable. See also notes for `hub.add_reader`{.interpreted-text role="meth"} above. ::: ::: method hub.remove(fd) Remove all callbacks for file descriptor `fd` from the loop. ::: ### Timer - Scheduling events ::: method timer.call_after(secs, callback, args=(), kwargs=(), priority=0) ::: ::: method timer.call_repeatedly(secs, callback, args=(), kwargs=(), priority=0) ::: ::: method timer.call_at(eta, callback, args=(), kwargs=(), priority=0) ::: --- # User Guide {#guide} Release : Date : ::: {.toctree maxdepth="1"} application tasks calling canvas workers daemonizing periodic-tasks routing monitoring security optimizing debugging concurrency/index signals testing extending configuration sphinx ::: --- # Monitoring and Management Guide {#guide-monitoring} ::: {.contents local=""} ::: ## Introduction There are several tools available to monitor and inspect Celery clusters. This document describes some of these, as well as features related to monitoring, like events and broadcast commands. ## Workers {#monitoring-workers} ### Management Command-line Utilities (`inspect`/`control`) {#monitoring-control} `celery`{.interpreted-text role="program"} can also be used to inspect and manage worker nodes (and to some degree tasks). To list all the commands available do: ``` console $ celery --help ``` or to get help for a specific command do: ``` console $ celery --help ``` #### Commands - **shell**: Drop into a Python shell. The locals will include the `celery` variable: this is the current app. Also all known tasks will be automatically added to locals (unless the `--without-tasks `{.interpreted-text role="option"} flag is set). Uses `Ipython`{.interpreted-text role="pypi"}, `bpython`{.interpreted-text role="pypi"}, or regular `python`{.interpreted-text role="program"} in that order if installed. You can force an implementation using `--ipython `{.interpreted-text role="option"}, `--bpython `{.interpreted-text role="option"}, or `--python `{.interpreted-text role="option"}. - **status**: List active nodes in this cluster > ``` console > $ celery -A proj status > ``` - **result**: Show the result of a task > ``` console > $ celery -A proj result -t tasks.add 4e196aa4-0141-4601-8138-7aa33db0f577 > ``` > > Note that you can omit the name of the task as long as the task > doesn\'t use a custom result backend. - **purge**: Purge messages from all configured task queues. > This command will remove all messages from queues configured in > the `CELERY_QUEUES`{.interpreted-text role="setting"} setting: > > ::: warning > ::: title > Warning > ::: > > There\'s no undo for this operation, and messages will be > permanently deleted! > ::: > > ``` console > $ celery -A proj purge > ``` > > You can also specify the queues to purge using the > [-Q]{.title-ref} option: > > ``` console > $ celery -A proj purge -Q celery,foo,bar > ``` > > and exclude queues from being purged using the [-X]{.title-ref} > option: > > ``` console > $ celery -A proj purge -X celery > ``` - **inspect active**: List active tasks > ``` console > $ celery -A proj inspect active > ``` > > These are all the tasks that are currently being executed. - **inspect scheduled**: List scheduled ETA tasks > ``` console > $ celery -A proj inspect scheduled > ``` > > These are tasks reserved by the worker when they have an > [eta]{.title-ref} or [countdown]{.title-ref} argument set. - **inspect reserved**: List reserved tasks > ``` console > $ celery -A proj inspect reserved > ``` > > This will list all tasks that have been prefetched by the worker, > and is currently waiting to be executed (doesn\'t include tasks > with an ETA value set). - **inspect revoked**: List history of revoked tasks > ``` console > $ celery -A proj inspect revoked > ``` - **inspect registered**: List registered tasks > ``` console > $ celery -A proj inspect registered > ``` - **inspect stats**: Show worker statistics (see `worker-statistics`{.interpreted-text role="ref"}) > ``` console > $ celery -A proj inspect stats > ``` - **inspect query_task**: Show information about task(s) by id. > Any worker having a task in this set of ids reserved/active will > respond with status and information. > > ``` console > $ celery -A proj inspect query_task e9f6c8f0-fec9-4ae8-a8c6-cf8c8451d4f8 > ``` > > You can also query for information about multiple tasks: > > ``` console > $ celery -A proj inspect query_task id1 id2 ... idN > ``` - **control enable_events**: Enable events > ``` console > $ celery -A proj control enable_events > ``` - **control disable_events**: Disable events > ``` console > $ celery -A proj control disable_events > ``` - **migrate**: Migrate tasks from one broker to another (**EXPERIMENTAL**). > ``` console > $ celery -A proj migrate redis://localhost amqp://localhost > ``` This command will migrate all the tasks on one broker to another. As this command is new and experimental you should be sure to have a backup of the data before proceeding. ::: note ::: title Note ::: All `inspect` and `control` commands supports a `--timeout `{.interpreted-text role="option"} argument, This is the number of seconds to wait for responses. You may have to increase this timeout if you\'re not getting a response due to latency. ::: #### Specifying destination nodes {#inspect-destination} By default the inspect and control commands operates on all workers. You can specify a single, or a list of workers by using the `--destination `{.interpreted-text role="option"} argument: ``` console $ celery -A proj inspect -d w1@e.com,w2@e.com reserved $ celery -A proj control -d w1@e.com,w2@e.com enable_events ``` ### Flower: Real-time Celery web-monitor {#monitoring-flower} Flower is a real-time web based monitor and administration tool for Celery. It\'s under active development, but is already an essential tool. Being the recommended monitor for Celery, it obsoletes the Django-Admin monitor, `celerymon` and the `ncurses` based monitor. Flower is pronounced like \"flow\", but you can also use the botanical version if you prefer. #### Features - Real-time monitoring using Celery Events > - Task progress and history > - Ability to show task details (arguments, start time, run-time, > and more) > - Graphs and statistics - Remote Control > - View worker status and statistics > - Shutdown and restart worker instances > - Control worker pool size and autoscale settings > - View and modify the queues a worker instance consumes from > - View currently running tasks > - View scheduled tasks (ETA/countdown) > - View reserved and revoked tasks > - Apply time and rate limits > - Configuration viewer > - Revoke or terminate tasks - HTTP API > - List workers > - Shut down a worker > - Restart worker's pool > - Grow worker's pool > - Shrink worker's pool > - Autoscale worker pool > - Start consuming from a queue > - Stop consuming from a queue > - List tasks > - List (seen) task types > - Get a task info > - Execute a task > - Execute a task by name > - Get a task result > - Change soft and hard time limits for a task > - Change rate limit for a task > - Revoke a task - OpenID authentication **Screenshots** ![](../images/dashboard.png){width="700px"} More [screenshots](https://github.com/mher/flower/tree/master/docs/screenshots): #### Usage You can use pip to install Flower: ``` console $ pip install flower ``` Running the flower command will start a web-server that you can visit: ``` console $ celery -A proj flower ``` The default port is , but you can change this using the [\--port](https://flower.readthedocs.io/en/latest/config.html#port) argument: ``` console $ celery -A proj flower --port=5555 ``` Broker URL can also be passed through the `--broker `{.interpreted-text role="option"} argument : ``` console $ celery --broker=amqp://guest:guest@localhost:5672// flower or $ celery --broker=redis://guest:guest@localhost:6379/0 flower ``` Then, you can visit flower in your web browser : ``` console $ open http://localhost:5555 ``` Flower has many more features than are detailed here, including authorization options. Check out the [official documentation](https://flower.readthedocs.io/en/latest/) for more information. ### celery events: Curses Monitor {#monitoring-celeryev} ::: versionadded 2.0 ::: [celery events]{.title-ref} is a simple curses monitor displaying task and worker history. You can inspect the result and traceback of tasks, and it also supports some management commands like rate limiting and shutting down workers. This monitor was started as a proof of concept, and you probably want to use Flower instead. Starting: ``` console $ celery -A proj events ``` You should see a screen like: ![](../images/celeryevshotsm.jpg) [celery events]{.title-ref} is also used to start snapshot cameras (see `monitoring-snapshots`{.interpreted-text role="ref"}: ``` console $ celery -A proj events --camera= --frequency=1.0 ``` and it includes a tool to dump events to `stdout`{.interpreted-text role="file"}: ``` console $ celery -A proj events --dump ``` For a complete list of options use `!--help`{.interpreted-text role="option"}: ``` console $ celery events --help ``` ## RabbitMQ {#monitoring-rabbitmq} To manage a Celery cluster it is important to know how RabbitMQ can be monitored. RabbitMQ ships with the [rabbitmqctl(1)](http://www.rabbitmq.com/man/rabbitmqctl.1.man.html) command, with this you can list queues, exchanges, bindings, queue lengths, the memory usage of each queue, as well as manage users, virtual hosts and their permissions. ::: note ::: title Note ::: The default virtual host (`"/"`) is used in these examples, if you use a custom virtual host you have to add the `-p` argument to the command, for example: `rabbitmqctl list_queues -p my_vhost …` ::: ### Inspecting queues {#monitoring-rmq-queues} Finding the number of tasks in a queue: ``` console $ rabbitmqctl list_queues name messages messages_ready \ messages_unacknowledged ``` Here [messages_ready]{.title-ref} is the number of messages ready for delivery (sent but not received), [messages_unacknowledged]{.title-ref} is the number of messages that\'s been received by a worker but not acknowledged yet (meaning it is in progress, or has been reserved). [messages]{.title-ref} is the sum of ready and unacknowledged messages. Finding the number of workers currently consuming from a queue: ``` console $ rabbitmqctl list_queues name consumers ``` Finding the amount of memory allocated to a queue: ``` console $ rabbitmqctl list_queues name memory ``` Tip : Adding the `-q` option to [rabbitmqctl(1)](http://www.rabbitmq.com/man/rabbitmqctl.1.man.html) makes the output easier to parse. ## Redis {#monitoring-redis} If you\'re using Redis as the broker, you can monitor the Celery cluster using the [redis-cli(1)]{.title-ref} command to list lengths of queues. ### Inspecting queues {#monitoring-redis-queues} Finding the number of tasks in a queue: ``` console $ redis-cli -h HOST -p PORT -n DATABASE_NUMBER llen QUEUE_NAME ``` The default queue is named [celery]{.title-ref}. To get all available queues, invoke: ``` console $ redis-cli -h HOST -p PORT -n DATABASE_NUMBER keys \* ``` ::: note ::: title Note ::: Queue keys only exists when there are tasks in them, so if a key doesn\'t exist it simply means there are no messages in that queue. This is because in Redis a list with no elements in it is automatically removed, and hence it won\'t show up in the [keys]{.title-ref} command output, and [llen]{.title-ref} for that list returns 0. Also, if you\'re using Redis for other purposes, the output of the [keys]{.title-ref} command will include unrelated values stored in the database. The recommended way around this is to use a dedicated [DATABASE_NUMBER]{.title-ref} for Celery, you can also use database numbers to separate Celery applications from each other (virtual hosts), but this won\'t affect the monitoring events used by for example Flower as Redis pub/sub commands are global rather than database based. ::: ## Munin {#monitoring-munin} This is a list of known Munin plug-ins that can be useful when maintaining a Celery cluster. - `rabbitmq-munin`: Munin plug-ins for RabbitMQ. > - `celery_tasks`: Monitors the number of times each task type has been executed (requires [celerymon]{.title-ref}). > - `celery_tasks_states`: Monitors the number of tasks in each state (requires [celerymon]{.title-ref}). > ## Events {#monitoring-events} The worker has the ability to send a message whenever some event happens. These events are then captured by tools like Flower, and `celery events`{.interpreted-text role="program"} to monitor the cluster. ### Snapshots {#monitoring-snapshots} ::: versionadded 2.1 ::: Even a single worker can produce a huge amount of events, so storing the history of all events on disk may be very expensive. A sequence of events describes the cluster state in that time period, by taking periodic snapshots of this state you can keep all history, but still only periodically write it to disk. To take snapshots you need a Camera class, with this you can define what should happen every time the state is captured; You can write it to a database, send it by email or something else entirely. `celery events`{.interpreted-text role="program"} is then used to take snapshots with the camera, for example if you want to capture state every 2 seconds using the camera `myapp.Camera` you run `celery events`{.interpreted-text role="program"} with the following arguments: ``` console $ celery -A proj events -c myapp.Camera --frequency=2.0 ``` #### Custom Camera {#monitoring-camera} Cameras can be useful if you need to capture events and do something with those events at an interval. For real-time event processing you should use `@events.Receiver`{.interpreted-text role="class"} directly, like in `event-real-time-example`{.interpreted-text role="ref"}. Here is an example camera, dumping the snapshot to screen: ``` python from pprint import pformat from celery.events.snapshot import Polaroid class DumpCam(Polaroid): clear_after = True # clear after flush (incl, state.event_count). def on_shutter(self, state): if not state.event_count: # No new events since last snapshot. return print('Workers: {0}'.format(pformat(state.workers, indent=4))) print('Tasks: {0}'.format(pformat(state.tasks, indent=4))) print('Total: {0.event_count} events, {0.task_count} tasks'.format( state)) ``` See the API reference for `celery.events.state`{.interpreted-text role="mod"} to read more about state objects. Now you can use this cam with `celery events`{.interpreted-text role="program"} by specifying it with the `-c `{.interpreted-text role="option"} option: ``` console $ celery -A proj events -c myapp.DumpCam --frequency=2.0 ``` Or you can use it programmatically like this: ``` python from celery import Celery from myapp import DumpCam def main(app, freq=1.0): state = app.events.State() with app.connection() as connection: recv = app.events.Receiver(connection, handlers={'*': state.event}) with DumpCam(state, freq=freq): recv.capture(limit=None, timeout=None) if __name__ == '__main__': app = Celery(broker='amqp://guest@localhost//') main(app) ``` ### Real-time processing {#event-real-time-example} To process events in real-time you need the following - An event consumer (this is the `Receiver`) - A set of handlers called when events come in. > You can have different handlers for each event type, or a > catch-all handler can be used (\'\*\') - State (optional) `@events.State`{.interpreted-text role="class"} is a convenient in-memory representation of tasks and workers in the cluster that\'s updated as events come in. It encapsulates solutions for many common things, like checking if a worker is still alive (by verifying heartbeats), merging event fields together as events come in, making sure time-stamps are in sync, and so on. Combining these you can easily process events in real-time: ``` python from celery import Celery def my_monitor(app): state = app.events.State() def announce_failed_tasks(event): state.event(event) # task name is sent only with -received event, and state # will keep track of this for us. task = state.tasks.get(event['uuid']) print('TASK FAILED: %s[%s] %s' % ( task.name, task.uuid, task.info(),)) with app.connection() as connection: recv = app.events.Receiver(connection, handlers={ 'task-failed': announce_failed_tasks, '*': state.event, }) recv.capture(limit=None, timeout=None, wakeup=True) if __name__ == '__main__': app = Celery(broker='amqp://guest@localhost//') my_monitor(app) ``` ::: note ::: title Note ::: The `wakeup` argument to `capture` sends a signal to all workers to force them to send a heartbeat. This way you can immediately see workers when the monitor starts. ::: You can listen to specific events by specifying the handlers: ``` python from celery import Celery def my_monitor(app): state = app.events.State() def announce_failed_tasks(event): state.event(event) # task name is sent only with -received event, and state # will keep track of this for us. task = state.tasks.get(event['uuid']) print('TASK FAILED: %s[%s] %s' % ( task.name, task.uuid, task.info(),)) with app.connection() as connection: recv = app.events.Receiver(connection, handlers={ 'task-failed': announce_failed_tasks, }) recv.capture(limit=None, timeout=None, wakeup=True) if __name__ == '__main__': app = Celery(broker='amqp://guest@localhost//') my_monitor(app) ``` ## Event Reference This list contains the events sent by the worker, and their arguments. ### Task Events {#event-reference-task} ::: event task-sent ::: #### task-sent signature : `task-sent(uuid, name, args, kwargs, retries, eta, expires, queue, exchange, routing_key, root_id, parent_id)` Sent when a task message is published and the `task_send_sent_event`{.interpreted-text role="setting"} setting is enabled. ::: event task-received ::: #### task-received signature : `task-received(uuid, name, args, kwargs, retries, eta, hostname, timestamp, root_id, parent_id)` Sent when the worker receives a task. ::: event task-started ::: #### task-started signature : `task-started(uuid, hostname, timestamp, pid)` Sent just before the worker executes the task. ::: event task-succeeded ::: #### task-succeeded signature : `task-succeeded(uuid, result, runtime, hostname, timestamp)` Sent if the task executed successfully. Run-time is the time it took to execute the task using the pool. (Starting from the task is sent to the worker pool, and ending when the pool result handler callback is called). ::: event task-failed ::: #### task-failed signature : `task-failed(uuid, exception, traceback, hostname, timestamp)` Sent if the execution of the task failed. ::: event task-rejected ::: #### task-rejected signature : `task-rejected(uuid, requeue)` The task was rejected by the worker, possibly to be re-queued or moved to a dead letter queue. ::: event task-revoked ::: #### task-revoked signature : `task-revoked(uuid, terminated, signum, expired)` Sent if the task has been revoked (Note that this is likely to be sent by more than one worker). - `terminated` is set to true if the task process was terminated, : and the `signum` field set to the signal used. - `expired` is set to true if the task expired. ::: event task-retried ::: #### task-retried signature : `task-retried(uuid, exception, traceback, hostname, timestamp)` Sent if the task failed, but will be retried in the future. ### Worker Events {#event-reference-worker} ::: event worker-online ::: #### worker-online signature : `worker-online(hostname, timestamp, freq, sw_ident, sw_ver, sw_sys)` The worker has connected to the broker and is online. - \`hostname\`: Nodename of the worker. - \`timestamp\`: Event time-stamp. - \`freq\`: Heartbeat frequency in seconds (float). - \`sw_ident\`: Name of worker software (e.g., `py-celery`). - \`sw_ver\`: Software version (e.g., 2.2.0). - \`sw_sys\`: Operating System (e.g., Linux/Darwin). ::: event worker-heartbeat ::: #### worker-heartbeat signature : `worker-heartbeat(hostname, timestamp, freq, sw_ident, sw_ver, sw_sys, active, processed)` Sent every minute, if the worker hasn\'t sent a heartbeat in 2 minutes, it is considered to be offline. - \`hostname\`: Nodename of the worker. - \`timestamp\`: Event time-stamp. - \`freq\`: Heartbeat frequency in seconds (float). - \`sw_ident\`: Name of worker software (e.g., `py-celery`). - \`sw_ver\`: Software version (e.g., 2.2.0). - \`sw_sys\`: Operating System (e.g., Linux/Darwin). - \`active\`: Number of currently executing tasks. - \`processed\`: Total number of tasks processed by this worker. ::: event worker-offline ::: #### worker-offline signature : `worker-offline(hostname, timestamp, freq, sw_ident, sw_ver, sw_sys)` The worker has disconnected from the broker. ### Mailbox Configuration (Advanced) Celery uses [kombu.pidbox.Mailbox]{.title-ref} internally to send control and broadcast commands to workers. ::: versionadded Kombu 5.6.0 ::: Advanced users can configure the behavior of this mailbox by customizing how it is created. The following parameters are now supported by \`Mailbox\`: - `durable` (default: `False`): If set to `True`, the control exchanges will survive broker restarts. - `exclusive` (default: `False`): If set to `True`, the exchanges will be usable by only one connection. ::: warning ::: title Warning ::: Setting both `durable=True` and `exclusive=True` is not permitted and will raise an error, as these two options are mutually incompatible in AMQP. ::: See `event_queue_durable`{.interpreted-text role="setting"} and `event_queue_exclusive`{.interpreted-text role="setting"} for advanced configuration. --- # Optimizing {#guide-optimizing} ## Introduction The default configuration makes a lot of compromises. It\'s not optimal for any single case, but works well enough for most situations. There are optimizations that can be applied based on specific use cases. Optimizations can apply to different properties of the running environment, be it the time tasks take to execute, the amount of memory used, or responsiveness at times of high load. ## Ensuring Operations In the book Programming Pearls, Jon Bentley presents the concept of back-of-the-envelope calculations by asking the question; > ❝ How much water flows out of the Mississippi River in a day? ❞ The point of this exercise[^1] is to show that there\'s a limit to how much data a system can process in a timely manner. Back of the envelope calculations can be used as a means to plan for this ahead of time. In Celery; If a task takes 10 minutes to complete, and there are 10 new tasks coming in every minute, the queue will never be empty. This is why it\'s very important that you monitor queue lengths! A way to do this is by `using Munin `{.interpreted-text role="ref"}. You should set up alerts, that\'ll notify you as soon as any queue has reached an unacceptable size. This way you can take appropriate action like adding new worker nodes, or revoking unnecessary tasks. ## General Settings {#optimizing-general-settings} ### Broker Connection Pools {#optimizing-connection-pools} The broker connection pool is enabled by default since version 2.5. You can tweak the `broker_pool_limit`{.interpreted-text role="setting"} setting to minimize contention, and the value should be based on the number of active threads/green-threads using broker connections. ### Using Transient Queues {#optimizing-transient-queues} Queues created by Celery are persistent by default. This means that the broker will write messages to disk to ensure that the tasks will be executed even if the broker is restarted. But in some cases it\'s fine that the message is lost, so not all tasks require durability. You can create a *transient* queue for these tasks to improve performance: ``` python from kombu import Exchange, Queue task_queues = ( Queue('celery', routing_key='celery'), Queue('transient', Exchange('transient', delivery_mode=1), routing_key='transient', durable=False), ) ``` or by using `task_routes`{.interpreted-text role="setting"}: ``` python task_routes = { 'proj.tasks.add': {'queue': 'celery', 'delivery_mode': 'transient'} } ``` The `delivery_mode` changes how the messages to this queue are delivered. A value of one means that the message won\'t be written to disk, and a value of two (default) means that the message can be written to disk. To direct a task to your new transient queue you can specify the queue argument (or use the `task_routes`{.interpreted-text role="setting"} setting): ``` python task.apply_async(args, queue='transient') ``` For more information see the `routing guide `{.interpreted-text role="ref"}. ## Worker Settings {#optimizing-worker-settings} ### Prefetch Limits {#optimizing-prefetch-limit} *Prefetch* is a term inherited from AMQP that\'s often misunderstood by users. The prefetch limit is a **limit** for the number of tasks (messages) a worker can reserve for itself. If it is zero, the worker will keep consuming messages, not respecting that there may be other available worker nodes that may be able to process them sooner[^2], or that the messages may not even fit in memory. The workers\' default prefetch count is the `worker_prefetch_multiplier`{.interpreted-text role="setting"} setting multiplied by the number of concurrency slots[^3] (processes/threads/green-threads). If you have many tasks with a long duration you want the multiplier value to be *one*: meaning it\'ll only reserve one task per worker process at a time. However \-- If you have many short-running tasks, and throughput/round trip latency is important to you, this number should be large. The worker is able to process more tasks per second if the messages have already been prefetched, and is available in memory. You may have to experiment to find the best value that works for you. Values like 50 or 150 might make sense in these circumstances. Say 64, or 128. If you have a combination of long- and short-running tasks, the best option is to use two worker nodes that are configured separately, and route the tasks according to the run-time (see `guide-routing`{.interpreted-text role="ref"}). ### Reserve one task at a time The task message is only deleted from the queue after the task is `acknowledged`{.interpreted-text role="term"}, so if the worker crashes before acknowledging the task, it can be redelivered to another worker (or the same after recovery). Note that an exception is considered normal operation in Celery and it will be acknowledged. Acknowledgments are really used to safeguard against failures that can not be normally handled by the Python exception system (i.e. power failure, memory corruption, hardware failure, fatal signal, etc.). For normal exceptions you should use task.retry() to retry the task. ::: seealso Notes at `faq-acks_late-vs-retry`{.interpreted-text role="ref"}. ::: When using the default of early acknowledgment, having a prefetch multiplier setting of *one*, means the worker will reserve at most one extra task for every worker process: or in other words, if the worker is started with `-c 10 `{.interpreted-text role="option"}, the worker may reserve at most 20 tasks (10 acknowledged tasks executing, and 10 unacknowledged reserved tasks) at any time. Often users ask if disabling \"prefetching of tasks\" is possible, and it is possible with a catch. You can have a worker only reserve as many tasks as there are worker processes, with the condition that they are acknowledged late (10 unacknowledged tasks executing for `-c 10 `{.interpreted-text role="option"}) For that, you need to enable `late acknowledgment`{.interpreted-text role="term"}. Using this option over the default behavior means a task that\'s already started executing will be retried in the event of a power failure or the worker instance being killed abruptly, so this also means the task must be `idempotent`{.interpreted-text role="term"} You can enable this behavior by using the following configuration options: ``` python task_acks_late = True worker_prefetch_multiplier = 1 ``` If your tasks cannot be acknowledged late you can disable broker prefetching by enabling `worker_disable_prefetch`{.interpreted-text role="setting"}. With this setting the worker fetches a new task only when an execution slot is free, preventing tasks from waiting behind long running ones on busy workers. This can also be set from the command line using `--disable-prefetch `{.interpreted-text role="option"}. This feature is currently only supported when using Redis as the broker. ### Memory Usage If you are experiencing high memory usage on a prefork worker, first you need to determine whether the issue is also happening on the Celery master process. The Celery master process\'s memory usage should not continue to increase drastically after start-up. If you see this happening, it may indicate a memory leak bug which should be reported to the Celery issue tracker. If only your child processes have high memory usage, this indicates an issue with your task. Keep in mind, Python process memory usage has a \"high watermark\" and will not return memory to the operating system until the child process has stopped. This means a single high memory usage task could permanently increase the memory usage of a child process until it\'s restarted. Fixing this may require adding chunking logic to your task to reduce peak memory usage. Celery workers have two main ways to help reduce memory usage due to the \"high watermark\" and/or memory leaks in child processes: the `worker_max_tasks_per_child`{.interpreted-text role="setting"} and `worker_max_memory_per_child`{.interpreted-text role="setting"} settings. You must be careful not to set these settings too low, or else your workers will spend most of their time restarting child processes instead of processing tasks. For example, if you use a `worker_max_tasks_per_child`{.interpreted-text role="setting"} of 1 and your child process takes 1 second to start, then that child process would only be able to process a maximum of 60 tasks per minute (assuming the task ran instantly). A similar issue can occur when your tasks always exceed `worker_max_memory_per_child`{.interpreted-text role="setting"}. **Footnotes** [^1]: The chapter is available to read for free here: [The back of the envelope](http://books.google.com/books?id=kse_7qbWbjsC&pg=PA67). The book is a classic text. Highly recommended. [^2]: RabbitMQ and other brokers deliver messages round-robin, so this doesn\'t apply to an active system. If there\'s no prefetch limit and you restart the cluster, there will be timing delays between nodes starting. If there are 3 offline nodes and one active node, all messages will be delivered to the active node. [^3]: This is the concurrency setting; `worker_concurrency`{.interpreted-text role="setting"} or the `celery worker -c`{.interpreted-text role="option"} option. --- # Periodic Tasks {#guide-beat} ::: {.contents local=""} ::: ## Introduction `celery beat`{.interpreted-text role="program"} is a scheduler; It kicks off tasks at regular intervals, that are then executed by available worker nodes in the cluster. By default the entries are taken from the `beat_schedule`{.interpreted-text role="setting"} setting, but custom stores can also be used, like storing the entries in a SQL database. You have to ensure only a single scheduler is running for a schedule at a time, otherwise you\'d end up with duplicate tasks. Using a centralized approach means the schedule doesn\'t have to be synchronized, and the service can operate without using locks. ## Time Zones {#beat-timezones} The periodic task schedules uses the UTC time zone by default, but you can change the time zone used using the `timezone`{.interpreted-text role="setting"} setting. An example time zone could be \`Europe/London\`: ``` python timezone = 'Europe/London' ``` This setting must be added to your app, either by configuring it directly using (`app.conf.timezone = 'Europe/London'`), or by adding it to your configuration module if you have set one up using `app.config_from_object`. See `celerytut-configuration`{.interpreted-text role="ref"} for more information about configuration options. The default scheduler (storing the schedule in the `celerybeat-schedule`{.interpreted-text role="file"} file) will automatically detect that the time zone has changed, and so will reset the schedule itself, but other schedulers may not be so smart (e.g., the Django database scheduler, see below) and in that case you\'ll have to reset the schedule manually. ::: admonition Django Users Celery recommends and is compatible with the `USE_TZ` setting introduced in Django 1.4. For Django users the time zone specified in the `TIME_ZONE` setting will be used, or you can specify a custom time zone for Celery alone by using the `timezone`{.interpreted-text role="setting"} setting. The database scheduler won\'t reset when timezone related settings change, so you must do this manually: ``` console $ python manage.py shell >>> from djcelery.models import PeriodicTask >>> PeriodicTask.objects.update(last_run_at=None) ``` Django-Celery only supports Celery 4.0 and below, for Celery 4.0 and above, do as follow: ``` console $ python manage.py shell >>> from django_celery_beat.models import PeriodicTask >>> PeriodicTask.objects.update(last_run_at=None) ``` ::: ## Entries {#beat-entries} To call a task periodically you have to add an entry to the beat schedule list. ``` python from celery import Celery from celery.schedules import crontab app = Celery() @app.on_after_configure.connect def setup_periodic_tasks(sender: Celery, **kwargs): # Calls test('hello') every 10 seconds. sender.add_periodic_task(10.0, test.s('hello'), name='add every 10') # Calls test('hello') every 30 seconds. # It uses the same signature of previous task, an explicit name is # defined to avoid this task replacing the previous one defined. sender.add_periodic_task(30.0, test.s('hello'), name='add every 30') # Calls test('world') every 30 seconds sender.add_periodic_task(30.0, test.s('world'), expires=10) # Executes every Monday morning at 7:30 a.m. sender.add_periodic_task( crontab(hour=7, minute=30, day_of_week=1), test.s('Happy Mondays!'), ) @app.task def test(arg): print(arg) @app.task def add(x, y): z = x + y print(z) ``` Setting these up from within the `~@on_after_configure`{.interpreted-text role="data"} handler means that we\'ll not evaluate the app at module level when using `test.s()`. Note that `~@on_after_configure`{.interpreted-text role="data"} is sent after the app is set up, so tasks outside the module where the app is declared (e.g. in a [tasks.py]{.title-ref} file located by `celery.Celery.autodiscover_tasks`{.interpreted-text role="meth"}) must use a later signal, such as `~@on_after_finalize`{.interpreted-text role="data"}. The `~@add_periodic_task`{.interpreted-text role="meth"} function will add the entry to the `beat_schedule`{.interpreted-text role="setting"} setting behind the scenes, and the same setting can also be used to set up periodic tasks manually: Example: Run the [tasks.add]{.title-ref} task every 30 seconds. ``` python app.conf.beat_schedule = { 'add-every-30-seconds': { 'task': 'tasks.add', 'schedule': 30.0, 'args': (16, 16) }, } app.conf.timezone = 'UTC' ``` ::: note ::: title Note ::: If you\'re wondering where these settings should go then please see `celerytut-configuration`{.interpreted-text role="ref"}. You can either set these options on your app directly or you can keep a separate module for configuration. If you want to use a single item tuple for [args]{.title-ref}, don\'t forget that the constructor is a comma, and not a pair of parentheses. ::: Using a `~datetime.timedelta`{.interpreted-text role="class"} for the schedule means the task will be sent in 30 second intervals (the first task will be sent 30 seconds after [celery beat]{.title-ref} starts, and then every 30 seconds after the last run). A Crontab like schedule also exists, see the section on [Crontab schedules](#crontab-schedules). Like with `cron`{.interpreted-text role="command"}, the tasks may overlap if the first task doesn\'t complete before the next. If that\'s a concern you should use a locking strategy to ensure only one instance can run at a time (see for example `cookbook-task-serial`{.interpreted-text role="ref"}). ### Available Fields {#beat-entry-fields} - [task]{.title-ref} > The name of the task to execute. > > Task names are described in the `task-names`{.interpreted-text > role="ref"} section of the User Guide. Note that this is not the > import path of the task, even though the default naming pattern is > built like it is. - [schedule]{.title-ref} > The frequency of execution. > > This can be the number of seconds as an integer, a > `~datetime.timedelta`{.interpreted-text role="class"}, or a > `~celery.schedules.crontab`{.interpreted-text role="class"}. You > can also define your own custom schedule types, by extending the > interface of `~celery.schedules.schedule`{.interpreted-text > role="class"}. - [args]{.title-ref} > Positional arguments (`list`{.interpreted-text role="class"} or > `tuple`{.interpreted-text role="class"}). - [kwargs]{.title-ref} > Keyword arguments (`dict`{.interpreted-text role="class"}). - [options]{.title-ref} > Execution options (`dict`{.interpreted-text role="class"}). > > This can be any argument supported by > `~celery.app.task.Task.apply_async`{.interpreted-text role="meth"} > \--[exchange]{.title-ref}, [routing_key]{.title-ref}, > [expires]{.title-ref}, and so on. - [relative]{.title-ref} > If [relative]{.title-ref} is true > `~datetime.timedelta`{.interpreted-text role="class"} schedules > are scheduled \"by the clock.\" This means the frequency is > rounded to the nearest second, minute, hour or day depending on > the period of the `~datetime.timedelta`{.interpreted-text > role="class"}. > > By default [relative]{.title-ref} is false, the frequency isn\'t > rounded and will be relative to the time when > `celery beat`{.interpreted-text role="program"} was started. ## Crontab schedules {#beat-crontab} If you want more control over when the task is executed, for example, a particular time of day or day of the week, you can use the `~celery.schedules.crontab`{.interpreted-text role="class"} schedule type: ``` python from celery.schedules import crontab app.conf.beat_schedule = { # Executes every Monday morning at 7:30 a.m. 'add-every-monday-morning': { 'task': 'tasks.add', 'schedule': crontab(hour=7, minute=30, day_of_week=1), 'args': (16, 16), }, } ``` The syntax of these Crontab expressions are very flexible. Some examples: +---------------------------------+------------------------------------+ | **Example** | **Meaning** | +---------------------------------+------------------------------------+ | `crontab()` | Execute every minute. | +---------------------------------+------------------------------------+ | `crontab(minute=0, hour=0)` | Execute daily at midnight. | +---------------------------------+------------------------------------+ | `crontab(minute=0, hour='*/3')` | Execute every three hours: | | | midnight, 3am, 6am, 9am, noon, | | | 3pm, 6pm, 9pm. | +---------------------------------+------------------------------------+ | `crontab(minute=0,` | Same as previous. | | | | | : | | | `hour='0,3,6,9,12,15,18,21')` | | +---------------------------------+------------------------------------+ | `crontab(minute='*/15')` | Execute every 15 minutes. | +---------------------------------+------------------------------------+ | `crontab(day_of_week='sunday')` | Execute every minute (!) at | | | Sundays. | +---------------------------------+------------------------------------+ | `crontab(minute='*',` | Same as previous. | | | | | : `hour='*',` | | | `day_of_week='sun')` | | +---------------------------------+------------------------------------+ | `crontab(minute='*/10',` | Execute every ten minutes, but | | | only between 3-4 am, 5-6 pm, and | | : `hour='3,17,22',` | 10-11 pm on Thursdays or Fridays. | | `day_of_week='thu,fri')` | | +---------------------------------+------------------------------------+ | `cro | Execute every even hour, and every | | ntab(minute=0, hour='*/2,*/3')` | hour divisible by three. This | | | means: at every hour *except*: | | | 1am, 5am, 7am, 11am, 1pm, 5pm, | | | 7pm, 11pm | +---------------------------------+------------------------------------+ | `crontab(minute=0, hour='*/5')` | Execute hour divisible by 5. This | | | means that it is triggered at 3pm, | | | not 5pm (since 3pm equals the | | | 24-hour clock value of \"15\", | | | which is divisible by 5). | +---------------------------------+------------------------------------+ | `cron | Execute every hour divisible by 3, | | tab(minute=0, hour='*/3,8-17')` | and every hour during office hours | | | (8am-5pm). | +---------------------------------+------------------------------------+ | `c | Execute on the second day of every | | rontab(0, 0, day_of_month='2')` | month. | +---------------------------------+------------------------------------+ | `crontab(0, 0,` | Execute on every even numbered | | | day. | | : `day_of_month='2-30/2')` | | +---------------------------------+------------------------------------+ | `crontab(0, 0,` | Execute on the first and third | | | weeks of the month. | | : `day_of_month='1-7,15-21')` | | +---------------------------------+------------------------------------+ | `cr | Execute on the eleventh of May | | ontab(0, 0, day_of_month='11',` | every year. | | | | | : `month_of_year='5')` | | +---------------------------------+------------------------------------+ | `crontab(0, 0,` | Execute every day on the first | | | month of every quarter. | | : `month_of_year='*/3')` | | +---------------------------------+------------------------------------+ See `celery.schedules.crontab`{.interpreted-text role="class"} for more documentation. ## Solar schedules {#beat-solar} If you have a task that should be executed according to sunrise, sunset, dawn or dusk, you can use the `~celery.schedules.solar`{.interpreted-text role="class"} schedule type: ``` python from celery.schedules import solar app.conf.beat_schedule = { # Executes at sunset in Melbourne 'add-at-melbourne-sunset': { 'task': 'tasks.add', 'schedule': solar('sunset', -37.81753, 144.96715), 'args': (16, 16), }, } ``` The arguments are simply: `solar(event, latitude, longitude)` Be sure to use the correct sign for latitude and longitude: --------------- ------------------- ---------------------- **Sign** **Argument** **Meaning** `+` `latitude` North `-` `latitude` South `+` `longitude` East `-` `longitude` West --------------- ------------------- ---------------------- Possible event types are: ---------------------------------- ------------------------------------ **Event** **Meaning** `dawn_astronomical` Execute at the moment after which the sky is no longer completely dark. This is when the sun is 18 degrees below the horizon. `dawn_nautical` Execute when there\'s enough sunlight for the horizon and some objects to be distinguishable; formally, when the sun is 12 degrees below the horizon. `dawn_civil` Execute when there\'s enough light for objects to be distinguishable so that outdoor activities can commence; formally, when the Sun is 6 degrees below the horizon. `sunrise` Execute when the upper edge of the sun appears over the eastern horizon in the morning. `solar_noon` Execute when the sun is highest above the horizon on that day. `sunset` Execute when the trailing edge of the sun disappears over the western horizon in the evening. `dusk_civil` Execute at the end of civil twilight, when objects are still distinguishable and some stars and planets are visible. Formally, when the sun is 6 degrees below the horizon. `dusk_nautical` Execute when the sun is 12 degrees below the horizon. Objects are no longer distinguishable, and the horizon is no longer visible to the naked eye. `dusk_astronomical` Execute at the moment after which the sky becomes completely dark; formally, when the sun is 18 degrees below the horizon. ---------------------------------- ------------------------------------ All solar events are calculated using UTC, and are therefore unaffected by your timezone setting. In polar regions, the sun may not rise or set every day. The scheduler is able to handle these cases (i.e., a `sunrise` event won\'t run on a day when the sun doesn\'t rise). The one exception is `solar_noon`, which is formally defined as the moment the sun transits the celestial meridian, and will occur every day even if the sun is below the horizon. Twilight is defined as the period between dawn and sunrise; and between sunset and dusk. You can schedule an event according to \"twilight\" depending on your definition of twilight (civil, nautical, or astronomical), and whether you want the event to take place at the beginning or end of twilight, using the appropriate event from the list above. See `celery.schedules.solar`{.interpreted-text role="class"} for more documentation. ## Starting the Scheduler {#beat-starting} To start the `celery beat`{.interpreted-text role="program"} service: ``` console $ celery -A proj beat ``` You can also embed [beat]{.title-ref} inside the worker by enabling the workers `-B `{.interpreted-text role="option"} option, this is convenient if you\'ll never run more than one worker node, but it\'s not commonly used and for that reason isn\'t recommended for production use: ``` console $ celery -A proj worker -B ``` Beat needs to store the last run times of the tasks in a local database file (named [celerybeat-schedule]{.title-ref} by default), so it needs access to write in the current directory, or alternatively you can specify a custom location for this file: ``` console $ celery -A proj beat -s /home/celery/var/run/celerybeat-schedule ``` ::: note ::: title Note ::: To daemonize beat see `daemonizing`{.interpreted-text role="ref"}. ::: ### Using custom scheduler classes {#beat-custom-schedulers} Custom scheduler classes can be specified on the command-line (the `--scheduler `{.interpreted-text role="option"} argument). The default scheduler is the `celery.beat.PersistentScheduler`{.interpreted-text role="class"}, that simply keeps track of the last run times in a local `shelve`{.interpreted-text role="mod"} database file. There\'s also the `django-celery-beat`{.interpreted-text role="pypi"} extension that stores the schedule in the Django database, and presents a convenient admin interface to manage periodic tasks at runtime. To install and use this extension: 1. Use `pip`{.interpreted-text role="command"} to install the package: > ``` console > $ pip install django-celery-beat > ``` 2. Add the `django_celery_beat` module to `INSTALLED_APPS` in your Django project\' `settings.py`{.interpreted-text role="file"}: INSTALLED_APPS = ( ..., 'django_celery_beat', ) Note that there is no dash in the module name, only underscores. 3. Apply Django database migrations so that the necessary tables are created: > ``` console > $ python manage.py migrate > ``` 4. Start the `celery beat`{.interpreted-text role="program"} service using the `django_celery_beat.schedulers:DatabaseScheduler` scheduler: > ``` console > $ celery -A proj beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler > ``` Note: You may also add this as the `beat_scheduler`{.interpreted-text role="setting"} setting directly. 5. Visit the Django-Admin interface to set up some periodic tasks. --- # Routing Tasks {#guide-routing} ::: note ::: title Note ::: Alternate routing concepts like topic and fanout is not available for all transports, please consult the `transport comparison table `{.interpreted-text role="ref"}. ::: ::: {.contents local=""} ::: ## Basics {#routing-basics} ### Automatic routing {#routing-automatic} The simplest way to do routing is to use the `task_create_missing_queues`{.interpreted-text role="setting"} setting (on by default). With this setting on, a named queue that\'s not already defined in `task_queues`{.interpreted-text role="setting"} will be created automatically. This makes it easy to perform simple routing tasks. Say you have two servers, [x]{.title-ref}, and [y]{.title-ref} that handle regular tasks, and one server [z]{.title-ref}, that only handles feed related tasks. You can use this configuration: task_routes = {'feed.tasks.import_feed': {'queue': 'feeds'}} With this route enabled import feed tasks will be routed to the [\"feeds\"]{.title-ref} queue, while all other tasks will be routed to the default queue (named [\"celery\"]{.title-ref} for historical reasons). Alternatively, you can use glob pattern matching, or even regular expressions, to match all tasks in the `feed.tasks` name-space: ``` python app.conf.task_routes = {'feed.tasks.*': {'queue': 'feeds'}} ``` If the order of matching patterns is important you should specify the router in *items* format instead: ``` python task_routes = ([ ('feed.tasks.*', {'queue': 'feeds'}), ('web.tasks.*', {'queue': 'web'}), (re.compile(r'(video|image)\.tasks\..*'), {'queue': 'media'}), ],) ``` ::: note ::: title Note ::: The `task_routes`{.interpreted-text role="setting"} setting can either be a dictionary, or a list of router objects, so in this case we need to specify the setting as a tuple containing a list. ::: After installing the router, you can start server [z]{.title-ref} to only process the feeds queue like this: ``` console user@z:/$ celery -A proj worker -Q feeds ``` You can specify as many queues as you want, so you can make this server process the default queue as well: ``` console user@z:/$ celery -A proj worker -Q feeds,celery ``` #### Changing the name of the default queue {#routing-changing-default-queue} You can change the name of the default queue by using the following configuration: ``` python app.conf.task_default_queue = 'default' ``` #### How the queues are defined {#routing-autoqueue-details} The point with this feature is to hide the complex AMQP protocol for users with only basic needs. However \-- you may still be interested in how these queues are declared. A queue named [\"video\"]{.title-ref} will be created with the following settings: ``` javascript {'exchange': 'video', 'exchange_type': 'direct', 'routing_key': 'video'} ``` The non-AMQP backends like [Redis]{.title-ref} or [SQS]{.title-ref} don\'t support exchanges, so they require the exchange to have the same name as the queue. Using this design ensures it will work for them as well. ### Manual routing {#routing-manual} Say you have two servers, [x]{.title-ref}, and [y]{.title-ref} that handle regular tasks, and one server [z]{.title-ref}, that only handles feed related tasks, you can use this configuration: ``` python from kombu import Queue app.conf.task_default_queue = 'default' app.conf.task_queues = ( Queue('default', routing_key='task.#'), Queue('feed_tasks', routing_key='feed.#'), ) app.conf.task_default_exchange = 'tasks' app.conf.task_default_exchange_type = 'topic' app.conf.task_default_routing_key = 'task.default' ``` `task_queues`{.interpreted-text role="setting"} is a list of `~kombu.entity.Queue`{.interpreted-text role="class"} instances. If you don\'t set the exchange or exchange type values for a key, these will be taken from the `task_default_exchange`{.interpreted-text role="setting"} and `task_default_exchange_type`{.interpreted-text role="setting"} settings. To route a task to the [feed_tasks]{.title-ref} queue, you can add an entry in the `task_routes`{.interpreted-text role="setting"} setting: ``` python task_routes = { 'feeds.tasks.import_feed': { 'queue': 'feed_tasks', 'routing_key': 'feed.import', }, } ``` You can also override this using the [routing_key]{.title-ref} argument to `Task.apply_async`{.interpreted-text role="meth"}, or `~celery.execute.send_task`{.interpreted-text role="func"}: > \>\>\> from feeds.tasks import import_feed \>\>\> > import_feed.apply_async(args=\[\'\'\], \... > queue=\'feed_tasks\', \... routing_key=\'feed.import\') To make server [z]{.title-ref} consume from the feed queue exclusively you can start it with the `celery worker -Q`{.interpreted-text role="option"} option: ``` console user@z:/$ celery -A proj worker -Q feed_tasks --hostname=z@%h ``` Servers [x]{.title-ref} and [y]{.title-ref} must be configured to consume from the default queue: ``` console user@x:/$ celery -A proj worker -Q default --hostname=x@%h user@y:/$ celery -A proj worker -Q default --hostname=y@%h ``` If you want, you can even have your feed processing worker handle regular tasks as well, maybe in times when there\'s a lot of work to do: ``` console user@z:/$ celery -A proj worker -Q feed_tasks,default --hostname=z@%h ``` If you have another queue but on another exchange you want to add, just specify a custom exchange and exchange type: ``` python from kombu import Exchange, Queue app.conf.task_queues = ( Queue('feed_tasks', routing_key='feed.#'), Queue('regular_tasks', routing_key='task.#'), Queue('image_tasks', exchange=Exchange('mediatasks', type='direct'), routing_key='image.compress'), ) ``` If you\'re confused about these terms, you should read up on AMQP. ::: seealso In addition to the `amqp-primer`{.interpreted-text role="ref"} below, there\'s [Rabbits and Warrens](http://web.archive.org/web/20160323134044/http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/), an excellent blog post describing queues and exchanges. There\'s also The [CloudAMQP tutorial]{.title-ref}, For users of RabbitMQ the [RabbitMQ FAQ](https://www.rabbitmq.com/faq.html) could be useful as a source of information. ::: ## Special Routing Options {#routing-special_options} ### RabbitMQ Message Priorities {#routing-options-rabbitmq-priorities} supported transports : RabbitMQ ::: versionadded 4.0 ::: Queues can be configured to support priorities by setting the `x-max-priority` argument: ``` python from kombu import Exchange, Queue app.conf.task_queues = [ Queue('tasks', Exchange('tasks'), routing_key='tasks', queue_arguments={'x-max-priority': 10}), ] ``` A default value for all queues can be set using the `task_queue_max_priority`{.interpreted-text role="setting"} setting: ``` python app.conf.task_queue_max_priority = 10 ``` A default priority for all tasks can also be specified using the `task_default_priority`{.interpreted-text role="setting"} setting: ``` python app.conf.task_default_priority = 5 ``` ### Redis Message Priorities {#amqp-primer} supported transports : Redis While the Celery Redis transport does honor the priority field, Redis itself has no notion of priorities. Please read this note before attempting to implement priorities with Redis as you may experience some unexpected behavior. To start scheduling tasks based on priorities you need to configure queue_order_strategy transport option. ``` python app.conf.broker_transport_options = { 'queue_order_strategy': 'priority', } ``` The priority support is implemented by creating n lists for each queue. This means that even though there are 10 (0-9) priority levels, these are consolidated into 4 levels by default to save resources. This means that a queue named celery will really be split into 4 queues. The highest priority queue will be named celery, and the the other queues will have a separator (by default [x06x16]{.title-ref}) and their priority number appended to the queue name. ``` python ['celery', 'celery\x06\x163', 'celery\x06\x166', 'celery\x06\x169'] ``` If you want more priority levels or a different separator you can set the priority_steps and sep transport options: ``` python app.conf.broker_transport_options = { 'priority_steps': list(range(10)), 'sep': ':', 'queue_order_strategy': 'priority', } ``` The config above will give you these queue names: ``` python ['celery', 'celery:1', 'celery:2', 'celery:3', 'celery:4', 'celery:5', 'celery:6', 'celery:7', 'celery:8', 'celery:9'] ``` That said, note that this will never be as good as priorities implemented at the broker server level, and may be approximate at best. But it may still be good enough for your application. ## AMQP Primer ### Messages A message consists of headers and a body. Celery uses headers to store the content type of the message and its content encoding. The content type is usually the serialization format used to serialize the message. The body contains the name of the task to execute, the task id (UUID), the arguments to apply it with and some additional meta-data \-- like the number of retries or an ETA. This is an example task message represented as a Python dictionary: ``` javascript {'task': 'myapp.tasks.add', 'id': '54086c5e-6193-4575-8308-dbab76798756', 'args': [4, 4], 'kwargs': {}} ``` ### Producers, consumers, and brokers {#amqp-producers-consumers-brokers} The client sending messages is typically called a *publisher*, or a *producer*, while the entity receiving messages is called a *consumer*. The *broker* is the message server, routing messages from producers to consumers. You\'re likely to see these terms used a lot in AMQP related material. ### Exchanges, queues, and routing keys {#amqp-exchanges-queues-keys} 1. Messages are sent to exchanges. 2. An exchange routes messages to one or more queues. Several exchange types exists, providing different ways to do routing, or implementing different messaging scenarios. 3. The message waits in the queue until someone consumes it. 4. The message is deleted from the queue when it has been acknowledged. The steps required to send and receive messages are: 1. Create an exchange 2. Create a queue 3. Bind the queue to the exchange. Celery automatically creates the entities necessary for the queues in `task_queues`{.interpreted-text role="setting"} to work (except if the queue\'s [auto_declare]{.title-ref} setting is set to `False`{.interpreted-text role="const"}). Here\'s an example queue configuration with three queues; One for video, one for images, and one default queue for everything else: ``` python from kombu import Exchange, Queue app.conf.task_queues = ( Queue('default', Exchange('default'), routing_key='default'), Queue('videos', Exchange('media'), routing_key='media.video'), Queue('images', Exchange('media'), routing_key='media.image'), ) app.conf.task_default_queue = 'default' app.conf.task_default_exchange_type = 'direct' app.conf.task_default_routing_key = 'default' ``` ### Exchange types {#amqp-exchange-types} The exchange type defines how the messages are routed through the exchange. The exchange types defined in the standard are [direct]{.title-ref}, [topic]{.title-ref}, [fanout]{.title-ref} and [headers]{.title-ref}. Also non-standard exchange types are available as plug-ins to RabbitMQ, like the [last-value-cache plug-in](https://github.com/squaremo/rabbitmq-lvc-plugin) by Michael Bridgen. #### Direct exchanges {#amqp-exchange-type-direct} Direct exchanges match by exact routing keys, so a queue bound by the routing key [video]{.title-ref} only receives messages with that routing key. #### Topic exchanges {#amqp-exchange-type-topic} Topic exchanges matches routing keys using dot-separated words, and the wild-card characters: `*` (matches a single word), and `#` (matches zero or more words). With routing keys like `usa.news`, `usa.weather`, `norway.news`, and `norway.weather`, bindings could be `*.news` (all news), `usa.#` (all items in the USA), or `usa.weather` (all USA weather items). ### Related API commands {#amqp-api} ::: method exchange.declare(exchange_name, type, passive, durable, auto_delete, internal) Declares an exchange by name. See `amqp:Channel.exchange_declare `{.interpreted-text role="meth"}. keyword passive : Passive means the exchange won\'t be created, but you can use this to check if the exchange already exists. keyword durable : Durable exchanges are persistent (i.e., they survive a broker restart). keyword auto_delete : This means the exchange will be deleted by the broker when there are no more queues using it. ::: ::: method queue.declare(queue_name, passive, durable, exclusive, auto_delete) Declares a queue by name. See `amqp:Channel.queue_declare `{.interpreted-text role="meth"} Exclusive queues can only be consumed from by the current connection. Exclusive also implies [auto_delete]{.title-ref}. ::: ::: method queue.bind(queue_name, exchange_name, routing_key) Binds a queue to an exchange with a routing key. Unbound queues won\'t receive messages, so this is necessary. See `amqp:Channel.queue_bind `{.interpreted-text role="meth"} ::: ::: method queue.delete(name, if_unused=False, if_empty=False) Deletes a queue and its binding. See `amqp:Channel.queue_delete `{.interpreted-text role="meth"} ::: ::: method exchange.delete(name, if_unused=False) Deletes an exchange. See `amqp:Channel.exchange_delete `{.interpreted-text role="meth"} ::: ::: note ::: title Note ::: Declaring doesn\'t necessarily mean \"create\". When you declare you *assert* that the entity exists and that it\'s operable. There\'s no rule as to whom should initially create the exchange/queue/binding, whether consumer or producer. Usually the first one to need it will be the one to create it. ::: ### Hands-on with the API {#amqp-api-hands-on} Celery comes with a tool called `celery amqp`{.interpreted-text role="program"} that\'s used for command line access to the AMQP API, enabling access to administration tasks like creating/deleting queues and exchanges, purging queues or sending messages. It can also be used for non-AMQP brokers, but different implementation may not implement all commands. You can write commands directly in the arguments to `celery amqp`{.interpreted-text role="program"}, or just start with no arguments to start it in shell-mode: ``` console $ celery -A proj amqp -> connecting to amqp://guest@localhost:5672/. -> connected. 1> ``` Here `1>` is the prompt. The number 1, is the number of commands you have executed so far. Type `help` for a list of commands available. It also supports auto-completion, so you can start typing a command and then hit the [tab]{.title-ref} key to show a list of possible matches. Let\'s create a queue you can send messages to: ``` console $ celery -A proj amqp 1> exchange.declare testexchange direct ok. 2> queue.declare testqueue ok. queue:testqueue messages:0 consumers:0. 3> queue.bind testqueue testexchange testkey ok. ``` This created the direct exchange `testexchange`, and a queue named `testqueue`. The queue is bound to the exchange using the routing key `testkey`. From now on all messages sent to the exchange `testexchange` with routing key `testkey` will be moved to this queue. You can send a message by using the `basic.publish` command: ``` console 4> basic.publish 'This is a message!' testexchange testkey ok. ``` Now that the message is sent you can retrieve it again. You can use the `basic.get` command here, that polls for new messages on the queue in a synchronous manner (this is OK for maintenance tasks, but for services you want to use `basic.consume` instead) Pop a message off the queue: ``` console 5> basic.get testqueue {'body': 'This is a message!', 'delivery_info': {'delivery_tag': 1, 'exchange': u'testexchange', 'message_count': 0, 'redelivered': False, 'routing_key': u'testkey'}, 'properties': {}} ``` AMQP uses acknowledgment to signify that a message has been received and processed successfully. If the message hasn\'t been acknowledged and consumer channel is closed, the message will be delivered to another consumer. Note the delivery tag listed in the structure above; Within a connection channel, every received message has a unique delivery tag, This tag is used to acknowledge the message. Also note that delivery tags aren\'t unique across connections, so in another client the delivery tag [1]{.title-ref} might point to a different message than in this channel. You can acknowledge the message you received using `basic.ack`: ``` console 6> basic.ack 1 ok. ``` To clean up after our test session you should delete the entities you created: ``` console 7> queue.delete testqueue ok. 0 messages deleted. 8> exchange.delete testexchange ok. ``` ## Routing Tasks {#routing-tasks} ### Defining queues {#routing-defining-queues} In Celery available queues are defined by the `task_queues`{.interpreted-text role="setting"} setting. Here\'s an example queue configuration with three queues; One for video, one for images, and one default queue for everything else: ``` python default_exchange = Exchange('default', type='direct') media_exchange = Exchange('media', type='direct') app.conf.task_queues = ( Queue('default', default_exchange, routing_key='default'), Queue('videos', media_exchange, routing_key='media.video'), Queue('images', media_exchange, routing_key='media.image') ) app.conf.task_default_queue = 'default' app.conf.task_default_exchange = 'default' app.conf.task_default_routing_key = 'default' ``` Here, the `task_default_queue`{.interpreted-text role="setting"} will be used to route tasks that doesn\'t have an explicit route. The default exchange, exchange type, and routing key will be used as the default routing values for tasks, and as the default values for entries in `task_queues`{.interpreted-text role="setting"}. Multiple bindings to a single queue are also supported. Here\'s an example of two routing keys that are both bound to the same queue: ``` python from kombu import Exchange, Queue, binding media_exchange = Exchange('media', type='direct') CELERY_QUEUES = ( Queue('media', [ binding(media_exchange, routing_key='media.video'), binding(media_exchange, routing_key='media.image'), ]), ) ``` ### Specifying task destination {#routing-task-destination} The destination for a task is decided by the following (in order): 1. The routing arguments to `Task.apply_async`{.interpreted-text role="func"}. 2. Routing related attributes defined on the `~celery.app.task.Task`{.interpreted-text role="class"} itself. 3. The `routers`{.interpreted-text role="ref"} defined in `task_routes`{.interpreted-text role="setting"}. It\'s considered best practice to not hard-code these settings, but rather leave that as configuration options by using `routers`{.interpreted-text role="ref"}; This is the most flexible approach, but sensible defaults can still be set as task attributes. ### Routers A router is a function that decides the routing options for a task. All you need to define a new router is to define a function with the signature `(name, args, kwargs, options, task=None, **kw)`: ``` python def route_task(name, args, kwargs, options, task=None, **kw): if name == 'myapp.tasks.compress_video': return {'exchange': 'video', 'exchange_type': 'topic', 'routing_key': 'video.compress'} ``` If you return the `queue` key, it\'ll expand with the defined settings of that queue in `task_queues`{.interpreted-text role="setting"}: ``` javascript {'queue': 'video', 'routing_key': 'video.compress'} ``` becomes \--\> ``` javascript {'queue': 'video', 'exchange': 'video', 'exchange_type': 'topic', 'routing_key': 'video.compress'} ``` You install router classes by adding them to the `task_routes`{.interpreted-text role="setting"} setting: ``` python task_routes = (route_task,) ``` Router functions can also be added by name: ``` python task_routes = ('myapp.routers.route_task',) ``` For simple task name -\> route mappings like the router example above, you can simply drop a dict into `task_routes`{.interpreted-text role="setting"} to get the same behavior: ``` python task_routes = { 'myapp.tasks.compress_video': { 'queue': 'video', 'routing_key': 'video.compress', }, } ``` The routers will then be traversed in order, it will stop at the first router returning a true value, and use that as the final route for the task. You can also have multiple routers defined in a sequence: ``` python task_routes = [ route_task, { 'myapp.tasks.compress_video': { 'queue': 'video', 'routing_key': 'video.compress', }, ] ``` The routers will then be visited in turn, and the first to return a value will be chosen. If you\'re using Redis or RabbitMQ you can also specify the queue\'s default priority in the route. ``` python task_routes = { 'myapp.tasks.compress_video': { 'queue': 'video', 'routing_key': 'video.compress', 'priority': 10, }, } ``` Similarly, calling [apply_async]{.title-ref} on a task will override that default priority. ``` python task.apply_async(priority=0) ``` ::: admonition Priority Order and Cluster Responsiveness It is important to note that, due to worker prefetching, if a bunch of tasks submitted at the same time they may be out of priority order at first. Disabling worker prefetching will prevent this issue, but may cause less than ideal performance for small, fast tasks. In most cases, simply reducing [worker_prefetch_multiplier]{.title-ref} to 1 is an easier and cleaner way to increase the responsiveness of your system without the costs of disabling prefetching entirely. Note that priorities values are sorted in reverse when using the redis broker: 0 being highest priority. ::: ### Broadcast Celery can also support broadcast routing. Here is an example exchange `broadcast_tasks` that delivers copies of tasks to all workers connected to it: ``` python from kombu.common import Broadcast app.conf.task_queues = (Broadcast('broadcast_tasks'),) app.conf.task_routes = { 'tasks.reload_cache': { 'queue': 'broadcast_tasks', 'exchange': 'broadcast_tasks' } } ``` Now the `tasks.reload_cache` task will be sent to every worker consuming from this queue. Here is another example of broadcast routing, this time with a `celery beat`{.interpreted-text role="program"} schedule: ``` python from kombu.common import Broadcast from celery.schedules import crontab app.conf.task_queues = (Broadcast('broadcast_tasks'),) app.conf.beat_schedule = { 'test-task': { 'task': 'tasks.reload_cache', 'schedule': crontab(minute=0, hour='*/3'), 'options': {'exchange': 'broadcast_tasks'} }, } ``` ::: admonition Broadcast & Results Note that Celery result doesn\'t define what happens if two tasks have the same task_id. If the same task is distributed to more than one worker, then the state history may not be preserved. It\'s a good idea to set the `task.ignore_result` attribute in this case. ::: --- # Security {#guide-security} ::: {.contents local=""} ::: ## Introduction While Celery is written with security in mind, it should be treated as an unsafe component. Depending on your [Security Policy](https://en.wikipedia.org/wiki/Security_policy), there are various steps you can take to make your Celery installation more secure. ## Areas of Concern ### Broker It\'s imperative that the broker is guarded from unwanted access, especially if accessible to the public. By default, workers trust that the data they get from the broker hasn\'t been tampered with. See [Message Signing](#message-signing) for information on how to make the broker connection more trustworthy. The first line of defense should be to put a firewall in front of the broker, allowing only white-listed machines to access it. Keep in mind that both firewall misconfiguration, and temporarily disabling the firewall, is common in the real world. Solid security policy includes monitoring of firewall equipment to detect if they\'ve been disabled, be it accidentally or on purpose. In other words, one shouldn\'t blindly trust the firewall either. If your broker supports fine-grained access control, like RabbitMQ, this is something you should look at enabling. See for example . If supported by your broker backend, you can enable end-to-end SSL encryption and authentication using `broker_use_ssl`{.interpreted-text role="setting"}. ### Client In Celery, \"client\" refers to anything that sends messages to the broker, for example web-servers that apply tasks. Having the broker properly secured doesn\'t matter if arbitrary messages can be sent through a client. *\[Need more text here\]* ### Worker The default permissions of tasks running inside a worker are the same ones as the privileges of the worker itself. This applies to resources, such as; memory, file-systems, and devices. An exception to this rule is when using the multiprocessing based task pool, which is currently the default. In this case, the task will have access to any memory copied as a result of the `fork`{.interpreted-text role="func"} call, and access to memory contents written by parent tasks in the same worker child process. Limiting access to memory contents can be done by launching every task in a subprocess (`fork`{.interpreted-text role="func"} + `execve`{.interpreted-text role="func"}). Limiting file-system and device access can be accomplished by using [chroot](https://en.wikipedia.org/wiki/Chroot), [jail](https://en.wikipedia.org/wiki/FreeBSD_jail), [sandboxing](https://en.wikipedia.org/wiki/Sandbox_(computer_security)), virtual machines, or other mechanisms as enabled by the platform or additional software. Note also that any task executed in the worker will have the same network access as the machine on which it\'s running. If the worker is located on an internal network it\'s recommended to add firewall rules for outbound traffic. ## Serializers {#security-serializers} The default serializer is JSON since version 4.0, but since it has only support for a restricted set of types you may want to consider using pickle for serialization instead. The [pickle]{.title-ref} serializer is convenient as it can serialize almost any Python object, even functions with some work, but for the same reasons [pickle]{.title-ref} is inherently insecure[^1], and should be avoided whenever clients are untrusted or unauthenticated. You can disable untrusted content by specifying a white-list of accepted content-types in the `accept_content`{.interpreted-text role="setting"} setting: ::: versionadded 3.0.18 ::: ::: note ::: title Note ::: This setting was first supported in version 3.0.18. If you\'re running an earlier version it will simply be ignored, so make sure you\'re running a version that supports it. ::: ``` python accept_content = ['json'] ``` This accepts a list of serializer names and content-types, so you could also specify the content type for json: ``` python accept_content = ['application/json'] ``` Celery also comes with a special [auth]{.title-ref} serializer that validates communication between Celery clients and workers, making sure that messages originates from trusted sources. Using [Public-key cryptography]{.title-ref} the [auth]{.title-ref} serializer can verify the authenticity of senders, to enable this read `message-signing`{.interpreted-text role="ref"} for more information. ## Message Signing Celery can use the `cryptography`{.interpreted-text role="pypi"} library to sign message using [Public-key cryptography]{.title-ref}, where messages sent by clients are signed using a private key and then later verified by the worker using a public certificate. Optimally certificates should be signed by an official [Certificate Authority](https://en.wikipedia.org/wiki/Certificate_authority), but they can also be self-signed. To enable this you should configure the `task_serializer`{.interpreted-text role="setting"} setting to use the [auth]{.title-ref} serializer. Enforcing the workers to only accept signed messages, you should set [accept_content]{.title-ref} to [\[\'auth\'\]]{.title-ref}. For additional signing of the event protocol, set [event_serializer]{.title-ref} to [auth]{.title-ref}. Also required is configuring the paths used to locate private keys and certificates on the file-system: the `security_key`{.interpreted-text role="setting"}, `security_certificate`{.interpreted-text role="setting"}, and `security_cert_store`{.interpreted-text role="setting"} settings respectively. You can tweak the signing algorithm with `security_digest`{.interpreted-text role="setting"}. If using an encrypted private key, the password can be configured with `security_key_password`{.interpreted-text role="setting"}. With these configured it\'s also necessary to call the `celery.setup_security`{.interpreted-text role="func"} function. Note that this will also disable all insecure serializers so that the worker won\'t accept messages with untrusted content types. This is an example configuration using the [auth]{.title-ref} serializer, with the private key and certificate files located in [/etc/ssl]{.title-ref}. ``` python app = Celery() app.conf.update( security_key='/etc/ssl/private/worker.key' security_certificate='/etc/ssl/certs/worker.pem' security_cert_store='/etc/ssl/certs/*.pem', security_digest='sha256', task_serializer='auth', event_serializer='auth', accept_content=['auth'] ) app.setup_security() ``` ::: note ::: title Note ::: While relative paths aren\'t disallowed, using absolute paths is recommended for these files. Also note that the [auth]{.title-ref} serializer won\'t encrypt the contents of a message, so if needed this will have to be enabled separately. ::: ## Intrusion Detection The most important part when defending your systems against intruders is being able to detect if the system has been compromised. ### Logs Logs are usually the first place to look for evidence of security breaches, but they\'re useless if they can be tampered with. A good solution is to set up centralized logging with a dedicated logging server. Access to it should be restricted. In addition to having all of the logs in a single place, if configured correctly, it can make it harder for intruders to tamper with your logs. This should be fairly easy to setup using syslog (see also [syslog-ng](https://en.wikipedia.org/wiki/Syslog-ng) and [rsyslog](http://www.rsyslog.com/)). Celery uses the `logging`{.interpreted-text role="mod"} library, and already has support for using syslog. A tip for the paranoid is to send logs using UDP and cut the transmit part of the logging server\'s network cable :-) ### Tripwire [Tripwire](http://tripwire.com/) is a (now commercial) data integrity tool, with several open source implementations, used to keep cryptographic hashes of files in the file-system, so that administrators can be alerted when they change. This way when the damage is done and your system has been compromised you can tell exactly what files intruders have changed (password files, logs, back-doors, root-kits, and so on). Often this is the only way you\'ll be able to detect an intrusion. Some open source implementations include: - [OSSEC](http://www.ossec.net/) - [Samhain](http://la-samhna.de/samhain/index.html) - [Open Source Tripwire](https://github.com/Tripwire/tripwire-open-source) - [AIDE](http://aide.sourceforge.net/) Also, the [ZFS](https://en.wikipedia.org/wiki/ZFS) file-system comes with built-in integrity checks that can be used. **Footnotes** [^1]: --- # Signals ::: {.contents local=""} ::: Signals allow decoupled applications to receive notifications when certain actions occur elsewhere in the application. Celery ships with many signals that your application can hook into to augment behavior of certain actions. ## Basics {#signal-basics} Several kinds of events trigger signals, you can connect to these signals to perform actions as they trigger. Example connecting to the `after_task_publish`{.interpreted-text role="signal"} signal: ``` python from celery.signals import after_task_publish @after_task_publish.connect def task_sent_handler(sender=None, headers=None, body=None, **kwargs): # information about task are located in headers for task messages # using the task protocol version 2. info = headers if 'task' in headers else body print('after_task_publish for task id {info[id]}'.format( info=info, )) ``` Some signals also have a sender you can filter by. For example the `after_task_publish`{.interpreted-text role="signal"} signal uses the task name as a sender, so by providing the `sender` argument to `~celery.utils.dispatch.signal.Signal.connect`{.interpreted-text role="class"} you can connect your handler to be called every time a task with name [\"proj.tasks.add\"]{.title-ref} is published: ``` python @after_task_publish.connect(sender='proj.tasks.add') def task_sent_handler(sender=None, headers=None, body=None, **kwargs): # information about task are located in headers for task messages # using the task protocol version 2. info = headers if 'task' in headers else body print('after_task_publish for task id {info[id]}'.format( info=info, )) ``` Signals use the same implementation as `django.core.dispatch`{.interpreted-text role="mod"}. As a result other keyword parameters (e.g., signal) are passed to all signal handlers by default. The best practice for signal handlers is to accept arbitrary keyword arguments (i.e., `**kwargs`). That way new Celery versions can add additional arguments without breaking user code. ## Signals {#signal-ref} ### Task Signals ::: signal before_task_publish ::: #### `before_task_publish` ::: versionadded 3.1 ::: Dispatched before a task is published. Note that this is executed in the process sending the task. Sender is the name of the task being sent. Provides arguments: - `body` > Task message body. > > This is a mapping containing the task message fields, see > `message-protocol-task-v2`{.interpreted-text role="ref"} and > `message-protocol-task-v1`{.interpreted-text role="ref"} for a > reference of possible fields that can be defined. - `exchange` > Name of the exchange to send to or a > `~kombu.Exchange`{.interpreted-text role="class"} object. - `routing_key` > Routing key to use when sending the message. - `headers` > Application headers mapping (can be modified). - `properties` > Message properties (can be modified) - `declare` > List of entities (`~kombu.Exchange`{.interpreted-text > role="class"}, `~kombu.Queue`{.interpreted-text role="class"}, or > `~kombu.binding`{.interpreted-text role="class"} to declare before > publishing the message. Can be modified. - `retry_policy` > Mapping of retry options. Can be any argument to > `kombu.Connection.ensure`{.interpreted-text role="meth"} and can > be modified. ::: signal after_task_publish ::: #### `after_task_publish` Dispatched when a task has been sent to the broker. Note that this is executed in the process that sent the task. Sender is the name of the task being sent. Provides arguments: - `headers` > The task message headers, see > `message-protocol-task-v2`{.interpreted-text role="ref"} and > `message-protocol-task-v1`{.interpreted-text role="ref"} for a > reference of possible fields that can be defined. - `body` > The task message body, see > `message-protocol-task-v2`{.interpreted-text role="ref"} and > `message-protocol-task-v1`{.interpreted-text role="ref"} for a > reference of possible fields that can be defined. - `exchange` > Name of the exchange or `~kombu.Exchange`{.interpreted-text > role="class"} object used. - `routing_key` > Routing key used. ::: signal task_prerun ::: #### `task_prerun` Dispatched before a task is executed. Sender is the task object being executed. Provides arguments: - `task_id` > Id of the task to be executed. - `task` > The task being executed. - `args` > The tasks positional arguments. - `kwargs` > The tasks keyword arguments. ::: signal task_postrun ::: #### `task_postrun` Dispatched after a task has been executed. Sender is the task object executed. Provides arguments: - `task_id` > Id of the task to be executed. - `task` > The task being executed. - `args` > The tasks positional arguments. - `kwargs` > The tasks keyword arguments. - `retval` > The return value of the task. - `state` > Name of the resulting state. ::: signal task_retry ::: #### `task_retry` Dispatched when a task will be retried. Sender is the task object. Provides arguments: - `request` > The current task request. - `reason` > Reason for retry (usually an exception instance, but can always be > coerced to `str`{.interpreted-text role="class"}). - `einfo` > Detailed exception information, including traceback (a > `billiard.einfo.ExceptionInfo`{.interpreted-text role="class"} > object). ::: signal task_success ::: #### `task_success` Dispatched when a task succeeds. Sender is the task object executed. Provides arguments - `result` : Return value of the task. ::: signal task_failure ::: #### `task_failure` Dispatched when a task fails. Sender is the task object executed. Provides arguments: - `task_id` > Id of the task. - `exception` > Exception instance raised. - `args` > Positional arguments the task was called with. - `kwargs` > Keyword arguments the task was called with. - `traceback` > Stack trace object. - `einfo` > The `billiard.einfo.ExceptionInfo`{.interpreted-text role="class"} > instance. #### `task_internal_error` Dispatched when an internal Celery error occurs while executing the task. Sender is the task object executed. ::: signal task_internal_error ::: Provides arguments: - `task_id` > Id of the task. - `args` > Positional arguments the task was called with. - `kwargs` > Keyword arguments the task was called with. - `request` > The original request dictionary. This is provided as the > `task.request` may not be ready by the time the exception is > raised. - `exception` > Exception instance raised. - `traceback` > Stack trace object. - `einfo` > The `billiard.einfo.ExceptionInfo`{.interpreted-text role="class"} > instance. #### `task_received` Dispatched when a task is received from the broker and is ready for execution. Sender is the consumer object. ::: signal task_received ::: Provides arguments: - `request` > This is a `~celery.worker.request.Request`{.interpreted-text > role="class"} instance, and not `task.request`. When using the > prefork pool this signal is dispatched in the parent process, so > `task.request` isn\'t available and shouldn\'t be used. Use this > object instead, as they share many of the same fields. ::: signal task_revoked ::: #### `task_revoked` Dispatched when a task is revoked/terminated by the worker. Sender is the task object revoked/terminated. Provides arguments: - `request` > This is a `~celery.app.task.Context`{.interpreted-text > role="class"} instance, and not `task.request`. When using the > prefork pool this signal is dispatched in the parent process, so > `task.request` isn\'t available and shouldn\'t be used. Use this > object instead, as they share many of the same fields. - `terminated` > Set to `True`{.interpreted-text role="const"} if the task was > terminated. - `signum` > Signal number used to terminate the task. If this is > `None`{.interpreted-text role="const"} and terminated is > `True`{.interpreted-text role="const"} then > `TERM`{.interpreted-text role="sig"} should be assumed. - `expired` Set to `True`{.interpreted-text role="const"} if the task expired. ::: signal task_unknown ::: #### `task_unknown` Dispatched when a worker receives a message for a task that\'s not registered. Sender is the worker `~celery.worker.consumer.Consumer`{.interpreted-text role="class"}. Provides arguments: - `name` Name of task not found in registry. - `id` The task id found in the message. - `message` > Raw message object. - `exc` > The error that occurred. ::: signal task_rejected ::: #### `task_rejected` Dispatched when a worker receives an unknown type of message to one of its task queues. Sender is the worker `~celery.worker.consumer.Consumer`{.interpreted-text role="class"}. Provides arguments: - `message` Raw message object. - `exc` > The error that occurred (if any). ### App Signals ::: signal import_modules ::: #### `import_modules` This signal is sent when a program (worker, beat, shell) etc, asks for modules in the `include`{.interpreted-text role="setting"} and `imports`{.interpreted-text role="setting"} settings to be imported. Sender is the app instance. ### Worker Signals ::: signal celeryd_after_setup ::: #### `celeryd_after_setup` This signal is sent after the worker instance is set up, but before it calls run. This means that any queues from the `celery worker -Q`{.interpreted-text role="option"} option is enabled, logging has been set up and so on. It can be used to add custom queues that should always be consumed from, disregarding the `celery worker -Q`{.interpreted-text role="option"} option. Here\'s an example that sets up a direct queue for each worker, these queues can then be used to route a task to any specific worker: ``` python from celery.signals import celeryd_after_setup @celeryd_after_setup.connect def setup_direct_queue(sender, instance, **kwargs): queue_name = '{0}.dq'.format(sender) # sender is the nodename of the worker instance.app.amqp.queues.select_add(queue_name) ``` Provides arguments: - `sender` Node name of the worker. - `instance` > This is the `celery.apps.worker.Worker`{.interpreted-text > role="class"} instance to be initialized. Note that only the > `app`{.interpreted-text role="attr"} and > `hostname`{.interpreted-text role="attr"} (nodename) attributes > have been set so far, and the rest of `__init__` hasn\'t been > executed. - `conf` > The configuration of the current app. ::: signal celeryd_init ::: #### `celeryd_init` This is the first signal sent when `celery worker`{.interpreted-text role="program"} starts up. The `sender` is the host name of the worker, so this signal can be used to setup worker specific configuration: ``` python from celery.signals import celeryd_init @celeryd_init.connect(sender='worker12@example.com') def configure_worker12(conf=None, **kwargs): conf.task_default_rate_limit = '10/m' ``` or to set up configuration for multiple workers you can omit specifying a sender when you connect: ``` python from celery.signals import celeryd_init @celeryd_init.connect def configure_workers(sender=None, conf=None, **kwargs): if sender in ('worker1@example.com', 'worker2@example.com'): conf.task_default_rate_limit = '10/m' if sender == 'worker3@example.com': conf.worker_prefetch_multiplier = 0 ``` Provides arguments: - `sender` Nodename of the worker. - `instance` > This is the `celery.apps.worker.Worker`{.interpreted-text > role="class"} instance to be initialized. Note that only the > `app`{.interpreted-text role="attr"} and > `hostname`{.interpreted-text role="attr"} (nodename) attributes > have been set so far, and the rest of `__init__` hasn\'t been > executed. - `conf` > The configuration of the current app. - `options` > Options passed to the worker from command-line arguments > (including defaults). ::: signal worker_init ::: #### `worker_init` Dispatched before the worker is started. ::: signal worker_before_create_process ::: #### `worker_before_create_process` Dispatched in the parent process, just before new child process is created in the prefork pool. It can be used to clean up instances that don\'t behave well when forking. ``` python @signals.worker_before_create_process.connect def clean_channels(**kwargs): grpc_singleton.clean_channel() ``` ::: signal worker_ready ::: #### `worker_ready` Dispatched when the worker is ready to accept work. ::: signal heartbeat_sent ::: #### `heartbeat_sent` Dispatched when Celery sends a worker heartbeat. Sender is the `celery.worker.heartbeat.Heart`{.interpreted-text role="class"} instance. ::: signal worker_shutting_down ::: #### `worker_shutting_down` Dispatched when the worker begins the shutdown process. Provides arguments: - `sig` > The POSIX signal that was received. - `how` > The shutdown method, warm or cold. - `exitcode` > The exitcode that will be used when the main process exits. ::: signal worker_process_init ::: #### `worker_process_init` Dispatched in all pool child processes when they start. Note that handlers attached to this signal mustn\'t be blocking for more than 4 seconds, or the process will be killed assuming it failed to start. ::: signal worker_process_shutdown ::: #### `worker_process_shutdown` Dispatched in all pool child processes just before they exit. Note: There\'s no guarantee that this signal will be dispatched, similarly to `finally`{.interpreted-text role="keyword"} blocks it\'s impossible to guarantee that handlers will be called at shutdown, and if called it may be interrupted during. Provides arguments: - `pid` > The pid of the child process that\'s about to shutdown. - `exitcode` > The exitcode that\'ll be used when the child process exits. ::: signal worker_shutdown ::: #### `worker_shutdown` Dispatched when the worker is about to shut down. ### Beat Signals ::: signal beat_init ::: #### `beat_init` Dispatched when `celery beat`{.interpreted-text role="program"} starts (either standalone or embedded). Sender is the `celery.beat.Service`{.interpreted-text role="class"} instance. ::: signal beat_embedded_init ::: #### `beat_embedded_init` Dispatched in addition to the `beat_init`{.interpreted-text role="signal"} signal when `celery beat`{.interpreted-text role="program"} is started as an embedded process. Sender is the `celery.beat.Service`{.interpreted-text role="class"} instance. ### Eventlet Signals ::: signal eventlet_pool_started ::: #### `eventlet_pool_started` Sent when the eventlet pool has been started. Sender is the `celery.concurrency.eventlet.TaskPool`{.interpreted-text role="class"} instance. ::: signal eventlet_pool_preshutdown ::: #### `eventlet_pool_preshutdown` Sent when the worker shutdown, just before the eventlet pool is requested to wait for remaining workers. Sender is the `celery.concurrency.eventlet.TaskPool`{.interpreted-text role="class"} instance. ::: signal eventlet_pool_postshutdown ::: #### `eventlet_pool_postshutdown` Sent when the pool has been joined and the worker is ready to shutdown. Sender is the `celery.concurrency.eventlet.TaskPool`{.interpreted-text role="class"} instance. ::: signal eventlet_pool_apply ::: #### `eventlet_pool_apply` Sent whenever a task is applied to the pool. Sender is the `celery.concurrency.eventlet.TaskPool`{.interpreted-text role="class"} instance. Provides arguments: - `target` > The target function. - `args` > Positional arguments. - `kwargs` > Keyword arguments. ### Logging Signals ::: signal setup_logging ::: #### `setup_logging` Celery won\'t configure the loggers if this signal is connected, so you can use this to completely override the logging configuration with your own. If you\'d like to augment the logging configuration setup by Celery then you can use the `after_setup_logger`{.interpreted-text role="signal"} and `after_setup_task_logger`{.interpreted-text role="signal"} signals. Provides arguments: - `loglevel` > The level of the logging object. - `logfile` > The name of the logfile. - `format` > The log format string. - `colorize` > Specify if log messages are colored or not. ::: signal after_setup_logger ::: #### `after_setup_logger` Sent after the setup of every global logger (not task loggers). Used to augment logging configuration. Provides arguments: - `logger` > The logger object. - `loglevel` > The level of the logging object. - `logfile` > The name of the logfile. - `format` > The log format string. - `colorize` > Specify if log messages are colored or not. ::: signal after_setup_task_logger ::: #### `after_setup_task_logger` Sent after the setup of every single task logger. Used to augment logging configuration. Provides arguments: - `logger` > The logger object. - `loglevel` > The level of the logging object. - `logfile` > The name of the logfile. - `format` > The log format string. - `colorize` > Specify if log messages are colored or not. ### Command signals ::: signal user_preload_options ::: #### `user_preload_options` This signal is sent after any of the Celery command line programs are finished parsing the user preload options. It can be used to add additional command-line arguments to the `celery`{.interpreted-text role="program"} umbrella command: ``` python from celery import Celery from celery import signals from celery.bin.base import Option app = Celery() app.user_options['preload'].add(Option( '--monitoring', action='store_true', help='Enable our external monitoring utility, blahblah', )) @signals.user_preload_options.connect def handle_preload_options(options, **kwargs): if options['monitoring']: enable_monitoring() ``` Sender is the `~celery.bin.base.Command`{.interpreted-text role="class"} instance, and the value depends on the program that was called (e.g., for the umbrella command it\'ll be a `~celery.bin.celery.CeleryCommand`{.interpreted-text role="class"}) object). Provides arguments: - `app` > The app instance. - `options` > Mapping of the parsed user preload options (with default values). ### Deprecated Signals ::: signal task_sent ::: #### `task_sent` This signal is deprecated, please use `after_task_publish`{.interpreted-text role="signal"} instead. --- # Documenting Tasks with Sphinx {#sphinx} This document describes how auto-generate documentation for Tasks using Sphinx. ## celery.contrib.sphinx ::: {.automodule members="" noindex=""} celery.contrib.sphinx ::: --- # Tasks {#guide-tasks} Tasks are the building blocks of Celery applications. A task is a class that can be created out of any callable. It performs dual roles in that it defines both what happens when a task is called (sends a message), and what happens when a worker receives that message. Every task class has a unique name, and this name is referenced in messages so the worker can find the right function to execute. A task message is not removed from the queue until that message has been `acknowledged`{.interpreted-text role="term"} by a worker. A worker can reserve many messages in advance and even if the worker is killed \-- by power failure or some other reason \-- the message will be redelivered to another worker. Ideally task functions should be `idempotent`{.interpreted-text role="term"}: meaning the function won\'t cause unintended effects even if called multiple times with the same arguments. Since the worker cannot detect if your tasks are idempotent, the default behavior is to acknowledge the message in advance, just before it\'s executed, so that a task invocation that already started is never executed again. If your task is idempotent you can set the `~Task.acks_late`{.interpreted-text role="attr"} option to have the worker acknowledge the message *after* the task returns instead. See also the FAQ entry `faq-acks_late-vs-retry`{.interpreted-text role="ref"}. Note that the worker will acknowledge the message if the child process executing the task is terminated (either by the task calling `sys.exit`{.interpreted-text role="func"}, or by signal) even when `~Task.acks_late`{.interpreted-text role="attr"} is enabled. This behavior is intentional as\... 1. We don\'t want to rerun tasks that forces the kernel to send a `SIGSEGV`{.interpreted-text role="sig"} (segmentation fault) or similar signals to the process. 2. We assume that a system administrator deliberately killing the task does not want it to automatically restart. 3. A task that allocates too much memory is in danger of triggering the kernel OOM killer, the same may happen again. 4. A task that always fails when redelivered may cause a high-frequency message loop taking down the system. If you really want a task to be redelivered in these scenarios you should consider enabling the `task_reject_on_worker_lost`{.interpreted-text role="setting"} setting. ::: warning ::: title Warning ::: A task that blocks indefinitely may eventually stop the worker instance from doing any other work. If your task does I/O then make sure you add timeouts to these operations, like adding a timeout to a web request using the `requests`{.interpreted-text role="pypi"} library: ``` python connect_timeout, read_timeout = 5.0, 30.0 response = requests.get(URL, timeout=(connect_timeout, read_timeout)) ``` `Time limits `{.interpreted-text role="ref"} are convenient for making sure all tasks return in a timely manner, but a time limit event will actually kill the process by force so only use them to detect cases where you haven\'t used manual timeouts yet. In previous versions, the default prefork pool scheduler was not friendly to long-running tasks, so if you had tasks that ran for minutes/hours, it was advised to enable the `-Ofair `{.interpreted-text role="option"} command-line argument to the `celery worker`{.interpreted-text role="program"}. However, as of version 4.0, -Ofair is now the default scheduling strategy. See `optimizing-prefetch-limit`{.interpreted-text role="ref"} for more information, and for the best performance route long-running and short-running tasks to dedicated workers (`routing-automatic`{.interpreted-text role="ref"}). If your worker hangs then please investigate what tasks are running before submitting an issue, as most likely the hanging is caused by one or more tasks hanging on a network operation. ::: \-- In this chapter you\'ll learn all about defining tasks, and this is the **table of contents**: ::: {.contents local="" depth="1"} ::: ## Basics {#task-basics} You can easily create a task from any callable by using the `@task`{.interpreted-text role="meth"} decorator: ``` python from .models import User @app.task def create_user(username, password): User.objects.create(username=username, password=password) ``` There are also many `options `{.interpreted-text role="ref"} that can be set for the task, these can be specified as arguments to the decorator: ``` python @app.task(serializer='json') def create_user(username, password): User.objects.create(username=username, password=password) ``` ### How do I import the task decorator? > The task decorator is available on your `@Celery`{.interpreted-text > role="class"} application instance, if you don\'t know what this is > then please read `first-steps`{.interpreted-text role="ref"}. > > If you\'re using Django (see `django-first-steps`{.interpreted-text > role="ref"}), or you\'re the author of a library then you probably > want to use the `@shared_task`{.interpreted-text role="func"} > decorator: > > ``` python > from celery import shared_task > > @shared_task > def add(x, y): > return x + y > ``` ### Multiple decorators > When using multiple decorators in combination with the task decorator > you must make sure that the [task]{.title-ref} decorator is applied > last (oddly, in Python this means it must be first in the list): > > ``` python > @app.task > @decorator2 > @decorator1 > def add(x, y): > return x + y > ``` ### Bound tasks A task being bound means the first argument to the task will always be the task instance (`self`), just like Python bound methods: ``` python logger = get_task_logger(__name__) @app.task(bind=True) def add(self, x, y): logger.info(self.request.id) ``` Bound tasks are needed for retries (using `Task.retry() <@Task.retry>`{.interpreted-text role="meth"}), for accessing information about the current task request, and for any additional functionality you add to custom task base classes. ### Task inheritance The `base` argument to the task decorator specifies the base class of the task: ``` python import celery class MyTask(celery.Task): def on_failure(self, exc, task_id, args, kwargs, einfo): print('{0!r} failed: {1!r}'.format(task_id, exc)) @app.task(base=MyTask) def add(x, y): raise KeyError() ``` ## Names {#task-names} Every task must have a unique name. If no explicit name is provided the task decorator will generate one for you, and this name will be based on 1) the module the task is defined in, and 2) the name of the task function. Example setting explicit name: ``` pycon >>> @app.task(name='sum-of-two-numbers') >>> def add(x, y): ... return x + y >>> add.name 'sum-of-two-numbers' ``` A best practice is to use the module name as a name-space, this way names won\'t collide if there\'s already a task with that name defined in another module. ``` pycon >>> @app.task(name='tasks.add') >>> def add(x, y): ... return x + y ``` You can tell the name of the task by investigating its `.name` attribute: ``` pycon >>> add.name 'tasks.add' ``` The name we specified here (`tasks.add`) is exactly the name that would\'ve been automatically generated for us if the task was defined in a module named `tasks.py`{.interpreted-text role="file"}: `tasks.py`{.interpreted-text role="file"}: ``` python @app.task def add(x, y): return x + y ``` ``` pycon >>> from tasks import add >>> add.name 'tasks.add' ``` ::: note ::: title Note ::: You can use the [inspect]{.title-ref} command in a worker to view the names of all registered tasks. See the [inspect registered]{.title-ref} command in the `monitoring-control`{.interpreted-text role="ref"} section of the User Guide. ::: ### Changing the automatic naming behavior {#task-name-generator-info} ::: versionadded 4.0 ::: There are some cases when the default automatic naming isn\'t suitable. Consider having many tasks within many different modules: project/ /__init__.py /celery.py /moduleA/ /__init__.py /tasks.py /moduleB/ /__init__.py /tasks.py Using the default automatic naming, each task will have a generated name like [moduleA.tasks.taskA]{.title-ref}, [moduleA.tasks.taskB]{.title-ref}, [moduleB.tasks.test]{.title-ref}, and so on. You may want to get rid of having [tasks]{.title-ref} in all task names. As pointed above, you can explicitly give names for all tasks, or you can change the automatic naming behavior by overriding `@gen_task_name`{.interpreted-text role="meth"}. Continuing with the example, [celery.py]{.title-ref} may contain: ``` python from celery import Celery class MyCelery(Celery): def gen_task_name(self, name, module): if module.endswith('.tasks'): module = module[:-6] return super().gen_task_name(name, module) app = MyCelery('main') ``` So each task will have a name like [moduleA.taskA]{.title-ref}, [moduleA.taskB]{.title-ref} and [moduleB.test]{.title-ref}. ::: warning ::: title Warning ::: Make sure that your `@gen_task_name`{.interpreted-text role="meth"} is a pure function: meaning that for the same input it must always return the same output. ::: ## Task Request {#task-request-info} `Task.request <@Task.request>`{.interpreted-text role="attr"} contains information and state related to the currently executing task. The request defines the following attributes: id : The unique id of the executing task. group : The unique id of the task\'s `group `{.interpreted-text role="ref"}, if this task is a member. chord : The unique id of the chord this task belongs to (if the task is part of the header). correlation_id : Custom ID used for things like de-duplication. args : Positional arguments. kwargs : Keyword arguments. origin : Name of host that sent this task. retries : How many times the current task has been retried. An integer starting at [0]{.title-ref}. is_eager : Set to `True`{.interpreted-text role="const"} if the task is executed locally in the client, not by a worker. eta : The original ETA of the task (if any). This is in UTC time (depending on the `enable_utc`{.interpreted-text role="setting"} setting). expires : The original expiry time of the task (if any). This is in UTC time (depending on the `enable_utc`{.interpreted-text role="setting"} setting). hostname : Node name of the worker instance executing the task. delivery_info : Additional message delivery information. This is a mapping containing the exchange and routing key used to deliver this task. Used by for example `Task.retry() <@Task.retry>`{.interpreted-text role="meth"} to resend the task to the same destination queue. Availability of keys in this dict depends on the message broker used. reply-to : Name of queue to send replies back to (used with RPC result backend for example). called_directly : This flag is set to true if the task wasn\'t executed by the worker. timelimit : A tuple of the current `(soft, hard)` time limits active for this task (if any). callbacks : A list of signatures to be called if this task returns successfully. errbacks : A list of signatures to be called if this task fails. utc : Set to true the caller has UTC enabled (`enable_utc`{.interpreted-text role="setting"}). ::: versionadded 3.1 ::: headers : Mapping of message headers sent with this task message (may be `None`{.interpreted-text role="const"}). reply_to : Where to send reply to (queue name). correlation_id : Usually the same as the task id, often used in amqp to keep track of what a reply is for. ::: versionadded 4.0 ::: root_id : The unique id of the first task in the workflow this task is part of (if any). parent_id : The unique id of the task that called this task (if any). chain : Reversed list of tasks that form a chain (if any). The last item in this list will be the next task to succeed the current task. If using version one of the task protocol the chain tasks will be in `request.callbacks` instead. ::: versionadded 5.2 ::: properties : Mapping of message properties received with this task message (may be `None`{.interpreted-text role="const"} or `{}`{.interpreted-text role="const"}) replaced_task_nesting : How many times the task was replaced, if at all. (may be `0`{.interpreted-text role="const"}) ### Example An example task accessing information in the context is: ``` python @app.task(bind=True) def dump_context(self, x, y): print('Executing task id {0.id}, args: {0.args!r} kwargs: {0.kwargs!r}'.format( self.request)) ``` The `bind` argument means that the function will be a \"bound method\" so that you can access attributes and methods on the task type instance. ## Logging {#task-logging} The worker will automatically set up logging for you, or you can configure logging manually. A special logger is available named \"celery.task\", you can inherit from this logger to automatically get the task name and unique id as part of the logs. The best practice is to create a common logger for all of your tasks at the top of your module: ``` python from celery.utils.log import get_task_logger logger = get_task_logger(__name__) @app.task def add(x, y): logger.info('Adding {0} + {1}'.format(x, y)) return x + y ``` Celery uses the standard Python logger library, and the documentation can be found `here `{.interpreted-text role="mod"}. You can also use `print`{.interpreted-text role="func"}, as anything written to standard out/-err will be redirected to the logging system (you can disable this, see `worker_redirect_stdouts`{.interpreted-text role="setting"}). ::: note ::: title Note ::: The worker won\'t update the redirection if you create a logger instance somewhere in your task or task module. If you want to redirect `sys.stdout` and `sys.stderr` to a custom logger you have to enable this manually, for example: ``` python import sys logger = get_task_logger(__name__) @app.task(bind=True) def add(self, x, y): old_outs = sys.stdout, sys.stderr rlevel = self.app.conf.worker_redirect_stdouts_level try: self.app.log.redirect_stdouts_to_logger(logger, rlevel) print('Adding {0} + {1}'.format(x, y)) return x + y finally: sys.stdout, sys.stderr = old_outs ``` ::: ::: note ::: title Note ::: If a specific Celery logger you need is not emitting logs, you should check that the logger is propagating properly. In this example \"celery.app.trace\" is enabled so that \"succeeded in\" logs are emitted: ``` python import celery import logging @celery.signals.after_setup_logger.connect def on_after_setup_logger(**kwargs): logger = logging.getLogger('celery') logger.propagate = True logger = logging.getLogger('celery.app.trace') logger.propagate = True ``` ::: ::: note ::: title Note ::: If you want to completely disable Celery logging configuration, use the `setup_logging`{.interpreted-text role="signal"} signal: ``` python import celery @celery.signals.setup_logging.connect def on_setup_logging(**kwargs): pass ``` ::: ### Argument checking {#task-argument-checking} ::: versionadded 4.0 ::: Celery will verify the arguments passed when you call the task, just like Python does when calling a normal function: ``` pycon >>> @app.task ... def add(x, y): ... return x + y # Calling the task with two arguments works: >>> add.delay(8, 8) # Calling the task with only one argument fails: >>> add.delay(8) Traceback (most recent call last): File "", line 1, in File "celery/app/task.py", line 376, in delay return self.apply_async(args, kwargs) File "celery/app/task.py", line 485, in apply_async check_arguments(*(args or ()), **(kwargs or {})) TypeError: add() takes exactly 2 arguments (1 given) ``` You can disable the argument checking for any task by setting its `~@Task.typing`{.interpreted-text role="attr"} attribute to `False`{.interpreted-text role="const"}: ``` pycon >>> @app.task(typing=False) ... def add(x, y): ... return x + y # Works locally, but the worker receiving the task will raise an error. >>> add.delay(8) ``` ### Hiding sensitive information in arguments {#task-hiding-sensitive-information} ::: versionadded 4.0 ::: When using `task_protocol`{.interpreted-text role="setting"} 2 or higher (default since 4.0), you can override how positional arguments and keyword arguments are represented in logs and monitoring events using the `argsrepr` and `kwargsrepr` calling arguments: ``` pycon >>> add.apply_async((2, 3), argsrepr='(, )') >>> charge.s(account, card='1234 5678 1234 5678').set( ... kwargsrepr=repr({'card': '**** **** **** 5678'}) ... ).delay() ``` ::: warning ::: title Warning ::: Sensitive information will still be accessible to anyone able to read your task message from the broker, or otherwise able intercept it. For this reason you should probably encrypt your message if it contains sensitive information, or in this example with a credit card number the actual number could be stored encrypted in a secure store that you retrieve and decrypt in the task itself. ::: ## Retrying {#task-retry} `Task.retry() <@Task.retry>`{.interpreted-text role="meth"} can be used to re-execute the task, for example in the event of recoverable errors. When you call `retry` it\'ll send a new message, using the same task-id, and it\'ll take care to make sure the message is delivered to the same queue as the originating task. When a task is retried this is also recorded as a task state, so that you can track the progress of the task using the result instance (see `task-states`{.interpreted-text role="ref"}). Here\'s an example using `retry`: ``` python @app.task(bind=True) def send_twitter_status(self, oauth, tweet): try: twitter = Twitter(oauth) twitter.update_status(tweet) except (Twitter.FailWhaleError, Twitter.LoginError) as exc: raise self.retry(exc=exc) ``` ::: note ::: title Note ::: The `Task.retry() <@Task.retry>`{.interpreted-text role="meth"} call will raise an exception so any code after the retry won\'t be reached. This is the `~@Retry`{.interpreted-text role="exc"} exception, it isn\'t handled as an error but rather as a semi-predicate to signify to the worker that the task is to be retried, so that it can store the correct state when a result backend is enabled. This is normal operation and always happens unless the `throw` argument to retry is set to `False`{.interpreted-text role="const"}. ::: The bind argument to the task decorator will give access to `self` (the task type instance). The `exc` argument is used to pass exception information that\'s used in logs, and when storing task results. Both the exception and the traceback will be available in the task state (if a result backend is enabled). If the task has a `max_retries` value the current exception will be re-raised if the max number of retries has been exceeded, but this won\'t happen if: - An `exc` argument wasn\'t given. > In this case the `~@MaxRetriesExceededError`{.interpreted-text > role="exc"} exception will be raised. - There\'s no current exception > If there\'s no original exception to re-raise the `exc` argument > will be used instead, so: > > ``` python > self.retry(exc=Twitter.LoginError()) > ``` > > will raise the `exc` argument given. ### Using a custom retry delay {#task-retry-custom-delay} When a task is to be retried, it can wait for a given amount of time before doing so, and the default delay is defined by the `~@Task.default_retry_delay`{.interpreted-text role="attr"} attribute. By default this is set to 3 minutes. Note that the unit for setting the delay is in seconds (int or float). You can also provide the [countdown]{.title-ref} argument to `~@Task.retry`{.interpreted-text role="meth"} to override this default. ``` python @app.task(bind=True, default_retry_delay=30 * 60) # retry in 30 minutes. def add(self, x, y): try: something_raising() except Exception as exc: # overrides the default delay to retry after 1 minute raise self.retry(exc=exc, countdown=60) ``` ### Automatic retry for known exceptions {#task-autoretry} ::: versionadded 4.0 ::: Sometimes you just want to retry a task whenever a particular exception is raised. Fortunately, you can tell Celery to automatically retry a task using [autoretry_for]{.title-ref} argument in the `@task`{.interpreted-text role="meth"} decorator: ``` python from twitter.exceptions import FailWhaleError @app.task(autoretry_for=(FailWhaleError,)) def refresh_timeline(user): return twitter.refresh_timeline(user) ``` If you want to specify custom arguments for an internal `~@Task.retry`{.interpreted-text role="meth"} call, pass [retry_kwargs]{.title-ref} argument to `@task`{.interpreted-text role="meth"} decorator: ``` python @app.task(autoretry_for=(FailWhaleError,), retry_kwargs={'max_retries': 5}) def refresh_timeline(user): return twitter.refresh_timeline(user) ``` This is provided as an alternative to manually handling the exceptions, and the example above will do the same as wrapping the task body in a `try`{.interpreted-text role="keyword"} \... `except`{.interpreted-text role="keyword"} statement: ``` python @app.task def refresh_timeline(user): try: twitter.refresh_timeline(user) except FailWhaleError as exc: raise refresh_timeline.retry(exc=exc, max_retries=5) ``` If you want to automatically retry on any error, simply use: ``` python @app.task(autoretry_for=(Exception,)) def x(): ... ``` ::: versionadded 4.2 ::: If your tasks depend on another service, like making a request to an API, then it\'s a good idea to use [exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff) to avoid overwhelming the service with your requests. Fortunately, Celery\'s automatic retry support makes it easy. Just specify the `~Task.retry_backoff`{.interpreted-text role="attr"} argument, like this: ``` python from requests.exceptions import RequestException @app.task(autoretry_for=(RequestException,), retry_backoff=True) def x(): ... ``` By default, this exponential backoff will also introduce random [jitter](https://en.wikipedia.org/wiki/Jitter) to avoid having all the tasks run at the same moment. It will also cap the maximum backoff delay to 10 minutes. All these settings can be customized via options documented below. ::: versionadded 4.4 ::: You can also set [autoretry_for]{.title-ref}, [max_retries]{.title-ref}, [retry_backoff]{.title-ref}, [retry_backoff_max]{.title-ref} and [retry_jitter]{.title-ref} options in class-based tasks: ``` python class BaseTaskWithRetry(Task): autoretry_for = (TypeError,) max_retries = 5 retry_backoff = True retry_backoff_max = 700 retry_jitter = False ``` ::: attribute Task.autoretry_for A list/tuple of exception classes. If any of these exceptions are raised during the execution of the task, the task will automatically be retried. By default, no exceptions will be autoretried. ::: ::: attribute Task.max_retries A number. Maximum number of retries before giving up. A value of `None` means task will retry forever. By default, this option is set to `3`. ::: ::: attribute Task.retry_backoff A boolean, or a number. If this option is set to `True`, autoretries will be delayed following the rules of [exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff). The first retry will have a delay of 1 second, the second retry will have a delay of 2 seconds, the third will delay 4 seconds, the fourth will delay 8 seconds, and so on. (However, this delay value is modified by `~Task.retry_jitter`{.interpreted-text role="attr"}, if it is enabled.) If this option is set to a number, it is used as a delay factor. For example, if this option is set to `3`, the first retry will delay 3 seconds, the second will delay 6 seconds, the third will delay 12 seconds, the fourth will delay 24 seconds, and so on. By default, this option is set to `False`, and autoretries will not be delayed. ::: ::: attribute Task.retry_backoff_max A number. If `retry_backoff` is enabled, this option will set a maximum delay in seconds between task autoretries. By default, this option is set to `600`, which is 10 minutes. ::: ::: attribute Task.retry_jitter A boolean. [Jitter](https://en.wikipedia.org/wiki/Jitter) is used to introduce randomness into exponential backoff delays, to prevent all tasks in the queue from being executed simultaneously. If this option is set to `True`, the delay value calculated by `~Task.retry_backoff`{.interpreted-text role="attr"} is treated as a maximum, and the actual delay value will be a random number between zero and that maximum. By default, this option is set to `True`. ::: ::: versionadded 5.3.0 ::: ::: attribute Task.dont_autoretry_for A list/tuple of exception classes. These exceptions won\'t be autoretried. This allows to exclude some exceptions that match `autoretry_for `{.interpreted-text role="attr"} but for which you don\'t want a retry. ::: ## Argument validation with Pydantic {#task-pydantic} ::: versionadded 5.5.0 ::: You can use [Pydantic](https://docs.pydantic.dev/) to validate and convert arguments as well as serializing results based on typehints by passing `pydantic=True`. ::: note ::: title Note ::: Argument validation only covers arguments/return values on the task side. You still have serialize arguments yourself when invoking a task with `delay()` or `apply_async()`. ::: For example: ``` python from pydantic import BaseModel class ArgModel(BaseModel): value: int class ReturnModel(BaseModel): value: str @app.task(pydantic=True) def x(arg: ArgModel) -> ReturnModel: # args/kwargs type hinted as Pydantic model will be converted assert isinstance(arg, ArgModel) # The returned model will be converted to a dict automatically return ReturnModel(value=f"example: {arg.value}") ``` The task can then be called using a dict matching the model, and you\'ll receive the returned model \"dumped\" (serialized using `BaseModel.model_dump()`): ``` python >>> result = x.delay({'value': 1}) >>> result.get(timeout=1) {'value': 'example: 1'} ``` ### Union types, arguments to generics Union types (e.g. `Union[SomeModel, OtherModel]`) or arguments to generics (e.g. `list[SomeModel]`) are **not** supported. In case you want to support a list or similar types, it is recommended to use `pydantic.RootModel`. ### Optional parameters/return values Optional parameters or return values are also handled properly. For example, given this task: ``` python from typing import Optional # models are the same as above @app.task(pydantic=True) def x(arg: Optional[ArgModel] = None) -> Optional[ReturnModel]: if arg is None: return None return ReturnModel(value=f"example: {arg.value}") ``` You\'ll get the following behavior: ``` python >>> result = x.delay() >>> result.get(timeout=1) is None True >>> result = x.delay({'value': 1}) >>> result.get(timeout=1) {'value': 'example: 1'} ``` ### Return value handling Return values will only be serialized if the returned model matches the annotation. If you pass a model instance of a different type, it will *not* be serialized. `mypy` should already catch such errors and you should fix your typehints then. ### Pydantic parameters There are a few more options influencing Pydantic behavior: ::: attribute Task.pydantic_strict By default, [strict mode](https://docs.pydantic.dev/dev/concepts/strict_mode/) is disabled. You can pass `True` to enable strict model validation. ::: ::: attribute Task.pydantic_context Pass [additional validation context](https://docs.pydantic.dev/dev/concepts/validators/#validation-context) during Pydantic model validation. The context already includes the application object as `celery_app` and the task name as `celery_task_name` by default. ::: ::: attribute Task.pydantic_dump_kwargs When serializing a result, pass these additional arguments to `dump_kwargs()`. By default, only `mode='json'` is passed. ::: ## List of Options {#task-options} The task decorator can take a number of options that change the way the task behaves, for example you can set the rate limit for a task using the `rate_limit`{.interpreted-text role="attr"} option. Any keyword argument passed to the task decorator will actually be set as an attribute of the resulting task class, and this is a list of the built-in attributes. ### General ::: {#task-general-options} ::: attribute Task.name The name the task is registered as. You can set this name manually, or a name will be automatically generated using the module and class name. See also `task-names`{.interpreted-text role="ref"}. ::: ::: ::: attribute Task.request If the task is being executed this will contain information about the current request. Thread local storage is used. See `task-request-info`{.interpreted-text role="ref"}. ::: ::: attribute Task.max_retries Only applies if the task calls `self.retry` or if the task is decorated with the `autoretry_for `{.interpreted-text role="ref"} argument. The maximum number of attempted retries before giving up. If the number of retries exceeds this value a `~@MaxRetriesExceededError`{.interpreted-text role="exc"} exception will be raised. ::: note ::: title Note ::: You have to call `~@Task.retry`{.interpreted-text role="meth"} manually, as it won\'t automatically retry on exception.. ::: The default is `3`. A value of `None`{.interpreted-text role="const"} will disable the retry limit and the task will retry forever until it succeeds. ::: ::: attribute Task.throws Optional tuple of expected error classes that shouldn\'t be regarded as an actual error. Errors in this list will be reported as a failure to the result backend, but the worker won\'t log the event as an error, and no traceback will be included. Example: ``` python @task(throws=(KeyError, HttpNotFound)): def get_foo(): something() ``` Error types: - Expected errors (in `Task.throws`) > Logged with severity `INFO`, traceback excluded. - Unexpected errors > Logged with severity `ERROR`, with traceback included. ::: ::: attribute Task.default_retry_delay Default time in seconds before a retry of the task should be executed. Can be either `int`{.interpreted-text role="class"} or `float`{.interpreted-text role="class"}. Default is a three minute delay. ::: ::: attribute Task.rate_limit Set the rate limit for this task type (limits the number of tasks that can be run in a given time frame). Tasks will still complete when a rate limit is in effect, but it may take some time before it\'s allowed to start. If this is `None`{.interpreted-text role="const"} no rate limit is in effect. If it is an integer or float, it is interpreted as \"tasks per second\". The rate limits can be specified in seconds, minutes or hours by appending [\"/s\"]{.title-ref}, [\"/m\"]{.title-ref} or [\"/h\"]{.title-ref} to the value. Tasks will be evenly distributed over the specified time frame. Example: [\"100/m\"]{.title-ref} (hundred tasks a minute). This will enforce a minimum delay of 600ms between starting two tasks on the same worker instance. Default is the `task_default_rate_limit`{.interpreted-text role="setting"} setting: if not specified means rate limiting for tasks is disabled by default. Note that this is a *per worker instance* rate limit, and not a global rate limit. To enforce a global rate limit (e.g., for an API with a maximum number of requests per second), you must restrict to a given queue. ::: ::: attribute Task.time_limit The hard time limit, in seconds, for this task. When not set the workers default is used. ::: ::: attribute Task.soft_time_limit The soft time limit for this task. When not set the workers default is used. ::: ::: attribute Task.ignore_result Don\'t store task state. Note that this means you can\'t use `~celery.result.AsyncResult`{.interpreted-text role="class"} to check if the task is ready, or get its return value. Note: Certain features will not work if task results are disabled. For more details check the Canvas documentation. ::: ::: attribute Task.store_errors_even_if_ignored If `True`{.interpreted-text role="const"}, errors will be stored even if the task is configured to ignore results. ::: ::: attribute Task.serializer A string identifying the default serialization method to use. Defaults to the `task_serializer`{.interpreted-text role="setting"} setting. Can be [pickle]{.title-ref}, [json]{.title-ref}, [yaml]{.title-ref}, or any custom serialization methods that have been registered with `kombu.serialization.registry`{.interpreted-text role="mod"}. Please see `calling-serializers`{.interpreted-text role="ref"} for more information. ::: ::: attribute Task.compression A string identifying the default compression scheme to use. Defaults to the `task_compression`{.interpreted-text role="setting"} setting. Can be [gzip]{.title-ref}, or [bzip2]{.title-ref}, or any custom compression schemes that have been registered with the `kombu.compression`{.interpreted-text role="mod"} registry. Please see `calling-compression`{.interpreted-text role="ref"} for more information. ::: ::: attribute Task.backend The result store backend to use for this task. An instance of one of the backend classes in [celery.backends]{.title-ref}. Defaults to [app.backend]{.title-ref}, defined by the `result_backend`{.interpreted-text role="setting"} setting. ::: ::: attribute Task.acks_late If set to `True`{.interpreted-text role="const"} messages for this task will be acknowledged **after** the task has been executed, not *just before* (the default behavior). Note: This means the task may be executed multiple times should the worker crash in the middle of execution. Make sure your tasks are `idempotent`{.interpreted-text role="term"}. The global default can be overridden by the `task_acks_late`{.interpreted-text role="setting"} setting. ::: ::: {#task-track-started} ::: attribute Task.track_started If `True`{.interpreted-text role="const"} the task will report its status as \"started\" when the task is executed by a worker. The default value is `False`{.interpreted-text role="const"} as the normal behavior is to not report that level of granularity. Tasks are either pending, finished, or waiting to be retried. Having a \"started\" status can be useful for when there are long running tasks and there\'s a need to report what task is currently running. The host name and process id of the worker executing the task will be available in the state meta-data (e.g., [result.info\[\'pid\'\]]{.title-ref}) The global default can be overridden by the `task_track_started`{.interpreted-text role="setting"} setting. ::: ::: ::: seealso The API reference for `~@Task`{.interpreted-text role="class"}. ::: ## States {#task-states} Celery can keep track of the tasks current state. The state also contains the result of a successful task, or the exception and traceback information of a failed task. There are several *result backends* to choose from, and they all have different strengths and weaknesses (see `task-result-backends`{.interpreted-text role="ref"}). During its lifetime a task will transition through several possible states, and each state may have arbitrary meta-data attached to it. When a task moves into a new state the previous state is forgotten about, but some transitions can be deduced, (e.g., a task now in the `FAILED`{.interpreted-text role="state"} state, is implied to have been in the `STARTED`{.interpreted-text role="state"} state at some point). There are also sets of states, like the set of `FAILURE_STATES`{.interpreted-text role="state"}, and the set of `READY_STATES`{.interpreted-text role="state"}. The client uses the membership of these sets to decide whether the exception should be re-raised (`PROPAGATE_STATES`{.interpreted-text role="state"}), or whether the state can be cached (it can if the task is ready). You can also define `custom-states`{.interpreted-text role="ref"}. ### Result Backends {#task-result-backends} If you want to keep track of tasks or need the return values, then Celery must store or send the states somewhere so that they can be retrieved later. There are several built-in result backends to choose from: SQLAlchemy/Django ORM, Memcached, RabbitMQ/QPid (`rpc`), and Redis \-- or you can define your own. No backend works well for every use case. You should read about the strengths and weaknesses of each backend, and choose the most appropriate for your needs. ::: warning ::: title Warning ::: Backends use resources to store and transmit results. To ensure that resources are released, you must eventually call `~@AsyncResult.get`{.interpreted-text role="meth"} or `~@AsyncResult.forget`{.interpreted-text role="meth"} on EVERY `~@AsyncResult`{.interpreted-text role="class"} instance returned after calling a task. ::: ::: seealso `conf-result-backend`{.interpreted-text role="ref"} ::: #### RPC Result Backend (RabbitMQ/QPid) The RPC result backend ([rpc://]{.title-ref}) is special as it doesn\'t actually *store* the states, but rather sends them as messages. This is an important difference as it means that a result *can only be retrieved once*, and *only by the client that initiated the task*. Two different processes can\'t wait for the same result. Even with that limitation, it is an excellent choice if you need to receive state changes in real-time. Using messaging means the client doesn\'t have to poll for new states. The messages are transient (non-persistent) by default, so the results will disappear if the broker restarts. You can configure the result backend to send persistent messages using the `result_persistent`{.interpreted-text role="setting"} setting. #### Database Result Backend Keeping state in the database can be convenient for many, especially for web applications with a database already in place, but it also comes with limitations. - Polling the database for new states is expensive, and so you should increase the polling intervals of operations, such as [result.get()]{.title-ref}. - Some databases use a default transaction isolation level that isn\'t suitable for polling tables for changes. In MySQL the default transaction isolation level is \`REPEATABLE-READ\`: meaning the transaction won\'t see changes made by other transactions until the current transaction is committed. Changing that to the [READ-COMMITTED]{.title-ref} isolation level is recommended. ### Built-in States {#task-builtin-states} ::: state PENDING ::: #### PENDING Task is waiting for execution or unknown. Any task id that\'s not known is implied to be in the pending state. ::: state STARTED ::: #### STARTED Task has been started. Not reported by default, to enable please see `@Task.track_started`{.interpreted-text role="attr"}. meta-data : [pid]{.title-ref} and [hostname]{.title-ref} of the worker process executing the task. ::: state SUCCESS ::: #### SUCCESS Task has been successfully executed. meta-data : [result]{.title-ref} contains the return value of the task. propagates : Yes ready : Yes ::: state FAILURE ::: #### FAILURE Task execution resulted in failure. meta-data : [result]{.title-ref} contains the exception occurred, and [traceback]{.title-ref} contains the backtrace of the stack at the point when the exception was raised. propagates : Yes ::: state RETRY ::: #### RETRY Task is being retried. meta-data : [result]{.title-ref} contains the exception that caused the retry, and [traceback]{.title-ref} contains the backtrace of the stack at the point when the exceptions was raised. propagates : No ::: state REVOKED ::: #### REVOKED Task has been revoked. propagates : Yes ### Custom states You can easily define your own states, all you need is a unique name. The name of the state is usually an uppercase string. As an example you could have a look at the `abortable tasks <~celery.contrib.abortable>`{.interpreted-text role="mod"} which defines a custom `ABORTED`{.interpreted-text role="state"} state. Use `~@Task.update_state`{.interpreted-text role="meth"} to update a task\'s state:. ``` python @app.task(bind=True) def upload_files(self, filenames): for i, file in enumerate(filenames): if not self.request.called_directly: self.update_state(state='PROGRESS', meta={'current': i, 'total': len(filenames)}) ``` Here I created the state [\"PROGRESS\"]{.title-ref}, telling any application aware of this state that the task is currently in progress, and also where it is in the process by having [current]{.title-ref} and [total]{.title-ref} counts as part of the state meta-data. This can then be used to create progress bars for example. ### Creating pickleable exceptions {#pickling_exceptions} A rarely known Python fact is that exceptions must conform to some simple rules to support being serialized by the pickle module. Tasks that raise exceptions that aren\'t pickleable won\'t work properly when Pickle is used as the serializer. To make sure that your exceptions are pickleable the exception *MUST* provide the original arguments it was instantiated with in its `.args` attribute. The simplest way to ensure this is to have the exception call `Exception.__init__`. Let\'s look at some examples that work, and one that doesn\'t: ``` python # OK: class HttpError(Exception): pass # BAD: class HttpError(Exception): def __init__(self, status_code): self.status_code = status_code # OK: class HttpError(Exception): def __init__(self, status_code): self.status_code = status_code Exception.__init__(self, status_code) # <-- REQUIRED ``` So the rule is: For any exception that supports custom arguments `*args`, `Exception.__init__(self, *args)` must be used. There\'s no special support for *keyword arguments*, so if you want to preserve keyword arguments when the exception is unpickled you have to pass them as regular args: ``` python class HttpError(Exception): def __init__(self, status_code, headers=None, body=None): self.status_code = status_code self.headers = headers self.body = body super(HttpError, self).__init__(status_code, headers, body) ``` ## Semipredicates {#task-semipredicates} The worker wraps the task in a tracing function that records the final state of the task. There are a number of exceptions that can be used to signal this function to change how it treats the return of the task. ### Ignore {#task-semipred-ignore} The task may raise `~@Ignore`{.interpreted-text role="exc"} to force the worker to ignore the task. This means that no state will be recorded for the task, but the message is still acknowledged (removed from queue). This can be used if you want to implement custom revoke-like functionality, or manually store the result of a task. Example keeping revoked tasks in a Redis set: ``` python from celery.exceptions import Ignore @app.task(bind=True) def some_task(self): if redis.ismember('tasks.revoked', self.request.id): raise Ignore() ``` Example that stores results manually: ``` python from celery import states from celery.exceptions import Ignore @app.task(bind=True) def get_tweets(self, user): timeline = twitter.get_timeline(user) if not self.request.called_directly: self.update_state(state=states.SUCCESS, meta=timeline) raise Ignore() ``` ### Reject {#task-semipred-reject} The task may raise `~@Reject`{.interpreted-text role="exc"} to reject the task message using AMQPs `basic_reject` method. This won\'t have any effect unless `Task.acks_late`{.interpreted-text role="attr"} is enabled. Rejecting a message has the same effect as acking it, but some brokers may implement additional functionality that can be used. For example RabbitMQ supports the concept of [Dead Letter Exchanges](http://www.rabbitmq.com/dlx.html) where a queue can be configured to use a dead letter exchange that rejected messages are redelivered to. Reject can also be used to re-queue messages, but please be very careful when using this as it can easily result in an infinite message loop. Example using reject when a task causes an out of memory condition: ``` python import errno from celery.exceptions import Reject @app.task(bind=True, acks_late=True) def render_scene(self, path): file = get_file(path) try: renderer.render_scene(file) # if the file is too big to fit in memory # we reject it so that it's redelivered to the dead letter exchange # and we can manually inspect the situation. except MemoryError as exc: raise Reject(exc, requeue=False) except OSError as exc: if exc.errno == errno.ENOMEM: raise Reject(exc, requeue=False) # For any other error we retry after 10 seconds. except Exception as exc: raise self.retry(exc, countdown=10) ``` Example re-queuing the message: ``` python from celery.exceptions import Reject @app.task(bind=True, acks_late=True) def requeues(self): if not self.request.delivery_info['redelivered']: raise Reject('no reason', requeue=True) print('received two times') ``` Consult your broker documentation for more details about the `basic_reject` method. ### Retry {#task-semipred-retry} The `~@Retry`{.interpreted-text role="exc"} exception is raised by the `Task.retry` method to tell the worker that the task is being retried. ## Custom task classes {#task-custom-classes} All tasks inherit from the `@Task`{.interpreted-text role="class"} class. The `~@Task.run`{.interpreted-text role="meth"} method becomes the task body. As an example, the following code, ``` python @app.task def add(x, y): return x + y ``` will do roughly this behind the scenes: ``` python class _AddTask(app.Task): def run(self, x, y): return x + y add = app.tasks[_AddTask.name] ``` ### Instantiation A task is **not** instantiated for every request, but is registered in the task registry as a global instance. This means that the `__init__` constructor will only be called once per process, and that the task class is semantically closer to an Actor. If you have a task, ``` python from celery import Task class NaiveAuthenticateServer(Task): def __init__(self): self.users = {'george': 'password'} def run(self, username, password): try: return self.users[username] == password except KeyError: return False ``` And you route every request to the same process, then it will keep state between requests. This can also be useful to cache resources, For example, a base Task class that caches a database connection: ``` python from celery import Task class DatabaseTask(Task): _db = None @property def db(self): if self._db is None: self._db = Database.connect() return self._db ``` #### Per task usage The above can be added to each task like this: ``` python from celery.app import task @app.task(base=DatabaseTask, bind=True) def process_rows(self: task): for row in self.db.table.all(): process_row(row) ``` The `db` attribute of the `process_rows` task will then always stay the same in each process. #### App-wide usage {#custom-task-cls-app-wide} You can also use your custom class in your whole Celery app by passing it as the `task_cls` argument when instantiating the app. This argument should be either a string giving the python path to your Task class or the class itself: ``` python from celery import Celery app = Celery('tasks', task_cls='your.module.path:DatabaseTask') ``` This will make all your tasks declared using the decorator syntax within your app to use your `DatabaseTask` class and will all have a `db` attribute. The default value is the class provided by Celery: `'celery.app.task:Task'`. ### Handlers Task handlers are methods that execute at specific points in a task\'s lifecycle. All handlers run **synchronously** within the same worker process and thread that executes the task. #### Execution timeline The following diagram shows the exact order of execution: ``` text Worker Process Timeline ┌───────────────────────────────────────────────────────────────┐ │ 1. before_start() ← Blocks until complete │ │ 2. run() ← Your task function │ │ 3. [Result Backend] ← State + return value persisted │ │ 4. on_success() OR ← Outcome-specific handler │ │ on_retry() OR │ │ │ on_failure() │ │ │ 5. after_return() ← Always runs last │ └───────────────────────────────────────────────────────────────┘ ``` ::: important ::: title Important ::: **Key points:** - All handlers run in the **same worker process** as your task - `before_start` **blocks** the task - `run()` won\'t start until it completes - Result backend is updated **before** `on_success`/`on_failure` - other clients can see the task as finished while handlers are still running - `after_return` **always** executes, regardless of task outcome ::: #### Available handlers ::: method before_start(self, task_id, args, kwargs) Run by the worker before the task starts executing. ::: note ::: title Note ::: This handler **blocks** the task: the `run`{.interpreted-text role="py:meth"} method will *not* begin until `before_start` returns. ::: ::: versionadded 5.2 ::: param task_id : Unique id of the task to execute. param args : Original arguments for the task to execute. param kwargs : Original keyword arguments for the task to execute. The return value of this handler is ignored. ::: ::: method on_success(self, retval, task_id, args, kwargs) Success handler. Run by the worker if the task executes successfully. ::: note ::: title Note ::: Invoked **after** the task result has already been persisted in the result backend. External clients may observe the task as `SUCCESS` while this handler is still running. ::: param retval : The return value of the task. param task_id : Unique id of the executed task. param args : Original arguments for the executed task. param kwargs : Original keyword arguments for the executed task. The return value of this handler is ignored. ::: ::: method on_retry(self, exc, task_id, args, kwargs, einfo) Retry handler. Run by the worker when the task is to be retried. ::: note ::: title Note ::: Invoked **after** the task state has been updated to `RETRY` in the result backend but **before** the retry is scheduled. ::: param exc : The exception sent to `retry`{.interpreted-text role="meth"}. param task_id : Unique id of the retried task. param args : Original arguments for the retried task. param kwargs : Original keyword arguments for the retried task. param einfo : `~billiard.einfo.ExceptionInfo`{.interpreted-text role="class"} instance. The return value of this handler is ignored. ::: ::: method on_failure(self, exc, task_id, args, kwargs, einfo) Failure handler. Run by the worker when the task fails. ::: note ::: title Note ::: Invoked **after** the task result has already been persisted in the result backend with `FAILURE` state. External clients may observe the task as failed while this handler is still running. ::: param exc : The exception raised by the task. param task_id : Unique id of the failed task. param args : Original arguments for the failed task. param kwargs : Original keyword arguments for the failed task. param einfo : `~billiard.einfo.ExceptionInfo`{.interpreted-text role="class"} instance. The return value of this handler is ignored. ::: ::: method after_return(self, status, retval, task_id, args, kwargs, einfo) Handler called after the task returns. ::: note ::: title Note ::: Executes **after** `on_success`/`on_retry`/`on_failure`. This is the final hook in the task lifecycle and **always** runs, regardless of outcome. ::: param status : Current task state. param retval : Task return value/exception. param task_id : Unique id of the task. param args : Original arguments for the task that returned. param kwargs : Original keyword arguments for the task that returned. param einfo : `~billiard.einfo.ExceptionInfo`{.interpreted-text role="class"} instance. The return value of this handler is ignored. ::: #### Example usage ``` python import time from celery import Task class MyTask(Task): def before_start(self, task_id, args, kwargs): print(f"Task {task_id} starting with args {args}") # This blocks - run() won't start until this returns def on_success(self, retval, task_id, args, kwargs): print(f"Task {task_id} succeeded with result: {retval}") # Result is already visible to clients at this point def on_failure(self, exc, task_id, args, kwargs, einfo): print(f"Task {task_id} failed: {exc}") # Task state is already FAILURE in backend def after_return(self, status, retval, task_id, args, kwargs, einfo): print(f"Task {task_id} finished with status: {status}") # Always runs last @app.task(base=MyTask) def my_task(x, y): return x + y ``` ### Requests and custom requests {#task-requests-and-custom-requests} Upon receiving a message to run a task, the `worker `{.interpreted-text role="ref"} creates a `request `{.interpreted-text role="class"} to represent such demand. Custom task classes may override which request class to use by changing the attribute `celery.app.task.Task.Request`{.interpreted-text role="attr"}. You may either assign the custom request class itself, or its fully qualified name. The request has several responsibilities. Custom request classes should cover them all \-- they are responsible to actually run and trace the task. We strongly recommend to inherit from `celery.worker.request.Request`{.interpreted-text role="class"}. When using the `pre-forking worker `{.interpreted-text role="ref"}, the methods `~celery.worker.request.Request.on_timeout`{.interpreted-text role="meth"} and `~celery.worker.request.Request.on_failure`{.interpreted-text role="meth"} are executed in the main worker process. An application may leverage such facility to detect failures which are not detected using `celery.app.task.Task.on_failure`{.interpreted-text role="meth"}. As an example, the following custom request detects and logs hard time limits, and other failures. ``` python import logging from celery import Task from celery.worker.request import Request logger = logging.getLogger('my.package') class MyRequest(Request): 'A minimal custom request to log failures and hard time limits.' def on_timeout(self, soft, timeout): super(MyRequest, self).on_timeout(soft, timeout) if not soft: logger.warning( 'A hard timeout was enforced for task %s', self.task.name ) def on_failure(self, exc_info, send_failed_event=True, return_ok=False): super().on_failure( exc_info, send_failed_event=send_failed_event, return_ok=return_ok ) logger.warning( 'Failure detected for task %s', self.task.name ) class MyTask(Task): Request = MyRequest # you can use a FQN 'my.package:MyRequest' @app.task(base=MyTask) def some_longrunning_task(): # use your imagination ``` ## How it works {#task-how-they-work} Here come the technical details. This part isn\'t something you need to know, but you may be interested. All defined tasks are listed in a registry. The registry contains a list of task names and their task classes. You can investigate this registry yourself: ``` pycon >>> from proj.celery import app >>> app.tasks {'celery.chord_unlock': <@task: celery.chord_unlock>, 'celery.backend_cleanup': <@task: celery.backend_cleanup>, 'celery.chord': <@task: celery.chord>} ``` This is the list of tasks built into Celery. Note that tasks will only be registered when the module they\'re defined in is imported. The default loader imports any modules listed in the `imports`{.interpreted-text role="setting"} setting. The `@task`{.interpreted-text role="meth"} decorator is responsible for registering your task in the applications task registry. When tasks are sent, no actual function code is sent with it, just the name of the task to execute. When the worker then receives the message it can look up the name in its task registry to find the execution code. This means that your workers should always be updated with the same software as the client. This is a drawback, but the alternative is a technical challenge that\'s yet to be solved. ## Tips and Best Practices {#task-best-practices} ### Ignore results you don\'t want {#task-ignore_results} If you don\'t care about the results of a task, be sure to set the `~@Task.ignore_result`{.interpreted-text role="attr"} option, as storing results wastes time and resources. ``` python @app.task(ignore_result=True) def mytask(): something() ``` Results can even be disabled globally using the `task_ignore_result`{.interpreted-text role="setting"} setting. ::: versionadded 4.2 ::: Results can be enabled/disabled on a per-execution basis, by passing the `ignore_result` boolean parameter, when calling `apply_async`. ``` python @app.task def mytask(x, y): return x + y # No result will be stored result = mytask.apply_async((1, 2), ignore_result=True) print(result.get()) # -> None # Result will be stored result = mytask.apply_async((1, 2), ignore_result=False) print(result.get()) # -> 3 ``` By default tasks will *not ignore results* (`ignore_result=False`) when a result backend is configured. The option precedence order is the following: 1. Global `task_ignore_result`{.interpreted-text role="setting"} 2. `~@Task.ignore_result`{.interpreted-text role="attr"} option 3. Task execution option `ignore_result` ### More optimization tips You find additional optimization tips in the `Optimizing Guide `{.interpreted-text role="ref"}. ### Avoid launching synchronous subtasks {#task-synchronous-subtasks} Having a task wait for the result of another task is really inefficient, and may even cause a deadlock if the worker pool is exhausted. Make your design asynchronous instead, for example by using *callbacks*. **Bad**: ``` python @app.task def update_page_info(url): page = fetch_page.delay(url).get() info = parse_page.delay(page).get() store_page_info.delay(url, info) @app.task def fetch_page(url): return myhttplib.get(url) @app.task def parse_page(page): return myparser.parse_document(page) @app.task def store_page_info(url, info): return PageInfo.objects.create(url, info) ``` **Good**: ``` python def update_page_info(url): # fetch_page -> parse_page -> store_page chain = fetch_page.s(url) | parse_page.s() | store_page_info.s(url) chain() @app.task() def fetch_page(url): return myhttplib.get(url) @app.task() def parse_page(page): return myparser.parse_document(page) @app.task(ignore_result=True) def store_page_info(info, url): PageInfo.objects.create(url=url, info=info) ``` Here I instead created a chain of tasks by linking together different `~celery.signature`{.interpreted-text role="func"}\'s. You can read about chains and other powerful constructs at `designing-workflows`{.interpreted-text role="ref"}. By default Celery will not allow you to run subtasks synchronously within a task, but in rare or extreme cases you might need to do so. **WARNING**: enabling subtasks to run synchronously is not recommended! ``` python @app.task def update_page_info(url): page = fetch_page.delay(url).get(disable_sync_subtasks=False) info = parse_page.delay(page).get(disable_sync_subtasks=False) store_page_info.delay(url, info) @app.task def fetch_page(url): return myhttplib.get(url) @app.task def parse_page(page): return myparser.parse_document(page) @app.task def store_page_info(url, info): return PageInfo.objects.create(url, info) ``` ## Performance and Strategies {#task-performance-and-strategies} ### Granularity {#task-granularity} The task granularity is the amount of computation needed by each subtask. In general it is better to split the problem up into many small tasks rather than have a few long running tasks. With smaller tasks you can process more tasks in parallel and the tasks won\'t run long enough to block the worker from processing other waiting tasks. However, executing a task does have overhead. A message needs to be sent, data may not be local, etc. So if the tasks are too fine-grained the overhead added probably removes any benefit. ::: seealso The book [Art of Concurrency](http://oreilly.com/catalog/9780596521547) has a section dedicated to the topic of task granularity [\[AOC1\]](#AOC1){.citation}. ::: ### Data locality {#task-data-locality} The worker processing the task should be as close to the data as possible. The best would be to have a copy in memory, the worst would be a full transfer from another continent. If the data is far away, you could try to run another worker at location, or if that\'s not possible - cache often used data, or preload data you know is going to be used. The easiest way to share data between workers is to use a distributed cache system, like [memcached](http://memcached.org/). ::: seealso The paper [Distributed Computing Economics](http://research.microsoft.com/pubs/70001/tr-2003-24.pdf) by Jim Gray is an excellent introduction to the topic of data locality. ::: ### State {#task-state} Since Celery is a distributed system, you can\'t know which process, or on what machine the task will be executed. You can\'t even know if the task will run in a timely manner. The ancient async sayings tells us that "asserting the world is the responsibility of the task". What this means is that the world view may have changed since the task was requested, so the task is responsible for making sure the world is how it should be; If you have a task that re-indexes a search engine, and the search engine should only be re-indexed at maximum every 5 minutes, then it must be the tasks responsibility to assert that, not the callers. Another gotcha is Django model objects. They shouldn\'t be passed on as arguments to tasks. It\'s almost always better to re-fetch the object from the database when the task is running instead, as using old data may lead to race conditions. Imagine the following scenario where you have an article and a task that automatically expands some abbreviations in it: ``` python class Article(models.Model): title = models.CharField() body = models.TextField() @app.task def expand_abbreviations(article): article.body.replace('MyCorp', 'My Corporation') article.save() ``` First, an author creates an article and saves it, then the author clicks on a button that initiates the abbreviation task: ``` pycon >>> article = Article.objects.get(id=102) >>> expand_abbreviations.delay(article) ``` Now, the queue is very busy, so the task won\'t be run for another 2 minutes. In the meantime another author makes changes to the article, so when the task is finally run, the body of the article is reverted to the old version because the task had the old body in its argument. Fixing the race condition is easy, just use the article id instead, and re-fetch the article in the task body: ``` python @app.task def expand_abbreviations(article_id): article = Article.objects.get(id=article_id) article.body.replace('MyCorp', 'My Corporation') article.save() ``` ``` pycon >>> expand_abbreviations.delay(article_id) ``` There might even be performance benefits to this approach, as sending large messages may be expensive. ### Database transactions {#task-database-transactions} Let\'s have a look at another example: ``` python from django.db import transaction from django.http import HttpResponseRedirect @transaction.atomic def create_article(request): article = Article.objects.create() expand_abbreviations.delay(article.pk) return HttpResponseRedirect('/articles/') ``` This is a Django view creating an article object in the database, then passing the primary key to a task. It uses the [transaction.atomic]{.title-ref} decorator, that will commit the transaction when the view returns, or roll back if the view raises an exception. There is a race condition because transactions are atomic. This means the article object is not persisted to the database until after the view function returns a response. If the asynchronous task starts executing before the transaction is committed, it may attempt to query the article object before it exists. To prevent this, we need to ensure that the transaction is committed before triggering the task. The solution is to use `~celery.contrib.django.task.DjangoTask.delay_on_commit`{.interpreted-text role="meth"} instead: ``` python from django.db import transaction from django.http import HttpResponseRedirect @transaction.atomic def create_article(request): article = Article.objects.create() expand_abbreviations.delay_on_commit(article.pk) return HttpResponseRedirect('/articles/') ``` This method was added in Celery 5.4. It\'s a shortcut that uses Django\'s `on_commit` callback to launch your Celery task once all transactions have been committed successfully. #### With Celery \<5.4 If you\'re using an older version of Celery, you can replicate this behaviour using the Django callback directly as follows: ``` python import functools from django.db import transaction from django.http import HttpResponseRedirect @transaction.atomic def create_article(request): article = Article.objects.create() transaction.on_commit( functools.partial(expand_abbreviations.delay, article.pk) ) return HttpResponseRedirect('/articles/') ``` ::: note ::: title Note ::: `on_commit` is available in Django 1.9 and above, if you are using a version prior to that then the [django-transaction-hooks](https://github.com/carljm/django-transaction-hooks) library adds support for this. ::: ## Example {#task-example} Let\'s take a real world example: a blog where comments posted need to be filtered for spam. When the comment is created, the spam filter runs in the background, so the user doesn\'t have to wait for it to finish. I have a Django blog application allowing comments on blog posts. I\'ll describe parts of the models/views and tasks for this application. ### `blog/models.py` The comment model looks like this: ``` python from django.db import models from django.utils.translation import ugettext_lazy as _ class Comment(models.Model): name = models.CharField(_('name'), max_length=64) email_address = models.EmailField(_('email address')) homepage = models.URLField(_('home page'), blank=True, verify_exists=False) comment = models.TextField(_('comment')) pub_date = models.DateTimeField(_('Published date'), editable=False, auto_add_now=True) is_spam = models.BooleanField(_('spam?'), default=False, editable=False) class Meta: verbose_name = _('comment') verbose_name_plural = _('comments') ``` In the view where the comment is posted, I first write the comment to the database, then I launch the spam filter task in the background. ### `blog/views.py` {#task-example-blog-views} ``` python from django import forms from django.http import HttpResponseRedirect from django.template.context import RequestContext from django.shortcuts import get_object_or_404, render_to_response from blog import tasks from blog.models import Comment class CommentForm(forms.ModelForm): class Meta: model = Comment def add_comment(request, slug, template_name='comments/create.html'): post = get_object_or_404(Entry, slug=slug) remote_addr = request.META.get('REMOTE_ADDR') if request.method == 'post': form = CommentForm(request.POST, request.FILES) if form.is_valid(): comment = form.save() # Check spam asynchronously. tasks.spam_filter.delay(comment_id=comment.id, remote_addr=remote_addr) return HttpResponseRedirect(post.get_absolute_url()) else: form = CommentForm() context = RequestContext(request, {'form': form}) return render_to_response(template_name, context_instance=context) ``` To filter spam in comments I use [Akismet](http://akismet.com/faq/), the service used to filter spam in comments posted to the free blog platform [Wordpress]{.title-ref}. [Akismet](http://akismet.com/faq/) is free for personal use, but for commercial use you need to pay. You have to sign up to their service to get an API key. To make API calls to [Akismet](http://akismet.com/faq/) I use the [akismet.py](http://www.voidspace.org.uk/downloads/akismet.py) library written by [Michael Foord](http://www.voidspace.org.uk/). ### `blog/tasks.py` {#task-example-blog-tasks} ``` python from celery import Celery from akismet import Akismet from django.core.exceptions import ImproperlyConfigured from django.contrib.sites.models import Site from blog.models import Comment app = Celery(broker='amqp://') @app.task def spam_filter(comment_id, remote_addr=None): logger = spam_filter.get_logger() logger.info('Running spam filter for comment %s', comment_id) comment = Comment.objects.get(pk=comment_id) current_domain = Site.objects.get_current().domain akismet = Akismet(settings.AKISMET_KEY, 'http://{0}'.format(domain)) if not akismet.verify_key(): raise ImproperlyConfigured('Invalid AKISMET_KEY') is_spam = akismet.comment_check(user_ip=remote_addr, comment_content=comment.comment, comment_author=comment.name, comment_author_email=comment.email_address) if is_spam: comment.is_spam = True comment.save() return is_spam ``` ::: {#citations} [AOC1]{#AOC1 .citation-label} : Breshears, Clay. Section 2.2.1, \"The Art of Concurrency\". O\'Reilly Media, Inc. May 15, 2009. ISBN-13 978-0-596-52153-0. ::: --- # Testing with Celery {#testing} Testing with Celery is divided into two parts: > - Unit & Integration: Using `celery.contrib.pytest`. > - Smoke / Production: Using > `pytest-celery `{.interpreted-text role="pypi"} \>= > 1.0.0 Installing the pytest-celery plugin will install the `celery.contrib.pytest` infrastructure as well, alongside the pytest plugin infrastructure. The difference is how you use it. ::: warning ::: title Warning ::: Both APIs are NOT compatible with each other. The pytest-celery plugin is Docker based and the `celery.contrib.pytest` is mock based. ::: To use the `celery.contrib.pytest` infrastructure, follow the instructions below. The pytest-celery plugin has its [own documentation](https://pytest-celery.readthedocs.io/). ## Tasks and unit tests To test task behavior in unit tests the preferred method is mocking. ::: admonition Eager mode The eager mode enabled by the `task_always_eager`{.interpreted-text role="setting"} setting is by definition not suitable for unit tests. When testing with eager mode you are only testing an emulation of what happens in a worker, and there are many discrepancies between the emulation and what happens in reality. Note that eagerly executed tasks don\'t write results to backend by default. If you want to enable this functionality, have a look at `task_store_eager_result`{.interpreted-text role="setting"}. ::: A Celery task is much like a web view, in that it should only define how to perform the action in the context of being called as a task. This means optimally tasks only handle things like serialization, message headers, retries, and so on, with the actual logic implemented elsewhere. Say we had a task like this: ``` python from .models import Product @app.task(bind=True) def send_order(self, product_pk, quantity, price): price = Decimal(price) # json serializes this to string. # models are passed by id, not serialized. product = Product.objects.get(product_pk) try: product.order(quantity, price) except OperationalError as exc: raise self.retry(exc=exc) ``` `Note`: A task being [bound](https://docs.celeryq.dev/en/latest/userguide/tasks.html#bound-tasks) means the first argument to the task will always be the task instance (self). which means you do get a self argument as the first argument and can use the Task class methods and attributes. You could write unit tests for this task, using mocking like in this example: ``` python from pytest import raises from celery.exceptions import Retry # for python 2: use mock.patch from `pip install mock`. from unittest.mock import patch from proj.models import Product from proj.tasks import send_order class test_send_order: @patch('proj.tasks.Product.order') # < patching Product in module above def test_success(self, product_order): product = Product.objects.create( name='Foo', ) send_order(product.pk, 3, Decimal(30.3)) product_order.assert_called_with(3, Decimal(30.3)) @patch('proj.tasks.Product.order') @patch('proj.tasks.send_order.retry') def test_failure(self, send_order_retry, product_order): product = Product.objects.create( name='Foo', ) # Set a side effect on the patched methods # so that they raise the errors we want. send_order_retry.side_effect = Retry() product_order.side_effect = OperationalError() with raises(Retry): send_order(product.pk, 3, Decimal(30.6)) ``` ## pytest {#pytest_plugin} ::: versionadded 4.0 ::: Celery also makes a `pytest`{.interpreted-text role="pypi"} plugin available that adds fixtures that you can use in your integration (or unit) test suites. ### Enabling Celery initially ships the plugin in a disabled state. To enable it, you can either: > - `pip install celery[pytest]` > - or add an environment variable > `PYTEST_PLUGINS=celery.contrib.pytest` > - or add `pytest_plugins = ("celery.contrib.pytest", )` to your root > conftest.py ### Marks #### `celery` - Set test app configuration. The `celery` mark enables you to override the configuration used for a single test case: ``` python @pytest.mark.celery(result_backend='redis://') def test_something(): ... ``` or for all the test cases in a class: ``` python @pytest.mark.celery(result_backend='redis://') class test_something: def test_one(self): ... def test_two(self): ... ``` ### Fixtures #### Function scope ##### `celery_app` - Celery app used for testing. This fixture returns a Celery app you can use for testing. Example: ``` python def test_create_task(celery_app, celery_worker): @celery_app.task def mul(x, y): return x * y celery_worker.reload() assert mul.delay(4, 4).get(timeout=10) == 16 ``` ##### `celery_worker` - Embed live worker. This fixture starts a Celery worker instance that you can use for integration tests. The worker will be started in a *separate thread* and will be shutdown as soon as the test returns. By default the fixture will wait up to 10 seconds for the worker to complete outstanding tasks and will raise an exception if the time limit is exceeded. The timeout can be customized by setting the `shutdown_timeout` key in the dictionary returned by the `celery_worker_parameters`{.interpreted-text role="func"} fixture. Example: ``` python # Put this in your conftest.py @pytest.fixture(scope='session') def celery_config(): return { 'broker_url': 'amqp://', 'result_backend': 'redis://' } def test_add(celery_worker): mytask.delay() # If you wish to override some setting in one test cases # only - you can use the ``celery`` mark: @pytest.mark.celery(result_backend='rpc') def test_other(celery_worker): ... ``` Heartbeats are disabled by default which means that the test worker doesn\'t send events for `worker-online`, `worker-offline` and `worker-heartbeat`. To enable heartbeats modify the `celery_worker_parameters`{.interpreted-text role="func"} fixture: ``` python # Put this in your conftest.py @pytest.fixture(scope="session") def celery_worker_parameters(): return {"without_heartbeat": False} ... ``` #### Session scope ##### `celery_config` - Override to setup Celery test app configuration. You can redefine this fixture to configure the test Celery app. The config returned by your fixture will then be used to configure the `celery_app`{.interpreted-text role="func"}, and `celery_session_app`{.interpreted-text role="func"} fixtures. Example: ``` python @pytest.fixture(scope='session') def celery_config(): return { 'broker_url': 'amqp://', 'result_backend': 'rpc', } ``` ##### `celery_parameters` - Override to setup Celery test app parameters. You can redefine this fixture to change the `__init__` parameters of test Celery app. In contrast to `celery_config`{.interpreted-text role="func"}, these are directly passed to when instantiating `~celery.Celery`{.interpreted-text role="class"}. The config returned by your fixture will then be used to configure the `celery_app`{.interpreted-text role="func"}, and `celery_session_app`{.interpreted-text role="func"} fixtures. Example: ``` python @pytest.fixture(scope='session') def celery_parameters(): return { 'task_cls': my.package.MyCustomTaskClass, 'strict_typing': False, } ``` ##### `celery_worker_parameters` - Override to setup Celery worker parameters. You can redefine this fixture to change the `__init__` parameters of test Celery workers. These are directly passed to `~celery.worker.WorkController`{.interpreted-text role="class"} when it is instantiated. The config returned by your fixture will then be used to configure the `celery_worker`{.interpreted-text role="func"}, and `celery_session_worker`{.interpreted-text role="func"} fixtures. Example: ``` python @pytest.fixture(scope='session') def celery_worker_parameters(): return { 'queues': ('high-prio', 'low-prio'), 'exclude_queues': ('celery'), } ``` ##### `celery_enable_logging` - Override to enable logging in embedded workers. This is a fixture you can override to enable logging in embedded workers. Example: ``` python @pytest.fixture(scope='session') def celery_enable_logging(): return True ``` ##### `celery_includes` - Add additional imports for embedded workers. You can override fixture to include modules when an embedded worker starts. You can have this return a list of module names to import, which can be task modules, modules registering signals, and so on. Example: ``` python @pytest.fixture(scope='session') def celery_includes(): return [ 'proj.tests.tasks', 'proj.tests.celery_signal_handlers', ] ``` ##### `celery_worker_pool` - Override the pool used for embedded workers. You can override fixture to configure the execution pool used for embedded workers. Example: ``` python @pytest.fixture(scope='session') def celery_worker_pool(): return 'prefork' ``` ::: warning ::: title Warning ::: You cannot use the gevent/eventlet pools, that is unless your whole test suite is running with the monkeypatches enabled. ::: ##### `celery_session_worker` - Embedded worker that lives throughout the session. This fixture starts a worker that lives throughout the testing session (it won\'t be started/stopped for every test). Example: ``` python # Add this to your conftest.py @pytest.fixture(scope='session') def celery_config(): return { 'broker_url': 'amqp://', 'result_backend': 'rpc', } # Do this in your tests. def test_add_task(celery_session_worker): assert add.delay(2, 2).get() == 4 ``` ::: warning ::: title Warning ::: It\'s probably a bad idea to mix session and ephemeral workers\... ::: ##### `celery_session_app` - Celery app used for testing (session scope). This can be used by other session scoped fixtures when they need to refer to a Celery app instance. ##### `use_celery_app_trap` - Raise exception on falling back to default app. This is a fixture you can override in your `conftest.py`, to enable the \"app trap\": if something tries to access the default or current_app, an exception is raised. Example: ``` python @pytest.fixture(scope='session') def use_celery_app_trap(): return True ``` If a test wants to access the default app, you would have to mark it using the `depends_on_current_app` fixture: ``` python @pytest.mark.usefixtures('depends_on_current_app') def test_something(): something() ``` --- # Workers Guide {#guide-workers} ::: {.contents local="" depth="1"} ::: ## Starting the worker {#worker-starting} ::: sidebar **Daemonizing** You probably want to use a daemonization tool to start the worker in the background. See `daemonizing`{.interpreted-text role="ref"} for help starting the worker as a daemon using popular service managers. ::: You can start the worker in the foreground by executing the command: ``` console $ celery -A proj worker -l INFO ``` For a full list of available command-line options see `~celery.bin.worker`{.interpreted-text role="mod"}, or simply do: ``` console $ celery worker --help ``` You can start multiple workers on the same machine, but be sure to name each individual worker by specifying a node name with the `--hostname `{.interpreted-text role="option"} argument: ``` console $ celery -A proj worker --loglevel=INFO --concurrency=10 -n worker1@%h $ celery -A proj worker --loglevel=INFO --concurrency=10 -n worker2@%h $ celery -A proj worker --loglevel=INFO --concurrency=10 -n worker3@%h ``` The `hostname` argument can expand the following variables: > - `%h`: Hostname, including domain name. > - `%n`: Hostname only. > - `%d`: Domain name only. If the current hostname is *george.example.com*, these will expand to: ---------- ---------------- ------------------------------ Variable Template Result `%h` `worker1@%h` *worker1@george.example.com* `%n` `worker1@%n` *worker1@george* `%d` `worker1@%d` *worker1@example.com* ---------- ---------------- ------------------------------ ::: admonition Note for `supervisor`{.interpreted-text role="pypi"} users The `%` sign must be escaped by adding a second one: [%%h]{.title-ref}. ::: ## Stopping the worker {#worker-stopping} Shutdown should be accomplished using the `TERM`{.interpreted-text role="sig"} signal. When shutdown is initiated the worker will finish all currently executing tasks before it actually terminates. If these tasks are important, you should wait for it to finish before doing anything drastic, like sending the `KILL`{.interpreted-text role="sig"} signal. If the worker won\'t shutdown after considerate time, for being stuck in an infinite-loop or similar, you can use the `KILL`{.interpreted-text role="sig"} signal to force terminate the worker: but be aware that currently executing tasks will be lost (i.e., unless the tasks have the `~@Task.acks_late`{.interpreted-text role="attr"} option set). Also as processes can\'t override the `KILL`{.interpreted-text role="sig"} signal, the worker will not be able to reap its children; make sure to do so manually. This command usually does the trick: ``` console $ pkill -9 -f 'celery worker' ``` If you don\'t have the `pkill`{.interpreted-text role="command"} command on your system, you can use the slightly longer version: ``` console $ ps auxww | awk '/celery worker/ {print $2}' | xargs kill -9 ``` ::: versionchanged 5.2 On Linux systems, Celery now supports sending `KILL`{.interpreted-text role="sig"} signal to all child processes after worker termination. This is done via [PR_SET_PDEATHSIG]{.title-ref} option of `prctl(2)`. ::: ### Worker Shutdown {#worker_shutdown} We will use the terms *Warm, Soft, Cold, Hard* to describe the different stages of worker shutdown. The worker will initiate the shutdown process when it receives the `TERM`{.interpreted-text role="sig"} or `QUIT`{.interpreted-text role="sig"} signal. The `INT`{.interpreted-text role="sig"} (Ctrl-C) signal is also handled during the shutdown process and always triggers the next stage of the shutdown process. #### Warm Shutdown {#worker-warm-shutdown} When the worker receives the `TERM`{.interpreted-text role="sig"} signal, it will initiate a warm shutdown. The worker will finish all currently executing tasks before it actually terminates. The first time the worker receives the `INT`{.interpreted-text role="sig"} (Ctrl-C) signal, it will initiate a warm shutdown as well. The warm shutdown will stop the call to `WorkController.start() `{.interpreted-text role="func"} and will call `WorkController.stop() `{.interpreted-text role="func"}. - Additional `TERM`{.interpreted-text role="sig"} signals will be ignored during the warm shutdown process. - The next `INT`{.interpreted-text role="sig"} signal will trigger the next stage of the shutdown process. ::: versionchanged 5.6 In previous versions of Celery, when the prefork pool was in use, heartbeats to the broker were not sent during warm shutdown. This caused the broker to terminate the connection, which meant that tasks were not able to complete. As of version 5.6, when the prefork pool is in use, heartbeats are now maintained during warm shutdown and tasks are able to complete before the worker terminates. ::: #### Cold Shutdown {#worker-cold-shutdown} Cold shutdown is initiated when the worker receives the `QUIT`{.interpreted-text role="sig"} signal. The worker will stop all currently executing tasks and terminate immediately. ::: {#worker-REMAP_SIGTERM} ::: note ::: title Note ::: If the environment variable `REMAP_SIGTERM` is set to `SIGQUIT`, the worker will also initiate a cold shutdown when it receives the `TERM`{.interpreted-text role="sig"} signal instead of a warm shutdown. ::: ::: The cold shutdown will stop the call to `WorkController.start() `{.interpreted-text role="func"} and will call `WorkController.terminate() `{.interpreted-text role="func"}. If the warm shutdown already started, the transition to cold shutdown will run a signal handler `on_cold_shutdown` to cancel all currently executing tasks from the MainProcess and potentially trigger the `worker-soft-shutdown`{.interpreted-text role="ref"}. #### Soft Shutdown {#worker-soft-shutdown} ::: versionadded 5.5 ::: Soft shutdown is a time limited warm shutdown, initiated just before the cold shutdown. The worker will allow `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} seconds for all currently executing tasks to finish before it terminates. If the time limit is reached, the worker will initiate a cold shutdown and cancel all currently executing tasks. If the `QUIT`{.interpreted-text role="sig"} signal is received during the soft shutdown, the worker will cancel all currently executing tasks but still wait for the time limit to finish before terminating, giving a chance for the worker to perform the cold shutdown a little more gracefully. The soft shutdown is disabled by default to maintain backward compatibility with the `worker-cold-shutdown`{.interpreted-text role="ref"} behavior. To enable the soft shutdown, set `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} to a positive float value. The soft shutdown will be skipped if there are no tasks running. To force the soft shutdown, *also* enable the `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} setting. ::: warning ::: title Warning ::: If the worker is not running any task but has ETA tasks reserved, the soft shutdown will not be initiated unless the `worker_enable_soft_shutdown_on_idle`{.interpreted-text role="setting"} setting is enabled, which may lead to task loss during the cold shutdown. When using ETA tasks, it is recommended to enable the soft shutdown on idle. Experiment which `worker_soft_shutdown_timeout`{.interpreted-text role="setting"} value works best for your setup to reduce the risk of task loss to a minimum. ::: For example, when setting `worker_soft_shutdown_timeout=3`, the worker will allow 3 seconds for all currently executing tasks to finish before it terminates. If the time limit is reached, the worker will initiate a cold shutdown and cancel all currently executing tasks. ``` console [INFO/MainProcess] Task myapp.long_running_task[6f748357-b2c7-456a-95de-f05c00504042] received [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 1/2000s [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 2/2000s [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 3/2000s ^C worker: Hitting Ctrl+C again will initiate cold shutdown, terminating all running tasks! worker: Warm shutdown (MainProcess) [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 4/2000s [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 5/2000s [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 6/2000s ^C worker: Hitting Ctrl+C again will terminate all running tasks! [WARNING/MainProcess] Initiating Soft Shutdown, terminating in 3 seconds [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 7/2000s [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 8/2000s [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 9/2000s [WARNING/MainProcess] Restoring 1 unacknowledged message(s) ``` - The next `QUIT`{.interpreted-text role="sig"} signal will cancel the tasks that are still running in the soft shutdown, but the worker will still wait for the time limit to finish before terminating. - The next (2nd) `QUIT`{.interpreted-text role="sig"} or `INT`{.interpreted-text role="sig"} signal will trigger the next stage of the shutdown process. #### Hard Shutdown {#worker-hard-shutdown} ::: versionadded 5.5 ::: Hard shutdown is mostly for local or debug purposes, allowing to spam the `INT`{.interpreted-text role="sig"} (Ctrl-C) signal to force the worker to terminate immediately. The worker will stop all currently executing tasks and terminate immediately by raising a `@WorkerTerminate`{.interpreted-text role="exc"} exception in the MainProcess. For example, notice the `^C` in the logs below (using the `INT`{.interpreted-text role="sig"} signal to move from stage to stage): ``` console [INFO/MainProcess] Task myapp.long_running_task[7235ac16-543d-4fd5-a9e1-2d2bb8ab630a] received [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 1/2000s [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 2/2000s ^C worker: Hitting Ctrl+C again will initiate cold shutdown, terminating all running tasks! worker: Warm shutdown (MainProcess) [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 3/2000s [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 4/2000s ^C worker: Hitting Ctrl+C again will terminate all running tasks! [WARNING/MainProcess] Initiating Soft Shutdown, terminating in 10 seconds [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 5/2000s [WARNING/ForkPoolWorker-8] long_running_task is running, sleeping 6/2000s ^C Waiting gracefully for cold shutdown to complete... worker: Cold shutdown (MainProcess) ^C[WARNING/MainProcess] Restoring 1 unacknowledged message(s) ``` ::: warning ::: title Warning ::: The log `Restoring 1 unacknowledged message(s)` is misleading as it is not guaranteed that the message will be restored after a hard shutdown. The `worker-soft-shutdown`{.interpreted-text role="ref"} allows adding a time window just between the warm and the cold shutdown that improves the gracefulness of the shutdown process. ::: ## Restarting the worker {#worker-restarting} To restart the worker you should send the [TERM]{.title-ref} signal and start a new instance. The easiest way to manage workers for development is by using \`celery multi\`: ``` console $ celery multi start 1 -A proj -l INFO -c4 --pidfile=/var/run/celery/%n.pid $ celery multi restart 1 --pidfile=/var/run/celery/%n.pid ``` For production deployments you should be using init-scripts or a process supervision system (see `daemonizing`{.interpreted-text role="ref"}). Other than stopping, then starting the worker to restart, you can also restart the worker using the `HUP`{.interpreted-text role="sig"} signal. Note that the worker will be responsible for restarting itself so this is prone to problems and isn\'t recommended in production: ``` console $ kill -HUP $pid ``` ::: note ::: title Note ::: Restarting by `HUP`{.interpreted-text role="sig"} only works if the worker is running in the background as a daemon (it doesn\'t have a controlling terminal). `HUP`{.interpreted-text role="sig"} is disabled on macOS because of a limitation on that platform. ::: ## Automatic re-connection on connection loss to broker ::: versionadded 5.3 ::: Unless `broker_connection_retry_on_startup`{.interpreted-text role="setting"} is set to False, Celery will automatically retry reconnecting to the broker after the first connection loss. `broker_connection_retry`{.interpreted-text role="setting"} controls whether to automatically retry reconnecting to the broker for subsequent reconnects. ::: versionadded 5.1 ::: If `worker_cancel_long_running_tasks_on_connection_loss`{.interpreted-text role="setting"} is set to True, Celery will also cancel any long running task that is currently running. ::: versionadded 5.3 ::: Since the message broker does not track how many tasks were already fetched before the connection was lost, Celery will reduce the prefetch count by the number of tasks that are currently running multiplied by `worker_prefetch_multiplier`{.interpreted-text role="setting"}. The prefetch count will be gradually restored to the maximum allowed after each time a task that was running before the connection was lost is complete. This feature is enabled by default, but can be disabled by setting False to `worker_enable_prefetch_count_reduction`{.interpreted-text role="setting"}. ## Process Signals {#worker-process-signals} The worker\'s main process overrides the following signals: -------------------------- ------------------------------------------------- `TERM`{.interpreted-text Warm shutdown, wait for tasks to complete. role="sig"} `QUIT`{.interpreted-text Cold shutdown, terminate ASAP role="sig"} `USR1`{.interpreted-text Dump traceback for all active threads. role="sig"} `USR2`{.interpreted-text Remote debug, see role="sig"} `celery.contrib.rdb`{.interpreted-text role="mod"}. -------------------------- ------------------------------------------------- ## Variables in file paths {#worker-files} The file path arguments for `--logfile `{.interpreted-text role="option"}, `--pidfile `{.interpreted-text role="option"}, and `--statedb `{.interpreted-text role="option"} can contain variables that the worker will expand: ### Node name replacements - `%p`: Full node name. - `%h`: Hostname, including domain name. - `%n`: Hostname only. - `%d`: Domain name only. - `%i`: Prefork pool process index or 0 if MainProcess. - `%I`: Prefork pool process index with separator. For example, if the current hostname is `george@foo.example.com` then these will expand to: - `--logfile=%p.log` -\> `george@foo.example.com.log`{.interpreted-text role="file"} - `--logfile=%h.log` -\> `foo.example.com.log`{.interpreted-text role="file"} - `--logfile=%n.log` -\> `george.log`{.interpreted-text role="file"} - `--logfile=%d.log` -\> `example.com.log`{.interpreted-text role="file"} ### Prefork pool process index {#worker-files-process-index} The prefork pool process index specifiers will expand into a different filename depending on the process that\'ll eventually need to open the file. This can be used to specify one log file per child process. Note that the numbers will stay within the process limit even if processes exit or if autoscale/`maxtasksperchild`/time limits are used. That is, the number is the *process index* not the process count or pid. - `%i` - Pool process index or 0 if MainProcess. > Where `-n worker1@example.com -c2 -f %n-%i.log` will result in > three log files: > > > - `worker1-0.log`{.interpreted-text role="file"} (main > > process) > > - `worker1-1.log`{.interpreted-text role="file"} (pool process > > 1) > > - `worker1-2.log`{.interpreted-text role="file"} (pool process > > 2) - `%I` - Pool process index with separator. > Where `-n worker1@example.com -c2 -f %n%I.log` will result in > three log files: > > > - `worker1.log`{.interpreted-text role="file"} (main process) > > - `worker1-1.log`{.interpreted-text role="file"} (pool process > > 1) > > - `worker1-2.log`{.interpreted-text role="file"} (pool process > > 2) ## Concurrency {#worker-concurrency} By default multiprocessing is used to perform concurrent execution of tasks, but you can also use `Eventlet `{.interpreted-text role="ref"}. The number of worker processes/threads can be changed using the `--concurrency `{.interpreted-text role="option"} argument and defaults to the number of CPUs available on the machine. ::: admonition Number of processes (multiprocessing/prefork pool) More pool processes are usually better, but there\'s a cut-off point where adding more pool processes affects performance in negative ways. There\'s even some evidence to support that having multiple worker instances running, may perform better than having a single worker. For example 3 workers with 10 pool processes each. You need to experiment to find the numbers that works best for you, as this varies based on application, work load, task run times and other factors. ::: ## Remote control {#worker-remote-control} ::: versionadded 2.0 ::: ::: sidebar **The `celery` command** The `celery`{.interpreted-text role="program"} program is used to execute remote control commands from the command-line. It supports all of the commands listed below. See `monitoring-control`{.interpreted-text role="ref"} for more information. ::: pool support : *prefork, eventlet, gevent, thread*, blocking:*solo* (see note) broker support : *amqp, redis* Workers have the ability to be remote controlled using a high-priority broadcast message queue. The commands can be directed to all, or a specific list of workers. Commands can also have replies. The client can then wait for and collect those replies. Since there\'s no central authority to know how many workers are available in the cluster, there\'s also no way to estimate how many workers may send a reply, so the client has a configurable timeout --- the deadline in seconds for replies to arrive in. This timeout defaults to one second. If the worker doesn\'t reply within the deadline it doesn\'t necessarily mean the worker didn\'t reply, or worse is dead, but may simply be caused by network latency or the worker being slow at processing commands, so adjust the timeout accordingly. In addition to timeouts, the client can specify the maximum number of replies to wait for. If a destination is specified, this limit is set to the number of destination hosts. ::: note ::: title Note ::: The `solo` pool supports remote control commands, but any task executing will block any waiting control command, so it is of limited use if the worker is very busy. In that case you must increase the timeout waiting for replies in the client. ::: ### The `~@control.broadcast`{.interpreted-text role="meth"} function {#worker-broadcast-fun} This is the client function used to send commands to the workers. Some remote control commands also have higher-level interfaces using `~@control.broadcast`{.interpreted-text role="meth"} in the background, like `~@control.rate_limit`{.interpreted-text role="meth"}, and `~@control.ping`{.interpreted-text role="meth"}. Sending the `rate_limit`{.interpreted-text role="control"} command and keyword arguments: ``` pycon >>> app.control.broadcast('rate_limit', ... arguments={'task_name': 'myapp.mytask', ... 'rate_limit': '200/m'}) ``` This will send the command asynchronously, without waiting for a reply. To request a reply you have to use the [reply]{.title-ref} argument: ``` pycon >>> app.control.broadcast('rate_limit', { ... 'task_name': 'myapp.mytask', 'rate_limit': '200/m'}, reply=True) [{'worker1.example.com': 'New rate limit set successfully'}, {'worker2.example.com': 'New rate limit set successfully'}, {'worker3.example.com': 'New rate limit set successfully'}] ``` Using the [destination]{.title-ref} argument you can specify a list of workers to receive the command: ``` pycon >>> app.control.broadcast('rate_limit', { ... 'task_name': 'myapp.mytask', ... 'rate_limit': '200/m'}, reply=True, ... destination=['worker1@example.com']) [{'worker1.example.com': 'New rate limit set successfully'}] ``` Of course, using the higher-level interface to set rate limits is much more convenient, but there are commands that can only be requested using `~@control.broadcast`{.interpreted-text role="meth"}. ## Commands ::: control revoke ::: ### `revoke`: Revoking tasks pool support : all, terminate only supported by prefork, eventlet and gevent broker support : *amqp, redis* command : `celery -A proj control revoke `{.interpreted-text role="program"} All worker nodes keeps a memory of revoked task ids, either in-memory or persistent on disk (see `worker-persistent-revokes`{.interpreted-text role="ref"}). ::: note ::: title Note ::: The maximum number of revoked tasks to keep in memory can be specified using the `CELERY_WORKER_REVOKES_MAX` environment variable, which defaults to 50000. When the limit has been exceeded, the revokes will be active for 10800 seconds (3 hours) before being expired. This value can be changed using the `CELERY_WORKER_REVOKE_EXPIRES` environment variable. Memory limits can also be set for successful tasks through the `CELERY_WORKER_SUCCESSFUL_MAX` and `CELERY_WORKER_SUCCESSFUL_EXPIRES` environment variables, and default to 1000 and 10800 respectively. ::: When a worker receives a revoke request it will skip executing the task, but it won\'t terminate an already executing task unless the [terminate]{.title-ref} option is set. ::: note ::: title Note ::: The terminate option is a last resort for administrators when a task is stuck. It\'s not for terminating the task, it\'s for terminating the process that\'s executing the task, and that process may have already started processing another task at the point when the signal is sent, so for this reason you must never call this programmatically. ::: If [terminate]{.title-ref} is set the worker child process processing the task will be terminated. The default signal sent is [TERM]{.title-ref}, but you can specify this using the [signal]{.title-ref} argument. Signal can be the uppercase name of any signal defined in the `signal`{.interpreted-text role="mod"} module in the Python Standard Library. Terminating a task also revokes it. **Example** ``` pycon >>> result.revoke() >>> AsyncResult(id).revoke() >>> app.control.revoke('d9078da5-9915-40a0-bfa1-392c7bde42ed') >>> app.control.revoke('d9078da5-9915-40a0-bfa1-392c7bde42ed', ... terminate=True) >>> app.control.revoke('d9078da5-9915-40a0-bfa1-392c7bde42ed', ... terminate=True, signal='SIGKILL') ``` ### Revoking multiple tasks ::: versionadded 3.1 ::: The revoke method also accepts a list argument, where it will revoke several tasks at once. **Example** ``` pycon >>> app.control.revoke([ ... '7993b0aa-1f0b-4780-9af0-c47c0858b3f2', ... 'f565793e-b041-4b2b-9ca4-dca22762a55d', ... 'd9d35e03-2997-42d0-a13e-64a66b88a618', ]) ``` The `GroupResult.revoke` method takes advantage of this since version 3.1. ### Persistent revokes {#worker-persistent-revokes} Revoking tasks works by sending a broadcast message to all the workers, the workers then keep a list of revoked tasks in memory. When a worker starts up it will synchronize revoked tasks with other workers in the cluster. The list of revoked tasks is in-memory so if all workers restart the list of revoked ids will also vanish. If you want to preserve this list between restarts you need to specify a file for these to be stored in by using the [\--statedb]{.title-ref} argument to `celery worker`{.interpreted-text role="program"}: ``` console $ celery -A proj worker -l INFO --statedb=/var/run/celery/worker.state ``` or if you use `celery multi`{.interpreted-text role="program"} you want to create one file per worker instance so use the [%n]{.title-ref} format to expand the current node name: ``` console celery multi start 2 -l INFO --statedb=/var/run/celery/%n.state ``` See also `worker-files`{.interpreted-text role="ref"} Note that remote control commands must be working for revokes to work. Remote control commands are only supported by the RabbitMQ (amqp) and Redis at this point. ::: control revoke_by_stamped_headers ::: ### `revoke_by_stamped_headers`: Revoking tasks by their stamped headers pool support : all, terminate only supported by prefork and eventlet broker support : *amqp, redis* command : `celery -A proj control revoke_by_stamped_headers `{.interpreted-text role="program"} This command is similar to `~@control.revoke`{.interpreted-text role="meth"}, but instead of specifying the task id(s), you specify the stamped header(s) as key-value pair(s), and each task that has a stamped header matching the key-value pair(s) will be revoked. ::: warning ::: title Warning ::: The revoked headers mapping is not persistent across restarts, so if you restart the workers, the revoked headers will be lost and need to be mapped again. ::: ::: warning ::: title Warning ::: This command may perform poorly if your worker pool concurrency is high and terminate is enabled, since it will have to iterate over all the running tasks to find the ones with the specified stamped header. ::: **Example** ``` pycon >>> app.control.revoke_by_stamped_headers({'header': 'value'}) >>> app.control.revoke_by_stamped_headers({'header': 'value'}, terminate=True) >>> app.control.revoke_by_stamped_headers({'header': 'value'}, terminate=True, signal='SIGKILL') ``` ### Revoking multiple tasks by stamped headers ::: versionadded 5.3 ::: The `revoke_by_stamped_headers` method also accepts a list argument, where it will revoke by several headers or several values. **Example** ``` pycon >> app.control.revoke_by_stamped_headers({ ... 'header_A': 'value_1', ... 'header_B': ['value_2', 'value_3'], }) ``` This will revoke all of the tasks that have a stamped header `header_A` with value `value_1`, and all of the tasks that have a stamped header `header_B` with values `value_2` or `value_3`. **CLI Example** ``` console $ celery -A proj control revoke_by_stamped_headers stamped_header_key_A=stamped_header_value_1 stamped_header_key_B=stamped_header_value_2 $ celery -A proj control revoke_by_stamped_headers stamped_header_key_A=stamped_header_value_1 stamped_header_key_B=stamped_header_value_2 --terminate $ celery -A proj control revoke_by_stamped_headers stamped_header_key_A=stamped_header_value_1 stamped_header_key_B=stamped_header_value_2 --terminate --signal=SIGKILL ``` ## Time Limits {#worker-time-limits} ::: versionadded 2.0 ::: pool support : *prefork/gevent (see note below)* ::: sidebar **Soft, or hard?** The time limit is set in two values, [soft]{.title-ref} and [hard]{.title-ref}. The soft time limit allows the task to catch an exception to clean up before it is killed: the hard timeout isn\'t catch-able and force terminates the task. ::: A single task can potentially run forever, if you have lots of tasks waiting for some event that\'ll never happen you\'ll block the worker from processing new tasks indefinitely. The best way to defend against this scenario happening is enabling time limits. The time limit ([\--time-limit]{.title-ref}) is the maximum number of seconds a task may run before the process executing it is terminated and replaced by a new process. You can also enable a soft time limit ([\--soft-time-limit]{.title-ref}), this raises an exception the task can catch to clean up before the hard time limit kills it: ``` python from myapp import app from celery.exceptions import SoftTimeLimitExceeded @app.task def mytask(): try: do_work() except SoftTimeLimitExceeded: clean_up_in_a_hurry() ``` Time limits can also be set using the `task_time_limit`{.interpreted-text role="setting"} / `task_soft_time_limit`{.interpreted-text role="setting"} settings. You can also specify time limits for client side operation using `timeout` argument of `AsyncResult.get()` function. ::: note ::: title Note ::: Time limits don\'t currently work on platforms that don\'t support the `SIGUSR1`{.interpreted-text role="sig"} signal. ::: ::: note ::: title Note ::: The gevent pool does not implement soft time limits. Additionally, it will not enforce the hard time limit if the task is blocking. ::: ### Changing time limits at run-time ::: versionadded 2.3 ::: broker support : *amqp, redis* There\'s a remote control command that enables you to change both soft and hard time limits for a task --- named `time_limit`. Example changing the time limit for the `tasks.crawl_the_web` task to have a soft time limit of one minute, and a hard time limit of two minutes: ``` pycon >>> app.control.time_limit('tasks.crawl_the_web', soft=60, hard=120, reply=True) [{'worker1.example.com': {'ok': 'time limits set successfully'}}] ``` Only tasks that starts executing after the time limit change will be affected. ## Rate Limits {#worker-rate-limits} ::: control rate_limit ::: ### Changing rate-limits at run-time Example changing the rate limit for the [myapp.mytask]{.title-ref} task to execute at most 200 tasks of that type every minute: ``` pycon >>> app.control.rate_limit('myapp.mytask', '200/m') ``` The above doesn\'t specify a destination, so the change request will affect all worker instances in the cluster. If you only want to affect a specific list of workers you can include the `destination` argument: ``` pycon >>> app.control.rate_limit('myapp.mytask', '200/m', ... destination=['celery@worker1.example.com']) ``` ::: warning ::: title Warning ::: This won\'t affect workers with the `worker_disable_rate_limits`{.interpreted-text role="setting"} setting enabled. ::: ## Max tasks per child setting {#worker-max-tasks-per-child} ::: versionadded 2.0 ::: pool support : *prefork* With this option you can configure the maximum number of tasks a worker can execute before it\'s replaced by a new process. This is useful if you have memory leaks you have no control over for example from closed source C extensions. The option can be set using the workers `--max-tasks-per-child `{.interpreted-text role="option"} argument or using the `worker_max_tasks_per_child`{.interpreted-text role="setting"} setting. ## Max memory per child setting {#worker-max-memory-per-child} ::: versionadded 4.0 ::: pool support : *prefork* With this option you can configure the maximum amount of resident memory a worker can execute before it\'s replaced by a new process. This is useful if you have memory leaks you have no control over for example from closed source C extensions. The option can be set using the workers `--max-memory-per-child `{.interpreted-text role="option"} argument or using the `worker_max_memory_per_child`{.interpreted-text role="setting"} setting. ## Autoscaling {#worker-autoscaling} ::: versionadded 2.2 ::: pool support : *prefork*, *gevent* The *autoscaler* component is used to dynamically resize the pool based on load: - The autoscaler adds more pool processes when there is work to do, : - and starts removing processes when the workload is low. It\'s enabled by the `--autoscale `{.interpreted-text role="option"} option, which needs two numbers: the maximum and minimum number of pool processes: ``` text --autoscale=AUTOSCALE Enable autoscaling by providing max_concurrency,min_concurrency. Example: --autoscale=10,3 (always keep 3 processes, but grow to 10 if necessary). ``` You can also define your own rules for the autoscaler by subclassing `~celery.worker.autoscale.Autoscaler`{.interpreted-text role="class"}. Some ideas for metrics include load average or the amount of memory available. You can specify a custom autoscaler with the `worker_autoscaler`{.interpreted-text role="setting"} setting. ## Queues {#worker-queues} A worker instance can consume from any number of queues. By default it will consume from all queues defined in the `task_queues`{.interpreted-text role="setting"} setting (that if not specified falls back to the default queue named `celery`). You can specify what queues to consume from at start-up, by giving a comma separated list of queues to the `-Q `{.interpreted-text role="option"} option: ``` console $ celery -A proj worker -l INFO -Q foo,bar,baz ``` If the queue name is defined in `task_queues`{.interpreted-text role="setting"} it will use that configuration, but if it\'s not defined in the list of queues Celery will automatically generate a new queue for you (depending on the `task_create_missing_queues`{.interpreted-text role="setting"} option). You can also tell the worker to start and stop consuming from a queue at run-time using the remote control commands `add_consumer`{.interpreted-text role="control"} and `cancel_consumer`{.interpreted-text role="control"}. ::: control add_consumer ::: ### Queues: Adding consumers The `add_consumer`{.interpreted-text role="control"} control command will tell one or more workers to start consuming from a queue. This operation is idempotent. To tell all workers in the cluster to start consuming from a queue named \"`foo`\" you can use the `celery control`{.interpreted-text role="program"} program: ``` console $ celery -A proj control add_consumer foo -> worker1.local: OK started consuming from u'foo' ``` If you want to specify a specific worker you can use the `--destination `{.interpreted-text role="option"} argument: ``` console $ celery -A proj control add_consumer foo -d celery@worker1.local ``` The same can be accomplished dynamically using the `@control.add_consumer`{.interpreted-text role="meth"} method: ``` pycon >>> app.control.add_consumer('foo', reply=True) [{u'worker1.local': {u'ok': u"already consuming from u'foo'"}}] >>> app.control.add_consumer('foo', reply=True, ... destination=['worker1@example.com']) [{u'worker1.local': {u'ok': u"already consuming from u'foo'"}}] ``` By now we\'ve only shown examples using automatic queues, If you need more control you can also specify the exchange, routing_key and even other options: ``` pycon >>> app.control.add_consumer( ... queue='baz', ... exchange='ex', ... exchange_type='topic', ... routing_key='media.*', ... options={ ... 'queue_durable': False, ... 'exchange_durable': False, ... }, ... reply=True, ... destination=['w1@example.com', 'w2@example.com']) ``` ::: control cancel_consumer ::: ### Queues: Canceling consumers You can cancel a consumer by queue name using the `cancel_consumer`{.interpreted-text role="control"} control command. To force all workers in the cluster to cancel consuming from a queue you can use the `celery control`{.interpreted-text role="program"} program: ``` console $ celery -A proj control cancel_consumer foo ``` The `--destination `{.interpreted-text role="option"} argument can be used to specify a worker, or a list of workers, to act on the command: ``` console $ celery -A proj control cancel_consumer foo -d celery@worker1.local ``` You can also cancel consumers programmatically using the `@control.cancel_consumer`{.interpreted-text role="meth"} method: ``` console >>> app.control.cancel_consumer('foo', reply=True) [{u'worker1.local': {u'ok': u"no longer consuming from u'foo'"}}] ``` ::: control active_queues ::: ### Queues: List of active queues You can get a list of queues that a worker consumes from by using the `active_queues`{.interpreted-text role="control"} control command: ``` console $ celery -A proj inspect active_queues [...] ``` Like all other remote control commands this also supports the `--destination `{.interpreted-text role="option"} argument used to specify the workers that should reply to the request: ``` console $ celery -A proj inspect active_queues -d celery@worker1.local [...] ``` This can also be done programmatically by using the `~celery.app.control.Inspect.active_queues`{.interpreted-text role="meth"} method: ``` pycon >>> app.control.inspect().active_queues() [...] >>> app.control.inspect(['worker1.local']).active_queues() [...] ``` ## Inspecting workers {#worker-inspect} `@control.inspect`{.interpreted-text role="class"} lets you inspect running workers. It uses remote control commands under the hood. You can also use the `celery` command to inspect workers, and it supports the same commands as the `@control`{.interpreted-text role="class"} interface. ``` pycon >>> # Inspect all nodes. >>> i = app.control.inspect() >>> # Specify multiple nodes to inspect. >>> i = app.control.inspect(['worker1.example.com', 'worker2.example.com']) >>> # Specify a single node to inspect. >>> i = app.control.inspect('worker1.example.com') ``` ### Dump of registered tasks {#worker-inspect-registered-tasks} You can get a list of tasks registered in the worker using the `~celery.app.control.Inspect.registered`{.interpreted-text role="meth"}: ``` pycon >>> i.registered() [{'worker1.example.com': ['tasks.add', 'tasks.sleeptask']}] ``` ### Dump of currently executing tasks {#worker-inspect-active-tasks} You can get a list of active tasks using `~celery.app.control.Inspect.active`{.interpreted-text role="meth"}: ``` pycon >>> i.active() [{'worker1.example.com': [{'name': 'tasks.sleeptask', 'id': '32666e9b-809c-41fa-8e93-5ae0c80afbbf', 'args': '(8,)', 'kwargs': '{}'}]}] ``` ### Dump of scheduled (ETA) tasks {#worker-inspect-eta-schedule} You can get a list of tasks waiting to be scheduled by using `~celery.app.control.Inspect.scheduled`{.interpreted-text role="meth"}: ``` pycon >>> i.scheduled() [{'worker1.example.com': [{'eta': '2010-06-07 09:07:52', 'priority': 0, 'request': { 'name': 'tasks.sleeptask', 'id': '1a7980ea-8b19-413e-91d2-0b74f3844c4d', 'args': '[1]', 'kwargs': '{}'}}, {'eta': '2010-06-07 09:07:53', 'priority': 0, 'request': { 'name': 'tasks.sleeptask', 'id': '49661b9a-aa22-4120-94b7-9ee8031d219d', 'args': '[2]', 'kwargs': '{}'}}]}] ``` ::: note ::: title Note ::: These are tasks with an ETA/countdown argument, not periodic tasks. ::: ### Dump of reserved tasks {#worker-inspect-reserved} Reserved tasks are tasks that have been received, but are still waiting to be executed. You can get a list of these using `~celery.app.control.Inspect.reserved`{.interpreted-text role="meth"}: ``` pycon >>> i.reserved() [{'worker1.example.com': [{'name': 'tasks.sleeptask', 'id': '32666e9b-809c-41fa-8e93-5ae0c80afbbf', 'args': '(8,)', 'kwargs': '{}'}]}] ``` ### Statistics {#worker-statistics} The remote control command `inspect stats` (or `~celery.app.control.Inspect.stats`{.interpreted-text role="meth"}) will give you a long list of useful (or not so useful) statistics about the worker: ``` console $ celery -A proj inspect stats ``` For the output details, consult the reference documentation of `~celery.app.control.Inspect.stats`{.interpreted-text role="meth"}. ## Additional Commands ::: control shutdown ::: ### Remote shutdown This command will gracefully shut down the worker remotely: ``` pycon >>> app.control.broadcast('shutdown') # shutdown all workers >>> app.control.broadcast('shutdown', destination='worker1@example.com') ``` ::: control ping ::: ### Ping This command requests a ping from alive workers. The workers reply with the string \'pong\', and that\'s just about it. It will use the default one second timeout for replies unless you specify a custom timeout: ``` pycon >>> app.control.ping(timeout=0.5) [{'worker1.example.com': 'pong'}, {'worker2.example.com': 'pong'}, {'worker3.example.com': 'pong'}] ``` `~@control.ping`{.interpreted-text role="meth"} also supports the [destination]{.title-ref} argument, so you can specify the workers to ping: ``` pycon >>> ping(['worker2.example.com', 'worker3.example.com']) [{'worker2.example.com': 'pong'}, {'worker3.example.com': 'pong'}] ``` ::: {#worker-enable-events} ::: control enable_events ::: ::: ::: control disable_events ::: ### Enable/disable events You can enable/disable events by using the [enable_events]{.title-ref}, [disable_events]{.title-ref} commands. This is useful to temporarily monitor a worker using `celery events`{.interpreted-text role="program"}/`celerymon`{.interpreted-text role="program"}. ``` pycon >>> app.control.enable_events() >>> app.control.disable_events() ``` ## Writing your own remote control commands {#worker-custom-control-commands} There are two types of remote control commands: - Inspect command > Does not have side effects, will usually just return some value > found in the worker, like the list of currently registered tasks, > the list of active tasks, etc. - Control command > Performs side effects, like adding a new queue to consume from. Remote control commands are registered in the control panel and they take a single argument: the current `!celery.worker.control.ControlDispatch`{.interpreted-text role="class"} instance. From there you have access to the active `~celery.worker.consumer.Consumer`{.interpreted-text role="class"} if needed. Here\'s an example control command that increments the task prefetch count: ``` python from celery.worker.control import control_command @control_command( args=[('n', int)], signature='[N=1]', # <- used for help on the command-line. ) def increase_prefetch_count(state, n=1): state.consumer.qos.increment_eventually(n) return {'ok': 'prefetch count incremented'} ``` Make sure you add this code to a module that is imported by the worker: this could be the same module as where your Celery app is defined, or you can add the module to the `imports`{.interpreted-text role="setting"} setting. Restart the worker so that the control command is registered, and now you can call your command using the `celery control`{.interpreted-text role="program"} utility: ``` console $ celery -A proj control increase_prefetch_count 3 ``` You can also add actions to the `celery inspect`{.interpreted-text role="program"} program, for example one that reads the current prefetch count: ``` python from celery.worker.control import inspect_command @inspect_command() def current_prefetch_count(state): return {'prefetch_count': state.consumer.qos.value} ``` After restarting the worker you can now query this value using the `celery inspect`{.interpreted-text role="program"} program: ``` console $ celery -A proj inspect current_prefetch_count ```