«« :: 2011 Jan :: »»


1 comments 2011 jan 26 (wed) 17:51  ::  Celery in a virtualenv with Supervisord

I'm using the combination of Celery and django-celery with RabbitMQ to do background video transcoding of videos uploaded to a Django web site. I use Supervisord to keep my gunicorn processes running, and I wanted to use to keep Celery running too. Unfortunately, it didn't work out of the box.

RabbitMQ just worked, no need for further comment there, except to say that it has a cool logo and is written in Erlang.

Celery comes with a supervisord/celeryd.conf file, but recommends that you use django-celery's celeryd.conf if you're using Django. This is my very slightly edited version of the file, with paths in directory and command to match the location of my project:

[program:celery]
command=/var/www/virtualenv/project/manage.py celeryd --loglevel=INFO
directory=/var/www/virtualenv/project
user=nobody
numprocs=1
...

When I reloaded the supervisord configs and tried to start celery it wouldn't work. I'd get one of the following error messages:

$ sudo supervisorctl status
celery          FATAL      Exited too quickly (process log may have details)
...

$ sudo supervisorctl status
celery          BACKOFF    Exited too quickly (process log may have details)
...

tail``ing ``/var/log/celeryd.log revealed the following:

...
Traceback (most recent call last):
  File "/var/www/virtualenv/project/manage.py", line 3, in <module>
    from django.core.management import execute_manager
ImportError: No module named django.core.management
...

I fixed if by customizing the Python path in manage.py so that it points to the Python bin in the virtualenv. This line:

#!/usr/bin/env python

becomes this line:

#!/var/www/stage/nikkeienv/bin/python

With that change and another sudo supervisorctl reload everything worked.

 

1 comments on this post

Aleck at 11:50 a.m. on Mon 16 Apr 2012

Thanks for the post. Since my manage.py is sourced from a repository, I didn't want to update it and break other instances of my project.

In the supervisor config /etc/supervisor/supervisord.conf
...
[include]
files=celeryd.conf celerybeat.conf celerycam.conf
...

in each of the celery conf files:
[program:celery]
command=/path/to/virtualenv/bin/python /path/to/virtualenv/project/mamange.py celeryd -E loglevl=INFO
environment=PYTHONPATH=/path/to/virtualenv/project
directory=/path/to/virtualenv/project
...

repeat pathing for celerybeat.conf and celerycam.conf

Comments for this post have been disabled