Many applications need some form of reliable background processing, usually in form a persistent queue. Instead of using a full-blown task queue like the popular celery-framework for Python, I like to use something that most projects already use anyway: PosgreSQL.
- For django-projects I like to use django-postgres-queue.
- For other projects I am experimenting with pq.
PostgreSQL’s provides two facilities that make implementing a task queue feasible in the database:
SKIP LOCKED and
NOTIFY/LISTEN. Both libraries use these features in their implementation.
I have not performed any benchmarks, because my typical workload is rather low, like sending password-reset emails. However, pq says: “In terms of performance, the implementation clock in at about 1,000 operations per second.” That’s plenty for many use cases.