Using PostgreSQL as a task queue

Many applications need some form of reliable background processing, usually in form a persistent queue. Instead of using a full-blown task queue like the popular celery-framework for Python, I like to use something that most projects already use anyway: PosgreSQL.

• For django-projects I like to use django-postgres-queue.
• For other projects I am experimenting with pq.

PostgreSQL’s provides two facilities that make implementing a task queue feasible in the database: SKIP LOCKED and NOTIFY/LISTEN. Both libraries use these features in their implementation.

I have not performed any benchmarks, because my typical workload is rather low, like sending password-reset emails. However, pq says: “In terms of performance, the implementation clock in at about 1,000 operations per second.” That’s plenty for many use cases.

See also

• Django 4.0 released
• Updated: How to extract a date from a PDF file using Python
• A new remote workflow for PyCharm (and other Jetbrains IDEs)
• How to extract a date from a PDF file using Python
• rxPY3: Always implement on_error and on_completed