Using Whenever GEM to overcome Postgres limitations

20 Jan 2015

One of the limitations I found with Postgres compared with Oracle is the lack of a scheduled request. To get round this limitation I used the whenever gemĀ https://github.com/javan/whenever. This routine gives you the ability to rub a Sidekiq worker as a cron job. One gotcha that you need to watch out for is Sidekiq creates multiple workers which can mean you get multiple instances of the same function carried out at the same time. This means you can end out with duplicate entries if this does not break your database constraints.

To limit this you can can specify the queue and the number of workers that can be created for your class. See example sidekiq.yml below

:verbose: false
:pidfile: ./tmp/pids/sidekiq.pid
:logfile: ./log/sidekiq.log
:concurrency: 25
:queues:
- default
- queue_analysis
- queue_scraper
:limits:
queue_analysis: 1
queue_scraper: 15

This means that something like a web scraping class can have multiple instances while the db class will just have one instance spawned.

Published on 20 Jan 2015 Find me on Twitter!