EDIT: The latest workling on github has all of the changes described here. There is no need to download my patch anymore. See github. In my previous post, I briefly described how simple it is to add background processing to your RoR app with Starling and the Workling plugin. Now, let's discuss the changes I made to the Workling plugin, and then get everything deployed and monitored. As good as the Workling plugin is, it had one limitation that hurt me. At Inquisix, I have 5 workers to handle importing contacts, stats/logging, emails, searching, etc. Most methods are pretty quick, but a large import process could take 5-10 minutes to complete. With the original Workling, that meant all my other background tasks would wait, and I didn't want that. The easiest solution for me was to modify Workling so each worker polled its queues in a thread. That way, if the contact worker was busy importing 1000 contacts, my emails still got delivered in a reasonable amount of time.
Most of the changes to Workling, are limited to the lib/workling/starling/poller.rb file. Here is a summary of the changes I made:
- Added threading
- Updated the way ClassAndMethodRouting builds the routing hash. I needed a routing hash per worker. Note, whatever you do don't ever try to call routing.build from inside another thread. Really strange things happen that were very difficult to track down. Ruby threads are nice, but I'm used to real threads.
- Added handling for MemCache exceptions. During development, I went to dinner and left the listener running without Starling. When I got back, my Mac was complaining the disk was full because my log was > 40Gig. Now, I explicitly catch MemCache exceptions and wait 30 seconds before trying again. The log will still grow, but at least not as fast. Besides, I will show you how to make sure Starling stays running.
- Keep your database connection alive. I found this out after leaving for the night. I thought I had a working system when I left, but nothing worked in the morning. Basically, ActiveRecord will drop your connection if you don't do anything for a while. Normally, this is not a problem in web apps because the connection is updated every time you access a page, but it doesn't work that way here. You have to do it yourself. Add this in your loop to be sure:
unless ActiveRecord::Base.connection.active? unless ActiveRecord::Base.connection.reconnect! RAILS_DEFAULT_LOGGER.fatal("FAILED - Database not available") break end end
Here is my workling patch. Now, I need to go back and update the tests, but the threads complicate that. It's a learning experience...
That's it for now. Next time, I will walk through the deployment and monitoring process as I have it.
EDIT: The latest workling on github has all of my changes in it. There is no need to download the patch.