Tarek N. Elsamni

Ruby on Rails Ninja

Ruby on Rails Ninja, Husband & Father for a girl. Founder of @VanellusCloud and currently working @Konnecti

More About Me

Senior Software Engineer, MSc. of Computer Science from University of Hertfordshire. Husband & Father for a girl. Founder of @VanellusCloud and currently working @Instabug

Close

Blog

Previous Next

Sidekiq performing background or delayed jobs with Rails on AWS Elastic BeansTalk

How it works and How I used it

To understand when to use a background scheduler we need to understand the following workflow, Imagine a project where you need to update some user data (change user location for ex.), then notify other users (friends) with the update made and finally send some emails to affected users. a classical implementation for such a work flow may be like this:

image

 

as shown in this workflow the user has to wait 3.7 seconds to get a response from the web server, also you can easily find out that the important part that should be executed before the user gets a response is the update database part and other steps (jobs) can be done later as it doesn’t affect the user itself. so a purpose workflow to implement this in a better way may be like this:

image

as shown in this workflow you can find out that the user will get a response on 0.3 second, AWESOME !!!

Implementing this workflow using Sidekiq will need a basic setup consisted of: Redis database to store queues and its data, workers running in the background to process queued jobs and finally the Sidekiq gem that will be used to queue jobs. In the following part I’ll try to show you how this architecture is implemented.

Basic Architecture

so as shown in the pervious part we’ll have queues (persisted in Redis database) and workers running the background to perform required jobs queued in the Redis database.

image

our code implementation will be divided as follows:

- Workers located at: app/workers

- Sidekiq initializers at: config/initializers/sidekiq.rb

- AWS Elastic BeansTalk configuration to get Sidekiq start with each deployment as: .ebextensions/sidekiq.config

Getting Things running

First of all we’ll need to setup Redis and get it up and running, you can easily get Redis installed and working in your development environment using homebrew using the following command :

$ brew install redis

Then you should see something like this:


==> Downloading http://redis.googlecode.com/files/redis-2.4.8.tar.gz

######################################################################## 100.0%
==> make -C src
==> Caveats
If this is your first install, automatically load on login with:
mkdir -p ~/Library/LaunchAgents
cp /usr/local/Cellar/redis/2.4.8/homebrew.mxcl.redis.plist ~/Library/LaunchAgents/
launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.redis.plist
If this is an upgrade and you already have the homebrew.mxcl.redis.plist loaded:
launchctl unload -w ~/Library/LaunchAgents/homebrew.mxcl.redis.plist
cp /usr/local/Cellar/redis/2.4.8/homebrew.mxcl.redis.plist ~/Library/LaunchAgents/
launchctl load -w ~/Library/LaunchAgents/homebrew.mxcl.redis.plist
To start redis manually:
redis-server /usr/local/etc/redis.conf
To access the server:
redis-cli
==> Summary
/usr/local/Cellar/redis/2.4.8: 9 files, 424K, built in 11 seconds

This is to install Redis at your development environment but for production I’d prefer using AWS Elastic Cache Redis instance to do so, Just login to your AWS console and follow these simple steps:

image

now in your Gemfile add Sidekiq gem and we’ll also add Sinatra gem to enable the Sidekiq dashboard (will show it later).

## Background Tasks

gem ‘sidekiq’

gem ‘sinatra’ ## to provide sidekiq web-view

Then bundle your Gemfile to fetch the required gems

$ bundle install

Now, we need to store Redis configurations (Redis server and port) to a yml file so we can define different settings for each environment (development, test, production, etc) as follows:

image

Now to initialize the Sidekiq on rails app starting based on the active environment we can use the following code:

image

https://gist.github.com/tareksamni/03325d7ac950c8872897

now you can easily write the following code to execute a function that’s a member of User model for example as follows:

u = User.find(1)

u.delay.notify_friends()

 This will run the the notify friends in background and it wont block the server from returning a response to the client.

delay method is automatically binded to all of your classes and its purpose is to run any chained methods in background using the default queue.

You can view current queued jobs using Sidekiq dashboard by opening a route for Sidekiq dashboard in your routes.rb file as follows :

mount Sidekiq::Web, at: '/sidekiq'

where ‘/sidekiq/ is the end point to mount the dashboard on so we can access it with:

http://localhost:3000/sidekiq

image

As you will find when you visit your Sidekiq dashboard, a lot of jobs are queued but not running/executed so you will need to run your workers to execute it jut by typing:

bundle exec sidekiq -q default

Deploying to AWS Elastic BeansTalk

Now everything is working perfectly and we move our notify functionality to a backend job being executed by running workers. but we need to get this implementation up and running on AWS Elastic BeansTalk .. You wont face any problem concerning our implementation as most of our code is handling Development/Productions environments so this typical code will run smoothly on AWS Elastic BeansTalk but the problem will be that you will need to restart your workers on each deployment so they can get updated with any code modifications have been made.

You can do so by logging in to your EC2 which has being used by AWS Elastic BeansTalk to kill old workers and start new ones. but this for sure kills the idea behind automating your deployments and add more hassle to your deployment procedure. Luckily Elastic Beanstalk suppports a pretty powerful configuration file syntax. Nevertheless it took me some time to figure this all out hopefully I can save you some time:

image

https://gist.github.com/tareksamni/c8ff8b8d1a6234c9137c

This code will create a new folder “/opt/elasticbeanstalk/hooks/appdeploy/post”

as this folder is not created by default, and it will also use other folders like the “pre” folder in the same path to hook on the deployment process and kill sidekiq workers before “pre” deployment process and then restart it in the after “post” deployment process.

I hope you learned something new from this blog post and I’d like to refer to some other blog posts which I used to create such a rich content:

http://blog.noizeramp.com/2013/04/21/using-sidekiq-with-elastic-beanstalk/

http://kamisama.me/2012/10/09/background-jobs-with-php-and-resque-part-1-introduction/

Back to Top

Twitter

Previous Next
Back to Top

Likes

Previous Next
Back to Top

Ask me anything

Previous Next
Back to Top

Submit

Previous Next
Back to Top

Instagram

Previous Next
Load More Photos
Back to Top