How to get 4x the performance out of Heroku with Unicorn

This is the first of a two part series on how we set up Railsonfire with Heroku. The second part will deal with Assets, Sprites and Amazon Cloudfront.

This guide is only relevant and tested with the Heroku Cedar stack.

I have had several debates over the last couple of months whether Heroku is the way to go and especially if it is expensive or not. They provide a great service, but their documentation makes them look pretty bad when it comes to price.

$35 for basically one single concurrent request seems very expensive, especially when starting out with a new project (although of course the first dyno is free).

Heroku provides plenty of resources, but as it only allows to listen on one port you can run only one thin instance (as recommended by their documentation).

What we need here is a webserver that listens on one port, but can work through several concurrent requests. Sounds like a job for Unicorn.

What is Unicorn

Unicorn is a ruby http server that starts one master process listening on one port and forks several worker processes. Every incoming client request is handed to a worker by the master and when finished the master returns the result to the client. Thus it only needs to listen on one port, but can work on several concurrent requests.

Defunkt wrote a nice blogpost about unicorn some time ago that goes into more detail how GitHub uses it.

Setup

To start using Unicorn all you have to do is:

1. Add the Gem to your Gemfile

2. Create a Procfile

3. Add unicorn config in config/unicorn.rb

4. Set the default Logger in production.rb to STDOUT, otherwise logging doesn't work.

Unicorn config

You can find all config parameters in the unicorn documentation.

Let's go quickly through the configuration we use

worker_processes: Setting the number of worker processes

timeout: Time after which a worker is restarted if unresponsive

preload_app: Load the application before forking workers. Set to true if you use NewRelic (which you should) or you won't see any data

before_fork/after_fork: Disconnect in before_fork and reconnect in after_fork for your Database, Resque or other services. Without those handlers there will be regular database errors.

NewRelic

Go to the NewRelic Page of your Heroku application and check your dynos and memory consumption.

Average Memory Consumption

Check your Memory Consumption in NewRelic and set the worker_processes accordingly. The average consumption is shown on the Dyno tab of your heroku dashboard.

One Heroku Dyno has 512Mb of Memory, so make sure your combined workers do not exceed that maximum amount, or your dyno will be shut down.

On the right hand side of that same tab you can see the number of dynos. Make sure it is the same as you set in worker_processes.

Benchmarks

I ran several tests with ApacheBench to determine how much the performance improved.

I ran 1000 Requests with 100 concurrent connections against the landing page of our staging application. The following graph shows the time the requests took combined with 1-4 workers.

Going from one process to several increases performance drastically, from then on it is still a boost to your application, but not as drastically. However you have to find the right spot on how many workers you want unicorn to fork depending on your application. Having too many may shut down your dyno due to memory constraints.

Conclusion

So in closing using Unicorn as your Heroku Webserver not only pays off, but should be put into the Heroku documentation at least as advanced information.

I actually talked to people and showed them our Unicorn setup, which convinced them that Heroku is not as expensive as it seems and especially when starting your project is a very viable alternative to having your own Server. By starting up one more dyno you can have several more concurrent requests.

If you have any questions regarding the setup or anything else you can send an email to flo@railsonfire.com, a Tweet to @Railsonfire or use the Olark Chat Box in the right hand corner.

Thanks

This post is very much built on Michael van Rooijen's Blogpost. Gists that helped with the setup were by leshill and jamiew

Backend Metrics done right

I love Mixpanel, Optimizely and Woopra. Diving into Google Analytics or Kissmetrics gives me regular nerdboners. But as nice as those services are the most important source for your metrics is and always will be your database.

The problem is this is by definition always very specific to your system, thus you need to implement it yourself.

To make this easy for us we implemented a gem called GSMetrics that lets us keep track of our data in Google Spreadsheets. We use it to run a daily job on Heroku to take all the data we are interrested in and write it into a Spreadsheet. We then analyse that data either directly on Google Spreadsheets or via other means.

I will go through the initial setup you have to do to get it running as well as a little rundown of how we use it with the Heroku Scheduler Addon.

Setup

To start using the gem you need to have a google Oauth2 token. You can simply run

gsmetrics

which will guide you through creating a new Oauth API token and give you a code block in the end that you can simply copy to create a new GSMetrics Session. The clientid, clientsecret and token used will be filled out in the code block already.

You then pass it the ID of the document you want to access (You can get that from the google docs url of that document) and it will show you all the different worksheets and give you the exact code to use for gsmetrics to access those worksheets.

You can read more about authorization with Google and OAuth2 in their Documentation

How does it work

The following code block is an example of how to use GSMetrics:

session = GSMetrics::Session.new(client_id,client_secret,refresh_token)
worksheet = session.worksheet(document_id, worksheet_id)
worksheet << 1
worksheet << 2
worksheet.append(5)
worksheet.save_row

It simply creates a Session in which you can access different worksheets in different documents. You simply add a data value via either << or append. As soon as your row is finished you call save_row. This will store your data in the first empty line of the worksheet and remove the local data so you can start with your next row.

If you add several new rows in a short period of time Google times out your requests. To solve that simply add a short delay (for example with sleep) into your code and google doesn't mind then. We used a two second delay after every five rows for the initial import of our data.

Heroku Scheduler FTW

We use the Heroku Scheduler Addon to run a job every day that goes through our database, takes alle the relevant data, aggregates it and calls gsmetrics.

This makes it incredibly easy to gather backend data and get meaningfull value out of it. Whenever we need to change our metrics it is a matter of minutes to change them, push the new change to github and in a matter of minutes it is tested and deployed (by Railsonfire of course).

Conclusion

GSMetrics is a really simple way to gather your data and analyse your database in every way you wish. You can run it locally with a copy of your production database or on your production system.

I would be really interrested in how you use GSMetrics or what you do to gather your backend metrics.

If you have any suggestions for new features or bugs you can either comment below, send me a pull request or an email to flo@railsonfire.com.

The all new Railsonfire Gem

To make the setup and interaction with Railsonfire as easy as possible we implemented a Ruby Gem. Right now you can set up Continuous Integration and Continuous Deployment for your project with the gem in less than a minute.

In the future we will develop the gem and our service to support developers from starting a project through development, testing and deployment. Our goal is to make these services a no-brainer to use so you can concentrate on developing your apps.

To use it to interact with Railsonfire a few simple steps are necessary.

What to do

1. gem install railsonfire
2. railsonfire new
3. git commit -a
4. git push origin master

Are there prerequisites?

• Logged in to Railsonfire and allowed access to GitHub
• Local git version controlled ruby project
• GitHub remotes added to local git repository

for Heroku?

• Authenticated with the Heroku gem
• Heroku remotes added to local git repository

What is exactly happening?

A more in depth description on what is exactly happening when you call the railsonfire gem.

gem install railsonfire

to install the latest version. Ruby 1.9.2 and 1.8.7 are supported

railsonfire new

You need to call this command in the folder of your ruby project. The gem will load all GitHub remotes and if you have several let you decide on which one to use. Then an ssh key is added to your GitHub Project so we can access it from our servers.

Authenticate

When adding a new project to railsonfire for the first time you will be asked for your Railsonfire token. The gem will try to open your Railsonfire Account Page, where you can find your Railsonfire token. Simply paste it into the command line.

Decide if you want heroku support

The gem will detect any heroku remotes and give you the option to deploy to them. If you decide so the gem will upload an ssh key to Heroku. You need to have authenticated with the Heroku gem before.

git commit -am "Adding railsonfire.yml file"

You have to add the railsonfire.yml that is created by the gem to your codebase. You can edit the file to set your test commands or the ruby versions you want to test with.

git push origin master

Simply push your changes to GitHub. From then on we will test and deploy all of your changes.

How to hack Seedcamp

Since the Mini Seedcamp London is starting today and the good people of ReplyDone asked us about tips for Seedcamp here are some that came up. If you got more please comment below and I will add them to the post.

• First and most importantly have an answer for the question "Why do you need the money". Having no answer or not a sufficient answer ("We may do some marketing") may be a deal breaker. Think about how exactly the money helps your business and why you need the money from Seedcamp for it. Also be honest. If you don't need the money you won't fool anyone there. If you need the network, but not the money say so.
• Read through the list of mentors and get an overview who everyone is. Write that down (maybe crowd sourced google doc). Think about the questions you have for all the mentor groups (Marketing, Sales, Investors, Tech, ...) so you got something to ask anybody.
• Go through the list of Startups and try to see which ones you would like to talk to the most.
• Think about and get Feedback on all the weak points in your Idea. e.g. Why use your service, what if "big company xyc" does this. The Investors will ask you those questions and if you don't have a sufficient answer they will ask the question again and again and again.
• When answering a question take your time. Think about the question and the underlying meaning of the question. For example we were asked if a huge company could do the same thing we do with their current infrastructure. I answered as a techie and said no and listed some techie reasons. Let's say they were not impressed. The real question was what if they step into this market and the answer should have been then the market is already pretty big since they are multi million dollar companies. Thus I only care about the other startups in that space and how I can be the one big enough to either partner with the big company or get bought. So stop thinking as a techie. At least a bit.
• Business Cards FTW. Prepare some case where you put all the business cards you will get into. You don't want to lose those. Use a CRM to store your contacts. Highrise has a free plan that my be sufficient for now.
• See which other Startups from your area go there and book your accomodations together. Drive there together and drive home together. We had great feedback sessions with the Blossom guys at Ljubljana. Very valuable. Also someone to take you home :).
• Don't forget to put some numbers about your market into the pitch. No one is interrested how your technology works, as long as it works. Everyone wants to know how you make money (at least the people that count there)
• Don't hesitate to tell someone "Then you will maybe never be my customer". We talked to one guy who would probably never use our service out of very legitimate reasons. In a private conversation you can discuss that, in a more public setting don't hesitate to say "Then you will never be my customer". Don't get dragged into an open discussion with someone who will not use your service. There is nothing to gain out of having that discussion publicly in front of everyone.
• Introduce yourself over the mailing list. If you are the first one (damn you Thomas Schranz from Blossom for beating me to it) everyone will read it. But provide some interresting or helpful information with your email, for example a crowd sourced mentor google doc.
• Have Fun and go to the parties. Talk to everyone at the parties and dance dance dance.
• Last but not least get Mike Butcher to sing a song with the name of your Startup in it. Or any song for that matter. You may not get money or fame from that, but damn that was fun.

Best of luck to all the new Seedcampers and hopefully this helped you somehow.

Mini Seedcamp Ljubljana

So we were invited to Mini Seedcamp Ljubljana from 13-15 July. Finderly and blossom.io were the other Viennese startups who joined us in three days of intense talks, mentoring, feedback and partying.

Seedcamp started on wednesday with a pitch training where every team could present the current version of their pitch, get feedback and, as we did, throw away their complete structure and redo the whole thing. Thanks to the incredible people there, especially Tomaz from vox.io, who advised us to stop presenting to a group of developers and think about how to present the idea to investors. This really opened our eyes on how to present to this crowd.

After the pitch training we went home and quickly redid our presentation to join the others again at a nice cocktail bar. Again great and relaxed conversations and lots of interesting people, which seemed like the recurring theme through all of Seedcamp.

Then came the big day. For two hours every team had their 5 minutes of fame. Some incredible technology like SiteCake or Slidemotion, but also great ideas like Farmeron, which makes Software as a Service Agricultural Management, were presented. You can see the list of all participants at the Seedcamp blog.

After all the Presentations we had time to lean back and watch Fred Destin talk about the Startup Lifecycle. Definitely worth taking a good hard look at. You can watch it on Prezi

Then all teams went into their respective rooms and waited for the mentors to come in, give feedback, advice and general words of wisdom. Five groups of four mentors each joined us to talk business, marketing, technology or strategy with us. It was easy, hard, fun, tough, interesting and definitely very tiring. The Seedcampers really got a very diverse team of mentors for the event. From TelCo CEO's to Social Media Marketing pro's to developers and investors. Really cool.

To round it all off Mike Butcher gave, in his very very funny style, an introduction on how to deal with the media, while Austrian beer was enjoyed by the audience.

We then had a great evening with food, plenty of cocktails, talks and dancing. Although the Startups impressed us with their ideas and hard work nothing impressed us more than some of the incredible dance skills shown in the early morning hours. It was a total blast.

On Friday several teams were invited for 15 minute talks with investors. Sadly we weren't invited (this time) but we'll be back. We had to leave for Vienna early and thus weren't able to make it to the Barcamp, but heard lot's of good stuff about it.

But the best part from our Seedcamp experience happened during the six hour train travel back to Vienna. Thanks to all the Feedback our thoughts were spinning in every direction and we got some great ideas for the future of our service. Some big changes are coming up in the next weeks, so stay tuned.

Thanks to Seedcamp for this great event and all the fun we had. See you soon.

All images are owned by Ivo Spigel and are used with permission. Take a look at his other Seedcamp pictures.