If you are using cucumber to test a multilingual app, you should check out this post for a step definition that you can use to test multilingual strings. However, most likely you would be using variables in your translation strings. Assuming you have pickle installed (if you don’t you should), you can use the step and step definition below to test strings with variable interpolation
Just like pickle, you can add any number of variables.
aka .. can’t throw hardware at it
Just when I decided to focus fulltime on my new startup SupportBee, our first product Muziboo decided to grow (no complaints though). We went from about 10k visits a day to about 30k visits a day in a month’s time. Muziboo was my first web programming experience and in the sprit of a real startup, I always postponed worrying about scaling. However, the cracks showed through this time. Here is what we observed
Below is a log of what we did to make the site faster
Better Server Monitoring and Logging
You can’t fix what you can’t measure. We researched a bit and installed munin. Apart from the standard plugins (disk, cpu, network) we setup plugins for beanstalk, rails and nginx. This gave us some pretty charts to understand how the system was behaving. Here is an example of what munin charts looks like
Not only can you see the current state of the system, you can also see data for past weeks/months/years. If you are not excited by installing munin, you can look at hosted solution like Server Density
MySQL slow query log for missing indexes
Rails does not (cannot) add indexes to your database. Rails does not (cannot) add indexes to your database automatically. You will have to figure out what indexes to add based on your queries. This is not a problem for small tables (hundreds or thousands of rows) but its a problem with bigger tables. Fortunately, its easy to find out queries that are not using the index. Apart from listing slow queries, MySQL can list all the queries that don’t use the index in slow query log. You can enable it by having this directive in your mysql configuration file
log-queries-not-using-indexes
We enabled this option and found out that we had a bunch of index less queries. As we added these indexes, mysql’s cpu usage started going down bringing the system to a much better shape. You can use mysql’s explain statement to understand your queries better and add indexes. Its a good idea to have explain’s output in your rails’ log during development. I found this snippet that does it for you
unless RAILS_ENV == 'production' module ActiveRecord module ConnectionAdapters class MysqlAdapter < AbstractAdapter def select_with_explain(sql, name = nil) explanation = execute_with_disable_logging('EXPLAIN ' + sql) e = explanation.all_hashes.first exp = e.collect{|k,v| " | #{k}: #{v} "}.join log(exp, 'Explain') select_without_explain(sql, name) end def execute_with_disable_logging(sql, name = nil) #:nodoc: #Run a query without logging @connection.query(sql) rescue ActiveRecord::StatementInvalid => exception if exception.message.split(":").first =~ /Packets out of order/ raise ActiveRecord::StatementInvalid, "'Packets out of order' error was received from the database. Please update your mysql bindings (gem install mysql) and read http://dev.mysql.com/doc/mysql/en/password-hashing.html for more information. If you're on Windows, use the Instant Rails installer to get the updated mysql bindings." else raise end end alias_method_chain :select, :explain end end end end
Put it in your config/initializers directory as mysql_explain.rb and restart your server. Now you should see output from explain in your logs before your sql queries. One point to note is that throwing memcache is not a substitute for having faster queries.
Beware of MySQL’s ORDER BY rand()
Even after adding indexes, one of our queries was not using them. On investigating further I found out that MySQL’s order by rand() is evil. I won’t go into the specifics now but you can read up more on stackoverflow. In short, order by rand() will always do a full table scan. Avoid it.
MySQLDump haz all the locks
Moving from mongrel to passenger
Muziboo was started when the world was just moving over from fastcgi to mongrel. Hence we were using a bunch of mongrels monitored by monit. However, every time you want to add more mongrels, you have to change your monit configuration file to start and monitor more mongrels. Also there is no easy way to reduce the number of mongrels when under less load. Hence we decided to move over to passenger. In passenger, you just have to change the number of max instances in the nginx configuration files and more app instances will be started under load. When the load is lesser, these instances are shut down. All this is automatically done for you.
Taming I/O during rsync
We use rsync to backup uploaded files uploaded by Muziboo’s user. We have about 1 TB of data and even though the backup is incremental, there is still considerable I/O to check which files have to be moved. To make sure that rsync does not overwhelm everything else, we used the following options
nice -20 ionice -c2 -n7 /usr/bin/rsync -avz -e ssh --times --size-only --stats ......
Nice and IONice help you keep the CPU and disk usage low. You can also use it for other cron/background jobs that you run.
With these changes in place, we have been able to accomodate all the growth on a single server (Quadcore + 12 GB RAM). I think with more growth we may have slowers mysql writes and at that point we may need to move to a separate db server.
Read some interesting comments on this post on Hacker News
Hosting a git repo yourself is a very simple exercise. If you don’t need (or don’t want to pay for) features offered by github, you can setup your server to host your git repo. All you need is a server with ssh access. Here are the steps that you need to follow
Login to your server
ssh tom@example.com
tom@example.com$ mkdir my_repo.git
tom@example.com$ cd my_repo.git
tom@example.com$ git init
On you local machine, where you have your code
$ cd my_code # cd to your code
$ git init
$ git add . # Adds everything. Use .gitignore to ignore some files
$ git remote add origin tom@example.com:my_repo.git
$ git push origin master
Now you can clone tom@example.com:my_repo.git on any number of machines. To not share password, put your public keys on tom@example.com
This approach gives you a basic git setup with ssh access (no http access). These steps should work on shared hosting too (as long as you have git available)
If you are using flexmock for testing your rails code and you are mocking an active record model using flexmock(:model, Model), you may get an error like
undefined method `destroyed?'
The problem is that flexmock defines a lot of methods/attributes like id and new_instance? etc but does not define a destroyed? method that rails now expects. Fortunately, this is easy to fix. Create a file ‘config/initializers/flexmock_extensions.rb’ and put this in
class FlexMock class MockContainerHelper def add_model_methods(mock, model_class, id) container = mock.flexmock_container mock_errors = container.flexmock("errors") mock_errors.should_receive(:count).and_return(0).by_default mock_errors.should_receive(:full_messages).and_return([]).by_default mock.should_receive(:id).and_return(id).by_default mock.should_receive(:to_params).and_return(id.to_s).by_default mock.should_receive(:new_record?).and_return(false).by_default mock.should_receive(:class).and_return(model_class).by_default mock.should_receive(:errors).and_return(mock_errors).by_default mock.should_receive(:destroyed?).and_return(false).by_default # HACK: Ruby 1.9 needs the following lambda so that model_class # is correctly bound below. lambda { } mock.should_receive(:is_a?).with(any).and_return { |other| other == model_class }.by_default mock.should_receive(:instance_of?).with(any).and_return { |other| other == model_class }.by_default mock.should_receive(:kind_of?).with(any).and_return { |other| model_class.ancestors.include?(other) }.by_default end end end
Basically, we are defining the method destroyed? and returning false everytime. Now your tests should work
Pickle is a great add on to cucumber as it provides steps for often used scenario. There is a step provided for verifying the association size of an active record instance. Therefore you can write something like
Then the user: “Joe” should have 1 friends
and it will check if joe.friends.size == 0. However this does not check the counter cache (if you have one). I modified it a bit to verify counter cache (changes in file features/step_definitions/pickle_steps.rb)
# assert size of association Then /^#{capture_model} should have (\d+) (\w+)$/ do |name, size, association| if(model!(name).respond_to?("#{association}_count")) model!(name).send("#{association}_count").should == size.to_i else model!(name).send(association).size.should == size.to_i end end
This assumes that you counter cache is named association_cache which is the convention in rails’ world.
YSlow is a popular tool for measuring and tweaking site performance. The tool recommends that Javascript be put at the bottom of the page (since javascript blocks the page until its loaded). However one of the challenges of doing this is that you cannot call any javascript function (that you have defined in your js files) before the files are loaded. This can be a serious limitation if you are doing a dynamic website where the header and footer (which is where the script tags are inserted) is static and all the dynamic content is in the middle of the page. If you have used any Ruby on Rails or similar framework, you know what I mean. Basically, its hard to put different code in the footer for different pages.
Lets take an example from Muziboo. I have a function in javascript to initialize the flash player Muziboo.Player.embedSingle(). If my script tags (for including the js) are in the header, I can make this call anywhere on the page and it will work. However if the javascript files are loaded last, this call has to be made after loading the javascript.
However you can overcome this limitation by using a javascript queue that holds the function calls till the external javascript files are loaded. Once the javascript is loaded, it can look at the queue and process the messages. So something like this would work
This idea is not new and is used by Google Analytics’ async tracking code. Lets see how to implement such a queue in javascript (using the prototype.js javascript library)
The queue is nothing but a simple javascript array. The process message simply pops out each message, executes it and then changes the push method on _muzq object to execute the message instead of storing it (since now the js is loaded and there is no need to wait). I am still trying to find out a better way than to use eval but this implementation works.
I just pushed some changes to my fork of the blackbook address book importer gem. Here is a list of changes
Sadly the yahoo address book importer is not working as yahoo as added a captcha before csv download. I will try to use an alternative view and fix the issue sometime soon. I have not tried aol and hotmail import.
If you need any help, please leave a comment here.
Yes … as unrealistic as it may sound, yesterday, I ran into an issue where something worked in Internet explorer but not in Firefox. I was trying to call an actionscript functions on a swf file from javascript. This is a fairly standard thing and is well documented on the web. I followed all the steps (registering a callback and embedding the swf using swfobject library) and it worked perfectly in IE. I could call my actionscript functions using javascript in IE but in firefox it would occasionally fail and I would see the following two messages in firebug
uncaught exception: [Exception... "Failure" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: http://localhost:9001/javascripts/swfobject.js?1259133537 :: anonymous :: line 4" data: no]
$(’movieName’).asFunc() undefined method
where movieName is the id of the swf file and asFunc is the function defined in actionscript using ExternalInterface.addCallback. After a lot of googling, I ran into this thread which explained the basic problem. Since my swf file was hidden in the footer of the page, the callbacks worked after I had once scrolled down and seen the footer. Yes as crazy as it sounds, that was really the problem. Firefox wants you to see the swf once before you make the function call. To make sure that the 1×1 sized swf file was always visible no matter where you are in the page, I added the following css rule
#movieName {
position: fixed;
top: 0;
left: 0;
}
This rule makes sure that no matter how much you scroll up or scroll down, the swf file is always at the same place (in this case, just at the starting of the page, relative to the browser window). You can read more about fixed positioning here. I am sure there must be a better explanation/solution for this problem but I have tried my approach and it atleast works. If you find a better solution, please post as a comment and I will update the post.
File column is a great plugin by Sebastian Kanthak for uploading images (and other files) in a rails’ site. The plugin has rmagick integration and can create thumbnails automatically for you. We use it for Muziboo and have been pretty happy with it (yes despite the rmagick dependency). We recently had to update the thumbnail dimensions for a few of our models. Its fairly simple to change the thumbnail sizes in the model and all new uploads are resized to new dimensions. However, the existing images are not automatically changed for you . Thankfully, on digging deeper into the code I found that there is pretty simple way to accomplish that. Lets say you have a user model (with a file_column field photo) and you want to update the thumbnails for all existing images. You just need to simply run the following commands
User.find(:all,:conditions => "photo IS NOT NULL").each do |u| u.photo = File.open(u.photo) u.save end
Ofcourse if you have a lot of users, you may wanna fetch them in batches. Also image resizing is a pretty CPU intensive operation so you may wanna watch out for that. Once the operation is over, you will have updated thumbnails for all your previously uploaded images.
Blackbook gem can be used to fetch contacts from gmail, yahoo and hotmail address books. However I recently found out that it would throw up a ‘Must be authenticated to access contacts’ error everytime I tried to import gmail contacts. On digging deeper, I found out that google has changed the contents of its cookie and blackbook uses this cookie to make sure that the user is authenticated. To fix the issue, you just need to make a one line change in the function scrape_contacts in the file lib/blackbook/importer/gmail.rb.
Change
unless agent.cookies.find{|c| c.name == 'GAUSR' && c.value == "mail:#{options[:username]}"} raise( Blackbook::BadCredentialsError, "Must be authenticated to access contacts." )
to
unless agent.cookies.find{|c| c.name == 'GAUSR' && (c.value =~ /mail.*:#{options[:username]}/)!=nil} raise( Blackbook::BadCredentialsError, "Must be authenticated to access contacts." )
If you want to install the updated gem instead of making the change, you can find it in my blackbook fork. My fork also includes the myspace address book import
