Switching SSH keys between Heroku accounts

At times, I have to switch between a few different Heroku accounts. Apart from having to login again, the other annoyance is having the right SSH key active. If you don’t have the right SSH key active, (i.e. if the SSH auth agent has more than one key added to it, or if it has no keys at all),  you’ll see errors that look like:

Your key with fingerprint: ... is not authorized to access <application-name>.
fatal: the remote end hung up unexpectedly

The proper solution for this involves configuring them in the SSH configuration files (~/.ssh/config or /etc/ssh/ssh_config). [See this article or ssh_config manpage for details]

But I’m loath to maintain all that configuration just for the sake of occasionally switching between accounts. Here’s what I usually do instead:

  1. Clear any active identities (removing all ambiguity about which SSH key should be picked up for auth)
    $ ssh-add -D
  2. ssh-add key for the account
    $ ssh-add ~/.ssh/an_account_key
  3. Push to Heroku
    $ git push heroku-remote master

Of course, this assumes that the key is already associated with your Heroku account. If you haven’t, you can do that (after heroku login) with:

$ heroku keys:add ~/.ssh/an_account_key


Note: On Linux, if you’re on GNOME, the gnome-keyring-daemon keeps adding keys back to the auth agent as you keep trying to remove them with ssh-add -D. So, it’ll look like the command is not working. The solution is to disable the damn thing (Google for it). I find the daemon annoying for the popups it keeps throwing at me, so personally, I’d be glad to see it gone.


While I use this technique mostly with Heroku, this is useful for any situations involving SSH and multiple keys. It’s useful when switching between SSH accounts, or switching between GitHub accounts (or other Git accounts), and in general, anything involving switching SSH keys for SSH auth. Of course, if you find yourself switching between the same set of accounts (keys) frequently, consider configuring them using ssh_config.


[PS: This post is based on one of my SO answers: http://stackoverflow.com/questions/13752908/managing-multiple-ssh-keys-on-heroku/13876622#13876622]

Take control of your Heroku Git repository

Have you ever wished to have a finer control over your Heroku git repository? There’s a neat little Heroku plugin that gives you just that: heroku-repo.

Around the first release of my current project, things were happening in a frenzy. There were plenty of last minute fixes, and each of them would get built and pushed to our staging instance. On occasions, when people couldn’t wait for the build to finish (wasn’t a particularly long build though), or when the build was a bit flaky, they’d push directly to the Heroku staging repo. At some point, Heroku started rejecting our pushes, because it’s git tree had diverged from ours. It wasn’t nice, but we resorted to doing force pushes to the Heroku git repo from then on.

Once the release frenzy was over, I started investigating the issue. I started by trying to get a local clone of the Heroku repo, to see if I’d find something there, but it kept timing out. I opened a shell on the staging instance and tried searching for a repo somewhere there, but in vain. Heroku doesn’t host the repo from there. I had pretty much decided to blow up the staging app and recreate it, when I found this plugin.

It’s a plugin for the Heroku toolbelt that, in a sense, gives you raw access to the git repo that your Heroku application uses. Using this, I managed to download the git repo locally, which took quite a while because the repo had grown to some insane size (was a few hundred MBs, nearing a GB). Digging into the repo, I found a huge pack file that was causing all the issues. I suspected it to be because of a huge binary accidentally checked in by a team member, but my git-fu isn’t really good enough to say for sure. Running a gc on the repo using the plugin didn’t really help. So, I was still left with having to blow up and recreate the staging app.

I thought it’d be nice if I could just reset the Heroku git repo, and start over instead of having to recreate the app (and then add all the addons to it, which was a bit of a hassle since there were paid addons and I’d have to contact the instance owner to re-enable them). So, to figure out where the repo is hosted and how the plugin manages access it, I went through the plugin source code. Turns out, the repo is hosted on S3, and Heroku toolbelt exposes the S3 URL to it’s plugins. Better yet, the plugin itself had an undocumented command to reset and upload an empty git repo back to S3 🙂

This is pretty amazing. I can now start over with a clean repo in case my Heroku repo is messed up for any reason. On top of that, I can now deploy an entirely new app into my Heroku instance without leaving any dangling commits (not that it’s a common usecase, or even a useful one). See this protip for details: https://coderwall.com/p/okrlzg.

To install the plugin, do:

$ heroku plugins:install https://github.com/lstoll/heroku-repo.git

Here’s a few commands that I found useful:

    • Download the Git repo as an archive (useful when you can’t clone from Heroku)
      $ heroku repo:download -a appname
    • GC the repo (on Heroku)
      $ heroku repo:gc -a appname
    • Reset the repo and upload an empty repo
      $ heroku repo:reset -a appname

The plugin has a few more useful commands. Do check it out on GitHub: https://github.com/lstoll/heroku-repo. Also, I’d recommend going through it’s source code to see how it works. I thought it was  pretty neat.

Bypassing SSL certificate checks during gem installation

I had to re-install rvm on my MacBook because my gemsets were a bit messed up, and I thought I should start over with a clean install. I just rm -rf‘ed ~/.rvm, and then went ahead and re-installed it according to the instructions at https://rvm.io/. Installed ruby 1.9.3, and the openssl package as described here. But after that, both gem install and bundle install started failing because the SSL certificate from https://rubygems.org couldn’t be verified. Couldn’t quite figure out what was wrong. But, after a bit of googling, found out a way to skip the SSL certificate checks.

To skip the SSL certificate checks, just add this line to your .gemrc

:ssl_verify_mode: 0

This causes the gem and bundle commands to skip SSL certificate verifications when fetching them from a HTTPS source.

Of course, you can also bypass the error by using a non HTTPS URLs for your gem sources in your Gemfile (when using bundler). So, something like:

source 'https://rubygems.org'

in your Gemfile, will become:

source 'http://rubygems.org'


Neither of these actually fix the problem. They just avoid SSL certificate checks, or use a non SSL source. I still don’t know what went wrong during the re-install.


Ruby 1.9.3 segfault in OSX Lion due to OpenSSL

I just set-up my new MacBook Pro, running OSX Lion (10.7.3), for Rails development. Installed rvm, installed Ruby 1.9.3 using rvm and did a gem install rails -v 3.2.3. Everything went fine, until I tried to create a new rails app.

rails new <app_name> would create the app structure, but would then fail with a segfault in http.rb. Here’s what the stacktrace looked like:

run  bundle install
/Users/CodeMangler/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:799: [BUG] Segmentation fault
ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-darwin11.3.0]

— Control frame information ———————————————–
c:0038 p:—- s:0217 b:0217 l:000216 d:000216 CFUNC :connect
c:0037 p:0011 s:0214 b:0214 l:001570 d:000213 BLOCK /Users/CodeMangler/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:799

So, the failure was while doing a bundle install at the end of creating a new rails application, which’s new in rails (to me anyway. It’s been a while since I last looked into rails).
Googling for ruby segfault in http.rb pointed towards issues with OpenSSL. Then I found this post that describes how to fix the issue: http://www.rojotek.com/blog/2012/01/20/how-to-get-openssl-in-ruby-1-9-3-working-on-osx-10-7-fixing-the-segmentation-fault-with-ruby-openssl/

It tuns out, you’ve to install the openssl package for rvm, and then, while installing ruby, point it to use this version of OpenSSL.

To quote steps from the post:

$ rvm pkg install openssl
$ rvm remove 1.9.3 # uninstall the existing version, if you've installed one
$ rvm install 1.9.3 --with-openssl-dir=$rvm_path/usr --with-gcc=clang

Do that, and you should be all set. I tried creating a new rails app after that, and it went through just fine. Also, from googling, it appears that if it’s a segfault in http.rb, it’s most likely due to OpenSSL.