Since I so seldom setup ssh I always forget the details, here is a good guide:
Setup SSH keys

An addition to this is that you may have to do ssh-add on the client to get it working.