Make sure the permissions of the key directory and keys are correct on the client. The ~/.ssh directory should only have execute, read and write permissions for the user. If not then change them:

User can execute, read and write

chmod 700 ~/.ssh

For the private keys and also the id_rsa, user can read and write

chmod 600 ~/.ssh/id_rsa

For the public keys, user can read and write, others can read

chmod 644 ~/.ssh/*.pub