Ubuntu: Why is this rsync + ssh cron job giving me 'Permission denied (publickey)' errors?



Question:

I make frequent backups to a local drive which I want to sync daily to a remote server.

The target server is configured for SSH key (no password) access only. Since my primary SSH key for that server is passphrase-protected, I've created a second SSH key (not passphrase protected) + user to use for unattended backups - this way I do not have to be present to enter my passphrase when cron runs.

I'm using cron and rsync, and all of the commands work individually, but fail when combined.

The furthest I've got while troubleshooting is running

env -i sh -c "rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/"  

which returns the error

Permission denied (publickey).  rsync: connection unexpectedly closed (0 bytes received so far) [sender]  rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.0]  

Any tips on how to troubleshoot this further?


Here's what I've tried so far and I'm out of ideas:

  1. Cron is definitely running ps aux | grep cron
  2. Nothing unusual in /var/log/syslog Sep 7 13:22:01 desktop CRON[6735]: (tom) CMD (sh /home/tom/Documents/Scripts/offsite-backup)

  3. SSH in Terminal to remote server as the backup user works ssh backups-user@XX.XX.XX.XX

  4. Running the command in Terminal works perfectly rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/
  5. Manually specifying the path to the backups-user key has no effect rsync -lrstRO --delete --exclude 'lost+found' -e 'ssh -i /home/tom/.ssh/backups-only' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/

  6. Replacing the non-functioning command with a simple test command works echo "Hello world" > ~/Desktop/test.txt

  7. Shouting/swearing at the computer had no effect (but made me feel better temporarily).


Edit 1:

Here's my crontab file and the script it calls.

...  # m h  dom mon dow   command  MAILTO=""  * * * * * sh /home/tom/Documents/Scripts/offsite-backup  

and

#!/bin/bash    rsync -lrstRO --delete --exclude 'lost+found' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/  

Edit 2:

Just to clarify, /var/log/auth.log on the target server contains the line Sep 11 08:23:01 <hostname> CRON[24421]: pam_unix(cron:session): session closed for user root This is confusing because I'm no longer running cron every minute locally, but a new entry still appears every minute in the server logs. Crontab files for all users (including root) on the server are empty & do nothing.

Also, user 'backups-only' was created only on the server and with limited rights, with a dedicated SSH key copied to my desktop machine. I'm assuming this is the way to go because everything works when running the commands manually.

The crontab file posted above is for me, user 'tom' on my desktop machine. My intent is to have it call the script which should log in to the server as user 'backups-only'. I just tried running the backup script (rather than the command inside it) and it successfully connected & worked. I ran it on my desktop as user 'tom', same user who created the cron job that won't work. Here's the output from the server log corresponding with that successful login

Sep 11 08:35:31 <hostname> sshd[25071]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key  Sep 11 08:35:32 <hostname> sshd[25071]: Accepted publickey for backups-only from <desktop IP> port 54242 ssh2: RSA e2:e6:07:27:c1:continues...  Sep 11 08:35:32 <hostname> sshd[25071]: pam_unix(sshd:session): session opened for user backups-only by (uid=0)  Sep 11 08:35:32 <hostname> systemd-logind[638]: New session 12 of user backups-only.  Sep 11 08:36:00 <hostname> sshd[25133]: Received disconnect from <desktop IP>: 11: disconnected by user  Sep 11 08:36:00 <hostname> sshd[25071]: pam_unix(sshd:session): session closed for user backups-only  


Solution:1

Since everything is working fine from the command line, the error Permission denied (publickey) means that the SSH part of rsync is using a different identity file than the specified username.

From Jan's comment on the original question, we can specify the identity file in the rsync command using -e 'ssh -i /path/to/identity.file' ....

Using the below command to start off with a fresh environment in cron and specifying the complete path to the file apparently solves the issue:

env -i sh -c "rsync -lrstRO --delete --exclude 'lost+found' -e 'ssh -i /home/tom/.ssh/backups-only' /Backups/auto-daily-backups/./ backups-only@XX.XX.XX.XX:/backups/desktop/"  

I'm still really interested in this finding. It probably has to do with cron, the fact that it starts with minimal environment variables, and the ssh-agent. I'll be setting up the same scenario ina a couple of days to test it out and report back.


Solution:2

Have you already tried the old trick of cleaning up the hosts files? I mean:

rm ~/.ssh/known_hosts  

It's worth trying as ssh will rebuild it and you will get rid of stale stuff. You can of course also remove the parts belonging to a given IP / Host.

More questions: Is your cron job running under your UID or is it running as user cron or root?


Solution:3

Use the rrsync script together with a dedicated ssh key as follows:

REMOTE server

mkdir ~/bin  gunzip /usr/share/doc/rsync/scripts/rrsync.gz -c > ~/bin/rrsync  chmod +x ~/bin/rrsync  

LOCAL computer

ssh-keygen -f ~/.ssh/id_remote_backup -C "Automated remote backup"      #NO passphrase  scp ~/.ssh/id_remote_backup.pub devel@10.10.10.83:/home/devel/.ssh  

REMOTE computer

cat id_remote_backup.pub >> authorized_keys  

Prepend to the newly added line the following

command="$HOME/bin/rrsync -ro ~/backups/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding  

So that the result looks like

command="$HOME/bin/rrsync -ro ~/backups/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAA...vp Automated remote backup  

LOCAL

Put in your crontab the following script with x permission:

#!/bin/sh  echo ""  echo ""  echo "CRON:" `date`  set -xv  rsync -e "ssh -i $HOME/.ssh/id_remote_backup" -avzP devel@10.10.10.83:/ /home/user/servidor   

Source: http://www.guyrutenberg.com/2014/01/14/restricting-ssh-access-to-rsync/


Solution:4

I think you haven't configured the sshd_config file properly. Verify that PermitRootLogin yes and PubkeyAuthentication yes for remote maintenance.


Note:If u also have question or solution just comment us below or mail us on toontricks1994@gmail.com
Previous
Next Post »