That was extremely helpful. Your server runs on EDT, which is GMT -0400.
At midnight on your server, it's 4:00 GMT.
Do you see the 4:00 AM backup times on files on your server? Or on your backup server? Does the date command on your backup server show time in GMT or also in EDT? or in something else?
My guess is that rsync is syncing timestamps in GMT. That may or may not be a switch in in rsync. And of course I could be wrong.
But it appears evident to me, anyway, that the time difference between GMT and EDT is the culprit here.
Jeff