Backups are primarily done through proxmox backup server taking snapshot of running lxc and vm's. These are stored on mirrored ZFS array and synchronized to both off-site location and cloud storage provider.

View all required configuration on page Proxmox Backup server.

3-2-1 backup principle

Original rule would be:
 - 3 copies of data
 - on 2 different media
 - with 1 copy being off-site

from here we implement the following storage strategy 3-2-2: 

 - Maintain 3 copies: 1 local, 1 off-site and 1 cloud copy.
 - Use 2 different storage media: HDD on both sites and Cloud storage
 - Keep 2 copies off site: secondary datacenter and Cloud.

Local backup

We run most all our services with proxmox. It comes with a companion software proxmox backup server that is used to backup all resources on interval. All virtual machines and LXC containers running in our infra are then managed by Proxmox Backup server. Every night snapshot are taken at between 02-05 from all servers hosts.  

View cluster backup configuration: https://apollo.schleppe:8006/#v1:0:18:4:::::32::21
View backups at: https://clio.schleppe:8007/#pbsDashboard
View syncthing: http://clio.schleppe:8384

Offsite backup

To ensure backups are not lost if the entire local datacenter is lost we want to syncronize all backups to an off-site location. Syncthing is used to duplicate everything on local proxmox backup server.

Tailscale is used networking, find configuration details at Proxmox Backup server - Tailscale.
Syncthing is used for file transfer, find configuration details at Proxmox Backup server - Syncthing.

Cloud backup

All proxmox backup files, including chunk, namespace and configuration files are syncronized to jottacloud using jotta-cli, view configuration details on page Proxmox Backup server - Jottacloud client.

There is an inherit flaw in this backing up deduplicated and chunked data, as an entire download is required to reconstruct backups for restoration. Read more about it on the page linked above.