Proxmox Backup server

1

(% class="row" %)

2

(((

3

(% class="col-xs-12 col-sm-8" %)

4

(((

5

(% class="wikigeneratedid" %)

6

Following provides setup steps, configuration explanation and application instructions for backup server. This box both generates backups and syncs them to remote locations. View general backup explanation page [[Server backup>>doc:infra.Backup.WebHome]] for high-level information.

7

8

(% class="wikigeneratedid" %)

9

Web GUI: [[https:~~/~~/clio.schleppe:8007/#pbsDashboard>>url:https://clio.schleppe:8007/#pbsDashboard]]

10

11

= Backup Server configuration =

12

13

Backup server is setup with:

14

15

* zfs storage

16

* access control - api tokens

* datastore

** sync jobs

** prune jobs

** verify jobs

** permissions

* timings and simulator

23

24

== ZFS storage array ==

25

26

There are currently 2 x 8TB WD drives. Current pool status:

(((

kevin@clio:~$ sudo zpool status pergamum

31

pool: pergamum

32

state: ONLINE

33

scan: scrub repaired 0B in 09:52:23 with 0 errors on Sun Mar 10 10:16:24 2024

34

config:

35

NAME STATE READ WRITE CKSUM

36

pergamum ONLINE 0 0 0

37

raidz1-0 ONLINE 0 0 0

38

scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1 ONLINE 0 0 0

39

sdc1 ONLINE 0 0 0

40

errors: No known data errors

)))

=== Creating and expanding zfs pool ===

(((

zpool create pergamum raidz /dev/disk/by-partuuid/9fab17e5-df2d-2448-b5d4-10193c673a6b /dev/disk/by-partuuid/f801ed37-1d6c-ee40-8b85-6bfc49aba0fb -f

50

zfs set mountpoint=/mnt/pergamum pergamum

51

(zpool import -c /etc/zfs/zpool.cache -aN)

52

zpool export pergamum

)))

(((

have not tried yet, but adding another set of disks for an additional top-level virtual device to our existing RAID-Z pool:

59

60

61

zpool add -n pergamum raidz DISK1 DISK2

~> NOTE! `-n` is dry run, remove to commit.

)))

== Access Control ==

Each client host that wants to backup their contents to the backup server should have their unique API token for authentication.

API Token:

* user: [[root@pam>>mailto:root@pam]]

76

* token name: CLIENT_NAME

* expire: never

* enabled: true

Permissions - Add a API Token Permission:

81

82

* path: /datastore/proxmox-backup/CLIENT_NAME

83

* api token: root@pam!CLIENT_NAME

84

* role: DatastoreBackup

85

* propagate: true

86

87

>Note! The path will not be define until after the Datastore namespace is define in the steps below

88

89

== Proxmox datastore ==

90

91

If none exists create the datastore. Ours point is named `proxmox-backup` and points to ZFS storage mounted at `/mnt/pergamum`. All references to `proxmox-backup` referes to what you named it as in the create step here.

=== Namespace ===

Namespaces is what we will use in a datastore to separate permissions to each host. It's important to create these for the API tokens create in Access Control section above.

96

97

=== Prune & Garbage collect ===

98

99

We don't require backups for every day of the year. Pruning lets you systematically delete older backups, retaining backups for the last given number of time intervals. There exists a fantastic simulator that can be used to experiment with different backup schedules and prune options: [[https:~~/~~/pbs.proxmox.com/docs/prune-simulator/>>https://pbs.proxmox.com/docs/prune-simulator/]]. The current configuration is:

100

101

* datastore: proxmox-backup

* namespace: root

* keep last: 4

* keep: hourly: -

* keep daily: 6

* keep weekly: 3

* keep monthly: 6

* keep yearly: 4

* max_depth: full

* prune schedule: 0/6:00

* enabled: true

=== Verify jobs ===

Current configuration is:

116

117

* local datastore: proxmox-backup

* namespace: root

* max-depth: full

* schedule: daily

* skip verified: true

122

* re-verify after: 30 days

=== Permissions ===

Permissions are explained in the Access Control section above, but it can be easier to configure permissions from the datastore. Navigate to the datastore Permission tab and add API Token Permission:

127

128

* path: /datastore/proxmox-backup/CLIENT_NAME

129

* API Token: root@pam!CLIENT_NAME

130

* Role: DatastoreBackup

* Propagate: true

= Tailscale =

Tailscale is used to create a network that uses wireguard to transparently between local and remote machines. To not require a third party a local instance of headscale is used as the tailscale login server.

136

137

138

curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/focal.noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null

139

curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/focal.tailscale-keyring.list | sudo tee /etc/apt/sources.list.d/tailscale.list

140

141

sudo apt-get update

142

sudo apt-get install tailscale

143

144

systemctl status tailscaled.service

145

sudo tailscale up --login-server SUBDOMAIN.schleppe.cloud

tailscale status

Connect to headscale login server:

150

151

152

$ sudo tailscale up --login-server https://SUBDOMAIN.schleppe.cloud

153

154

To authenticate, visit:

155

156

https://SUBDOMAIN.schleppe.cloud/register/nodekey:fe30125f6dc09b2ac387a3b06c3ebc2678f031d07bd87bb76d91cd1890226c9f

Success.

View more info in the docs: [[https:~~/~~/earvingad.github.io/posts/headscale/>>https://earvingad.github.io/posts/headscale/]]

162

163

= Jottacloud client =

164

165

Cloud backup provider used is jottacloud. They provide a cli to easily add directories to sync to their cloud backup storage.

166

NOTE! This setup still uses user `kevin` and not the correct jottad user.

(((

# install jotta-cli

sudo curl -fsSL https://repo.jotta.cloud/public.asc -o /usr/share/keyrings/jotta.gpg

172

echo "deb [signed-by=/usr/share/keyrings/jotta.gpg] https://repo.jotta.cloud/debian debian main" | sudo tee /etc/apt/sources.list.d/jotta-cli.list

173

sudo apt-get update

174

sudo apt-get install jotta-cli

175

176

# configure runtime environment

177

sudo useradd -m jottad

178

sudo usermod -a -G jottad backup

)))

Create systemd file: `/usr/lib/systemd/user/jottad.service ` and enable with :

(((

{{code language="ini" layout="LINENUMBERS" title="/usr/lib/systemd/user/jottad.service"}}

188

[Unit]

189

Description=Jotta client daemon

[Service]

Type=notify

# Group=backup

# UMask=0002

# EnvironmentFile=-%h/.config/jotta-cli/jotta-cli.env

197

ExecStart=/usr/bin/jottad stdoutlog datadir %h/.jottad/

Restart=on-failure

[Install]

WantedBy=default.target

)))

== Flaws ==

Since proxmox backup server uses chunks for deduplicating data a complete file list is required. This makes it impossible to download a single file representing a VM or LXC, all files must be downloaded and imported into proxmox backup server for reconstruction.

208

209

It also seems like there are a LOT of files shifting - being added and deleted. Making the diff uploaded to jottacloud huge.

= Syncthing =

TODO

= Client Configuration =

216

217

Configure Backup on the Datacenter or PVE host level in the proxmox web GUI. If a backup storage is already added input the following preferences:

218

219

* selection mode: include selected VMs

220

* send email to: EMAIL_ADDRESS

221

* email: on failure only

222

* mode: snapshot

223

* enabled: true

224

* job comment: ~{~{guestname}}, ~{~{node}}, ~{~{vmid}}

225

226

= Debugging/issues live here =

227

228

== Permission denied anything for certain backups ==

229

230

When trying to restore a VM I noticed that it was very outdated. Before doing anything I got a `Permission denied (os error 13)` error message. I checked the permissions of the storage mount in proxmox cluster, generated new API key, removed and re-added the storage to node getting permission denied, and what gave it away I also got it when running the CLI command from proxmox-backup-server host.

231

232

233

kevin@clio:~$ sudo proxmox-backup-client snapshot forget -ns apollo -repository proxmox-backup 'vm/201/2023-07-31T01:31:18Z'

234

[sudo] password for kevin:

235

Password for "root@pam": *****************

236

fingerprint: **:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**

237

Are you sure you want to continue connecting? (y/n): y