Backups
This service was neglected for far too long. Mostly because it was either bothersome to backup a live server without corrupting data, or tedious creating specific scripts that copies parts of the data needed. One issue was also storage.
The first attempt was to create a script that could copy the most necessary data from respective vm. All else was ignored as it could easily be replicated or deemed redundant. The initial testing went okay, but it created a security hole where if the container that ran the script would have been compromised, it would result in all servers connected to that backup system to be accessible. This was not desirable.
However, due to how Proxmox works, they had several ways to backup whole vms/containers. For vms, ignoring the live snapshots, which was unstable, the stop-backup-start was alright. Yes, it threw out the current users, but not for long. When the backup had started, it made a quick snapshot of the system and then started it again, sometimes it isn’t even noticable.
For containers, they work better with snapshots, and hence they are always online.
There is at least one extra copy of the backup. Everything is stored on a CIFSNFS-attached NAS.
Was this the best way to do this? For this particular system it works. However, if redundancy is required, one maybe need to look more into more expensive alternatives to be able to get what one wants.