rsnapshot style pull backups with rsync and overlay for local deduplication

[felix-engelmann]

Inspired by issue #1811 and a security preference for pull based backups, we would like to know how bad the following solution is and especially how mount --bypass-lock behaves when the repo changes. All of this still relates to the stable 1.* releases.

There is a server on the public internet that we want to back up to a raspberry with a large disk attached.

For a backup cycle, we find the latest archive in the repo:

LATEST_ARCHIVE="$(borg list --last 1 --short "$REPOSITORY")"

create a temporary directory and mount it there:

mkdir yesterday_mnt
borg mount --bypass-lock server.borg::$LATEST_ARCHIVE yesterday_mnt

then we create an empty overlay on top of the borg mount:

mkdir merged new_sync work
mount -t overlay overlay -o lowerdir=yesterday_mnt/merged,upperdir=new_sync,workdir=work merged

The merged overlay folder now presents yesterdays snapshot, but we can change it however we like. This is done by rsync that syncs the local view of merged with our server:

rsync -aAX -v --delete --numeric-ids --relative --delete-excluded --exclude '/dev' root@server:/ merged/

With an adjusted directory looking like the server in the merged directory, we can make a local borg backup:

borg create -p --stats $REPOSITORY::'nlogn-full-{now:%Y-%m-%d_%H-%M}' merged

The benefit of this is reducing the required disk space locally because the data is transparently served from the last archive.
The remaining question is: does mount break when another borg process creates a new archive and therefore is it safe to use --bypass-lock. We understand that destructive operations such as compact or delete of the mounted archive will break things, but if we can assure that only borg create is running in parallel, is that somewhat safe? Empirically, we put the repo under an overlay and mounted it from the merged. Except of the lock file, nothing changed, which gives us confidence that the mount operation is mostly read-only and with --bypass-lock can run in the background.
We are not too much worried regarding performance, as the rsync check will query every file in the mounted archive, which is probably the slowest part of the process.

Before that approach, I tried the serve --append-only but did not manage to recover a previous manifest after a archive deletion. The chunks were present, but the most recent manifest had the archive removed.

[ThomasWaldmann]

IIRC, the reason to add --bypass-lock to mount was to be able to work with repositories on r/o media - in that case, the repo of course does not change.

In borg 1.x mount works by building an in-memory filesystem from the relevant archives. If you do a full-repo mount, it will happen on-demand, when you cd into one of the "archive directories" inside the mounted fs. For a single archive mount, it will happen immediately.

Chunk data will then be loaded from the repo as required, e.g. if you open and read a file in the mounted fs.

That code is written for a locked (or at least not-changing) repository.

Whether that works with a changing repository, will depend on the change. Guess adding an archive will do no harm, but borg delete/prune/compact/check might cause problems with the mounted fs.

[ThomasWaldmann]

Raspberry PI as backup server: I would not trust that hw enough for that TBH, but my experiences with that hw are from some years ago and maybe the current hw is more stable.

In any case, you will need enough RAM for borg (see the docs for a formula). If you run out of RAM and you can't upgrade RAM, you have a problem. Especially on "asymetric" systems that can be a problem (like: litte RAM, but huge disk with huge repository).

[ThomasWaldmann]

overlayfs: I don't use that, so can't comment on quality/features/feasability.

For what do you use new_sync dir?

About borg mount: it has limited support for metadata, see the docs. E.g. it does not support ACLs.

rsync: IIRC it works based on mtime (and size?) by default. Check if that is good enough for you.

Maybe also exclude /proc and /sys.

[ThomasWaldmann]

In general:

What you plan to do is rather complex, compared to just using borg push as usual. Complex systems can break more easily.

borg create has a "files cache", which accelerates its operation by a lot for unchanged files, by checking each src files' ctime/inode/size. Guess you will need to tweak that using --files-cache=... for your approach because your inode numbers won't be stable, also not sure about ctime.

[ThomasWaldmann]

About append-only: that is not how it is supposed to work, check our docs how to use append-only in case of emergency.