From cb481b81c53a6f27b4de2a3af71e3e41e72c204a Mon Sep 17 00:00:00 2001 From: Thomas Cook Date: Thu, 22 Jan 2026 11:00:04 -0600 Subject: [PATCH 1/2] switch to relative paths for proper rendering on readthedocs paths were broken on readthedocs because full paths to github pages were there, switched to relative paths for links for proper rendering on readthedocs. --- docs/tools/backup-and-restore.md | 253 +++++++++++++------------------ 1 file changed, 103 insertions(+), 150 deletions(-) diff --git a/docs/tools/backup-and-restore.md b/docs/tools/backup-and-restore.md index 053b07a..10ba98e 100644 --- a/docs/tools/backup-and-restore.md +++ b/docs/tools/backup-and-restore.md @@ -1,48 +1,48 @@ -# Backup and Restore Instructions: +# Backup and Restore Instructions ## Overview -Backup and restore scripts are provided that will backup your graphistry environment to remote blob storage (AWS s3, Azure Blob Storage or Google Cloud Storage). Graphistry uses [restic](https://restic.net/) backup utility which provides fast and secure incremental backups to remote blob storage. More details can be found in the [restic documentation](https://restic.readthedocs.io/en/stable/index.html). +Backup and restore scripts are provided that will backup your Graphistry environment to remote blob storage (AWS S3, Azure Blob Storage, or Google Cloud Storage). Graphistry uses [restic](https://restic.net/), a fast and secure incremental backup utility for remote blob storage. More details can be found in the [restic documentation](https://restic.readthedocs.io/en/stable/index.html).
-> **Warning** -> On a restore, the existing postgres database and data directory are irrevocably lost. If think you may need the postgres and/or data directory on the restore server, either run backup.sh from the restore server or manually copy the data directory and export from the postgres database. See `${FROM_PATH}/etc/scripts/copy-db-local.sh` for details on manually exporting postgres. - -
+> **Warning** +> On restore, the existing Postgres database and data directory are irrevocably lost. If you think you may need the Postgres database and/or data directory on the restore server, either run `backup.sh` from the restore server or manually copy the data directory and export from the Postgres database. +> +> See `${FROM_PATH}/etc/scripts/copy-db-local.sh` for details on manually exporting Postgres. +
-### [Configuration](https://github.com/graphistry/graphistry-cli/blob/master/docs/backup-and-restore.md#configuration) -### [Backup](https://github.com/graphistry/graphistry-cli/blob/master/docs/backup-and-restore.md#backup) -### [Restore](https://github.com/graphistry/graphistry-cli/blob/master/docs/backup-and-restore.md#restore) -### [Scheduling Backups](https://github.com/graphistry/graphistry-cli/blob/master/docs/backup-and-restore.md#scheduling-backups) +### [Configuration](#configuration) +### [Backup](#backup) +### [Restore](#restore) +### [Scheduling Backups](#scheduling-backups) -
+
## Configuration -The following environment variables are required for depending on the cloud provider: +The following environment variables are required depending on the cloud provider. -[**AWS:**](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#amazon-s3) +### AWS -We recommend adding these to ~/.bashrc or ~/.profile and re-starting the shell or sourcing the file to pick up the changes: +See the [restic Amazon S3 documentation](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#amazon-s3). -``` +We recommend adding these to `~/.bashrc` or `~/.profile`, then restarting the shell or sourcing the file: + +```bash export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export RESTIC_REPOSITORY=s3:s3.amazonaws.com// export RESTIC_PASSWORD= +```` -``` - -[**Azure:**](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#microsoft-azure-blob-storage) +### Azure -We recommend adding these to ~/.bashrc or ~/.profile and re-starting the shell or sourcing the file to pick up the changes: - -``` -# we recommend adding these to ~/.bashrc or ~/.profile +See the [restic Azure Blob Storage documentation](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#microsoft-azure-blob-storage). +```bash export AZURE_ACCOUNT_NAME= export AZURE_ACCOUNT_KEY= @@ -51,240 +51,193 @@ export AZURE_ACCOUNT_KEY= export AZURE_ACCOUNT_NAME= export AZURE_ACCOUNT_SAS= -# and - -export RESTIC_REPOSITORY=azure::/ +export RESTIC_REPOSITORY=azure::/ export RESTIC_PASSWORD= - ``` -[**Google Cloud Storage Authentication**](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#google-cloud-storage): +### Google Cloud Storage -Restic supports using a Service Account to access the storage bucket, see instructions in the link above, or you can follow these steps from the [Google CLI](https://cloud.google.com/sdk/gcloud): +See the [restic Google Cloud Storage documentation](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html#google-cloud-storage). -``` -# define the project: -project= +You may also use the [Google Cloud CLI](https://cloud.google.com/sdk/gcloud): -# define the bucket: +```bash +project= bucket=gs:/// -# define the default compute service account credential file path: GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gs-secret-restic-key.json -# get the default service account email address: -service_acct=$(gcloud --project=${project} iam service-accounts list --format=json | jq -r '.[] | select(.displayName=="Compute Engine default service account")| .email') +service_acct=$(gcloud --project=${project} iam service-accounts list --format=json \ + | jq -r '.[] | select(.displayName=="Compute Engine default service account") | .email') -# get the service account key file: -gcloud iam service-accounts keys create ${GOOGLE_APPLICATION_CREDENTIALS} --iam-account=${service_acct} - -# grant the default service account required permissions to access the bucket: -gsutil iam ch serviceAccount:${service_account}:objectCreator,objectViewer,objectAdmin ${bucket} +gcloud iam service-accounts keys create ${GOOGLE_APPLICATION_CREDENTIALS} \ + --iam-account=${service_acct} +gsutil iam ch serviceAccount:${service_acct}:objectCreator,objectViewer,objectAdmin ${bucket} ``` -We recommend adding these to ~/.bashrc or ~/.profile and re-starting the shell or sourcing the file to pick up the changes: - -``` +Add the following to your shell environment: +```bash export GOOGLE_PROJECT_ID=123123123123 export GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gs-secret-restic-key.json export RESTIC_REPOSITORY=gs::/ export RESTIC_PASSWORD= - ```
## Backup -backup config options: +Backup configuration options: -| variable | default if unset | description | -|-------------------|---------------------------------------|--------------------------------| -| FROM_PATH | /home/ubuntu/graphistry | graphistry install dir | -| DATA_DIR | ${FROM_PATH}/data | override if symlinked data dir | -| LOCAL_SUDO_DISK | sudo | set to empty str to override | -| LOCAL_SUDO_DOCKER | sudo | set to empty str to override | -| DRY_RUN | False | dry run only | -| RESTIC_TAGS | "graphistry" | space separated list of tags | +| Variable | Default if unset | Description | +| ----------------- | ----------------------- | --------------------------------------- | +| FROM_PATH | /home/ubuntu/graphistry | Graphistry install directory | +| DATA_DIR | ${FROM_PATH}/data | Override if data directory is symlinked | +| LOCAL_SUDO_DISK | sudo | Set to empty string to disable sudo | +| LOCAL_SUDO_DOCKER | sudo | Set to empty string to disable sudo | +| DRY_RUN | False | Dry run only | +| RESTIC_TAGS | "graphistry" | Space-separated list of snapshot tags | -

- -1. Make sure you have defined RESTIC_REPOSTITORY, RESTIC_PASSWORD and the other authentication variables required for the cloud provider in the [configuration](https://github.com/graphistry/graphistry-cli/blob/master/docs/backup-and-restore.md#configuration) section above. +
-2. ssh into the graphistry server e.g. `ssh -i ubuntu@IP_addr` +1. Define `RESTIC_REPOSITORY`, `RESTIC_PASSWORD`, and provider-specific credentials as described in the [Configuration](#configuration) section. +2. SSH into the Graphistry server: -3. cd to the scripts directory in + ```bash + ssh -i ubuntu@ + ``` +3. Change to the scripts directory: -``` -# AWS: +```bash +# AWS cd /home/ubuntu/graphistry/compose/etc/scripts -# Azure +# Azure cd /var/graphistry/compose/etc/scripts - ``` -4. Run the backup script: +4. Run the backup script: -``` +```bash # AWS ./backup.sh # Azure -FROM_PATH=/var/graphistry/ ./backup.sh +FROM_PATH=/var/graphistry ./backup.sh ``` ### Additional examples -``` -# dry-run: will only print the commands and execute restic with --dry-run option flag for testing purposes - -DRY_RUN=True ./backup.sh - -# adding tags to a back (useful if you are using the same repo for multiple graphistry servers) this example creates three tags attached to the restic snapshot: server1, dev and nightly: +```bash +# Dry run +DRY_RUN=True ./backup.sh +# Add snapshot tags RESTIC_TAGS="server_1 dev nightly" ./backup.sh -# if your data directory is a symlink, you need to override the DATA_DIR with the path to mount point as restic does not follow symlinks - +# Override data directory if symlinked DATA_DIR=/mnt/data ./backup.sh - ``` -
## Restore -| variable | default if unset | description | -|-------------------|---------------------------------------|--------------------------------| -| TO_PATH | /home/ubuntu/graphistry | graphistry install dir | -| DATA_DIR | ${FROM_PATH}/data | override if symlinked data dir | -| LOCAL_SUDO_DISK | sudo | set to empty str to override | -| LOCAL_SUDO_DOCKER | sudo | set to empty str to override | -| DRY_RUN | False | dry run only | -| RESTIC_TAGS | "graphistry" | space separated list of tags | +Restore configuration options: -

+| Variable | Default if unset | Description | +| ----------------- | ----------------------- | --------------------------------------- | +| TO_PATH | /home/ubuntu/graphistry | Graphistry install directory | +| DATA_DIR | ${FROM_PATH}/data | Override if data directory is symlinked | +| LOCAL_SUDO_DISK | sudo | Set to empty string to disable sudo | +| LOCAL_SUDO_DOCKER | sudo | Set to empty string to disable sudo | +| DRY_RUN | False | Dry run only | +| RESTIC_TAGS | "graphistry" | Space-separated list of snapshot tags | -1. ssh into the graphistry server e.g. `ssh -i ubuntu@IP_addr` +
-2. cd to the scripts directory in +1. SSH into the Graphistry server. +2. Change to the scripts directory: -``` -# AWS: +```bash +# AWS cd /home/ubuntu/graphistry/compose/etc/scripts -# Azure +# Azure cd /var/graphistry/compose/etc/scripts - ``` -3. Run the backup script: - +3. Run the restore script: -``` +```bash # AWS - ./restore.sh ``` -
- -``` +```bash # Azure - -TO_PATH=/var/graphistry/ ./restore.sh +TO_PATH=/var/graphistry ./restore.sh ```
## Scheduling Backups -Any scheduler can be used, but below are some examples of setting up cron to schedule nightly or weekly backups. - -``` -# list your current crontab entries: - -crontab -l -``` - -
+Any scheduler can be used. Below are cron examples. +```bash +crontab -l +crontab -e ``` -# edit your contrab -crontab -e -``` +### AWS cron examples -
- -### AWS crontab examples -``` -# add the following line to run backup either daily: - -0 0 * * * /home/ubuntu/graphistry/compose/etc/scripts/backup.sh - -# add the following line to run backup either weekly: +```bash +# Daily +0 0 * * * /home/ubuntu/graphistry/compose/etc/scripts/backup.sh +# Weekly 0 0 * * 0 /home/ubuntu/graphistry/compose/etc/scripts/backup.sh ``` -
+### Azure cron examples -### Azure crontab examples +```bash +# Daily +0 0 * * * TO_PATH=/var/graphistry /var/graphistry/compose/etc/scripts/backup.sh +# Weekly +0 0 * * 0 TO_PATH=/var/graphistry /var/graphistry/compose/etc/scripts/backup.sh ``` -# add the following line to run backup either daily: - -0 0 * * * TO_PATH=/var/graphistry/ /var/graphistry/compose/etc/scripts/backup.sh -# add the following line to run backup either weekly: +### Wrapper script example -0 0 * * 0 TO_PATH=/var/graphistry/ /var/graphistry/compose/etc/scripts/backup.sh -``` - -#### Creating a script that has several variables that your crontab entry will call - -``` +```bash #!/bin/bash export AZURE_ACCOUNT_NAME= export AZURE_ACCOUNT_KEY= -export RESTIC_REPOSITORY=azure::/ +export RESTIC_REPOSITORY=azure::/ export RESTIC_PASSWORD= export FROM_PATH=/var/graphistry - export RESTIC_TAGS="server2 prod nightly" ${FROM_PATH}/compose/etc/scripts/backup.sh - ``` -make the script executable: +Make executable: -``` +```bash chmod +x ~/scripts/my_backup_script.sh ``` -add the path to the above to your crontab: - -``` -crontab -e - -# then add the following for nightly backups: +Add to crontab: +```bash 0 0 * * * ~/scripts/my_backup_script.sh - ``` - - - - - - From 95350838d4e05aeeb231665ed7e4f8d220614f1b Mon Sep 17 00:00:00 2001 From: Thomas Cook Date: Thu, 22 Jan 2026 19:06:08 -0600 Subject: [PATCH 2/2] fix sphinx build error - added pytz to requirements --- infra/requirements-python.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/infra/requirements-python.txt b/infra/requirements-python.txt index 7049a0b..feffb2b 100644 --- a/infra/requirements-python.txt +++ b/infra/requirements-python.txt @@ -1 +1,2 @@ -linkify-it-py==2.0.3 \ No newline at end of file +linkify-it-py==2.0.3 +pytz==2025.2