This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision | ||
manuals:vps:datasets [2016/10/30 14:46] – toms | manuals:vps:datasets [2022/01/17 19:51] – Reorganize, move mounts away Aither | ||
---|---|---|---|
Line 1: | Line 1: | ||
+ | < | ||
====== Datasets ====== | ====== Datasets ====== | ||
+ | |||
+ | Dataset is a term from the ZFS filesystem that we're using everywhere. You can imagine it as | ||
+ | a formatted partition on disk containg directories and files. For example, btrfs has | ||
+ | a similar concept called // | ||
The dataset in vpsAdmin directly represents the ZFS dataset on the hard drive. Datasets | The dataset in vpsAdmin directly represents the ZFS dataset on the hard drive. Datasets | ||
- | are used for VPS and NAS data. The concept of a dataset replaces NAS | + | are used for VPS (each VPS has its own dataset) |
- | exports. A VPS dataset can be used the same way as an NAS. | + | the same way as an NAS, but are located in different locations (VPS details and the NAS menu). |
+ | The operations you can carry out with them are the same, such as creating snapshots, restoring | ||
+ | to snapshots or mounting datasets to VPS. | ||
{{: | {{: | ||
Line 10: | Line 17: | ||
quotas and ZFS properties for various data/apps. | quotas and ZFS properties for various data/apps. | ||
- | VPS datasets are located in the VPS details and NAS datasets are in the NAS menu. | + | VpsAdmin |
- | The operations you can carry out with them are the same. VpsAdmin | + | |
- | creating | + | |
{{: | {{: | ||
Line 51: | Line 56: | ||
is applied and the subdataset quotas can have any settings. | is applied and the subdataset quotas can have any settings. | ||
- | ===== Snapshots ===== | + | ===== Mount NAS Dataset to a VPS ==== |
+ | For VPS using [[manuals: | ||
+ | and mount it inside the VPS using NFS. | ||
- | Backups are made using ZFS snapshots, which can be seen in the Backups | + | For VPS using [[information: |
- | menu. They can be created in the very same menu. The created VPS snapshots cannot be | + | |
- | deleted, you have to wait until they are automatically overwritten by further daily backups. | + | |
- | {{:navody: | + | ===== Mounts ===== |
+ | For VPS using [[information:openvz|OpenVZ Legacy]], vpsAdmin mounts can be used to | ||
+ | mount any dataset or snapshot. | ||
- | VPS backups are made every day at 1:00 AM, when one node | + | If the VPS is running on [[manuals:vps: |
- | creates a snapshot of all the datasets at once. Then the snapshots are moved to | + | are used only to mount VPS subdatasets. NAS datasets and snapshots are mounted |
- | backuper.prg. | + | using [[manuals: |
- | Attention! NAS **is not backed up** to backuper.prg. Snapshots are | + | Mounts can be seen in the VPS details: |
- | local only and their only purpose is protection against the damage or unwanted deletion of data. | + | |
- | + | ||
- | ===== Mounts ===== | + | |
- | + | ||
- | Mounts can be seen in the VPS details. Both datasets | + | |
- | and snapshots can be mounted. Any dataset or snapshot can be mounted to any | + | |
- | VPS. Mounts of individual snapshots replace a permanent backup mount to | + | |
- | / | + | |
{{: | {{: | ||
- | Each snapshot can only have one mount at any given moment, datasets | + | We do not recommend nesting mount points in the incorrect order. The situation when |
- | have no such limitation. | + | a '' |
- | + | ||
- | I do not recommend nesting mount points in the incorrect order. The situation when | + | |
- | a “one/two” dataset is mounted above the “one” dataset has not been solved. | + | |
{{: | {{: | ||
Line 86: | Line 82: | ||
persistent between VPS restarts. | persistent between VPS restarts. | ||
- | ===== Restoring Backups | + | ===== More Information |
- | + | * [[manuals:vps:backups|Backups]] | |
- | Restoring a VPS from a backup (snapshot) works the same way as it has until now. Restoring | + | * [[manuals:vps:exports|Exports |
- | always works on the dataset level. If a VPS has subdatasets, | + | |
- | from the backup, subdatasets are not restored. I.e. it is possible to restore | + | |
- | any dataset and this doesn’t have any effect on other datasets. During the restore process, | + | |
- | all snapshots are stored thanks to the fact that backups in the backuper are branched. | + | |
- | + | ||
- | You can only make snapshots of an NAS **manually**. Since it is not backed up to the | + | |
- | backuper, the restore process behaves the same way as '' | + | |
- | an older snapshot **deletes** all newer snapshots. It is an **irreversible** operation. | + | |
- | + | ||
- | In order to restore data from a backup on an NAS without deleting snapshots, mount the selected | + | |
- | snapshot to a VPS and copy the data. | + | |
- | + | ||
- | ===== Downloading Backups ===== | + | |
- | + | ||
- | Backups can be either downloaded through an online interface or [[navody:vps:api#cli|a CLI]]. | + | |
- | The CLI has the advantage of not having to wait for an e-mail with a link to the backup download location – | + | |
- | we can start the download immediately or automate the whole process. We can either download | + | |
- | a tar.gz archive or the ZFS data stream directly (even incrementally). | + | |
- | + | ||
- | ==== Incremental | + | |
- | + | ||
- | An incremental backup only contains the data that has changed since the previous snapshot. | + | |
- | In order to help the client identify which snapshots can be downloaded incrementally, | + | |
- | each snapshot contains a //history indicator// (number). Snapshots with the same | + | |
- | identifier can be moved incrementally. The history flow can be interrupted by a VPS reinstallation | + | |
- | or using a backup to restore. Afterwards, the history identifier is increased by 1 and | + | |
- | the full backup needs to be downloaded again. | + | |
- | + | ||
- | The history identifier is shown in the table with a list of snapshots in the Backups menu. | + | |
- | + | ||
- | ==== Downloading the Backup as a File ==== | + | |
- | + | ||
- | < | + | |
- | $ vpsfreectl snapshot download --help | + | |
- | snapshot download [SNAPSHOT_ID] | + | |
- | + | ||
- | Command options: | + | |
- | -f, --format FORMAT | + | |
- | -I, --from-snapshot SNAPSHOT_ID | + | |
- | -d, --[no-]delete-after | + | |
- | -F, --force | + | |
- | -o, --output FILE Save the download to FILE | + | |
- | -q, --quiet | + | |
- | -r, --resume | + | |
- | -s, --[no-]send-mail | + | |
- | -x, --max-rate N | + | |
- | --[no-]checksum | + | |
- | </ | + | |
- | + | ||
- | If the snapshot ID isn’t passed on to the program as an argument, it displays an interactive | + | |
- | prompt: | + | |
- | + | ||
- | < | + | |
- | $ vpsfreectl snapshot download | + | |
- | Dataset 14 | + | |
- | | + | |
- | VPS #198 | + | |
- | (2) @2015-12-01T09: | + | |
- | (3) @2015-12-01T09: | + | |
- | (4) @2015-12-01T11: | + | |
- | (5) @2015-12-01T11: | + | |
- | (6) @2015-12-01T11: | + | |
- | (7) @2015-12-01T11: | + | |
- | (8) @2015-12-01T12: | + | |
- | (9) @2015-12-01T12: | + | |
- | (10) @2015-12-01T12: | + | |
- | (11) @2015-12-01T12: | + | |
- | (12) @2016-02-29T09: | + | |
- | (13) @2016-02-29T10: | + | |
- | (14) @2016-02-29T10: | + | |
- | Pick a snapshot for download: | + | |
- | + | ||
- | </ | + | |
- | + | ||
- | We will be downloading the 4th snapshot ('' | + | |
- | + | ||
- | < | + | |
- | Pick a snapshot for download: 4 | + | |
- | The download is being prepared... | + | |
- | Downloading to 198__2015-12-01T12-25-56.tar.gz | + | |
- | Time: 00:01:37 Downloading 0.3 GB: [=====================================================================================] 100% 992 kB/s | + | |
- | </ | + | |
- | + | ||
- | Using the '' | + | |
- | stream or an incremental data stream. Under default settings, the tar.gz archive | + | |
- | is downloaded. | + | |
- | + | ||
- | We can either let vpsAdmin name the file | + | |
- | ([[https:// | + | |
- | or choose our own location using the '' | + | |
- | the output is redirected to stdout. | + | |
- | + | ||
- | The program enables pausing the download (you need to use '' | + | |
- | again. If the '' | + | |
- | the program asks the user whether it should resume the download or start | + | |
- | over. | + | |
- | + | ||
- | < | + | |
- | The download can only start if the prepared file has not been deleted on the server | + | |
- | in the meantime (the '' | + | |
- | download attempt. | + | |
- | </ | + | |
- | + | ||
- | ==== ZFS Data Stream ==== | + | |
- | + | ||
- | < | + | |
- | $ vpsfreectl snapshot send --help | + | |
- | snapshot send SNAPSHOT_ID | + | |
- | + | ||
- | Command options: | + | |
- | -I, --from-snapshot SNAPSHOT_ID | + | |
- | -d, --[no-]delete-after | + | |
- | -q, --quiet | + | |
- | -s, --[no-]send-mail | + | |
- | -x, --max-rate N | + | |
- | --[no-]checksum | + | |
- | </ | + | |
- | + | ||
- | The difference from '' | + | |
- | in an uncompressed form so that we can mount it directly from | + | |
- | '' | + | |
- | + | ||
- | < | + | |
- | $ vpsfreectl snapshot send <id> | zfs recv -F < | + | |
- | </ | + | |
- | + | ||
- | An incremental data stream can be requested using the '' | + | |
- | + | ||
- | < | + | |
- | $ vpsfreectl snapshot send <id2> -- --from-snapshot <id1> | zfs recv -F < | + | |
- | </ | + | |
- | + | ||
- | ==== Automated Backup Downloads ==== | + | |
- | + | ||
- | Automated backup downloads are performed using the '' | + | |
- | '' | + | |
- | while the latter uses the dataset ID. | + | |
- | + | ||
- | These commands require ZFS to be installed, zpool to be created and root permissions. | + | |
- | The program can be run directly under root, otherwise it will use sudo when running. | + | |
- | + | ||
- | Upon startup, snapshots with the current history identifier are downloaded, as long as they | + | |
- | do not exist locally yet. If possible, they are downloaded incrementally. In order for | + | |
- | incremental transfer to work, the program must find the snapshot which is present locally and | + | |
- | on the server at the same time. This means that backups have to be downloaded at least | + | |
- | once every 14 days since the newest local snapshot gets deleted from the server after that time period | + | |
- | and the program won’t be able to resume downloading backups – there won’t be | + | |
- | any common snapshot. | + | |
- | + | ||
- | < | + | |
- | $ vpsfreectl backup vps --help | + | |
- | backup vps [VPS_ID] FILESYSTEM | + | |
- | + | ||
- | Command options: | + | |
- | -p, --pretend | + | |
- | -r, --[no-]rotate | + | |
- | -m, --min-snapshots N Keep at least N snapshots (30) | + | |
- | -M, --max-snapshots N Keep at most N snapshots (45) | + | |
- | -a, --max-age N Delete snapshots older then N days (30) | + | |
- | -x, --max-rate N | + | |
- | -q, --quiet | + | |
- | -s, --safe-download | + | |
- | --retry-attemps N Retry N times to recover from download error (10) | + | |
- | -i, --init-snapshots N | + | |
- | --[no-]checksum | + | |
- | -d, --[no-]delete-after | + | |
- | --no-snapshots-as-error | + | |
- | --[no-]sudo | + | |
- | </ | + | |
- | + | ||
- | If the program does not receive the VPS/Dataset ID as an argument, it either asks the user for it | + | |
- | or it tries to identify the ID itself. The '' | + | |
- | provided. It should contain the name of the dataset where we want to store the backups. | + | |
- | + | ||
- | <note warning> | + | |
- | The first argument of the program is the **VPS/ | + | |
- | is used, since the ID is not identical to the dataset name, but both are usually | + | |
- | numbers. | + | |
- | </ | + | |
- | + | ||
- | Before we actually run the program, the '' | + | |
- | shows us what the program would do, i.e. which snapshots it would download and potentially | + | |
- | delete. | + | |
- | + | ||
- | The '' | + | |
- | make room for the new ones. Unless we change other settings, we will have at least | + | |
- | 30 snapshots (which currently means 30 daily histories) and a maximum of 45 snapshots | + | |
- | (if we create some ourselves) and snapshots older than 30 days will be | + | |
- | deleted. | + | |
- | + | ||
- | The content of the '' | + | |
- | should not create more subdatasets/ | + | |
- | subdatasets, | + | |
- | + | ||
- | === Usage Example === | + | |
- | + | ||
- | < | + | |
- | $ vpsfreectl backup vps storage/ | + | |
- | (1) VPS #198 | + | |
- | (2) VPS #199 | + | |
- | (3) VPS #202 | + | |
- | Pick a dataset to backup: 2 | + | |
- | Will download 8 snapshots: | + | |
- | @2016-03-07T18: | + | |
- | @2016-03-07T18: | + | |
- | @2016-03-07T18: | + | |
- | @2016-03-10T10: | + | |
- | @2016-03-10T10: | + | |
- | @2016-03-10T11: | + | |
- | @2016-03-10T14: | + | |
- | @2016-03-10T14: | + | |
- | + | ||
- | Performing a full receive of @2016-03-07T18: | + | |
- | The download is being prepared... | + | |
- | Time: 00:00:56 Downloading 0.3 GB: [====================================================================================] 100% 1755 kB/s | + | |
- | Performing an incremental receive of @2016-03-07T18:12:58 - @2016-03-10T14: | + | |
- | The download is being prepared... | + | |
- | Time: 00:00:00 Downloading 0.0 GB: [=======================================================================================] 100% 0 kB/s | + | |
- | </ | + | |
- | + | ||
- | We can notice that the program downloads the first snapshot in full size and | + | |
- | all the following ones incrementally. | + | |
- | + | ||
- | A list of snapshots can be displayed using '' | + | |
- | < | + | |
- | $ sudo zfs list -r -t snapshot storage/ | + | |
- | NAME | + | |
- | storage/ | + | |
- | storage/ | + | |
- | storage/ | + | |
- | storage/ | + | |
- | storage/ | + | |
- | storage/ | + | |
- | storage/ | + | |
- | storage/ | + | |
- | </ | + | |
- | + | ||
- | We can access our own data using the special '' | + | |
- | < | + | |
- | $ ls -1 / | + | |
- | 2016-03-07T18: | + | |
- | 2016-03-07T18: | + | |
- | 2016-03-07T18: | + | |
- | 2016-03-10T10: | + | |
- | 2016-03-10T10: | + | |
- | 2016-03-10T11: | + | |
- | 2016-03-10T14: | + | |
- | 2016-03-10T14: | + | |
- | </ | + | |
- | + | ||
- | Cron can be used to download backups regularly. The crontab record can look like | + | |
- | this: | + | |
- | + | ||
- | < | + | |
- | MAILTO=your@email | + | |
- | + | ||
- | # Example of job definition: | + | |
- | # .---------------- minute (0 - 59) | + | |
- | # | .------------- hour (0 - 23) | + | |
- | # | | .---------- day of month (1 - 31) | + | |
- | # | | | .------- month (1 - 12) OR jan, | + | |
- | # | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun, | + | |
- | # | | | | | | + | |
- | # * * * * * user-name command to be executed | + | |
- | 0 7 * * * root vpsfreectl backup | + | |
- | </ | + | |
- | + | ||
- | This means that the program runs every day at 7:00 AM (at this point, the backups | + | |
- | in vpsFree should already have been be moved to backuper.prg) the backups will be downloaded using a maximum speed of | + | |
- | 1 MB/s. Cron will send us the output of the command to the email set in the | + | |
- | '' | + | |
- | This is why the program has the '' | + | |
- | errors are printed. | + | |
- | + | ||
- | < | + | |
- | If we download the backups using the root user, but '' | + | |
- | by a standard user, the program has to be | + | |
- | [[navody: | + | |
- | </ | + | |
- | + | ||
- | === Downloading a Full Backup Using a Slow/ | + | |
- | + | ||
- | With default settings, the '' | + | |
- | since it doesn’t download the data to a file, but it directly sends it to ZFS. If we only have | + | |
- | a slow and unreliable connection at our disposal, it can happen that the download | + | |
- | fails and it is necessary to start over. However, we can use the | + | |
- | '' | + | |
- | does it send it to ZFS. Because | + | |
- | resumed at any point. The disadvantage of this procedure is that it requires twice as much space on the hard drive since the data | + | |
- | is simultaneously stored in a temporary file and the ZFS dataset. The temporary file is created | + | |
- | in the folder from which the program is running. | + | |
- | + | ||
- | Another problem can be encountered after a long period of downloading. This is because when the program is first started, | + | |
- | it downloads all snapshots | + | |
- | downloading the oldest snapshot takes too long, it can be deleted from the | + | |
- | server, which causes us to be unable to use it for incremental downloads later on and | + | |
- | we have to download the full backup again. To solve cases like this one, there is the | + | |
- | '' | + | |
- | most recent snapshots. The safest method is using '' | + | |
- | we have 14 days to finish the download (the last pause can occur after 7 | + | |
- | days). However, this is no cure-all since if the program is closed and run again on a different day, | + | |
- | the last snapshot will be different and the download process will start over unless | + | |
- | '' | + | |
- | + | ||
- | === Detecting Missing Backups === | + | |
- | + | ||
- | Sometimes it can happen that the daily backup doesn’t occur and so the program doesn’t have | + | |
- | anything to download. This situation typically isn’t considered an error – all the | + | |
- | snapshots have simply been downloaded and the program doesn’t have anything to do. However, if | + | |
- | backups are downloaded automatically using Cron, we have no way of finding out that | + | |
- | no backups are being downloaded. This is why the program has the | + | |
- | '' | + | |
- | download, it returns an error. Errors are not hidden by the '' | + | |
- | so Cron will send it to us via email and we will find out about the outage. | + | |
- | + | ||
- | === Downloading backups with a standard user account using sudo === | + | |
- | + | ||
- | If we don’t want to install or run '' | + | |
- | run under an unprivileged user as well. In this case, sudo is used in order to work with ZFS. | + | |
- | + | ||
- | In the following example, we will install and use the program under the user | + | |
- | '' | + | |
- | + | ||
- | < | + | |
- | # useradd -m -d / | + | |
- | # su vpsfree | + | |
- | $ gem install --user-install vpsfree-client | + | |
- | </ | + | |
- | + | ||
- | Add the following lines to ''/ | + | |
- | + | ||
- | < | + | |
- | Defaults: | + | |
- | vpsfree ALL=(root) NOPASSWD: /sbin/zfs | + | |
- | </ | + | |
- | + | ||
- | The user '' | + | |
- | which is necessary if we want to run it using Cron. | + | |
- | + | ||
- | Now we’ll try to run the program manually and then move it to the crontab. Let’s try | + | |
- | requesting and saving an authentication token: | + | |
- | + | ||
- | < | + | |
- | # su vpsfree | + | |
- | $ vpsfreectl --auth token --new-token --token-lifetime permanent --save user current | + | |
- | </ | + | |
- | + | ||
- | If you get an error stating that the program doesn’t exist, you will need to specify the whole | + | |
- | path or add the correct directory to '' | + | |
- | '' | + | |
- | ''/ | + | |
- | + | ||
- | When we have a working client, we can download the first backup to the dataset that we | + | |
- | have created. In this example, VPS #123 will be backed up to the '' | + | |
- | dataset. | + | |
- | + | ||
- | < | + | |
- | # su vpsfree | + | |
- | $ sudo zfs create -p storage/ | + | |
- | $ vpsfreectl backup vps 123 storage/ | + | |
- | </ | + | |
- | + | ||
- | We will use Cron for regular downloads of further backups. | + | |
- | Open the '' | + | |
- | + | ||
- | < | + | |
- | PATH=/ | + | |
- | MAILTO=your@email | + | |
- | HOME=/ | + | |
- | + | ||
- | 0 7 * * * vpsfree vpsfreectl backup vps storage/ | + | |
- | </ | + | |
- | + | ||
- | '' | + | |
- | we no longer need to provide the VPS ID for the program – the program stores it when it runs the first time. | + | |
- | + | ||
- | === Downloading Backups Under a Standard User by Delegating Permissions === | + | |
- | + | ||
- | Solaris/ | + | |
- | to various users. In this case, the program does not need root permissions at all, and neither does it need | + | |
- | sudo. | + | |
- | + | ||
- | We will assign the required permissions to the user '' | + | |
- | + | ||
- | < | + | |
- | # zfs create storage/ | + | |
- | # zfs allow vpsfree create, | + | |
- | </ | + | |
- | + | ||
- | In order for the user to be able to create subdatasets and connect them, the user needs permissions on | + | |
- | the directory and file levels: | + | |
- | + | ||
- | < | + | |
- | # chown vpsfree: | + | |
- | </ | + | |
- | + | ||
- | < | + | |
- | It is necessary to modify the kernel settings in FreeBSD so that it allows users to mount: | + | |
- | + | ||
- | < | + | |
- | # sysctl vfs.usermount=1 | + | |
- | </ | + | |
- | </ | + | |
- | + | ||
- | Now we can start downloading the backups. We use the '' | + | |
- | program doesn’t try to use sudo. | + | |
- | + | ||
- | < | + | |
- | # su vpsfree | + | |
- | $ vpsfreectl backup vps 123 storage/ | + | |
- | </ | + | |
- | + | ||
- | + | ||
- | ==== General Options ==== | + | |
- | * '' | + | |
- | * '' | + | |
- | * '' | + | |
- | * '' | + | |
- | * '' | + | |
- | + | ||
- | ===== Restoring Downloaded Backups ===== | + | |
- | + | ||
- | Restoring VPS from a downloaded backup so far isn’t automated in any way. One of the | + | |
- | possible ways is mounting the VPS dataset that we want to restore to a different VPS | + | |
- | (Playground) and copying the data. This method is described in the | + | |
- | [[navody: | + | |