HPC Data Recovery#

The following VSC compute clusters keep regular data snapshots on their shared storage systems. This can include the storage of VSC_HOME, VSC_DATA and VSC_SCRATCH in your account, as well as project folders or Virtual Organizations.

KU Leuven

Tier-2 wICE
Tier-2 Mindwell

VUB

Tier-2 Anansi
Tier-2 Hydra
Tier-1 sofia

Restoring old files/folders#

Restoring a file or directory from one of the snapshots is relatively easy. Snapshots are located somewhere in the file system of the cluster. Once you locate your snapshot (see below) you can restore your own files to a previous version just by copying the file from one of the snapshots to its desired location.

Some sites might provide specific tools to carry out the restoration of files from snapshots. Please follow the instructions below that correspond to your cluster of choice.

KU Leuven/UHasselt

The snapshots can be found in two ways. Either from the top-level folder on $VSC_HOME and $VSC_DATA:

/user/leuven/.snapshot for $VSC_HOME
/data/leuven/.snapshot for $VSC_DATA

Or equivalently, from any nested subfolder inside $VSC_HOME and $VSC_DATA. For instance, if a file is missing from projects/simulations folder, you may retrieve it from:

$VSC_DATA/projects/simulations/.snapshot

In these .snapshot directories, a number of directories can be found with names of the form hourly.<timestamp>, daily.<timestamp> or weekly.<timestamp>; where timestamp reflects the moment that snapshot was taken. For example, the directory weekly.2019-06-09_0000 will contain the snapshot taken at midnight on June 9, 2019.

The directory structure of all the directories mirrors that of /user/leuven and /data/leuven respectively, so the most recent backup of your home directory can be found in /user/leuven/.snapshot/hourly.2019-06-12_0205/301/vsc30124.

In this directory, you will find all your files in the state they were in when that particular snapshot was made, and you can simply copy them using the familiar bash shell operations.

Note

You can indeed retrieve older states of your files using the snapshots, taken strictly at fixed time intervals. However, for a full control over the history of your critical files, you may consider using version control. Good example of that would be using Github or the private Gitlab services offered at KU Leuven.

VUB

The command tool mysnapshots provides a simple interface to restore a file or directory from a snapshots.

First, change your current directory (with cd) to the directory that contained (or still contains) the file/directory to be restored. If that parent directory does not exist anymore, you can jump up to the last parent directory that still exists. In the worse case where a lot of data is lost, this location might be the top directory of $VSC_DATA or $VSC_SCRATCH for instance.

$ cd $VSC_SCRATCH/labo
$ ls -l
total 8
-rw-rw-r-- 1 vsc10930 vsc10930 2261  5 nov  2024 job.sh
-rw-rw-r-- 1 vsc10930 vsc10930  907  5 nov  2024 results_dir

The command mysnapshots always works on the snapshots of the current working directory. You can use it without arguments to list the available snapshots for that directory ordered by date/time.

$ mysnapshots
Snapshots of $VSC_SCRATCH/labo:
2025-05-08T17:02:44  2025-05-08T22:02:08  2025-05-09T08:01:56  2025-05-09T13:02:02

The output of the previous command shows the timestamps of the four snapshots with data for the working directory. Now if we want to know which files/directories are available in a specific snapshot, we can execute mysnapshots with the timestamp of that snapshot in the given YYYY-MM-DDTHH:mm:ss format as argument:

$ mysnapshots 2025-05-09T13:02:02
Inventory of $VSC_SCRATCH/labo at 2025-05-09T13:02:02:
job.sh      results_dir   old_data

In the example above, mysnapshots shows three files/directories in the snapshot 2025-05-09T13:02:02. Of these, job.sh and results_dir are still present in the current directory, whereas old_data is no longer present.

You can restore any of these entries to their previous state at 2025-05-09T13:02:02 by calling mysnapshots with the timestamps of the snapshot and the path to this item as argument. For instance, we can recover the missing entry old_data from this snapshot:

$ mysnapshots 2025-05-09T13:02:02 old_data
Restoring file '$VSC_SCRATCH/labo/old_data' from snapshot at 2025-05-09T13:02:02
Restoration complete
$ ls -l
total 12
-rw-rw-r-- 1 vsc10930 vsc10930 2261  5 nov  2024 job.sh
-rw-rw-r-- 1 vsc10930 vsc10930  907  5 nov  2024 results_dir
-rw-rw-r-- 1 vsc10930 vsc10930   97  9 mei  2025 old_data

Warning

By default, mysnapshots will refuse to overwrite existing files/directories in your working directory. Use the -f option to force the restoration of existing items. Beware that restoring an already existing directory with the -f option will also restore all files and folders within it as they were at that point in time. More recent files will be removed and all updates since then will be lost.

Tier-1 sofia