.. _KU Leuven storage: ################# KU Leuven storage ################# Storage on the clusters hosted at KU Leuven is organized :ref:`in a similar way as on other VSC clusters `. Currently, we use an NFS filesystem for home and data complemented with Lustre and IBM Storage Scale (GPFS) parallel filesystems for scratch storage. .. note:: We use environment variables to point at your different storage locations, as demonstrated in the tables below. These paths are constructed based on the 5 digits in your VSC ID, which are represented by "x" characters below. As mentioned in :ref:`data location`, you should always (try to) use the environment variables rather than the paths shown in the tables. VSC home and data storage ------------------------- The table below describes the VSC home and data storage locations, which can be acessed on all VSC clusters. These are intended for long term storage of files that are not accessed at high frequency by compute jobs. +-----------------------+----------------------------------+--------+------------------+-------+---------------+ | Variable | Path | Type | Access |Backup | Default quota | +=======================+==================================+========+==================+=======+===============+ | ``$VSC_HOME`` | ``/user/leuven/xxx/vscxxxxx`` | NFS | All VSC clusters | Yes | 3 GiB | +-----------------------+----------------------------------+--------+------------------+-------+---------------+ | ``$VSC_DATA`` | ``/data/leuven/xxx/vscxxxxx`` | NFS | All VSC clusters | Yes | 75 GiB | +-----------------------+----------------------------------+--------+------------------+-------+---------------+ Note that for ``$VSC_HOME`` and ``$VSC_DATA``: - data is protected by snapshots, which means it is possible to :ref:`recover data ` that was accidentally deleted or modified, - quota for VSC accounts that do *not* have Leuven / Hasselt as home institute are determined by the policy of the user's home institute. Note that you can check the institute of your VSC account on the `VSC account page`_. .. _leuven_scratch: Parallel scratch storage ------------------------ For workflows requiring frequent read or write operations (especially those involving intermediate files), it is recommended to use scratch storage. Compared to NFS, the GPFS and Lustre filesystems are better designed to handle intensive serial and parallel input/output (IO) operations. :ref:`Genius ` and :ref:`wICE ` share the same (Lustre based) scratch storage, while :ref:`Mindwell ` comes with its own (GPFS based) scratch storage: +-----------------------+----------------------------------+--------+---------------+-------+---------------+ | Variable | Path | Type | Access |Backup | Default quota | +=======================+==================================+========+===============+=======+===============+ |``$VSC_SCRATCH`` | ``/scratch/leuven/xxx/vscxxxxx`` | Lustre | Genius, wICE | No | 500 GiB | | | +--------+---------------+-------+---------------+ | | | GPFS | Mindwell | No | 500 GiB | +-----------------------+----------------------------------+--------+---------------+-------+---------------+ On each node, the ``$VSC_SCRATCH`` environment variable will point to the scratch storage associated with the node (GPFS scratch on the Mindwell nodes, Lustre scratch on the nodes of Genius and wICE). .. warning:: It is *crucial* that intensive IO operations in your compute jobs are done on the scratch storage associated with the cluster where the job is running. In other words, compute jobs running on Genius and wICE have to use Lustre and jobs running on Mindwell have to use GPFS. Compute jobs that do not comply can be cancelled by the system administrators without prior notice. If you are looking for more information on this, please read the page on :ref:`parallel filesystems ` .. note:: VSC accounts whose home institute is not Leuven / Hasselt need to `contact the servicedesk `_ to receive scratch storage, as it is not set up by default. .. _leuven_lustre_gpfs_transfer: Transferring data between Lustre and GPFS ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To facilitate data transfers between the Lustre and GPFS storage, Lustre is accessible from Mindwell and GPFS is accessible from wICE and from the (Genius) login nodes. Note that GPFS is not reachable from the Genius *compute* nodes. Two more environment variables (``$VSC_SCRATCH_LUSTRE1`` and ``$VSC_SCRATCH_GPFS1``) have been defined for this purpose, so that you can easily find the mount location of your scratch directory on the "other" parallel file system. For instance, to copy a file from your Lustre scratch to your GPFS scratch, you could go about it as follows: .. code-block:: bash # If initiating the transfer from Genius or wICE: cp ${VSC_SCRATCH}/myfile ${VSC_SCRATCH_GPFS1} # If initiating the transfer from Mindwell: cp ${VSC_SCRATCH_LUSTRE1}/myfile ${VSC_SCRATCH} As a best practice, data transfers between Lustre and GPFS should be performed through 'transfer' jobs submitted to, for example, the ``interactive`` partition of :ref:`wICE ` or :ref:`Mindwell `. Short transfers which don't take more than a couple of minutes can also be performed from a Genius login node. For more advanced examples, please read the :ref:`recommendations for managing data on multiple filesystems `. Globus endpoints have been defined on both Lustre and GPFS filesystems, so you can use the :ref:`globus platform` for these data transfers. For transferring large volumes of data (> 1 TB), however, we recommend using 'transfer' jobs instead of Globus for performance reasons. Automatic scratch cleanup ^^^^^^^^^^^^^^^^^^^^^^^^^ ``$VSC_SCRATCH`` at KU Leuven is not for long term storage, as files not accessed for more than 30 days are automatically removed. The reasoning behind this is that as long as you are actively using a file (so accessing it), the file will not be removed. This policy can however cause confusion when files are initially transferred to a scratch directory. If you use for instance the ``mv`` command, the file is not actually accessed. As a result, if the last access timestamp of the original file is a long time in the past, the file on scratch will be considered to be inactive and automatically removed. A similar thing happens when using ``rsync`` with the option to preserve timestamps. To avoid this problem, the ``cp`` command (without ``-a`` argument) should be used to copy files to the scratch directory, followed by removing the sources (if needed) using the ``rm`` command upon a successful transfer. Node scratch ------------ If your jobs require temporary storage that does not need to be shared across compute nodes, you may also consider using the local node disks: +-----------------------+----------------------------------+--------+---------------+-------+---------------+ |Variable | Path | Type | Access |Backup | Default quota | +=======================+==================================+========+===============+=======+===============+ |``$VSC_SCRATCH_NODE`` | ``/tmp`` | ext4 | Genius | No | 200 GiB | | | | +---------------+ +---------------+ | | | | wICE | | 600 GiB | | | | +---------------+ +---------------+ | | | | Mindwell | | 600 GiB | +-----------------------+----------------------------------+--------+---------------+-------+---------------+ Though limited in storage capacity, ``$VSC_SCRATCH_NODE`` has the advantage that no network traffic is involved. The contents of this temporary storage location are always removed when the job ends. Therefore, results need to be copied elsewhere if needed.