FOSS toolchain#

The foss toolchain consists entirely of free and open source software components. When building third-party software, or developing your own, load the module for the toolchain:

$ module load foss/<version>

where <version> should be replaced by the one to be used, e.g., 2014a. See the documentation on the software module system for more details.

Compilers: GNU#

Three GCC compilers are available:

  • C: gcc

  • C++: g++

  • Fortran: gfortran

For example, to compile/link a Fortran program fluid.f90 to an executable fluid with architecture specific optimization for processors that support AVX2 instructions, use:

$ gfortran -O2  -march=haswell  -o fluid  fluid.f90

Documentation on GCC other compiler flags and options is available on the project’s website for the GCC documentation.

Note

Do not forget to load the toolchain module first!

Optimizing for a CPU architecture#

To optimize your application or library for specific CPU architectures, use the appropriate option listed in the table below.

CPU architecture

compiler option

Ivy Bridge

-march=ivybridge

Sandy Bridge

-march=sandybridge

Haswell

-march=haswell

Broadwell

-march=broadwell

Skylake

-march=skylake-avx512

Cascade Lake

-march=cascadelake

Napels/Rome (AMD)

-march=znver2

detect host CPU

-march=native

Note

GCC doesn’t support applications with multiple code paths, so you have to build multiple versions optimized for specific architectures. Dispatching can be done at runtime by checking the value of the $VSC_ARCH_LOCAL environment variable.

GCC OpenMP#

The compiler switch to use to compile/link OpenMP C/C++ or Fortran code is -fopenmp. For example, to compile/link a OpenMP C program scattter.c to an executable scatter, use:

$ gcc -fopenmp  -O2  -o scatter  scatter.c

Note

The OpenMP runtime library used by GCC is of inferior quality when compared to Intel’s, so developers are strongly encouraged to use the Intel toolchain when developing/building OpenMP software.

Running an OpenMP job#

When submitting a job, remember to specify as many processes per node as the number of threads the executable is supposed to run. This can be done using the ppn resource specification, e.g., -l nodes=1:ppn=10 for an executable that should be run with 10 OpenMP threads.

Warning

The number of threads should not exceed the number of cores on a compute node.

Communication library: Open MPI#

For the foss toolchain, Open MPI is used as the communications library. To compile/link MPI programs, wrappers are supplied, so that the correct headers and libraries are used automatically. These wrappers are:

  • C: mpicc

  • C++: mpic++

  • Fortran: mpif77, mpif90

The compiler wrappers take the same options as the corresponding compilers.

Using the MPI compilers from Open MPI#

For example, to compile/link a C program thermo.c to an executable thermodynamics:

$ mpicc -O2  -o thermodynamics  thermo.c

Extensive documentation is provided on the Open MPI documentation site.

Note

Do not forget to load the toolchain module first.

Running an Open MPI program#

Note that an MPI program must be run with the exact same version of the toolchain as it was originally build with. The listing below shows a PBS job script thermodynamics.pbs that runs the thermodynamics executable.

#!/bin/bash -l

module load foss/<version>
cd $PBS_O_WORKDIR
mpirun ./thermodynamics

The hosts and number of processes is retrieved from the queue system, that gets this information from the resource specification for that job.

FOSS mathematical libraries#

The FOSS toolchain contains the basic HPC mathematical libraries, it offers:

Version numbers#

2021a

2020b

2020a

2019a

2018b

2018a

2017b

2017a

2016b

2016a

GCC

10.3.0

10.2.0

9.3.0

8.2.0

7.3.0

6.4.0

6.4.0

6.3

5.4

4.9.3

Open MPI

4.1.1

4.0.5

4.0.3

3.1.3

3.1.1

2.1.2

2.1.1

2.0.2

1.10.3

1.10.2

UCX

1.10.0

1.9.0

OpenBLAS

0.3.15

0.3.12

0.3.9

0.3.5

0.3.1

0.2.20

0.2.20

0.2.19

0.2.18

0.2.15

ScaLAPACK

2.1.0

2.1.0

2.1.0

2.0.2

2.0.2

2.0.2

2.0.2

2.0.2

2.0.2

2.0.2

FFTW

3.3.9

3.3.8

3.3.8

3.3.8

3.3.8

3.3.7

3.3.6

3.3.6

3.3.4

3.3.4

binutils

2.36.1

2.35

2.34

2.32

2.30

2.28

2.28

2.27

2.26

2.25

Further information on FOSS components#