.. _Vulcan cluster:

Vulcan cluster
==============

.. _Vulcan overview:

Overview
--------

The `NEC Cluster (Vulcan) <https://www.hlrs.de/solutions/systems/vulcan>`__
is a heterogeneous university cluster with AMD and NVIDIA GPUs,
available to the University of Stuttgart for research, visualization, AI, and big data.

See page `NEC Cluster Hardware and Architecture
<https://kb.hlrs.de/platforms/index.php/NEC_Cluster_Hardware_and_Architecture_(vulcan)>`__
for more information. See the `HLRS status page
<https://websrv.hlrs.de/cgi-bin/hwwweather?task=viewmachine&machine=vulcan>`__
for maintenance information.

+----------------------+---------------------------------+--------+-------+------------------+-------------+-----------------+
| Entgeltordnung       | Intended use                    | Nodes  | Cores | RAM              | GPU         | Node list       |
+======================+===========+=====================+========+=======+==================+=============+=================+
| smp Knoten           | (Pre- / Post-) processing       | 1      | 2x20  | :qty:`1536 <GB>` | 1x K6000    | ``smp``         |
+----------------------+---------------------------------+--------+-------+------------------+-------------+-----------------+
| Cascadelake 384 GB   | Data processing                 | 96     | 2x20  | :qty:`384 <GB>`  |             | ``clx``         |
+----------------------+---------------------------------+--------+-------+------------------+-------------+-----------------+
| CS500-Spark          | Data processing & storage       | 8      | 2x20  | :qty:`384 <GB>`  |             | ``clx``         |
+----------------------+---------------------------------+--------+-------+------------------+-------------+-----------------+
| CS Storm             | AI and big data                 | 8      | 2x18  | :qty:`768 <GB>`  | 8x V100     | ``clx-ai``      |
+----------------------+---------------------------------+--------+-------+------------------+-------------+-----------------+
| Haswell 128 GB       | Data processing                 | 76     | 2x10  | :qty:`128 <GB>`  |             | ``hsw128gb20c`` |
+----------------------+---------------------------------+--------+-------+------------------+-------------+-----------------+
| Haswell 256 GB       | Data processing                 | 4      | 2x10  | :qty:`256 <GB>`  |             | ``hsw256gb20c`` |
+----------------------+---------------------------------+--------+-------+------------------+-------------+-----------------+
| Haswell 128 GB       | Data processing                 | 76     | 2x12  | :qty:`128 <GB>`  |             | ``hsw128gb24c`` |
+----------------------+---------------------------------+--------+-------+------------------+-------------+-----------------+
| Haswell 256 GB       | Data processing                 | 16     | 2x12  | :qty:`256 <GB>`  |             | ``hsw256gb24c`` |
+----------------------+---------------------------------+--------+-------+------------------+-------------+-----------------+
| SKL-192GB            | Data processing                 | 88     | 2x20  | :qty:`192 <GB>`  |             | ``skl``         |
+----------------------+---------------------------------+--------+-------+------------------+-------------+-----------------+
| Graphikserver AMD    | Visualization                   | 6      | 2x4   | :qty:`96 <GB>`   | 1x WX 8200  | ``visamd``      |
+----------------------+---------------------------------+--------+-------+------------------+-------------+-----------------+
| Graphikserver NVIDIA | Visualization                   | 1      | 2x4   | :qty:`96 <GB>`   | 1x RTX 4000 | ``visnv``       |
+----------------------+---------------------------------+--------+-------+------------------+-------------+-----------------+
| VISP 100             | Data processing & Visualization | 10     | 2x8   | :qty:`256 <GB>`  | 1x P100     | ``visp100``     |
+----------------------+---------------------------------+--------+-------+------------------+-------------+-----------------+

.. _Vulcan partitions:

Partitions and nodes
--------------------

Vulcan uses the PBS scheduling system!

smp Knoten
^^^^^^^^^^
There is 1 node equipped with 2 :cpu-intel-sheet2:`Intel Xeon Gold 6148
<120489/intel-xeon-gold-6148-processor-27-5m-cache-2-40-ghz>`
(20 cores, 40 threads, :qty:`2.40 <GHz>`, :qty:`150 <W>`), :qty:`1536 <GB>` of RAM, :qty:`15 <TB>` SSD.
Each node has 1 `NVIDIA Quadro K6000
<https://www.nvidia.com/content/PDF/data-sheet/NV_DS_Quadro_K6000_OCT13_NV_US_LR.pdf>`__
(:qty:`288 <Gbps>`, :qty:`225 <W>`, :qty:`12 <GB>` GDDR5).

Cascadelake 384 GB
^^^^^^^^^^^^^^^^^^
There are 96 nodes equipped with 2 :cpu-intel-sheet2:`Intel Xeon Gold 6248
<192446/intel-xeon-gold-6248-processor-27-5m-cache-2-50-ghz>`
(20 cores, 40 threads, :qty:`2.50 <GHz>`, :qty:`150 <W>`), :qty:`384 <GB>` of RAM.
There are no GPUs.

CS500-Spark
^^^^^^^^^^^
There are 8 nodes equipped with 2 :cpu-intel-sheet2:`Intel Xeon Gold 6230
<192437/intel-xeon-gold-6230-processor-27-5m-cache-2-10-ghz>`
(20 cores, 40 threads, :qty:`2.10 <GHz>`, :qty:`125 <W>`), :qty:`384 <GB>` of RAM, :qty:`1.8 <TB>` NVMe SSD.
There are no GPUs.

CS Storm
^^^^^^^^
There are 8 nodes equipped with 2 :cpu-intel-sheet2:`Intel Xeon Gold 6240
<192443/intel-xeon-gold-6240-processor-24-75m-cache-2-60-ghz>`
(18 cores, 36 threads, :qty:`2.60 <GHz>`, :qty:`150 <W>`), :qty:`768 <GB>` of RAM, :qty:`7.3 <TB>` NVMe SSD, :qty:`220 <GB>` SSD.
Each node has 8 :gpu-nvidia-sheet2:`NVIDIA Tesla V100 SXM2
<volta/pdf/volta-v100-datasheet-update-us-1165301-r5.pdf>`
(:qty:`300 <Gbps>`, :qty:`300 <W>`, :qty:`32 <GB>` HBM2).

Haswell 128 GB 20 cores
^^^^^^^^^^^^^^^^^^^^^^^
There are 76 nodes equipped with 2 :cpu-intel-sheet2:`Intel Xeon E5-2660v3
<81706/intel-xeon-processor-e52660-v3-25m-cache-2-60-ghz>`
(10 cores, 20 threads, :qty:`2.60 <GHz>`, :qty:`105 <W>`), :qty:`128 <GB>` of RAM.
There are no GPUs.

Haswell 256 GB 20 cores
^^^^^^^^^^^^^^^^^^^^^^^
There are 4 nodes equipped with 2 :cpu-intel-sheet2:`Intel Xeon E5-2660v3
<81706/intel-xeon-processor-e52660-v3-25m-cache-2-60-ghz>`
(10 cores, 20 threads, :qty:`2.60 <GHz>`, :qty:`105 <W>`), :qty:`256 <GB>` of RAM.
There are no GPUs.

Haswell 128 GB 24 cores
^^^^^^^^^^^^^^^^^^^^^^^
There are 152 nodes equipped with 2 :cpu-intel-sheet2:`Intel Xeon E5-2668v3
<81908/intel-xeon-processor-e5-2680-v3-30m-cache-2-50-ghz>`
(12 cores, 24 threads, :qty:`2.50 <GHz>`, :qty:`120 <W>`), :qty:`128 <GB>` of RAM.
There are no GPUs.

Haswell 256 GB 24 cores
^^^^^^^^^^^^^^^^^^^^^^^
There are 16 nodes equipped with 2 :cpu-intel-sheet2:`Intel Xeon E5-2668v3
<81908/intel-xeon-processor-e5-2680-v3-30m-cache-2-50-ghz>`
(12 cores, 24 threads, :qty:`2.50 <GHz>`, :qty:`120 <W>`), :qty:`128 <GB>` of RAM.
There are no GPUs.

SKL-192GB
^^^^^^^^^
There are 16 nodes equipped with 2 :cpu-intel-sheet1:`Intel Xeon Gold 6138
<120476/intel-xeon-gold-6138-processor-27-5m-cache-2-00-ghz>`
(20 cores, 40 threads, :qty:`2.00 <GHz>`, :qty:`125 <W>`), :qty:`192 <GB>` of RAM.
There are no GPUs.

Graphikserver AMD
^^^^^^^^^^^^^^^^^
There are 6 nodes equipped with 2 :cpu-intel-sheet1:`Intel Xeon Silver 4112
<123551/intel-xeon-silver-4112-processor-8-25m-cache-2-60-ghz.html>`
(4 cores, 8 threads, :qty:`2.60 <GHz>`, :qty:`80 <W>`), :qty:`96 <GB>` of RAM.
Each node has 1 `AMD Radeon Pro WX 8200
<https://www.amd.com/en/support/professional-graphics/radeon-pro/radeon-pro-wx-x200-series/radeon-pro-wx-8200>`__
(:qty:`512 <Gbps>`, :qty:`230 <W>`, :qty:`8 <GB>` HBM2).

Graphikserver NVIDIA
^^^^^^^^^^^^^^^^^^^^
There is 1 node equipped with 2 :cpu-intel-sheet2:`Intel Xeon Silver 4112
<123551/intel-xeon-silver-4112-processor-8-25m-cache-2-60-ghz>`
(4 cores, 8 threads, :qty:`2.60 <GHz>`, :qty:`80 <W>`), :qty:`96 <GB>` of RAM.
Each node has 1 :gpu-nvidia-sheet1:`NVIDIA Quadro RTX 4000
<design-visualization/quadro-product-literature/quadro-rtx-4000-datasheet-us-nvidia-1060942-r2-web.pdf>`
(:qty:`400 <Gbps>`, :qty:`160 <W>`, :qty:`8 <GB>` GDDR6).

VISP 100
^^^^^^^^
There are 16 nodes equipped with 2 :cpu-intel-sheet2:`Intel Xeon E5-2667v4
<92979/intel-xeon-processor-e52667-v4-25m-cache-3-20-ghz>`
(8 cores, 16 threads, :qty:`3.20 <GHz>`, :qty:`135 <W>`), :qty:`256 <GB>` of RAM, :qty:`4.1 <TB>` SSD.
Each node has 1 :gpu-nvidia-sheet1:`NVIDIA Tesla P100
<Data-Center/tesla-p100/pdf/nvidia-tesla-p100-PCIe-datasheet.pdf>`
(:qty:`549 <Gbps>`, :qty:`250 <W>`, :qty:`12 <GB>` CoWoS HBM2).

.. _Vulcan access:

Access
------

Please refer to the :ref:`HLRS access policy <HLRS Applying for compute time>`.

Once access is granted, refer to the
`Vulcan user documentation <https://kb.hlrs.de/platforms/index.php/Vulcan>`__.

.. _Vulcan publications:

Publications
------------

* :cite:`zhong23a`: benchmarking of Tensorflow :abbr:`DNN (Deep Neural Network)` on CS-Storm
  coupled to :abbr:`FEM (Finite Elements Method)` on Hawk
* :cite:`zhou19a`: benchmarking of MPI communication strategies on Vulcan and Hazel Hen
* :cite:`bispo21a`: guidelines for NVIDIA accelerators on various hardware, including Vulcan
* :cite:`hpcg20a`: Vulcan ranked in 123\ :sup:`rd` place of the conjugate gradient benchmark