.. _bwUniCluster: bwUniCluster ============ .. _bwUniCluster overview: Overview -------- `bwUniCluster 2.0 `__ is a Tier-3, heterogeneous regional cluster with NVIDIA GPUs, available to the University of Stuttgart for general purpose and teaching. For research purposes, consider using :ref:`bwForCluster` instead, which has more resources. See `cluster status page `__ for outage notifications. See the `bwUniCluster2.0/Hardware and Architecture `__ page for more information. +----------------+----------+---------+-------+------------------+-----------------------+------------------+------+----------------+-----------------------+-------------------+ | Compute node | Nodes | Sockets | Cores | Clock speed | RAM | Local SSD | Bus | Accelerators | VRAM | Interconnect | +================+==========+=========+=======+==================+=======================+==================+======+================+=======================+===================+ | Thin | 200 + 60 | 2 | 40 | :qty:`2.1 ` | :qty:`96[--]192 ` | :qty:`0.96 ` | SATA | \- | \- | :qty:`100 ` | +----------------+----------+---------+-------+------------------+-----------------------+------------------+------+----------------+-----------------------+-------------------+ | HPC | 260 | 2 | 40 | :qty:`2.1 ` | :qty:`96 ` | :qty:`0.96 ` | SATA | \- | \- | :qty:`100 ` | +----------------+----------+---------+-------+------------------+-----------------------+------------------+------+----------------+-----------------------+-------------------+ | IceLake | 272 | 2 | 64 | :qty:`2.6 ` | :qty:`256 ` | :qty:`1.8 ` | NVMe | \- | \- | :qty:`200 ` | +----------------+----------+---------+-------+------------------+-----------------------+------------------+------+----------------+-----------------------+-------------------+ | Fat | 6 | 4 | 80 | :qty:`2.1 ` | :qty:`3 ` | :qty:`4.8 ` | NVMe | \- | \- | IB HDR | +----------------+----------+---------+-------+------------------+-----------------------+------------------+------+----------------+-----------------------+-------------------+ | GPU x4 | 14 | 2 | 40 | :qty:`2.1 ` | :qty:`384 ` | :qty:`3.2 ` | NVMe | 4x V100 | :qty:`32 ` | IB HDR | +----------------+----------+---------+-------+------------------+-----------------------+------------------+------+----------------+-----------------------+-------------------+ | GPU x8 | 10 | 2 | 40 | :qty:`2.6 ` | :qty:`768 ` | :qty:`15 ` | NVMe | 8x V100 | :qty:`32 ` | IB HDR | +----------------+----------+---------+-------+------------------+-----------------------+------------------+------+----------------+-----------------------+-------------------+ | IceLake GPU x4 | 15 | 2 | 64 | :qty:`2.5 ` | :qty:`512 ` | :qty:`6.4 ` | NVMe | 4x A100 / H100 | :qty:`80 [/] 94 ` | :qty:`200 ` | +----------------+----------+---------+-------+------------------+-----------------------+------------------+------+----------------+-----------------------+-------------------+ | Login | 4 | 2 | 40 | | :qty:`384 ` | | | | | :qty:`100 ` | +----------------+----------+---------+-------+------------------+-----------------------+------------------+------+----------------+-----------------------+-------------------+ .. _bwUniCluster partitions: Partitions and nodes -------------------- This cluster uses queues instead of partitions. ``dev_*`` partitions are only used for development, i.e. debugging or performance optimization. +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ | queue | node | default resources | minimum resources | maximum resources | +=====================+================+=========================+===================+================================================+ | ``dev_single`` | thin | | time=10, | | | time=30, nodes=1, mem=180000mb, | | | | | mem-per-cpu=1125mb | | | ntasks-per-node=40, (threads-per-core=2) | | | | | | | 6 nodes are reserved for this queue | +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ | ``single`` | thin | | time=30, | | | time=72:00:00, nodes=1, mem=180000mb, | | | | | mem-per-cpu=1125mb | | | ntasks-per-node=40, (threads-per-core=2) | +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ | ``dev_multiple`` | hpc | | time=10, | nodes=2 | | time=30, nodes=4, mem=90000mb, | | | | | mem-per-cpu=1125mb | | | ntasks-per-node=40, (threads-per-core=2) | | | | | | | 8 nodes are reserved for this queue | +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ | ``multiple`` | hpc | | time=30, | nodes=2 | | time=72:00:00, mem=90000mb, nodes=80, | | | | | mem-per-cpu=1125mb | | | ntasks-per-node=40, (threads-per-core=2) | +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ | ``dev_multiple_il`` | IceLake | | time=10, | nodes=2 | | time=30, nodes=8, mem=249600mb, | | | | | mem-per-cpu=1950mb | | | ntasks-per-node=64, (threads-per-core=2) | | | | | | | 8 nodes are reserved for this queue | +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ | ``multiple_il`` | IceLake | | time=10, | nodes=2 | | time=72:00:00, nodes=80, mem=249600mb, | | | | | mem-per-cpu=1950mb | | | ntasks-per-node=64, (threads-per-core=2) | +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ | ``dev_gpu_4_a100`` | IceLake + A100 | | time=10, | | | time=30, nodes=1, mem=510000mb, | | | | | mem-per-gpu=127500mb, | | | ntasks-per-node=64, (threads-per-core=2) | | | | | cpus-per-gpu=16 | | | +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ | ``gpu_4_a100`` | IceLake + A100 | | time=10, | | | time=48:00:00, nodes=9, mem=510000mb, | | | | | mem-per-gpu=127500mb, | | | ntasks-per-node=64, (threads-per-core=2) | | | | | cpus-per-gpu=16 | | | +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ | ``gpu_4_h100`` | IceLake + H100 | | time=10, | | | time=48:00:00, nodes=5, mem=510000mb, | | | | | mem-per-gpu=127500mb, | | | ntasks-per-node=64, (threads-per-core=2) | | | | | cpus-per-gpu=16 | | | +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ | ``fat`` | fat | | time=10, | mem=180001mb | | time=72:00:00, nodes=1, mem=3000000mb, | | | | | mem-per-cpu=18750mb | | | ntasks-per-node=80, (threads-per-core=2) | +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ | ``dev_gpu_4`` | gpu4 | | time=10, | | | time=30, nodes=1, mem=376000, | | | | | mem-per-gpu=94000mb, | | | ntasks-per-node=40, (threads-per-core=2) | | | | | cpus-per-gpu=10 | | | 1 node is reserved for this queue | +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ | ``gpu_4`` | gpu4 | | time=10, | | | time=48:00:00, mem=376000, nodes=14, | | | | | mem-per-gpu=94000mb, | | | ntasks-per-node=40, (threads-per-core=2) | | | | | cpus-per-gpu=10 | | | +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ | ``gpu_8`` | gpu8 | | time=10, | | | time=48:00:00, mem=752000, nodes=10, | | | | | mem-per-cpu=94000mb, | | | ntasks-per-node=40, (threads-per-core=2) | | | | | cpus-per-gpu=10 | | | +---------------------+----------------+-------------------------+-------------------+------------------------------------------------+ Source: `bwUniCluster2.0/Batch Queues `__. .. Note: to convert a wiki table to a reST table, use the following command: $ pandoc table.wiki --to markdown --columns=158 Thin nodes ^^^^^^^^^^ There are 260 nodes equipped with 2 :cpu-intel-sheet2:`Intel Xeon Gold 6230 <192437/intel-xeon-gold-6230-processor-27-5m-cache-2-10-ghz>` (20 cores, 40 threads, :qty:`2.1 `, :qty:`125 `), :qty:`96 ` or :qty:`192 ` of RAM, InfiniBand HDR (:qty:`100 ` blocking) interconnect. There are no GPUs. HPC nodes ^^^^^^^^^ There are 260 nodes equipped with 2 :cpu-intel-sheet2:`Intel Xeon Gold 6230 <192437/intel-xeon-gold-6230-processor-27-5m-cache-2-10-ghz>` (20 cores, 40 threads, :qty:`2.1 `, :qty:`125 `), :qty:`96 ` of RAM, InfiniBand HDR (:qty:`100 `) interconnect. There are no GPUs. IceLake nodes ^^^^^^^^^^^^^ There are 272 nodes equipped with 2 :cpu-intel-sheet1:`Intel Xeon Platinum 8358 <212282/intel-xeon-platinum-8358-processor-48m-cache-2-60-ghz>` (32 cores, 64 threads, :qty:`2.6 `, :qty:`250 `), :qty:`256 ` of RAM, InfiniBand HDR (:qty:`200 `) interconnect. There are no GPUs. Fat nodes ^^^^^^^^^ There are 6 nodes equipped with 4 :cpu-intel-sheet2:`Intel Xeon Gold 6230 <192437/intel-xeon-gold-6230-processor-27-5m-cache-2-10-ghz>` (20 cores, 40 threads, :qty:`2.1 `, :qty:`125 `) and :qty:`3 ` of RAM, InfiniBand HDR interconnect. There are no GPUs. GPU x4 nodes ^^^^^^^^^^^^ There are 14 nodes equipped with 2 :cpu-intel-sheet2:`Intel Xeon Gold 6230 <192437/intel-xeon-gold-6230-processor-27-5m-cache-2-10-ghz>` (20 cores, 40 threads, :qty:`2.1 `, :qty:`125 `) and :qty:`384 ` of RAM, InfiniBand HDR interconnect. Each node has 4 :gpu-nvidia-sheet2:`NVIDIA Tesla V100 SXM2 ` (:qty:`300 `, :qty:`300 `, :qty:`32 ` HBM2). GPU x8 nodes ^^^^^^^^^^^^ There are 10 nodes equipped with 2 :cpu-intel-sheet2:`Intel Xeon Gold 6248 <192446/intel-xeon-gold-6248-processor-27-5m-cache-2-50-ghz>` (20 cores, 40 threads, :qty:`2.6 `, :qty:`150 `), :qty:`768 ` of RAM, InfiniBand HDR interconnect. Each node has 8 :gpu-nvidia-sheet2:`NVIDIA Tesla V100 SXM2 ` (:qty:`300 `, :qty:`300 `, :qty:`32 ` HBM2). IceLake GPU x4 nodes ^^^^^^^^^^^^^^^^^^^^ There are 15 nodes equipped with 2 :cpu-intel-sheet1:`Intel Xeon Platinum 8358 <212282/intel-xeon-platinum-8358-processor-48m-cache-2-60-ghz>` (32 cores, 64 threads, :qty:`2.5 `, :qty:`250 `), :qty:`512 ` of RAM, InfiniBand HDR (:qty:`200 `) interconnect. Each node has either 4 `NVIDIA Tesla A100 `__ (:qty:`80 ` HBM2e) or 4 `NVIDIA Tesla H100 `__ (:qty:`94 ` HBM). Login nodes ^^^^^^^^^^^ There are 4 nodes equipped with 2 CPUs (20 cores, 40 threads, :qty:`2.6 `, :qty:`150 `), :qty:`384 ` of RAM, InfiniBand HDR (:qty:`100 ` blocking) interconnect. .. _bwUniCluster filesystems: Filesystems ----------- * `bwUniCluster2.0/Filesystems `__ .. _bwUniCluster access: Access ------ For University of Stuttgart personnel, applications are processed by the HLRS. Follow the instructions outlined in the HLRS page `bwUniCluster Access `__. You need to communicate your personal information and write a short abstract of your research project or teaching project. Once your application is approved, you will need to register an account at KIT and fill out a questionnaire. The review phase takes a few working days. Be advised that entitlements are time-limited: 1 year for students, or contract end date for academic staff. No reminder will be sent before entitlements are revoked by TIK. Students need to ask for an extension before the cutoff date. Academic staff whose contract gets renewed need to ask for an extension before the end date of the old contract (in the e-mail, mention the new contract end date). To check your entitlements, log into `bwIDM `__, open the "Shibboleth" tab and look for ``http://bwidm.de/entitlement/bwUniCluster``. Afterwards, create an account on the bwHPC service by following the instructions in `Registration/bwUniCluster `__. You need 2FA, and SMS are not an option. If you don't have a YubiKey or a device capable of managing software tokens, it is possible to use the KeePassXC software instead (see :ref:`2FA TOTP`). Once access is granted, refer to the `bwUniCluster2.0 user documentation `__. See also :doc:`/_pages/_appendix/using_bwunicluster` for building software and submitting jobs. .. _bwUniCluster obligations: Obligations ----------- Use of the cluster must be acknowledged in scientific publications. Citation details of these publications must be communicated to the bwHPC-S5 project (publications@bwhpc.de). For details, refer to `bwUniCluster2.0/Acknowledgement `__. .. _bwUniCluster publications: Publications ------------ * :cite:`kuron19b`: ICP ESPResSo simulations on bwUniCluster * :cite:`zeman21a`: ICP GROMACS simulations on Hazel Hen with support from bwHPC * `all publications enabled by bwHPC `__