new server, EPYC 7351P based, hardware info

My significant other and I got ourselves a new home server.

You’ll notice that the motherboard is far from maxed out in our current build. That is quite intentional as we want the possibility of future expansion.

My installation notes, with regards to hardware, follow.

CPU

While the motherboard can accommodate Dual AMD EPYC™ 7000-Series Processors, we currently run it with one AMD EPYC 7351P 16-Core Processor only.

Yes, expanding to Dual CPU will mean removing the 7351P, since P models can only operate in a 1P configuration. But any CPU upgrade will anyway be to the generation after 7351P, 1P or 2P will be decided at the time of upgrade.

Note to self: be sure to flash mobo firmware update before installing a CPU model that was not yet released at the time of last flashing.

CPU cooler

Is a Noctua NH-U14S TR4-SP3

RAM

number description total
4 Samsung M393A4K40BB2-CTD 128 GiB

An Epyc-CPU has 8 DDR4-RAM channels (each 64 / 72 bits wide), which are connected to 8 DIMM-Slots per CPU-Socket (i.e., one DIMM-Slot per channel) on the Super Micro H11DSi-NT motherboard, totaling 16 DIMM-Slots. Initially, just four 32 GByte registered DDR4-DIMMs with ECC were chosen:

  • RAM was very expensive mid 2018
  • Epyc-CPUs and this Super Micro motherboard support equipping only 4 channels per CPU (instead of all 8).
  • An upgrade path for more RAM still with a single CPU should be kept open, without having to swap out the extant DIMMs.
  • The original CPU has just 16 cores, as the Threadripper 1950 CPU, which is clocked higher albeit owning only 4 DDR4-RAM channels. While situations may arise, in which this CPU with 4 memory channels equipped saturates available memory bandwidth, AMD also proposes and produces 32 core Epycs (i.e., also with 4 cores per memory channel) and more recently even a 32 core Threadripper 2 (although benchmarks have shown that 8 cores per memory channel only makes sense for extremely compute-intensive applications).
  • To maximise the provided memory bandwidth, the DDR4 speed grade “2666” was chosen, alleviating the initial 4 channel configuration a bit. This is the highest available DDR4 speed grade for registered ECC RAM mid 2018, bearing a slightly higher price than “2400”-DIMMs.
  • For a couple of reasons DIMMs with the largest generally available RAM-chips (16 GBit mid 2018) are preferable (always!).
  • The largest DIMM size was chosen, which is still reasonably priced: 32 GByte (mid 2018, this criterion excluded any higher capacity and LR-DIMMs, which really make sense in configurations with more than one DIMM per memory channel, anyway).
  • A configuration with eight 16 GByte DIMMs would have provided the full memory bandwidth at a 10 - 15% lower price, but would also rendered these DIMMs superfluous when upgrading.
  • Due to their good standing among overclockers and being on Super Micro’s RAM compatibility list for this motherboard, Samsung DIMMs utilising Samsung’s “B-dies” were chosen (instead of the slightly cheaper “C-die” DIMMs from Samsung).

Following the usual “rabbits & foxes”-cycle for RAM prices, it should become cheap again in 2019; this is nicely visible for commodity RAM, but also for “server RAM” (a different DIMM!). Thus do not forget to check prices and buy 4 additional DIMMs of exactly the same model in Q3/2019. Avoid slipping into Q4 (orders should be submitted early September, latest) as Q3 is a traditionally “weak” quarter with low prices, while Q4 with Thanksgiving in October and Christmas in December is always “strong”.

Merci olf!

Motherboard

While I mostly see other vendors when I do my day job, Super Micro servers are used by some of the customers I get sent to. Personally I much prefer Super Micro machines, so as motherboard a H11DSi-NT was chosen.

update firmwares

on 2018-08-17 I went to the Super Micro resources for H11DSi-NT and followed the links

  • Update Your BIOS
  • IPMI Firmware

Re-flashing updated the versions to:

Firmware Revision 01.34
Firmware Build Time 06/05/2018
BIOS Version 1.1a
BIOS Build Time 05/02/2018
Redfish Version 1.0.1
CPLD Version 04.00.14

Original fan thresholds

Default values for all fans were identical for all fans. Values of one of them follow;

[root@guest01 ~]$ ipmitool -H supermicro-bmc -U ADMIN -f ~/.ipmi-supermicro-bmc -I lanplus sensor get FAN6
Locating sensor record...
Sensor ID              : FAN6 (0x46)
 Entity ID             : 29.6
 Sensor Type (Threshold)  : Fan
 Sensor Reading        : 1100 (+/- 0) RPM
 Status                : ok
 Lower Non-Recoverable : 300.000
 Lower Critical        : 500.000
 Lower Non-Critical    : 700.000
 Upper Non-Critical    : 25300.000
 Upper Critical        : 25400.000
 Upper Non-Recoverable : 25500.000
 Positive Hysteresis   : 100.000
 Negative Hysteresis   : 100.000
 Assertion Events      : 
 Assertions Enabled    : lcr- lnr- ucr+ unr+ 
 Deassertions Enabled  : lcr- lnr- ucr+ unr+ 

Case

The case is a LIAN LI, model PC-A79.

noise level

original fans

The fans LIAN LI included with the case were, as expected, too noisy for my taste.

Noctua replacement fans

I replaced the fans that came mounted in the case with 6 Noctua fans of type NF-S12A PWM connected to PWM headers on the motherboard and 1 of type NF-S12A FLX connected to a molex adapter.

Once the Noctua fans (all mounted with their included rubber pegs instead of screws) had replaced the factory fans from LIAN LI I and I adjusted the lower fan RPM thresholds and hunted down all sources of vibration of the case, the noise level of the server became much more bearable in my small home office.

Cooling Fan mappings

I’ve connected my fans as follows to the motherboard PWM connectors

connector location
FAN 1 roof exhaust rear fan
FAN 2 not connected
FAN 3 front bottom HDD fan
FAN 4 roof exhaust front fan
FAN 5 CPU fan
FAN 6 rear exhaust fan
FAN A front middle HDD fan
FAN B front top HDD fan
molex adapter side panel fan

I’ve opted to only connect one fan in the side panel, the lower one. The upper one would reach in the space where the CPU cooler is. I might move the fan on the CPU cooler at some point. For now the upper side panel opening is covered with cardboard.

While the side panel has a neat connector that automatically shuts the side panel fans off when it is removed, that connector has only 2 contacts. Since PWM uses 4 lead, I used the NF-S12A FLX there nd connected it with both the low noise adapter and a molex adapter.

Storage

NVMe SSD

Samsung 970 Pro M.2 2280 PCIe 3.0 x4 32Gb/s 3D MLC NAND (MZ-V7P1T0BW)

hard drives

3 Seagate IronWolf Pro NAS ST8000NE0004 7.200U/min 256MB 3.5” (8.9cm) SATA 6Gb/s One of them was dead on arrival and I’m awaiting a replacement drive.

TPM

Supermicro AOM-TPM-9655V TPM security module

Power Supply

850 Watt Enermax Revolution87+ Modular 80+ Gold

Optical Drive

Pioneer BDR-209EB Blu-ray BDXL Recorder, SATA, 16x/16x/40x