new server, EPYC 7351P based, hardware info

Table of Contents

My significant other and I got ourselves a new home server.

You’ll notice that the motherboard is far from maxed out in our current build. That is quite intentional as we want the possibility of future expansion.

My installation notes, with regards to hardware, follow.

CPU

While the motherboard can accommodate Dual AMD EPYC™ 7000-Series Processors, we currently run it with one AMD EPYC 7351P 16-Core Processor only.

Yes, expanding to Dual CPU will mean removing the 7351P, since P models can only operate in a 1P configuration. But any CPU upgrade will anyway be to the generation after 7351P, 1P or 2P will be decided at the time of upgrade.

TODO: March / April 2020; evaluate available Epyc 2 CPUs, expect prices settled and CPU available around September 2020.

Note to self: be sure to flash mobo firmware update before installing a CPU model that was not yet released at the time of last flashing.

Update 2019-08-09: Evaluation of available Epyc “Rome” CPUs for “2P” / dual-socket configurations in August 2020, followed by either buying two in early September 2020 or a re-evaluation in August 2021 and buying early September 2021.

CPU cooler

Is a Noctua NH-U14S TR4-SP3

RAM

number	description	total
4	Samsung M393A4K40BB2-CTD	128 GiB

An Epyc-CPU has 8 DDR4-RAM channels (each 64 / 72 bits wide), which are connected to 8 DIMM-Slots per CPU-Socket (i.e., one DIMM-Slot per channel) on the Super Micro H11DSi-NT motherboard, totaling 16 DIMM-Slots. Initially, just four 32 GByte registered DDR4-DIMMs with ECC were chosen:

RAM was very expensive mid 2018
Epyc-CPUs and this Super Micro motherboard support equipping only 4 channels per CPU (instead of all 8).
An upgrade path for more RAM still with a single CPU should be kept open, without having to swap out the extant DIMMs.
The original CPU has just 16 cores, as the Threadripper 1950 CPU, which is clocked higher albeit owning only 4 DDR4-RAM channels. While situations may arise, in which this CPU with 4 memory channels equipped saturates available memory bandwidth, AMD also proposes and produces 32 core Epycs (i.e., also with 4 cores per memory channel) and more recently even a 32 core Threadripper 2 (although benchmarks have shown that 8 cores per memory channel only makes sense for extremely compute-intensive applications).
To maximise the provided memory bandwidth, the DDR4 speed grade “2666” was chosen, alleviating the initial 4 channel configuration a bit. This is the highest available DDR4 speed grade for registered ECC RAM mid 2018, bearing a slightly higher price than “2400”-DIMMs.
For a couple of reasons DIMMs with the largest generally available RAM-chips (8 GBit mid 2018) are preferable (always!).
The largest DIMM size was chosen, which was still reasonably priced: 32 GByte (mid 2018 this criterion excluded any higher capacity R- and LR-DIMMs; LR-DIMMs really make sense in configurations with more than one DIMM per memory channel or more ranks than this 2Rx4 configuration of each DIMM, anyway).
A configuration with eight 16 GByte RDIMMs would have provided the full memory bandwidth at a 10 - 15% lower price, but would also rendered these DIMMs superfluous when upgrading.
Due to their good standing among overclockers and being on Super Micro’s RAM compatibility list for this motherboard, Samsung DIMMs utilising Samsung’s “B-dies” were chosen (instead of the slightly cheaper “C-die” DIMMs from Samsung).

Following the usual “rabbits & foxes”-cycle for RAM prices, it should become cheap again in 2019; this is nicely visible for commodity RAM, but also for “server RAM” (a different DIMM!). Thus do not forget to check prices and buy 4 additional DIMMs of exactly the same model in Q3/2019. Avoid slipping into Q4 (orders should be submitted early September, latest) as Q3 is a traditionally “weak” quarter with low prices, while Q4 with Thanksgiving in October and Christmas in December is always “strong”.

Merci olf!

RAM upgrade September 2019

In September 2019, as planned above, another 128 GiB RAM was added. Samsung M393A4K40BB2-CTD modules were used again.

Now all 8 RAM slots of CPU1 are populated.

Motherboard

While I mostly see other vendors when I do my day job, Super Micro servers are used by some of the customers I get sent to. Personally I much prefer Super Micro machines, so as motherboard a H11DSi-NT was chosen.

update firmwares

on 2018-08-17 I went to the Super Micro resources for H11DSi-NT and followed the links

Update Your BIOS
IPMI Firmware

Re-flashing updated the versions to:


Firmware Revision	01.34
Firmware Build Time	06/05/2018
BIOS Version	1.1a
BIOS Build Time	05/02/2018
Redfish Version	1.0.1
CPLD Version	04.00.14

Original fan thresholds

Default values for all fans were identical for all fans. Values of one of them follow;

[root@guest01 ~]$ ipmitool -H supermicro-bmc -U ADMIN -f ~/.ipmi-supermicro-bmc -I lanplus sensor get FAN6
Locating sensor record...
Sensor ID              : FAN6 (0x46)
 Entity ID             : 29.6
 Sensor Type (Threshold)  : Fan
 Sensor Reading        : 1100 (+/- 0) RPM
 Status                : ok
 Lower Non-Recoverable : 300.000
 Lower Critical        : 500.000
 Lower Non-Critical    : 700.000
 Upper Non-Critical    : 25300.000
 Upper Critical        : 25400.000
 Upper Non-Recoverable : 25500.000
 Positive Hysteresis   : 100.000
 Negative Hysteresis   : 100.000
 Assertion Events      : 
 Assertions Enabled    : lcr- lnr- ucr+ unr+ 
 Deassertions Enabled  : lcr- lnr- ucr+ unr+

Case

The case is a LIAN LI, model PC-A79.

noise level

original fans

The fans LIAN LI included with the case were, as expected, too noisy for my taste.

Noctua replacement fans

I replaced the fans that came mounted in the case with 6 Noctua fans of type NF-S12A PWM connected to PWM headers on the motherboard and 1 of type NF-S12A FLX connected to a molex adapter.

Once the Noctua fans (all mounted with their included rubber pegs instead of screws) had replaced the factory fans from LIAN LI I and I adjusted the lower fan RPM thresholds and hunted down all sources of vibration of the case, the noise level of the server became much more bearable in my small home office.

Cooling Fan mappings

I’ve connected my fans as follows to the motherboard PWM connectors

connector	location
FAN 1	roof exhaust rear fan
FAN 2	not connected
FAN 3	front bottom HDD fan
FAN 4	roof exhaust front fan
FAN 5	CPU fan
FAN 6	rear exhaust fan
FAN A	front middle HDD fan
FAN B	front top HDD fan
molex adapter	side panel fan

I’ve opted to only connect one fan in the side panel, the lower one. The upper one would reach in the space where the CPU cooler is. I might move the fan on the CPU cooler at some point. For now the upper side panel opening is covered with cardboard.

While the side panel has a neat connector that automatically shuts the side panel fans off when it is removed, that connector has only 2 contacts. Since PWM uses 4 lead, I used the NF-S12A FLX there nd connected it with both the low noise adapter and a molex adapter.