new server, EPYC 7351P based, hardware info
Table of Contents
My significant other and I got ourselves a new home server.
You’ll notice that the motherboard is far from maxed out in our current build. That is quite intentional as we want the possibility of future expansion.
My installation notes, with regards to hardware, follow.
While the motherboard can accommodate Dual AMD EPYC™ 7000-Series Processors, we currently run it with one AMD EPYC 7351P 16-Core Processor only.
Yes, expanding to Dual CPU will mean removing the 7351P, since P models can only operate in a 1P configuration. But any CPU upgrade will anyway be to the generation after 7351P, 1P or 2P will be decided at the time of upgrade.
TODO: March / April 2020; evaluate available Epyc 2 CPUs, expect prices settled and CPU available around September 2020.
Note to self: be sure to flash mobo firmware update before installing a CPU model that was not yet released at the time of last flashing.
Update 2019-08-09: Evaluation of available Epyc “Rome” CPUs for “2P” / dual-socket configurations in August 2020, followed by either buying two in early September 2020 or a re-evaluation in August 2021 and buying early September 2021.
See also at anandtech:
|4||Samsung M393A4K40BB2-CTD||128 GiB|
An Epyc-CPU has 8 DDR4-RAM channels (each 64 / 72 bits wide), which are connected to 8 DIMM-Slots per CPU-Socket (i.e., one DIMM-Slot per channel) on the Super Micro H11DSi-NT motherboard, totaling 16 DIMM-Slots. Initially, just four 32 GByte registered DDR4-DIMMs with ECC were chosen:
- RAM was very expensive mid 2018
- Epyc-CPUs and this Super Micro motherboard support equipping only 4 channels per CPU (instead of all 8).
- An upgrade path for more RAM still with a single CPU should be kept open, without having to swap out the extant DIMMs.
- The original CPU has just 16 cores, as the Threadripper 1950 CPU, which is clocked higher albeit owning only 4 DDR4-RAM channels. While situations may arise, in which this CPU with 4 memory channels equipped saturates available memory bandwidth, AMD also proposes and produces 32 core Epycs (i.e., also with 4 cores per memory channel) and more recently even a 32 core Threadripper 2 (although benchmarks have shown that 8 cores per memory channel only makes sense for extremely compute-intensive applications).
- To maximise the provided memory bandwidth, the DDR4 speed grade “2666” was chosen, alleviating the initial 4 channel configuration a bit. This is the highest available DDR4 speed grade for registered ECC RAM mid 2018, bearing a slightly higher price than “2400”-DIMMs.
- For a couple of reasons DIMMs with the largest generally available RAM-chips (8 GBit mid 2018) are preferable (always!).
- The largest DIMM size was chosen, which was still reasonably priced: 32 GByte (mid 2018 this criterion excluded any higher capacity R- and LR-DIMMs; LR-DIMMs really make sense in configurations with more than one DIMM per memory channel or more ranks than this 2Rx4 configuration of each DIMM, anyway).
- A configuration with eight 16 GByte RDIMMs would have provided the full memory bandwidth at a 10 - 15% lower price, but would also rendered these DIMMs superfluous when upgrading.
- Due to their good standing among overclockers and being on Super Micro’s RAM compatibility list for this motherboard, Samsung DIMMs utilising Samsung’s “B-dies” were chosen (instead of the slightly cheaper “C-die” DIMMs from Samsung).
Following the usual “rabbits & foxes”-cycle for RAM prices, it should become cheap again in 2019; this is nicely visible for commodity RAM, but also for “server RAM” (a different DIMM!). Thus do not forget to check prices and buy 4 additional DIMMs of exactly the same model in Q3/2019. Avoid slipping into Q4 (orders should be submitted early September, latest) as Q3 is a traditionally “weak” quarter with low prices, while Q4 with Thanksgiving in October and Christmas in December is always “strong”.
RAM upgrade September 2019
In September 2019, as planned above, another 128 GiB RAM was added. Samsung M393A4K40BB2-CTD modules were used again.
Now all 8 RAM slots of CPU1 are populated.
While I mostly see other vendors when I do my day job, Super Micro servers are used by some of the customers I get sent to. Personally I much prefer Super Micro machines, so as motherboard a H11DSi-NT was chosen.
on 2018-08-17 I went to the Super Micro resources for H11DSi-NT and followed the links
- Update Your BIOS
- IPMI Firmware
Re-flashing updated the versions to:
|Firmware Build Time||06/05/2018|
|BIOS Build Time||05/02/2018|
Original fan thresholds
Default values for all fans were identical for all fans. Values of one of them follow;
[root@guest01 ~]$ ipmitool -H supermicro-bmc -U ADMIN -f ~/.ipmi-supermicro-bmc -I lanplus sensor get FAN6 Locating sensor record... Sensor ID : FAN6 (0x46) Entity ID : 29.6 Sensor Type (Threshold) : Fan Sensor Reading : 1100 (+/- 0) RPM Status : ok Lower Non-Recoverable : 300.000 Lower Critical : 500.000 Lower Non-Critical : 700.000 Upper Non-Critical : 25300.000 Upper Critical : 25400.000 Upper Non-Recoverable : 25500.000 Positive Hysteresis : 100.000 Negative Hysteresis : 100.000 Assertion Events : Assertions Enabled : lcr- lnr- ucr+ unr+ Deassertions Enabled : lcr- lnr- ucr+ unr+
The case is a LIAN LI, model PC-A79.
The fans LIAN LI included with the case were, as expected, too noisy for my taste.
Noctua replacement fans
I replaced the fans that came mounted in the case with 6 Noctua fans of type NF-S12A PWM connected to PWM headers on the motherboard and 1 of type NF-S12A FLX connected to a molex adapter.
Once the Noctua fans (all mounted with their included rubber pegs instead of screws) had replaced the factory fans from LIAN LI I and I adjusted the lower fan RPM thresholds and hunted down all sources of vibration of the case, the noise level of the server became much more bearable in my small home office.
Cooling Fan mappings
I’ve connected my fans as follows to the motherboard PWM connectors
|FAN 1||roof exhaust rear fan|
|FAN 2||not connected|
|FAN 3||front bottom HDD fan|
|FAN 4||roof exhaust front fan|
|FAN 5||CPU fan|
|FAN 6||rear exhaust fan|
|FAN A||front middle HDD fan|
|FAN B||front top HDD fan|
|molex adapter||side panel fan|
I’ve opted to only connect one fan in the side panel, the lower one. The upper one would reach in the space where the CPU cooler is. I might move the fan on the CPU cooler at some point. For now the upper side panel opening is covered with cardboard.
While the side panel has a neat connector that automatically shuts the side panel fans off when it is removed, that connector has only 2 contacts. Since PWM uses 4 lead, I used the NF-S12A FLX there nd connected it with both the low noise adapter and a molex adapter.
FAN-A/B is for add-on card or chassis fan and is controlled by system temp. FAN 1~6 is controlled by CPU temp.
https://www.supermicro.com/support/faqs/faq.cfm?faq=13798 applies to X9, unsure if it also applies to H11.
Samsung 970 Pro M.2 2280 PCIe 3.0 x4 32Gb/s 3D MLC NAND (MZ-V7P1T0BW)
3 Seagate IronWolf Pro NAS ST8000NE0004 7.200U/min 256MB 3.5" (8.9cm) SATA 6Gb/s One of them was dead on arrival and I’m awaiting a replacement drive.
Supermicro AOM-TPM-9655V TPM security module
850 Watt Enermax Revolution87+ Modular 80+ Gold