HyperCloud Enables Density Upgrades without Throughput Penalties – Confirmed by IBM
November 19, 2012 at 6:00 AM
In previous blog entries, we highlighted the application benefits and significant return on investment realized by utilizing 16GB and 32GB HyperCloud™ HCDIMMs on Intel® Xeon® Processor E5-2600 series server platforms. In this post, we will reveal the best strategy for purchasing a server and scaling vertically later by adding additional memory.
There are many reasons to consider purchasing servers that are only partially populated with memory- 1 DIMM per channel (DPC) or 2 DPC. One reason is that memory requirements cannot initially be defined exactly for a given workload; the usual way these approximations are made is from historical DRAM capacity per server data. Another reason is that DRAM typically becomes less expensive over time so some users tend to put in the minimum amount of memory then scale up later at a much lesser expense. And lastly, users want to scale up their server's memory over time as the demand calls for it.
According to IBM's Redbook for the System x3650 M4 (http://www.redbooks.ibm.com/abstracts/tips0850.html), LRDIMMs cannot reach maximum operating speed when fully populated. See Fig.1 below for a summary.
Fig. 1 Maximum Memory Speeds
At 1.35V, 32GB LRDIMMs can be used to populate the server to only 256GB or 1 DPC at 1333MHz. Upgrading to 2 or 3 DPC inevitably requires a speed downgrade by at least one level to 1066MHz. At 1.5V, a maximum of two channels can be populated for a total of 512GB at 1333MHz and upgrading a LRDIMM solution to a third channel requires, again, a downgrade in speed to 1066MHz.
HyperCloud™ is rated at 1333MHz for 1 DPC, 2DPC, or 3DPC configurations. 16GB HCDIMMs are in mass production and are qualified on IBM and HP servers, and the 32GB HCDIMM will be available at 1333MHz in the near future (it is already available at 1066MHz on the IBM System x3650 M4 server). The key takeaway here is that a sever populated with 32GB HyperCloud™ can be started with 1 or 2 DPCs and can be easily scaled to 3 DPC (which is 768GB with 32GB HCDIMMs) running at the maximum operating speed of 1333MHz.
Why is the Scaling Factor so Important?
A leading virtualization provider, VMware, illustrates how users can cope with the ever increasing amount of data by segmenting them into virtual machines. Generally speaking, each virtual node requires about 2GB of DRAM to provide efficient operation. The maximum number of virtual nodes (virtual machines or VMs) per core is 320 in the new vSphere™ editions (4 /5.1), but most users utilize far less than that. If we assume 16 virtual nodes per core the memory requirement adds up to 32GB per core and since a typical Romley based CPU has 8 to 10 cores, you would need up to 320GB per CPU, or 80GB per channel (320GB/4 channels).
Using 32GB DIMMs (which is not expected to increase for the DDR3 generation due to the lack of 8Gb mono DRAM die), three DIMMs are required to meet this 80GB per channel memory density requirement. Initial 1 or 2 DPC configurations with LRDIMM will hit even bigger performance disadvantages when scaling to 3 DPC. The server will downgrade the speed from 1333MHz to 1066MHz when the user tries to scale up the server making LRDIMM a poor choice. This problem is visualized in Fig. 2 below.
Fig. 2 Throughput decreases when upgrading with LRDIMM
The reason the effective throughput is mentioned in Fig. 2 above is explained in another one of our blog posts that detail that even though HCDIMMs and LRDIMMs have identical data rates, HCDIMMs actually have a significantly higher data throughput than LRDIMMs.(http://www.netlist.com/hypercloud/hcdimm-white-papers/effective-data-rates-on-fully-populated-3dpc-servers-lrdimms-dirty-little-secret-what-you-see-is-not-what-you-get/).
Coming back to the VM example above,VMware has shown in its user information guide that memory is the most critical parameter in a server system to perform optimal hardware usage (www.vmware.com/pdf/esx3_memory.pdf). The virtualization software claims typically less than 6% of the available CPU, network I/O and disk I/O resources are being utilized. This is strikingly different for the memory--here up to 40% of the resources are being utilized. VMware states that experience with production workloads has shown that on modern, x86-based hardware, memory is often the most used resource. In other words,
All of these trends lead to the conclusion that additional memory is one of the most important investments that can be made to get more out of a virtualized infrastructure
Increased memory requirements means populating servers at the full 3 DPC with highest density DIMMs.Since no 32GB dual-Rank RDIMM solution exists for DDR3, the only options are 32GB LRDIMM or 32GB HyperCloud™ HCDIMM. Only HyperCloud™ allows the user to scale up the server to the highest densities without a performance penalty.
By using HyperCloud™ DIMMs, a penalty-free migration path from 1 or 2 DPC to 3 DPC is preserved. This path does not exist with LRDIMMs because of that speed and throughput penalty. Starting with an LRDIMM solution and populating additional channels later can turn out to be costly to productivity because as memory speed is lowered the overall performance of the server is also decreased. HyperCloud™ HCDIMMs are the only solution to provide the highest densities at the highest speeds regardless of populating at 1 DPC, 2 DPC, or 3 DPC. As such, HCDIMMs provide an easy and painless memory scalability path.