Cray Unveils ClusterStor E1000 Storage Arrays: HDDs and SSDs, 1.6 TB/s per Rack
by Anton Shilov on October 30, 2019 7:00 PM EST
Cray on Wednesday introduced its new ClusterStor E1000 highly-scalable storage system, which is designed for next generation exascale supercomputers as well as future datacenters that will require massive storage performance while running converged workloads. The ClusterStor E1000 uses Cray’s new global file storage system as well as a variety of storage media, including all-flash setups and mixes of hard drives and SSDs.
From a hardware point of view, Cray’s ClusterStor E1000 relies on a proprietary highly-parallel internal architecture, which in turn is based around uses purpose-engineered AMD EPYC (Rome)-based PCIe 4.0 system with 24 NVMe U.2 SSDs. The cluster then connects to an external HPC system using Cray’s 200 Gbps Slingshot, Infiniband EDR/HDR, or 100/200 Gbps Ethernet. The key peculiarity of the ClusterStor architecture is its flexibility and scalability: it can use a wide variety of hardware and storage media to offer the a range of different performance and capacity options. The highly-parallel architecture also enables ClusterStor E1000 to be used for converged workloads without any performance degradation.
On the software side of things, the ClusterStor E1000 uses Cray's next-generation global file storage system as well as their ClusterStor Data Services system, which automatically aligns the data flow in the file system with the workflow by shifting I/O operations between different tiers of storage as appropriate. At all times, applications ‘think’ that they are dealing with a high-performance all-flash array, whereas ClusterStor E1000 uses both SSDs and HDDs to offer high-enough levels of performance and maximum storage capacity. The CDS supports scripted, scheduled or policy driven placement of data to provide optimal performance for different workloads.
Cray’s entry-level ClusterStor E1000 will offer about 60 TB of usable capacity while providing around 30 GB/s throughput. When scaled to its highest performance levels, the ClusterStor E1000 will deliver up to 1.6 TB/s sequential read/write speed and up to 50 million IOPS per rack. Clients with more than one rack will naturally get higher performance.
For general customers, Cray’s ClusterStor E1000 systems will be available starting Q1 2020. Pricing will depend on exact configurations. Specially configured ClusterStor E1000 external storage systems will be used by Cray’s upcoming Aurora, Frontier, and El Capitan exascale supercomputers, which will feature over 1.3 ExaBytes of total storage space. Furthermore, the National Energy Research Scientific Computing Center (NERSC) will use a 30 PB all-flash ClusterStor E1000.
By launching its new ClusterStor E1000 storage platform, Cray concludes a complete redesign of its product portfolio that also encompasses Shasta supercomputers, Slingshot interconnects, and software.
Related Reading:
- Hewlett Packard Enterprise to Acquire Cray for $1.3 Billion
- US Dept. of Energy Announces Frontier Supercomputer: Cray and AMD to Build 1.5 Exaflop Machine
- Intel’s Xeon & Xe Compute Accelerators to Power Aurora Exascale Supercomputer
- Cray Adds AMD EPYC Processors to CS500 Cluster Supercomputers
Source: Cray
Facebook
Twitter
RSS
11 Comments
View All Comments
rpg1966 - Wednesday, October 30, 2019 - link
50,000 IOPS per rack?Cellar Door - Wednesday, October 30, 2019 - link
Must depend on the drivers used but that does seem low.Santoval - Wednesday, October 30, 2019 - link
Bandwidth scales easily, IOPS do not. If you stripe a number of SSDs via RAID 0 their bandwidth scales almost linearly - provided the interconnect (the required number of PCIe 3.0/4.0 lanes) is not saturated of course. However the IOPS are barely affected in RAID 0 mode. Depending on the RAID implementation the IOPS might slightly improve, they might remain the same or even get a little worse.It also depends on what IOPS we are talking about : 4K-QD1 IOPS will either not break a sweat or get a bit worse due to striping overhead. Longer QDs and 4K+ IOPS should see a gain from RAID 0 though. The same apparently applies to Cray's implementation, though I don't think they would ever use RAID 0, which is unsuitable for professional storage solutions due to high risk of data loss.
danielfranklin - Thursday, October 31, 2019 - link
Nope you're wrong. My QLC NVME SSD is faster, the benchmark i dont remotely understand said so.This "Cray" thing didnt even mention M.2 or RGB so you know they arent the real deal.
/s
Seriously though, they must mean 5 Million or something, as much as we understand how IO works under different types of load, EVERYONE quotes best case and this thing can easily do more than 50,000 if all the stars are aligned.
50,000 "Kilo" IOPS would probably be closer...
deil - Thursday, October 31, 2019 - link
With 2 drives you gain a bit as you don't have to scan gigabytes of index file AND you look on other thing. You look at bursts, that would make your ssd's overheat within 2-20 minutes. Thing is that with such massive raid, locating things is the issue. I think they guarantee 50k on Q1 and usual speed is way above that but sometimes spikes down to 50k/s. In this case it does not matter that it can burst to 5M iops on favorable conditions, they need to know if it will hold up the sustained load.danielfranklin - Thursday, October 31, 2019 - link
So it was a misprint, bit I like your idea of quoting Q1 and worst case specs.Imagine a world where SSD and other storage manufacturers allowed you to have any understanding of min or sustained performance from the spec sheet alone, you could almost understand where the product sat in the market, shocking!
xelc - Thursday, October 31, 2019 - link
You're right, it is 50,000 "Kilo" IOPS.From press release: Unprecedented storage performance: ClusterStor E1000 systems can deliver up to 1.6 terabytes per second and up to 50 million I/O operations per second per rack – more than double compared to other parallel storage systems in the market today.
abufrejoval - Wednesday, October 30, 2019 - link
As much as I like Anandtech for the personal compute stuff, when it comes to HPC, there is nothing like the source: https://www.nextplatform.com/2019/10/30/cray-revam...gfkBill - Wednesday, October 30, 2019 - link
Maybe just post a link to the press release next time guys, if you can't even copy and paste, let alone add any commentary :(It scales to 50 *million* IOPS per rack, not 50,000. Thought that sounded odd.
30GB/s from 60TB is pretty impressive. Fair bit of flash involved there, presumably!
close - Thursday, October 31, 2019 - link
[Checks author]... Anton, yeah, checks out. Copy/pasta.