Today Nimbus Data, one of the first companies to venture into enterprise flash storage in 2003, is announcing its latest generation ExaDrive product. Following on from the success of the 100 TB ExaDrive DC, the company is releasing a new ExaDrive NL series, from 16 TB to 64 TB. This new series uses enterprise grade QLC, compared to TLC in the DC variant, to provide a more cost effective solution with endurance designed to eclipses any other QLC drive on the market.

Enterprise Storage Comes in Layers

For most enterprise storage needs, tiered storage has become the norm. For random access workloads with high bandwidth requirements, NVMe storage or DRAM-like alternatives are preferred, while on the other end where capacity is king, large mechanical hard drives based on SATA or SAS take center stage, with up to ninety 18 TB drives in a single 4U operating as ‘Nearline’ storage as a data retrieval tier or cold storage. There are a number of companies designing controllers, flash, new platter technology, and all between in order to offers something either at the extremes of these storage needs, or to create new caching tiers in the middle. The idea is to trade off latency, bandwidth and price, for capacity as we move up from those bulky spinning rust things.

This leads us to Nimbus Data. The company started offering flash storage solutions for the enterprise as early as 2003, before flash storage was really a thing – at the time the products were small but offered amazing performance over hard drives, and it was several years before other companies started producing similar products for the market. The company has been profitable for years, never relying on any venture capital or seed funding, and has over 200+ customers, including big names over the years such as eBay, Paypal, and Apple. There are also two big CDNs in the mix, which unfortunately they can’t disclose, but these are the sorts of customers that Nimbus Data is courting with its ExaDrive product range.

Two years ago, Nimbus Data showcased the ExaDrive DC, a 100 TB 3.5-inch drive designed for like-for-like swap for the 14-18 TB drives that populated most of the nearline storage options on the market. The problem as always with NAND flash is that cost per TB is higher than mechanical storage, but the DC was aimed to offer much greater density and better power consumption in the same chassis. The ExaDrive DC internally has four separate PCBs, each with a custom flash controller with firmware written by Nimbus Data to manage the status of the flash cells. These are grouped into a backplane inside the 3.5-inch chassis, where an FPGA does overall management and monitoring of the drive. The drive comes with SAS 2.0 and SATA 3.0 6 Gbps interfaces, and the size of the drive combined with the speed of the interface means that Nimbus Data has been offering the drive with an unlimited endurance warranty for five years. Writing sequential data to this drive, at around 400 MBps, would take 70 hours for context.


From LinusTechTips' ExaDrive DC 100TB Teardown

These drives were priced in the region of $40,000 a piece, however the goal was to be able to replace what could be 20U of 540+ mechanical hard drives, around 8 PB of storage, with one 4U with 9 PB. The metrics that Nimbus Data’s customers cared about were centered around density, better latency, endurance (mechanical hard drives are often rated at 550 TB/year, which is actually a limit on the motor, not the platter), power per TB, and ultimately operational expenses over the lifetime of the server (OpEx), including rack space, power, cooling, and maintenance. The drives were expensive for not only being new on the market, but they also used TLC NAND, as it was easy to come by at the time. Some of Nimbus Data’s customers loved the product, but requested if a cheaper version was possible. Enter the ExaDrive NL.

The ExaDrive NL is almost an identical product to the DC variants, except instead of using TLC NAND, this product uses QLC NAND. This allows the amount of NAND to be used to drop, saving some expense.

The main criticism for QLC NAND is that it is rated at a lower endurance than TLC NAND, at users who deploy QLC at an enterprise level at scale might be put off by this property. Nimbus Data tells us that going into this project, they understood why this was a concern for its customers. As a result, they are using some of the best enterprise-grade QLC available on the market, which I’m told is rated for 1500 P/E cycles minimum.

Also to that, Nimbus Data says that competing products in this space that use QLC often come up across controller limitations when growing in size. Modern NAND controllers can only reliably address 4-16 TB of NAND, and so going beyond that requires adjusting the block size which increases write amplification, reducing the effective endurance of the NAND and lifetime of the drive. It also increases the load on the controller, causing extra power consumption. Due to the ExaDrive’s design, using four PCBs of NAND each with its own controller and an FPGA intermediary before the SAS/SATA interface, the ExaDrive NL maintains high endurance and low write amplification as well as a standard 4K block size. The drive also has overprovisioning, around the 8% level as is industry standard.

The new ExaDrive NL versions will be offered in capacities including 16 TB, 32 TB, and 64 TB, with SATA and dual-port SAS interfaces. These drives don’t have an unlimited endurance 5-year warranty like the DC 100 TB model, but are rated up to 75 PB within the 5 years, which is good for 0.6 DWPD. However, writing 60% of the 64 TB drive at SATA speeds will take 1.7 days anyway, indicating that this drive should be good for a long time.

Performance is rated at 500 MB/s read and 460 MB/s write with sequential 128 KB data, and up to 100K read/write IOPs with 4K data. The drives do operate an SLC cache, and Nimbus Data was keen to point out that these drives ‘in typical scenarios’ will never go outside that SLC cache window. Also, the fact that the design is four PCBs of NAND each with its own controller means that write performance is almost identical to read performance, which negates one of the common criticisms with QLC. Latency is in the 50-100 microsecond range. Power consumption is rated at 10W idle and 14W typical for the 64 TB model, and the 3.5-inch enclosure is ventilated to assist this.

One of the comments Nimbus Data made is that in these big 3.5-inch JBOD designs that can hold 90 drives in a 4U, as hard drives spin up and get warm, the drives furthest from the front of the chassis end up with warm air on them, and then the CPU behind that also gets even warmer air. The lower power consumption of NAND helps with that, and the additional ventilation should also keep the drives cooler as required. Also, those vertical drop design chassis are apparently more suited to NAND – Nimbus Data’s CEO in our briefing call stated that in speaking to customers, he learned that vertically mounted HDD reliability is reduced ~5-10%, something which a NAND SSD wouldn’t have a problem with. Also, with regards array rebuild time if these drives were in an array, we were told that due to using NAND that despite the large drive size, rebuilding a 64 TB SSD is quicker than any of the 18 TB HDDs.

The key targets Nimbus Data is hoping to hit with the new ExaDrive NL series is that nearline / CDN type storage, but at a better latency, better density per rack unit, better power per TB, and better OpEx per TB over the lifetime of the hardware. Nimbus Data predicts an 80% reduction in power and cooling costs per TB with its product, as well as potential savings in software licensing (fewer servers needed), reduced admin costs (fewer servers to manage), and a better blending of SSD/HDD as needed with respect to performance and cost. The other metric Nimbus Data is proud of is weight per TB, with an 18 TB HDD weighing ~1.5 lbs, whereas the 64 TB NL drive weighs ~0.87 lbs, an 87% reduction per TB.

In order to assuage concerns about compatibility, Nimbus Data is quoting that the drives have already been tested in a number of high-profile and popular server and storage designs, including PowerEdge Rx40 devices and PERC HBAs, HPE DL300 Gen10 with SmartArray Gen10 storage adaptors, Cisco UCS designs and also Lenovo ThinkSystem and Supermicro builds. A full list is in the slide deck we’ve republished here.

We were told that Nimbus Data currently sells its large 3.5-inch SSD drives mostly into edge deployments, where data density is critical, and these new NL drives will help deliver to customers currently on spinning HDDs that required a better price per TB ratio before jumping to NAND. Pricing will be customer-specific, however the 16 TB model will start through authorized resellers and system manufacturers for under $3000.

Update: Pricing has been revealed as:

  • 64 TB: $10900
  • 32 TB: $5600
  • 16 TB: $2900

Pricing will be updated quarterly as NAND pricing changes.

We’re hoping to get the new ExaDrive NL and perhaps the DC in to test. Stay tuned for that.

Gallery: ExaDrive NL

Related Reading

 

Comments Locked

25 Comments

View All Comments

  • stanleyipkiss - Wednesday, August 26, 2020 - link

    Amazingly, can both afford and want two to replace a 12 disk 8TB Synology NAS. It would be simpler and easier to maintain and no raid rebuilds.
  • nathanddrews - Wednesday, August 26, 2020 - link

    I mean, it's expensive, but I surprised at how reasonable it seems!
  • Samus - Thursday, August 27, 2020 - link

    True, it's only around 5x the price of 64TB Enterprise HDD storage. That actually isn't bad when you consider the storage density, performance and expected reliability. Footprint is going to be everything going forward as HDD's have really hit a wall (2x storage density in the last decade)
  • phoenix_rizzen - Thursday, September 3, 2020 - link

    When a 4 TB Dell-branded SATA drive costs over $800 CAD, $3800 CAD for 16 TB of SSD doesn't seem that bad. :)

    https://www.dell.com/en-ca/shop/4tb-72k-rpm-sata-6...
  • imaheadcase - Wednesday, August 26, 2020 - link

    I believe though you couldn't upgrade to these. Unless they have changed in last couple years, synology is pretty specific on size and types of hardrives you can install. I mean its a huge selective, but enterprise stuff is not supported in its configs for most part. If you are hell bent on installing a $3k+ drive i would contact them to make sure they synology OS even supports that oddity of size.
  • nimbusdata - Friday, August 28, 2020 - link

    Hi there! We have tested ExaDrive NL in Synology DS1819+ NAS - works without any issues. We plan to publish a more complete compatibility list in the coming weeks. Thank you for your comment!
  • ksec - Thursday, August 27, 2020 - link

    Yes. Although I would like to see more HDD improvement so I could fit 2.5" HDD first at a reasonable cost before moving to SSD. Which is still quite expensive.

    I am still waiting for Qnap's ZFS Hero OS.
  • frbeckenbauer - Wednesday, August 26, 2020 - link

    Interesting that the QLC version is only available in lower capacity, even though QLC offers higher density.
  • FreckledTrout - Wednesday, August 26, 2020 - link

    Over provisioning to keep the workload rating reasonable?
  • Ian Cutress - Wednesday, August 26, 2020 - link

    ...for now at least. It could also depends on thermals and things, or QLC supply. I think one QLC wafer is good for ~55 TB? The DC range also had 64 TB versions, so perhaps they're just going for what fits right for the customer base. Or, if you want the densest, you have to pay the premium - right now, competition for them is limited.

Log in

Don't have an account? Sign up now