HPE Issues HPD7 Fix for SSDs That Will Fail at 40000 Hours

4
HPE 846432 B21 EO1600JVYPP 1.6TB SAS SSD
HPE 846432 B21 EO1600JVYPP 1.6TB SAS SSD

HPE has a fix out now for an issue they have been working through with customers that impacts four SSD models that can fail at 40,000 hours. A few months ago, HPE found that some SAS SSDs it utilized would fail at 32,768 hours of operation or 3 years, 270 days 8 hours exactly. Here is a link to that bulletin. It turns out, this is a different issue where drives will fail at 40,000 hours or 4 years, 206 days, 16 hours. Here is the new HPE bulletin.

HPE SSDs Failing at 40,000 Hours

There are four SSDs that will fail at 40,000 hours which are 800GB and 1.6TB SAS SSDs. Here is the official list:

PE Model Number
HPE SKU
HPE SKU DESCRIPTION
HPE Spare Part SKU
HPE Firmware Fix Date
EK0800JVYPN
846430-B21
HPE 800GB 12G SAS WI-1 SFF SC SSD
846622-001
3/20/2020
EO1600JVYPP
846432-B21
HPE 1.6TB 12G SAS WI-1 SFF SC SSD
846623-001
3/20/2020
MK0800JVYPQ
846434-B21
HPE 800GB 12G SAS MU-1 SFF SC SSD
846624-001
3/20/2020
MO1600JVYPR
846436-B21
HPE 1.6TB 12G SAS MU-1 SFF SC SSD
846625-001
3/20/2020

HPE says that the drives need to be updated with firmware HPD7 or later. You can see the drive in our cover image is HPD6 which is still vulnerable.

The hardest part about these failures is that the death timer starts at the time of power on. As a result, if you have an array of these drives that go from 39,999 hours to 40,000 hours, you will see the entire array fail at the same time. Even if you mixed EO16000JVYPP and MO1600JVYPR drives in an array, and they were all powered on and ran together, this could again lead to catastrophic failure.

HPE says that “this issue is not unique to HPE and potentially affects other manufacturers and customers that purchased these drives.” They also say that “HPE was notified by a Solid State Drive (SSD) manufacturer of a firmware defect affecting certain SAS SSD models.” (Source: HPE) As a result, this could, in theory, impact the same SSD models sold under different brands. While HPE is being forthcoming and has firmware updates and mitigations 6-7 months beforehand, but the fact we have not heard from other vendors is concerning.

HPE says that given the shipping dates of the drives, no drives should fail until October 2020 at the earliest so there is time to patch. HPE also has the HPD7 firmware and tools to detect and update drives in-place. It says most updates should be able to happen without requiring a reboot and will require minimal I/O. It certainly sounds like updating is a lot less scary than seeing an entire array fail.

HPE SSDs Failing at 32,768 Hours

Just to keep a complete list going, here are the drives that were impacted by the 32,768-hour firmware bug:

HPE Model Number
HPE SKU
HPE SKU DESCRIPTION
HPE Spare Part SKU
HPE Firmware Fix Date
VO0480JFDGT
816562-B21
HP 480GB 12Gb SAS 2.5 RI PLP SC SSD
817047-001
11/22/2019
VO0960JFDGU
816568-B21
HP 960GB 12Gb SAS 2.5 RI PLP SC SSD
817049-001
11/22/2019
VO1920JFDGV
816572-B21
HP 1.92TB 12Gb SAS 2.5 RI PLP SC SSD
817051-001
11/22/2019
VO3840JFDHA
816576-B21
HP 3.84TB 12Gb SAS 2.5 RI PLP SC SSD
817053-001
11/22/2019
MO0400JFFCF
822555-B21
HP 400GB 12Gb SAS 2.5 MU PLP SC SSD S2
822784-001
11/22/2019
MO0800JFFCH
822559-B21
HP 800GB 12Gb SAS 2.5 MU PLP SC SSD S2
822786-001
11/22/2019
MO1600JFFCK
822563-B21
HP 1.6TB 12Gb SAS 2.5 MU PLP SC SSD S2
822788-001
11/22/2019
MO3200JFFCL
822567-B21
HP 3.2TB 12Gb SAS 2.5 MU PLP SC SSD S2
822790-001
11/22/2019
VO000480JWDAR
875311-B21
HPE 480GB SAS SFF RI SC DS SSD
875681-001
12/9/2019
VO000960JWDAT
875313-B21
HPE 960GB SAS SFF RI SC DS SSD
875682-001
12/9/2019
VO001920JWDAU
875326-B21
HPE1.92TB SAS RI SFF SC DS SSD
875684-001
12/9/2019
VO003840JWDAV
875330-B21
HPE 3.84TB SAS RI SFF SC DS SSD
875686-001
12/9/2019
VO007680JWCNK
870144-B21
HPE 7.68TB SAS 12G RI SFF SC DS SSD
870460-001
12/9/2019
VO015300JWCNL
870148-B21
HPE 15.3TB SAS 12G RI SFF SC DS SSD
870462-001
12/9/2019
VK000960JWSSQ
P06584-B21
HPE 960GB SAS RI SFF SC DS SSD
P08608-001
12/9/2019
VK001920JWSSR
P06586-B21
HPE 1.92TB SAS RI SFF SC DS SSD
P08609-001
12/9/2019
VK003840JWSST
P06588-B21
HPE 3.84TB SAS RI SFF SC DS SSD
P08610-001
12/9/2019
VK003840JWSST
P11329-B21
HPE 3.84TB SAS RI LFF SCC DS SPL SSD
P11360-001
12/9/2019
VK007680JWSSU
P06590-B21
HPE 7.68TB SAS RI SFF SC DS SSD
P08611-001
12/9/2019
VO015300JWSSV
P06592-B21
HPE 15.3TB SAS RI SFF SC DS SSD
P08612-001
12/9/2019

Although these types of firmware issues are very dangerous, we have to give credit to HPE for proactively getting the word and fixes out to minimize impacts.

4 COMMENTS

  1. Dell had the exact same issues. Those are Sandisk SSD’s.
    Are these Sandisk too?

    We got affected after missing a few servers we thought had several weeks left. Three of them died on a Friday :-(

  2. So, what’s the next expiration date for those drives? Also I’m wondering whether or not one could reset the counter by just successively pulling the drives? Once is never, twice is always. I wouldn’t trust those disks going forward.

  3. I love to use good quality desktop ssd parts on servers to avoid this scenarios. I can’t remember the amount of times a “certified” part has given me more headaches than regular decent joe sixpack piece of hardware. For this case, not a single Intel SSD 400 series on small production servers has failed me, and so with sandforce2 based sata ssd drives (already phased out most of them).

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.