ned Productions Consulting

Technology musings by Niall Douglas
ned Productions Consulting
(an expert advice and services company based in Ireland)

Monday 7th September 2015 2.50pm

Link shared:

Came back from a short holiday in Dingle on the Irish west coast to find yet another failed hard drive, this being the third failed drive this year! Thankfully, still no data loss thanks to ZFS, but losing three hard drives in a single year is definitely a record :(

The failing drive this time is the second Western Digital Red 3Tb WD30EFRX to go this year, and the cause is the same:

  5 Reallocated_Sector_Ct   0x0033   189   189   140    Pre-fail  Always       -       323
  7 Seek_Error_Rate         0x002e   197   197   000    Old_age   Always       -       486
  9 Power_On_Hours          0x0032   077   077   000    Old_age   Always       -       17436

At 17k hours powered on, this is the oldest drive in the array after the previous WD Red failed (which at the time was the oldest WD Red drive). I'm getting the sense that these drives tend to quickly die with age, so I checked and BackBlaze finds an average 7.93% failure rate per three months with little change across quarters (i.e. it's constant). Make that annual, and you will lose about one third of your WD Red 3Tb drives per year.

I have six WD Red 3Tb drives with these powered on hours in order of array position: 6958, 12718; 17437 (failed), 17423; 3745, 3549. That makes their average age 10305, or 1.175 years. That suggests statistically I should have lost 2.35 drives on average, so the fact I have lost two is about right.

WD replace the drives under warranty for three years, after that you are on your own. So in this sense the rate of failure has cost me time and inconvenience and stress, but not much money … yet. Nevertheless, this rate of failure and the BackBlaze results suggests handing over more of my money to Western Digital for crappy unreliable drives makes no sense despite them being called "NAS ready" drives. In other words, it's time to change my standard drive choice.

Back when I started the ZFS pool three years ago, there was very little empirical information to work with regarding choice of drives, though I knew Seagate drives at the time were appallingly unreliable thanks to very high shop returns from infant mortality (BackBlaze has since found you will lose three out of four of that generation of Seagate drives per year!). Mixing in different models might seem a way of spreading your bets, but ZFS gets upset if the drives in a mirror aren't exactly the same size if you do a swap, so per mirror pair your life is easiest if they are identical which makes spreading your bets difficult as losing a mirror pair means losing everything. One does of course never buy the mirror pair at the same time lest you get two of a bad batch, so I always bought with a three month gap. And due to Seagate drives at the time being known to be awful, I chose the WD Red 3Tb as it was (and still is) the largest three platter drive available for a reasonable price - the NAS anti-vibration sensors etc were a nice value add, but to be honest ZFS makes even the cheapest desktop drives no worse than fancy and expensive server grade drives. The reason for choosing three platter designs was under the assumption of fewer moving parts probably means more reliability … usually.

The BackBlaze statistics weren't available then, but they are now and there is good year and a half worth of data in there. So now I know that the 3Tb Hitachi Deskstar 5K3000 - launched same time as the WD Red 3Tb - is nearly ten times more reliable. Unfortunately, Hitachi's hard drive division was purchased by WD and they instantly cancelled that model which is no longer available, so the next best low idle power drive not crazy expensive model known to BackBlaze is the Seagate Desktop HDD.15 ST4000DM000 4TB with a 2.87% average rate of failure per quarter, or a one in ten annual rate of failure which is about three times better than the WD Red 3Tb.

That Seagate is still three times more likely to fail than the HGST Megascale 4000.B drive, but it costs at least €280 versus €140 for the Seagate. Which pretty much seals the deal: the ST4000DM000 4Tb drive is going to be my new standard ZFS array drive model choice from now on. Let's just hope I can swap that in before the other WD Red drive in that mirror pair fails!