Vulnerant omnes, ultima necat

If you just thought of maybe setting up a USB Linux RAID (a RAID made out of many external USB drives), you might have looked it up on the Web and saw many posts claiming that is a very bad idea. Since I have maintained and grown (from three to more than twenty disks) such a RAID over the past ten years or so, I can testify that it is indeed a task that can be prone to failure and many problems of all kinds — if you don’t plan ahead and don’t take into account some « quirks ». Also, be certain that, even if « done right », I would never recommend setting up a USB RAID in a any high-availability context, because, well, there *are* quirks out there and new ones are coming up on a regular basis. But for casual needs where maintenance skills are readily available — especially at home –, this can be the perfect solution for a relatively cheap, redundant and growable storage.

Advantages of a USB RAID

The main advantages of a USB RAID versus a pure SATA RAID:

• Much easier to physically maintain. USB is external, so it is much easier to simply plug new drives in than having do open your computer everytime you want to remove or add a disk.
• Much easier to grow. Since there is a maximum amount of SATA ports + power supply on any generic computer, if you want to go beyond that you’ll probably have to rely on expensive external enclosures to add more disks than your system can support, while you can add a great many USB drives without too much of a hassle if you take some precautions (see Mind the number of drives below).
• Much easier to move. My own USB RAID is presently sitting on its fourth computer: to move a USB RAID from a system to another one, you mereley have to unplug it from the old system and plug it in the new system.
• Can be much cheaper. If your budget is limited and you don’t mind risking losing drives more often (after all, that’s what a RAID is for…!), you can buy external USB drives for much less than internal ones (because, apparently, they just slap refurbished or just anything deemed « working » in an enclosure without much care) — see Mind the hardware below for more insight about this.

Disadvantages of a USB RAID

The main disadvantages of a USB RAID versus a pure SATA RAID:

• Less reliable. A USB RAID is sensitive to power fluctuations and all kinds of USB hardware and driver bugs.
• Can be messy. Cables. A lot of cables.
• Takes room. The drives are outside the computer: where do you put them all if you have a lot?

Read this

If not done already, read this opinion about USB RAID on the Linux-raid kernel list community-managed reference for Linux software RAID.

It exposes plenty of good and sound arguments against USB RAID, but I disagree that you should never do it: it’s all about common sense, knowledge, experience and context. Simply backup any sensitive data outside of the RAID to have complete peace of mind.

So below I present « rules » or at least advices as to what is important to consider before setting up a USB Linux RAID.

1. Mind the hardware

Over the years, I’ve tried all sorts of hardware:

• factory-enclosed external 3.5″ USB drives,
• 3.5″ SATA drives that I put myself inside enclosures,
• external USB-powered 2.5″ USB drives coming with their own enclosure,
• internal 2.5″ SATA drives simply plugged in with a USB/SATA cable,
• internal 2.5″ SATA drives inside a disk docking station.

Even though a RAID offers the latitude to dab at cheap drives, it is also good to try and minimize the maintenance over time.

With no surprise, factory-sealed external USB drives are the ones that have the shortest life expectation (in my case, the shortest-lived lasted only three weeks), but that’s highly variable (my oldest array drives are also factory-sealed external USB drives). I also noticed that cheap USB enclosures can be more likely to fail than the disks themselves (I once bought a bunch of cheap enclosures along with 3.5″ SATA HDDs and all the enclosures failed at some point while the HDDs themselves were still alright).

But all in all, a simple good USB/SATA adaptor offers a cheap and reliable way to connect a SATA drive to USB, no enclosure needed. This solution also offers the advantage of bypassing any sleeping function that could be embedded in a factory-enclosed drive.

A bit of caution (non-USB related): read carefully the warning on the Linux-raid kernel list community-managed reference for Linux software RAID as well as their explanation about the problems many recent drives can have with a software RAID.

Of course, SSDs should be much more reliable than HDDs (in addition to being much faster), but obviously substantially more expensive (for now), so it’s up to you to decide if they’re worth it depending on your needs and budget. I currently have three SSDs, the oldest being 3 years old, and none of them has yet given me a single problem. I also found that NAS-optimized disks (like this one) are very reliable and should be preferred for the most hassle-free experience possible.

What about multiple-bay enclosures/docking stations?

They can be a good solution to optimise space management, but make sure the bays are totally independant. For example, I’ve tried one 4-bay SATA to USB docking station… and was a bit disappointed, because even though it was advertised as « hot swappable », in reality it will disconnect all drives already in whenever I add or remove a disk. Even though all the disks automatically reconnect right after the change, it still causes an unwanted disruption in the array (unless you stop the array beforehand, of course). Most cheaper USB multiple bay stations behave the same. Also, this one I tried renames the disks (and changes their ids) in a inconstant manner which makes very dificult to know which disk is which (because otherwise you simply label each disk with its hardware id from lsblk and it becomes a piece of cake to identify them when they fail). But this other product has really independant bays and worked perfectly for me. Such enclosures do provide two very compelling advantages: 1- they allow connecting multiple drives with a single USB cord, and 2- procure both adequate and stable power for all the drives inside.

2. Mind the power

Speaking of power, basically, in the end you’ll have to chose between USB-powered 2.5″ drives or externally powered 3.5″ drives. The 3.5″ drives work on 12V, so they need to be externally powered, while 2.5″ drives work on 5V and can therefore be USB-powered. While needing only one cable to plug a 2.5″ drive is tempting, if you go USB-powered, then you’ll have to deal with the power needs of all your USB-powered drives VS what power is actually available on your system.

For modern drives, something that can work quite well is using a powered USB hub offering at least 5W/port (so, for, say, a 7-port hub, make sure it can offer at least 35W of power). However, a safer bet is probably to go for externally powered 3.5″ drives: that way, each drive has its own power source and that makes for a much more stable system (because even slight variations in power can make a drive drop of the array). Or, as said in the previous section, you can also try a USB multi-bay station.

So all in all a safe bet for your hardware is to go for 3.5″ SATA drives that you connect with good quality SATA/USB adaptor + 12V power supply. Pricewise, buying a 3.5″ SATA drive + the powered USB adaptor is in par with a single factory-enclosed 2.5″ USB drive, but is likely to be more reliable.

3. Mind the heat

Do not underestimate the amount of heat a drive (any drive) can produce: if a drive overheats, it will fail and get kicked out of the array. A simple mistake could be, for example, to stack the drives onto each other. So make sure all drives have enough air around them do dissipate their heat. I found that simply laying the drives side by side in a row (with the help of some racking system) was alright.

4. Mind the number of drives per USB bus/channel (not port!)

I was stunned when I tried to add a new drive to my array and the kernel spat a « max number of USB devices reached » error (or something alike). To my knowledge of the time, the maximum number of USB devices on a single port could go up to at least 127, so what was wrong? Well, –> this <– was wrong. It happens that 127 is merely a theoritical number (and it is for USB 2.0; this number is 7,906 for USB 3.0): in practice however, manufacturers put all sorts of limitations that heavily dimish the number of USB devices you can actually connect to a system, and USB hubs happen to slash even more this number.

You have to be aware that depending on your motherboard, many USB ports will actually be connected to internal hubs so that they are on the same bus, and you have to count the number of devices per bus. In my experience, it’s safe to connect up to 10 devices per bus (but feel free to experiment more with your own system).

A good way to work around this limitation is to add USB extension cards with dedicated channels (such as this one, for example). Cheaper extension cards will merely act like USB hubs and won’t allow you to increase the overall number of devices you can connect, while a card with dedicated channels should allow at least 10 more devices per channel.

5. Mind UASP (UAS)

USB 3.0 brought faster speeds, but there was also something else: UASP (USB Attached SCSI Protocol). It is a protocol used along with some USB 3.0 hardware for speeding things up even more than out-of-the-box USB 3.0; it is meant to replace BOT (Bulk-Only Transport) — you can read here for more info — which can still be used over USB 3.0. In order to work, UASP needs to be supported both by the USB device and the OS. While it sounds great in theory, in practice there are bugs related to UASP coming both from some manufacturers’ USB/SATA interfaces as well as from the Linux UAS driver. The possible bad results are variable: those bugs can actually slow down transfers or, worst, completly freeze any access to any USB drive. Sometimes those problems arise only in specific contexts (such as high-stressed drives, for example). And, yes, it all happened to me.

As of February 2022, the best solution I found to avoid any possible UASP-related problem was to disable UAS for every UAS-enabled drive (or adaptor) plugged to my system. You can follow this guide to help you deal with it.

6. Mind the sleep

Like hinted in another section, some portable USB drive enclosures can come with an embedded sleeping function that will activate when they are idle for a certain time. Probably the best way to prevent this altogether is with a simple cron job that will regularly (e.g. every minute) issue a touch on each drive. I personnally use the following script:

#!/bin/bash

for LETTER in {a..z}
do
        /bin/touch --no-create /dev/sd${LETTER} &>/dev/null
done

7. Mind the ids

Like mentioned in another section, it is of first importance to be able to know which drive is which. Unfortunately, mdadm deals only with the linux-mapped devices (/dev/sdxxx), whose names are utterly unreliable between boots and between connects and disconnects. So the only appropiate way to identify a disk is to refer to some harware info taken from the disk itself. Personnally, I typically use the SERIAL field from lsblk to physically tag my drives, so when mdadm tells me that /dev/sdc (for example) has been kicked out, I can easily find which is it with lsblk. For example, I commonly use this script to list useful info about all my drives:

#/bin/bash
lsblk --nodeps --output name,vendor,model,serial,size | grep sd[a-z]