CPU speed & cores, RAM, GPU (VRAM & cores) and PCI-E speed are all recognised as potential bottlenecks within a 4K pipeline, but, regardless of resolution, storage speed (drive I/O) is one aspect of the NLE ecosystem that is often overlooked as a limiting factor on performance when editing and grading. The main goal of your drive setup is to meet the sustained transfer rates required by bandwidth hungry multiple HD and 4K video streams. 4K+ formats, lightly compressed mezzanine codecs and data-rich DNG all require increasing amounts of storage space, so consider speed, capacity and reliability when configuring your NLE Workstation.
| Storage Type | |||
|---|---|---|---|
| Solid State Drives | Hard Disk Drive | ||
| Pros | Fastest storage available, silent | Big capacity, low cost per GB | |
| Cons | Highest cost per GB | Slow, audible | |
Storage Capacity
The data capacity of a storage device are generally measured in gigabytes (GB) or terabytes (TB), but familiarize yourself with the terms petabytes (PB) exabytes (EB) and zettabytes (ZB), because it won’t be too long before your NLE system will require such exotic storage requirements.
Storage Speed
A drive’s speed relates to a specific amount of data transferred over a unit of time (MB/s). The speed of a drive is stated as two figures; its capacity to read and its capacity to write data. Read speeds for some drives are nearly 2x faster than write, so choose carefully. Beyond the theoretical speed of the physical drive, the type of host connection can also be a limiting factor. Speed and connection are important factors when considering a drive set up for video, but regardless of the stated read and write speed, if you ask a drive to perform both operations concurrently, we meet our first problem - drive latency. In simple terms think of this as weight of traffic. Latency becomes a noticeable issue when you have a one-way road suddenly becoming a two way thoroughfare, and limited connection speed similar to moving from four lanes down to two; with heavy traffic everything slows down. To overcome this problem NLE workstations benefit from the fastest possible connection protocol and multiple drive setups with volumes tasked with specific functions such as the OS drive, Media Cache, Project drive and Export drives. Everyone gets their own private multi-lane toll road and the only speed limit (until we try to send more traffic along this route) is the power of the engine. Happy days! That’s the concept. Let’s look at the practice and take a closer look at the two main storage options currently available, HDD and SSD.
Hard Disk Drives
Hard disks drives (HDD for short) have been around since the 1950s although back then they could only store 3.5MB, and now can store up to 6TB - more than 17,000 times the capacity. This makes them the cheapest form of storage. Hard drives have got a lot faster too, transferring at around 150MB/sec, although this is till a fraction of the speed of an SSD, making them best used for bulk storage such as old projects. As mechanical devices, hard drives make a bit of noise, so are best avoided for a system located in a recording studio. If you need lots of storage, why not consider a NAS box, you can locate this small external box in another room, so you don't have to listen to the hard drives clattering away.
Bigger is Better
Large capacity HDDs are therefore the rule when it comes to video storage. As mechanical devices with moving parts, spinning hard drives are also more prone to failure making them best suited for large arrays that incorporate redundancy. The spin is measured, like vinyl LPs, in RPM (revolutions per minute) and only 7200rpm disks or faster should be considered for video editing. Using a dedicated RAID controller and multiple HDDs it’s possible to achieve much higher speeds so RAID arrays are still widely used and trusted in video, broadcast and film post production. Redundancy and fill degredation increases the overall cost per GB, but as of 2016, HDDs in RAID still represent a clear economic benefit for users who need multiple TBs of fast, protected storage with minimum down time.
Solid State Drives
Solid state drives (SSD), using Flash memory, are the newest form of storage device, recording data on chips rather than a spinning disk platter. This gives them a significant edge when it comes to performance, although compared to traditional hard drives you do pay a higher cost per GB. Being solid state devices with no moving parts, SSDs are immune to mechanical failure, consume less power than traditional hard drives and are lighter, tougher and silent, making them the ideal choice for a workstation PC or laptop. For these same reasons modern cameras use solid state flash media to record video and audio.
SSDs are available in two types, SATA and PCI-E. SATA drives connect to the motherboard using a SATA cable, which limits their performance to a maximum of 600MB/sec, although this is still much faster than a HDD. PCI-E drives are faster still, with speeds over 2,000MB/sec possible from some drives, and are available in three main types, M.2, U.2 and AIC. Tiny M.2 drives connect directly to a slot on the motherboard but due to their small size are currently limited to a maximum capacity of 512GB. However they are an excellent choice for OS and Media Cache Drives. U.2 drives look similar to SATA drives, using a cable to connect to the motherboard while AIC (Add-In-Cards) need to be fitted to a 4x or faster PCI-E 3.0 slot on the motherboard. For the ultimate Project Drive, choose a PCI-E SSD that supports NVME 'Non-Volatile Memory Express', a new performance boosting instruction set specifically written for SSDs to overcome the limitations of the older AHCI instruction set.
External Storage
So far we’ve described storage connections based on internal drive set-ups. Editing via external drives is a good choice for those who need to work in a mobile environment or use shared storage, but with the same caveat that the overall drive performance and connection speed will dictate your workflow. Once you start connecting to external drives, you need to consider speed of the connection you’re using, with Fibre Channel being the fastest, followed by Thunderbolt, USB3 and eSATA.
Thunderbolt, developed by Intel promises much needed performance boost for external video storage and expansion solutions. The next gen Thunderbolt 3 is an ideal connection for SSDs that can take full advantage of the 40Gb/sec bandwidth. Support is slowly building on the PC workstation platform with a limited range of mother boards available (an AIC is required so the number of available PCI-E lanes becomes an important factor). Laptops are ahead of the curve in this respect, offering direct plug-and-play connectivity and the ability to daisy chain multiple devices.
For external drives used in a non concurrent fashion such as shuttle drives (portable HDD or SSD in external chassis) as used in the field or on-set to backup and transport camera footage from location to editorial, disk I/O latency is not a problem. When making multiple copies the benefit of a fast connection are obvious. For mobile users there are rugged drives with protective bumpers marketed to the pro video and photo user, which offer some added protection, but never entrust your camera originals to one single drive.
For back up and deep archive, DLT (Digital Linear Tape) is still the weapon of choice for hardware based storage, but cloud storage has shown rapid growth over the last few years to become a trusted solution for major post production facilities. As it becomes more accessible and affordable, many smaller production houses and indie producers are adopting the coud as part of their 3-2-1 back up procedure, using it as both a second format and the off-site destination (See #LifeHacks below).
Optimum NLE Workstation Drive Configuration
The optimum storage configuration is the one that works for your workflow, but as a general guide we recommend the use of four volumes using individual SSDs in the following disk set up for video editing:
| Optimum Storage Configuration | ||
|---|---|---|
| OS Drive | SATA SSD 120GB Min | Windows Boot drive + Editing Software |
| Media Cache | M.2 SSD 512GB Max | Cache and Preview |
| Project Drive | PCI-E SSD 800GB+ | Media Assets & Project Files |
| Export and Backup | HDD 4TB+ | |
This is the standard configuration as used in our 3XS Evolve NLE Workstation. If required, the addition of a RAID controller and multiple drives could be employed to scale up the performance of the volumes.
Adobe Premier Pro Workflows
Different applications and workflows put different stress on the system I/O. Trace tool analysis has shown that applications such as Adobe Premier Pro conduct most data transfers sequentially at a queue depth of between two and eight in 128KB transfers.
Using the powerful Iometer benchmarking tool we've tested the main types of storage drive at these settings, with two variations. The first graphs shows the drives editing, during which the drive is mostly tasked with reading, while the second graph shows the drives exporting during which the drive is mostly tasked with writing.
Editing in Premier Pro
HDD
SATA SSD
PCI-E SSD
Exporting from Premier Pro
HDD
SATA SSD
PCI-E SSD
This powerful benchmark is able to reproduce the two main types of drive activity that occur when editing and for video work, choosing the fastest storage device available makes perfect sense. Choosing an SSD for the OS drive is the ultimate upgrade: not only will it make Windows boot faster, but your applications load quicker and your system should feel zippy!
As part of a well-balanced system, multiple HDDs in RAID or a PCI-E SSDs dedicated to Media Cache and Project Drives offer the highest I/O performance and will allow your system to access more video streams, realtime playback and resolutions beyond HD.
Pro Video #LifeHacks
As an extension of the 2nd Law of Thermodynamics, Sod’s Law dictates that all hardware will fail. With HDD it’s sensible to remember the mantra ‘use a disk, take a risk’, but SSDs are not immune. Make sure you observe industry good practice regarding data backup and volume redundancy. Back up, back up, back up. The Rule of Three. Three check sum copies of the project and all media, stored on Two different formats, with One copy stored off-site.
Huge frames or big data codecs need to be pushed through the gate at the same speed if we want to achieve real-time fps playback. Large file = faster disk I/O throughput.