Original Link: https://www.anandtech.com/show/8985/synology-ds2015xs-review-an-armbased-10g-nas



Introduction and Testbed Setup

Synology is one of the most popular COTS (commercial off-the-shelf) NAS vendors in the SMB / SOHO market segment. The NAS models introduced by them in 2014 were mostly based on Intel Rangeley (the Atom-based SoCs targeting the storage and communication market). However, in December, they sprang a surprise by launching the DS2015xs, an ARM-based model with dual 10GbE ports. We covered the launch of the Synology DS2015xs in December, and provided some details about the Annapurna Labs AL514 SoC in it.

ARM-based SoCs for SMB / SOHO NAS units typically support up to 4 bays and come with dual GbE links. Intel's offerings have had a virtual monopoly in the other tiers of the market. Synology's DS2015xs, with its native 10G capabilities, brings in another contender into the market.

The DS2015xs is 8-bay NAS unit presented as a step-up from the DS1815+. While the DS1815+ can expand up to a total of 18 bays with two DX513 expansion chassis, the DS2015xs is compatible with the 12-bay DX1215 expander (for a total of 20 bays). The main step-up from the DS1815+ is the presence of two built-in 10G SFP+ links (supporting direct-attach copper cables). The gallery below takes us around the unit's chassis.

The specifications of the Synology DS2015xs are provided in the table below

Synology DS2015xs Specifications
Processor Annapurna Labs AL514 SoC (Quad-Core Cortex-A15 @ 1.7 GHz)
RAM 4 GB
Drive Bays 8x 3.5"/2.5" SATA II / III HDD / SSD (Hot-Swappable)
Network Links 2x 1 GbE RJ-45 + 2x 10GbE SFP+
External I/O Peripherals 2x USB 3.0, 1x Infiniband for Expansion Bay
Expansion Slots N/A
VGA / Display Out N/A
Full Specifications Link Synology DS2015xs Specifications
Price USD 1400

The Synology DS2015xs runs the latest DiskStation Manager OS, which, subjectively speaking, is one of the best COTS NAS operating systems in the market. Geared towards both novice and power users, it also provides SSH access. Some additional aspects can be gleaned through SSH. For example, the unit runs on Linux kernel version 3.2.40. The AL514 SoC has hardware acceleration for cryptography and two in-built USB 3.0 ports. There are also four network links (we know from external inspection that two are 10GbE, while the others are 1GbE) with unified drivers for both types of interfaces.

In the rest of the review, we will first take a look at the performance of the unit as a direct-attached storage device. This is followed by benchmark numbers for both single and multi-client scenarios across a number of different client platforms as well as access protocols. We have a separate section devoted to the performance of the NAS with encrypted shared folders. Prior to all that, we will take a look at our testbed setup and testing methodology.

Testbed Setup and Testing Methodology

The Synology DS2015xs can take up to 8 drives. Users can opt for different RAID types depnding on their requirements. We expect typical usage to be with multiple volumes in a RAID-5 or RAID-6 disk group. However, to keep things consistent across different NAS units, we benchmarked a SHR volume with single disk redundancy (RAID-5). Tower / desktop form factor NAS units are usually tested with Western Digital RE drives (WD4000FYYZ). However, the presence of 10-GbE on the DS2015xs meant that SSDs had to be used to bring out the maximum possible performance. Therefore, evaluation of the unit was done by setting up a RAID-5 volume with eight OCZ Vector 4 120 GB SSDs. Our testbed configuration is outlined below.

AnandTech NAS Testbed Configuration
Motherboard Asus Z9PE-D8 WS Dual LGA2011 SSI-EEB
CPU 2 x Intel Xeon E5-2630L
Coolers 2 x Dynatron R17
Memory G.Skill RipjawsZ F3-12800CL10Q2-64GBZL (8x8GB) CAS 10-10-10-30
OS Drive OCZ Technology Vertex 4 128GB
Secondary Drive OCZ Technology Vertex 4 128GB
Tertiary Drive OCZ Z-Drive R4 CM88 (1.6TB PCIe SSD)
Other Drives 12 x OCZ Technology Vertex 4 64GB (Offline in the Host OS)
Network Cards 6 x Intel ESA I-340 Quad-GbE Port Network Adapter
Chassis SilverStoneTek Raven RV03
PSU SilverStoneTek Strider Plus Gold Evolution 850W
OS Windows Server 2008 R2
Network Switch Netgear ProSafe GSM7352S-200

The above testbed runs 25 Windows 7 VMs simultaneously, each with a dedicated 1 Gbps network interface. This simulates a real-life workload of up to 25 clients for the NAS being evaluated. All the VMs connect to the network switch to which the NAS is also connected (with link aggregation, as applicable). The VMs generate the NAS traffic for performance evaluation.

Thank You!

We thank the following companies for helping us out with our NAS testbed:

DAS Evaluation Setup and Methodology

In addition to our standard NAS evaluation suite, the Synology DS2015xs also warrants investigation under ideal network conditions as a direct-attached storage unit. The presence of 10G network links in the unit has prompted Synology to market it as a fast DAS unit for video production workflows. In order to evaluate this aspect, we augmented our DAS testbed with an Emulex OneConnect OCe11102(R)-N 2-port 10GbE SFP+ PCIe NIC and a OCZ Vector 120 GB SSD.

The Emulex PCIe NIC doesn't support teaming under Windows 8.1. Therefore, we had to install Windows Server 2012 R2 on the additional SSD to make our DAS testbed dual-boot for evaluating NAS units. The DHCP Server feature was also activated on the teamed port to which the NAS's 10G ports were connected. On the NAS side, the ports were set up for teaming too and configured to receive an IP address from a DHCP server. The MTU for the interface was configured to be 9000 bytes. The details of the tests that were run in this mode will be presented along with the performance numbers in the next section.



Direct-Attached Storage Performance

The presence of 10G ports on the Synology DS2015xs presents some interesting use-cases. As an example, video production houses have a need for high-speed storage. Usually, direct-attached storage units suffice. Thunderbolt is popular for this purpose - for both single-user modes as well as part of a SAN. However, as 10G network interfaces become common and affordable, there is scope for NAS units to act as a direct-attached storage units also. In order to evaluate the DAS performance of the Synology DS2015xs, we utilized the DAS testbed augmented with an appropriate CNA (converged network adapter), as described in the previous section. To get an idea of the available performance for different workloads, we ran a couple of quick artificial benchmarks along with a subset of our DAS test suite.

CIFS

In the first case, we evaluate the performance of a CIFS share created in a RAID 5 volume. One of the aspects to note is that the direct link between the NAS and the testbed is configured with a MTU of 9000 (compared to the default of 1500 used for the NAS benchmarks).

Synology DS2015xs - CrystalDiskMark Benchmark

Our first artificial benchmark is CrystalDiskMark, which tackles sequential accesses as well as 512 KB and 4KB random accesses. For 4K accesses, we have a repetition of the benchmark at a queue depth of 32. As the screenshot above shows, Synology DS2015xs manages around 675 MBps reads and 703 MBps writes. The write benchmark number corresponds correctly to the claims made by Synology in their marketing material, but the 675 MBps read speeds are a far cry from the promised 1930 MBps. We moved on to ATTO, another artificial benchmark, to check if the results were any different.

Synology DS2015xs - ATTO Benchmark

ATTO Disk Benchmark tackles sequential accesses with different block sizes. We configured a queue depth of 10 and a master file size of 4 GB for accesses with block sizes ranging from 512 bytes to 8 MB and the results are presented above. In this benchmark, we do see 1 MB block sizes giving read speeds of around 1214 MBps.

Synology DS2015xs - 2x10 Gbps LACP - RAID-5 CIFS DAS Performance (MBps)
  Read Write
Photos 594.69 363.47
Videos 915.95 500.09
Blu-ray Folder 949.32 543.93

For real-world performance evaluation, we wrote and read back multi-gigabyte folders of photos, videos and Blu-ray files. The results are presented in the table below. These numbers show that it is possible to achieve speeds close to 1 GBps for real-life workloads. The advantage of a unit like the DS2015xs is that the 10G interfaces can be used as a DAS interface, while the other two 1G ports can connect the unit to the rest of the network for sharing the contents seamlessly with other computers.

iSCSI

We configured a block-level (Single LUN on RAID) iSCSI LUN in RAID-5 using all available disks. Network settings were retained from the previous evaluation environment. The same benchmarks were repeated in this configuration also.

Synology DS2015xs - CrystalDiskMark Benchmark

Synology DS2015xs - ATTO Benchmark

The iSCSI performance seems to be a bit off compared to what we got with CIFS. Results from the real-world performance evaluation suite are presented in the table below. These numbers track what we observed in the artificial benchmarks too.

Synology DS2015xs - 2x10 Gbps LACP - RAID-5 iSCSI DAS Performance (MBps)
  Read Write
Photos 535.28 532.54
Videos 770.41 483.97
Blu-ray Folder 734.51 505.3

Performance Analysis

The performance numbers that we obtained with teamed ports (20 Gbps) were frankly underwhelming. The more worrisome aspect was that we couldn't replicate Synology's claims of upwards of 1900 MBps throughput for reads. In order to determine if there were any issues with our particular setup, we wanted to isolate the problem to either the disk subsystem on the NAS side or the network configuration. Unfortunately, Synology doesn't provide any tools to evaluate them separately. For optimal functioning, 10G links require careful configuration on either side.

iPerf is the tool of choice for many when it comes to ensuring that the network segment is operating optimally. Unfortunately, iPerf for DSM requires an optware package that is not yet available for the Alpine platform. On the positive side, Synology had uploaded the tool chain for Alpine on SourceForge - this helped us to cross-compile iPerf from source for the DS2015xs. Armed with iPerf on both the NAS and the testbed side, we proceeded to evaluate the links operating simultaneously without the teaming overhead.

The screenshot above shows that the two links together saturated at around 5 Gbps (out of the theoretically possible 20 Gbps), but the culprit was our cross-compiled iPerf executable (with each instance completely saturating one core - 25% of the CPU).

In the CIFS case, the smbd process is not multi-threaded, and this severely affects the utilization of the 10G links fully.

In the iSCSI case, the iscsi_trx process also seems to saturate one CPU core, leading to similar results for 10G link utilization.

On the whole, the 10G links are getting utilized, but not to the full possible extent. The utilization is definitely more than, say, four single GbE links teamed together, but the presence of two 10G links had us expecting more from the unit as a DAS.



Single Client Performance - CIFS & iSCSI on Windows

The single client CIFS and iSCSI performance of the Synology DS2015xs was evaluated on the Windows platforms using Intel NASPT and our standard robocopy benchmark. This was run from one of the virtual machines in our NAS testbed. All data for the robocopy benchmark on the client side was put in a RAM disk (created using OSFMount) to ensure that the client's storage system shortcomings wouldn't affect the benchmark results. It must be noted that all the shares / iSCSI LUNs are created in a RAID-5 volume. One important aspect to note here is that only the Synology DS2015xs uses SSDs. The other NAS units in each of the graphs below were benchmarked with hard drives.

HD Video Playback - CIFS

2x HD Playback - CIFS

4x HD Playback - CIFS

HD Video Record - CIFS

HD Playback and Record - CIFS

Content Creation - CIFS

Office Productivity - CIFS

File Copy to NAS - CIFS

File Copy from NAS - CIFS

Dir Copy to NAS - CIFS

Dir Copy from NAS - CIFS

Photo Album - CIFS

robocopy (Write to NAS) - CIFS

robocopy (Read from NAS) - CIFS

We created a 250 GB iSCSI LUN / target and mapped it on to a Windows VM in our testbed. The same NASPT benchmarks were run and the results are presented below. The observations we had in the CIFS subsection above hold true here too.

HD Video Playback - iSCSI

2x HD Playback - iSCSI

4x HD Playback - iSCSI

HD Video Record - iSCSI

HD Playback and Record - iSCSI

Content Creation - iSCSI

Office Productivity - iSCSI

File Copy to NAS - iSCSI

File Copy from NAS - iSCSI

Dir Copy to NAS - iSCSI

Dir Copy from NAS - iSCSI

Photo Album - iSCSI

robocopy (Write to NAS) - iSCSI

robocopy (Read from NAS) - iSCSI

The DS2015xs loses out to the Asustor AS7008T and sometimes, even the DS1815+, when it comes to certain benchmarks. This shows that when it comes to access from a single GbE-equipped client, x86-based units can perform better despite being handicapped by hard drives.



Single Client Performance - CIFS and NFS on Linux

A CentOS 6.2 virtual machine was used to evaluate NFS and CIFS performance of the NAS when accessed from a Linux client. We chose IOZone as the benchmark for this case. In order to standardize the testing across multiple NAS units, we mount the CIFS and NFS shares during startup with the following /etc/fstab entries.

//<NAS_IP>/PATH_TO_SMB_SHARE /PATH_TO_LOCAL_MOUNT_FOLDER cifs rw,username=guest,password= 0 0

<NAS_IP>:/PATH_TO_NFS_SHARE /PATH_TO_LOCAL_MOUNT_FOLDER nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2, sec=sys,mountaddr <NAS_IP>,mountvers=3,mountproto=udp,local_lock=none,addr=<NAS_IP> 0 0

The following IOZone command was used to benchmark the CIFS share:

IOZone -aczR -g 2097152 -U /PATH_TO_LOCAL_CIFS_MOUNT -f /PATH_TO_LOCAL_CIFS_MOUNT/testfile -b <NAS_NAME>_CIFS_EXCEL_BIN.xls > <NAS_NAME>_CIFS_CSV.csv

IOZone provides benchmark numbers for a multitude of access scenarios with varying file sizes and record lengths. Some of these are very susceptible to caching effects on the client side. This is evident in some of the graphs in the gallery below.

Readers interested in the hard numbers can refer to the CSV program output here.

The NFS share was also benchmarked in a similar manner with the following command:

IOZone -aczR -g 2097152 -U /nfs_test_mount/ -f /nfs_test_mount/testfile -b <NAS_NAME>_NFS_EXCEL_BIN.xls > <NAS_NAME>_NFS_CSV.csv

The IOZone CSV output can be found here for those interested in the exact numbers.

A summary of the bandwidth numbers for various tests averaged across all file and record sizes is provided in the table below. As noted previously, some of these numbers are skewed by caching effects. A reference to the actual CSV outputs linked above make the entries affected by this effect obvious.

Synology DS2015xs - Linux Client Performance (MBps)
IOZone Test CIFS NFS
Init Write 86 82
Re-Write 85 82
Read 50 120
Re-Read 50 122
Random Read 33 70
Random Write 80 82
Backward Read 32 58
Record Re-Write 56 1719*
Stride Read 45 117
File Write 85 83
File Re-Write 85 82
File Read 35 95
File Re-Read 36 97
*: Benchmark number skewed due to caching effect


Multi-Client Performance - CIFS on Windows

We put the Synology DS2015xs through some IOMeter tests with a CIFS share being accessed from up to 25 VMs simultaneously. The following four graphs show the total available bandwidth and the average response time while being subject to different types of workloads through IOMeter. The tool also reports various other metrics of interest such as maximum response time, read and write IOPS, separate read and write bandwidth figures etc. Detailed listings of the IOMeter benchmark numbers (including IOPS and maximum response times) for each configuration are linked below:

Synology DS2015xs - 2x 10G Multi-Client CIFS Performance - 100% Sequential Reads

 

Synology DS2015xs - 2x 10G Multi-Client CIFS Performance - Max Throughput - 50% Reads

 

Synology DS2015xs - 2x 10G Multi-Client CIFS Performance - Random 8K - 70% Reads

 

Synology DS2015xs - 2x 10G Multi-Client CIFS Performance - Real Life - 65% Reads

We see that the sequential accesses get saturated around 700 MBps, similar to what we found in our evaluation of the unit as a DAS. In the Random 8K 70% Reads case, we see a sudden drop after more than five clients come into the mix - we believe it has to do with the smbd processes saturating the CPU cores completely. On the positive side, we find that the bandwidth numbers and response times are excellent across the board, better than all the other NAS units that we are comparing against.



Multi-Client iSCSI Evaluation

As virtualization becomes more and more popular even in home / power user settings, the importance of the iSCSI feature set of any COTS NAS can't be overstated. Starting with our ioSafe 1513+ review, we have started devoting a separate section (in the reviews of NAS units targeting SMBs and SMEs) to the evaluation of iSCSI performance. Since we have already looked at the way iSCSI LUNs are implemented in DSM in the ioSafe 1513+ review, it won't be discussed in detail.

We evaluated the performance of the Synology DS2015xs with file-based LUNs as well as configuring a RAID-5 disk group with single as well as multiple LUNs. The standard IOMeter benchmarks that we used for multi-client CIFS evaluation were utilized for iSCSI evaluation also. The main difference to note is that the CIFS evaluation was performed on a mounted network share, while the iSCSI evaluation was done on a 'clean physical disk' (from the viewpoint of the virtual machine).

Performance Numbers

The four IOMeter traces were run on the physical disk manifested by mapping the iSCSI target on each VM. The benchmarking started with one VM accessing the NAS. The number of VMs simultaneously playing out the trace was incremented one by one till we had all 25 VMs in the fray. Detailed listings of the IOMeter benchmark numbers (including IOPS and maximum response times) for each configuration are linked below:

Synology DS2015xs - LUNs (Regular Files) - Multi-Client Performance - 100% Sequential Reads

 

Synology DS2015xs - LUNs (Regular Files) - Multi-Client Performance - Max Throughput - 50% Reads

 

Synology DS2015xs - LUNs (Regular Files) - Multi-Client Performance - Random 8K - 70% Reads

 

Synology DS2015xs - LUNs (Regular Files) - Multi-Client Performance - Real Life - 65% Reads

The observations we had in the previous section hold true here also. In general, the performance of the DS2015xs iSCSI volumes is better than the others that are presented as comparison points, even considering the SSD vs. HDD aspect.



Encryption Support Evaluation

Consumers looking for encryption capabilities can opt to encrypt a iSCSI share with TrueCrypt or some in-built encryption mechanism in the client OS. However, if requirements dictate that the data must be shared across multiple users / computers, relying on encryption in the NAS is the best way to move forward. Most NAS vendors use the industry-standard 256-bit AES encryption algorithm. One approach is to encrypt only a particular shared folder while the other approach is to encrypt the full volume. Synology supports only folder-level encryption for now in DSM.

On the hardware side, encryption support can be in the form of specialized hardware blocks in the SoC (common in ARM / PowerPC based NAS units). In x86-based systems, accelerated encryption support is dependent on whether the AES-NI instruction is available on the host CPU. On the Annapurna Labs Alpine platform, we have a dedicated crypto block.

HD Video Playback - Encrypted CIFS

2x HD Playback - Encrypted CIFS

4x HD Playback - Encrypted CIFS

HD Video Record - Encrypted CIFS

HD Playback and Record - Encrypted CIFS

Content Creation - Encrypted CIFS

Office Productivity - Encrypted CIFS

File Copy to NAS - Encrypted CIFS

File Copy from NAS - Encrypted CIFS

Dir Copy to NAS - Encrypted CIFS

Dir Copy from NAS - Encrypted CIFS

Photo Album - Encrypted CIFS

robocopy (Write to NAS) - Encrypted CIFS

robocopy (Read from NAS) - Encrypted CIFS

The graphs above show that there is a performance impact only for writes to encrypted volumes. The impact is not as bad as that of software encryption, but the slowdown is definitely noticeable. This indicates an imbalance between the cryptography engine's encryption rate and the the rate at which data can be written to the storage media.



Miscellaneous Aspects and Final Words

In order to keep testing consistent across all 8-bay units, we performed all our expansion / rebuild testing as well as power consumption evaluation with the unit configured in RAID-5. The disks used for benchmarking (OCZ Vector 4 SSDs) were also used in this section. The table below presents the average power consumption of the unit as well as time taken for various RAID-related activities.

Synology DS2015xs RAID Expansion and Rebuild / Power Consumption
Activity Duration (HH:MM:SS) Avg. Power (W)
Single Disk Init 00:01:00 18.64 W
JBOD to RAID-1 Migration 00:19:14 20.91 W
RAID-1 (2D) to RAID-5 (3D) Migration 00:25:07 23.95 W
RAID-5 (3D) to RAID-5 (4D) Expansion 00:23:48 26.92 W
RAID-5 (4D) to RAID-5 (5D) Expansion 00:24:19 29.86 W
RAID-5 (5D) to RAID-5 (6D) Expansion 00:24:20 32.82 W
RAID-5 (6D) to RAID-5 (7D) Expansion 00:22:49 35.8 W
RAID-5 (7D) to RAID-5 (8D) Expansion 00:20:03 38.79 W
RAID-5 (8D) Rebuild 00:13:12 32.77 W

We don't have any comparison graphs for RAID rebuild duration / power consumption since other 8-bay units were subject to RAID operations with 4 TB hard drives. The usage of 120 GB SSDs prevents us from making any concrete conclusions about the efficiency of Synology's expansion, migration and rebuild operations.

Concluding Remarks

Coming to the business end of the review, it is clear that the presence of two 10G ports enables the DS2015xs to achieve high transfer rates. Compared to the last 10G desktop NAS we evaluated (based on an Intel Xeon platform), the power consumption is much lower At $1400, the DS2015xs is affordably priced, considering the 10G features built into it. DSM's business-oriented features (very important for the DS2015xs's target market) are stable and industry-leading.

The above advantages aside, there are certain areas of concern / scope for improvement: Synology needs to supply a way to test the network performance (provide an optimized iPerf package) isolated from the storage subsystem. The SoC vendor, Annapurna Labs is in the process of being purchased by Amazon. It is not clear what effect that would have on the DS2015xs's successors. It is likely that other ARM server vendors might want to make a play for future products in the lineup.

It is impossible not to conclude this review without a car analogy for the DS2015xs. The presence of dual 10G ports and eight native SATA III ports might delude consumers into thinking that they are purchasing a Ferrari. Unfortunately, consumers only get a souped up Volkswagen engine inside. That said, with tempered expectations, the $1400 Synology DS2015xs can fulfill the role of a cost-effective power-efficient high performance NAS.

Log in

Don't have an account? Sign up now