Review: Seagate 1200 Reviewed by: J.Reynolds Provided by: Seagate Firmware version: 04

 

Welcome to
Myce’s belated review of the Seagate 1200 400GB SAS Enterprise SSD (hereafter
referred to as the Seagate 1200).

Belated,
especially as the Seagate 1200 has recently been succeeded by the 1200.2 (which
we hope to review soon).

There were a
number of reasons for the delay.

Firstly, the sample drive that was sent to Myce came with an
early version of the firmware (01), which I found problems with.  My thanks to
Seagate for sorting this out (especially to Andy Beckwith) and updating the
firmware level to 04.

Secondly, the
problems were with me.

Suffice to say I am now able to bring you the review using a
recent version of the OakGate system, which now includes OakGate’s native 12
Gbit/sec driver.  I am keen to put the review into our catalogue of enterprise
reviews, even if it is late, as the Seagate 1200 is an important drive from a
big player in the enterprise storage market.

Another advantage of performing this review belatedly is that
the Seagate 1200 could be subjected to Myce’s comprehensive power consumption
testing procedures, as announced here,
and the Seagate 1200 gives us our first set of results for an SAS drive.

I will now take
this opportunity to show you a couple of the new features of our OakGate
system:

Firstly, the
upgraded Dashboard capability, which enables one to monitor in flight
performance -

Here we see
the Seagate 1200 performing mixed 4K random Reads/Writes (70% Reads, 30%
Writes).

Secondly, the
Peripheral Control screen, together with a display of the drive’s in flight
power consumption being taken from the integrated Quarch hardware.

 

Market Positioning and
Specification

Market
Positioning

This is how Seagate
positions the Seagate 1200 series of drives –

 

Specification

Here is Seagate’s
specification for the Seagate 1200 series –

Product
Image

Here is a
picture of the Seagate 1200 that I tested –

 

The Seagate 1200 uses a Marvell controller with Seagate’s
proprietary firmware, 21nm eMLC NAND manufactured by Samsung, and has a robust
all metal case that feels as if it is built to last.

Now let's
head to the next page, to look at Myce’s Enterprise Testing Methodology.....

 

Please click here
to view or download a detailed introduction to Myce’s Enterprise Class Solid
State Storage (‘SSS’) Testing Methodology as a PDF.

Put briefly:

All testing
is performed on an OakGate Technology test unit

We perform
two sets of Performance Tests:

1.          
A full set of the Storage Network Industry Association’s (‘SNIA’) tests
with mandatory parameters, as specified in their Solid State Storage
Performance Test Specification Enterprise V1.0 – SNIA
SSS PTS Version 1.0.

2.          
A set of tests, known as the ‘Myce/OakGate Full Characterisation Test
Set’, that provides readers with a fuller characterisation of the solution.

We also review other important factors such as Data
Reliability and Failover features.

A word about SNIA testing – before striking a partnership
with OakGate Technology I spent some time researching how I might implement
SNIA testing using freely available tools such as IOMeter and FIO.  I arrived
at the conclusion that whilst it was theoretically possible it was impractical. 
The reason for this is as without the automation offered by a test bench, such
as the OakGate Unit, the only way to meet the SSS PTS requirements is to run
the maximum number of test cycles and then to manually look back at the results
to determine when/if steady state has been achieved in the workload specific
test cycle, and then harvest the data from the qualifying Measurement Window. This
means that the test runs would always take a maximum elapsed time, and there
would be a great deal of human effort required to review, gather, and report
upon the data.  I empathise with, acknowledge, and respect the efforts of other
reviewers who endeavour to meet the SNIA’s principles in their testing - I am
privileged and thankful to be able to use a superb test bench which automates
the whole process and allows me to meet the SNIA’s specification in full.

Before we
move on, let’s remind ourselves of some basics –

When
reviewing the performance of an SSS solution there are three basic metrics that
we look at:

1.          
IOPS – the number of Input/Output Operations per Second

2.          
Bandwidth – the number of bytes transferred per second (usually measured
in Megabytes per second, ‘MB/s’)

3.          
Latency – the amount of time each IO request will take to complete
(usually, in the context of SSS solutions, measured in Microseconds, which are
millionths of a second).

It is true to say that IOPS and Bandwidth had all been
growing rapidly before the advent of SSS solutions, but Latency can only be significantly
decreased by eliminating mechanical devices, and thus Latency is the single
most important aspect that SSS solutions deliver to enhance performance.

Latency in a technical environment is synonymous with delay.
In the context of an SSS solution it is the amount of time between an IO
request being made, and when the request is serviced.

Bandwidth, also commonly referred to as ‘Throughput’, is the
amount of data that can be transferred from a storage device to a host, in a
given amount of time.  In the context of SSS solutions it is typically measured
in Megabytes per second (MB/s). 

A great enterprise SSS solution offers an effective balance
of all three metrics.  High IOPS and Bandwidth is simply not enough if Latency
(the delay in an IO operation) is too high. As we will see in the test results
presented below, as Latency increases IOPS will inevitably decrease.

Queue Depth is the average amount of IO requests
outstanding.  If you are running an application and the Average Queue Depth is
one or higher and CPU utilisation is low, then the application’s performance is
most probably suffering from a ‘Storage Bottleneck’.

Another important factor to remember is that SSS performance
is influenced by previous workloads, not just the current workload, and
especially by what has previously been written to the drive. As specified in
the SNIA SSS PTS the goal of all good Enterprise level testing is to provide
consistent circumstances, so that results can be compared fairly across
different SSS solutions – it is for this reason that all of our tests start
with a purge of the drive, so that it starts in a ‘Fresh Out of the Box’ (FOB)
state.  Most tests then have a pre-conditioning phase where the drive is put
into a ‘Steady State’ before the test phase begins. Put briefly, a ‘Steady
State’ is achieved when the performance of the drive no longer varies over time
and settles into a consistent level of performance for the workload in hand. You
can find a detailed explanation of ‘Steady State’ and how it is determined in
the SNIA tests in our Enterprise Testing Methodology paper, which can be viewed
or downloaded as a PDF by clicking here.

For interest, here are some generally accepted
assumptions that differentiate the use and therefore the approach to testing
Enterprise/Server and Consumer/Client SSS solutions:

Enterprise/Server SSS assumptions:

1.          
The drive is always full

2.          
The drive is being accessed 100% of the time (i.e. the drive gets no
idle time)

3.          
Failure is catastrophic for many users

4.          
The Enterprise market chooses SSS solutions based on their performance
in steady state, and that steady state, full, and worst case are not the same
thing

Consumer/Client SSS assumptions:

1.          
The drive typically has less than 50% of its user space occupied

2.          
The drive is accessed around 8 hours per day, 5 days per week, and
typically data is written far less frequently

3.          
Failure is catastrophic for a single user

4.          
The consumer/client market generally chooses SSS solutions based on
their performance in the FOB state

Comprehensive
power consumption testing is performed using Quarch hardware as documented here.

Esther Spanjer, Director, Enterprise
Business Development EMEA at Sandisk, said, 'I am happy to commend Myce for
their high level of professionalism and cooperation during the review process',
Ms. Spanjer added, 'I wish them every success in their partnership with OakGate
Technology and their initiative to provide authoritative performance reviews
for the Enterprise Solid State Storage market'

Now let's
head to the next page, to look at the results of our SNIA IOPS (Input/Output
Operations per Second) Test.....

IOPS performance will typically vary greatly depending on
the nature of the IO traffic, including the mixture of Read and Write
operations, and the mixture of Block Sizes (the size of the IO operation’s data
packet, also referred to as IO Size). This test is designed to benchmark the
IOPS performance profile for random IO operations for 56 different combinations
of Read/Write mix % and Block Sizes when in a Steady State, which are of
interest to most users.

All of the SNIA’s test specifications define a ‘required’
set of parameters that must be run for the test and then allow the operator to
elect to run additional tests with different parameters of their choice. It is
the mandatory test with the required parameters that we run. Note that all of
the mandatory SNIA tests must be conducted with fully random data

As previously mentioned, a key
principle of SNIA testing is to provide a consistent basis for comparing
different solutions from different manufacturers.

Here are the
results -

 

You can see
here a visual confirmation that Steady State Convergence was determined at the
end of Round 5.

 

Here is a 3D and tabular presentation of the results. Users
can simply refer to the grid to obtain the R/W mix and Block Size value of
interest.  For example, Online Transaction Processing applications
typically run at a Block Size of 8K and a Read/Write Mix of 65/35, and users
can quickly understand how the device might perform under Steady State for
these access characteristics.

You can see that the 4K 100% Read IOPS result is 86,851 (that
is well below Seagate’s specification of 110,000) and that the 4K 100% Write
IOPS result is an excellent 40,693 (that exceeds Seagate’s specification of
40,000). 

Product
Comparison

For interest
we present a comparison of the 4K 100% Writes and Reads results with those of
the other Enterprise SSDs we have tested -

 

 

Now let's
head to the next page, where we look at the results of the SNIA Write
Saturation Test.....

 

This test
performs random 4K writes.

The objective of this test is to observe the time evolution of
the drive’s performance, as a function of time, from a ‘factory fresh’, ‘fresh
out of the box’ (‘FOB’) state.  When a drive is in a FOB state (e.g. after it
has been purged by, for example a SATA Secure Erase or SCSI Format), we can
expect an initial period of time when writes can easily be accommodated by
clean/empty blocks, but once all of the clean blocks have been written to once
and the drive’s controller must first clean blocks (with erase write
operations) before it can write new data, then we can expect a slow down.  The
slow-down is usually quite dramatic and is commonly referred to as the ‘write cliff’. 

The Write Saturation Test is easy to run as it
requires no steady state determination – it can be easily run in freely
available software, such as IOMeter.

Here are the results -

 

You can see here a steep fall followed by a gradual drop in
Write IOPS performance as the Seagate 1200 reaches a Steady State. The fall that
begins at around Round 17 is the ‘write cliff’. 

Note that the test was halted, as specified in the SNIA SSS
PTS, when 4 x the User Capacity had been written to the drive. You can see that
the Seagate 1200 is settling towards a steady state at around the 42,000 IOPS
level, which is excellent.   

 

You can also
see that the latency graph line is a mirror image of the IOPS graph line.

Now let's
head to the next page, to look at the SNIA Throughput Test.....

The test is designed to measure the sequential Read and
Write IO performance for two Block Sizes, when under Steady State conditions. 
One can easily compare the results produced by this test with box-top numbers,
which are usually stated as “Up to xxx MB/S”.

Here are the
results -

You can see
here that Steady State was achieved for both Write IO sizes by the end of Round
5.

-

You can see
here that Steady State for both Read IO sizes was achieved by the end of Round
6.

Here are the
average values recorded in the measurement window –

These are excellent results but the highest level of read
throughput we see here for reads is 639.24 MB/s and this is falling short of
Seagate’s specification of 750MB/s for maximum read rate with a block size of 128K
(I’m guessing that this shortfall is a function of the more recent firmware
that I used to test the drive).

 

Product
Comparison

For interest
we present a comparison of the 1024K sequential reads and writes (single port)
performance in comparison with those of the other Enterprise SSDs we have
tested -

 

 

Now let's
head to the next page, to look at the results of the SNIA Latency Test.....

 

The Latency Test measures average and maximum response times
using random IOs at specified Block Sizes and Read/Write mixes, taken under
steady state conditions.  The test runs at a Queue Depth of 1 (1 outstanding
IO), thus the results give the baseline response time for a single IO request.

The test also
reports maximum latency values, which can be helpful to see if there might be
processes within the drive that may cause max Latency values to become larger.

Here are the
results -

You can see
here that Steady State was achieved in Round 10 through Round 14 (the
‘Measurement Window’).

 

These are the
Average and Maximum Latency Values observed in the Measurement Window (measured
in Milliseconds).

 

 

Here is a 3D
graph showing, at a glance, the Maximum Latency values for each combination of
Read/Write Mix and IO Size.

 

 

Here is a 3D
graph showing, at a glance, the Average Latency values for each combination of
Read/Write Mix and IO Size.  The 100% writes results are excellent. 

Product
Comparison

For interest
we present a comparison of the 4K 65% Reads 35% Writes latency results in
comparison with those of the other Enterprise SSDs we have tested -

 

 

Now let's
head to the next page, to look at the results for the Myce/OakGate 4K Read and
Write Latency Tests......

 

These tests steadily increase the random 4K IO demand in
terms of IOPS, and report the drive's response in terms of Average IOPS, Average
Latency and Maximum Latency.  It is designed to show a drive’s maximum IOPS
capability and report the all important Latency numbers for each level of IOPS
demanded.  The Maximum latency numbers give us an insight into the occurrence
of Latency peaks that could cause an unexpected response from time to time.

Here are the
results –

Firstly, here
is a graph showing the result for the initial Pre-Conditioning step (4K Random
Writes) –

 

A quick aside - for SAS drives we run the preconditioning step
at a queue depth of 128.  The following graph shows what happened with firmware
version 01, with which the drive was first supplied to Myce -

You can see that the drive crashes to a low level of
performance after around 1300 seconds.  This crashed state would then persist
until the system was rebooted.  The problem went away once firmware version 04
had been applied to the drive.

 

 

4K Latency Read Test

You can see that the drive can no longer meet the increase
in IOPS demand at 92,000 IOPS, which is a lot less than Seagate’s specification
of 110,000. Perhaps Seagate’s specification is for an earlier version of
firmware.

 

You can see a
small and gradual increase in read latency up to the maximum IOPS mark. 

 

You can see a
few max latency spikes.

Let’s have a
close look at the distribution of the Latency results at the 54,000 IOPS level
(at one of the spikes) –

As this is the first time in this review, that we are
looking at a High Resolution Latency Histogram, here’s an explanation – The X
axis to the left is the count of the IOs in the observation period (in a Round)
that had a Latency of the value along the Y axis (please note that the X axis
is logarithmic to allow the low order counts of the huge number of IOs that
have been measured to be visible); the Y axis is the Latency value measured in
Microseconds; The X axis to the right is the % of the Total IOs observed that
have a Latency <= to a given Latency value; the rate of getting to 100% is
highlighted by the red graph line.

You can see
that 99.9% of the Latency values are <= 820 Microseconds (0.82 Milliseconds)
and there are relatively few outliers, so the quality of service is good.

 

4K Latency Write Test

You can see
that the Seagate 1200 begins to fail to meet the increase in demand at around
the 40,000 IOPS level (at the 40,000 level of demand the response is actually
below 39,000)

 

Here we can
see that Average Write Latency stays below 50 microseconds until a demand of 35,000
IOPS.

The maximum
write latency results are excellent, with no spikes until a demand of 38,000.

Now let’s
have a look at the distribution of the Latency Values at the 38,000 IOPS Mark –

You can see
that 99.9% of the Latency Values are <= 2.72 milliseconds.  The quality of
service is excellent.

 

Now let's
head to the next page, to look at the results for the Myce/Oakgate Reads and
Writes Tests.....

 

The tests are
designed to show the Random and Sequential, Read and Write, performance metrics
for different combinations of Queue Depth and IO size. 

Here are the
results -

Random Reads

You can see
there is no scalability beyond a queue depth of 32.

 

 

 

Random Writes

You can see a
healthy peak for 4K random writes and there is little or no scalability beyond
a queue depth of 2.

 

 

 

Sequential Reads

 

 

 

Sequential Writes

 

 

 

Now let's
head to the next page, to look at the results for the Myce/Oakgate 4K Mixed Reads/Writes
Tests.....

 

This test is
designed to show the performance metrics for different combinations of Queue
Depth and Read/Write mix (the % of Reads and the % of Writes making up the IO
traffic) 

4K Mixed R/W Test

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Now let's
head to the next page, to look at the results of the Myce/OakGate Entropy Tests.....

 

These tests are designed to show performance metrics for
different combinations of Queue Depth and Entropy % (Entropy % is the degree to
which the data is random and therefore incompressible).  Testing with different
Entropy % levels has become important with the advent of controllers, such as
those from LSI Sandforce, that compress data before writing it to NAND.
Controllers that compress data can be expected to perform better with highly
compressible data (i.e. data with low Entropy).

The first
test performs 5 minutes of Random 4K writes for each combination of Queue Depth
and Entropy %.

The second
test does the same thing for a mixture of Read and Write traffic (70% Reads,
30% Writes).    

4K Entropy Write Test

 

 

 

You can see there is little or no variance in performance to
be found in any of the Entropy tests, as the degree of random data increases
(and this comment applies to all of the test results for the Myce/OakGate
Entropy Tests). We can therefore conclude that the Seagate 1200 does not
compress data.

 

 

 

4K Entropy 70% Reads 30% Writes
Test

As we saw no
evidence of compression in the 4K Entropy Write Test we skip the presentation
of the 70/30 entropy results.

 

 

 

Now let's
head to the next page, to look at Power Consumption and Data Reliability.....

 

Power
Consumption

I believe most people know that data centres are already one
of the major consumers of electricity in the industrialised world; indeed it is
estimated that currently 2% of all electricity consumption goes into IT
applications.  According to the European Union the energy consumption of data
centres was 46 Terawatt hours in 2006 and is set to rise to 93 TW hrs by 2020. This
is equivalent to one hundred million 100W light bulbs burning 24 hours a day,
365 days a year.

Typically 40% of the power consumed by data centres is for
the IT load and 35% is for cooling the system.  Generally speaking, if a drive
consumes more power it will produce more heat – so power consumption is indeed
a double edged sword.  It is no surprise then that a significant proportion of
a data centre’s power consumption goes on servers.  I understand cloud based
applications, such as Facebook, are the primary cause of the growth in servers
and the demand for storage space.

If you are a Facebook user, like me and the Reynolds sibs, and
you reside in Europe – this is most probably where your data is click here.  Some
interesting Facebook statistics – Facebook has more than 1 Billion monthly
active users, it generates 1 Trillion page views per month and more than 219
Billion photos have been uploaded since launch – amazing!  Here is an
interesting video showing the remarkable scale of Facebook’s largest North
American data centre click
here.

My thanks to Anna of Intel for pointing me to the following
Info-graphs -

 

 

Power Testing

We present
our standard set of power consumption tests.

SNIA Write
Saturation

 

This test allows us to observe the power consumption
characteristics as the drive passes from a fresh ‘out of the box’ state to one
where blocks must first be cleaned before they can be written to.  It also
allows us to form a view on the amount of power that is being consumed by the
cleaning of the blocks (for 4K random writes). We can see a slight increase in
power consumption after hitting the write cliff at round 17, but roughly
speaking, by round 105 we can see that 6,000mW is required to sustain around
42,000 IOPS whereas before the write cliff, the same level of power was
sustaining nearly 110,000 IOPS. Thus we can deduce that roughly 68,000/110,000
of the 6,000mW (i.e. roughly 6/10 of the power) is consumed by housekeeping. 

4K Latency
Test - Reads

This test allows us to observe how power consumption
characteristics vary as the demand for random 4K reads (in terms of IOPS) is
increased.  You can see that the demand for power increases gradually and in a
linear fashion.  We could use these two lines to calculate the sweetspot where
one would consume the least power per IO, however as the rate of increase in
power consumption is not as steep as the increase in IOPS we know intuitively
that the best power consumption to IO ratio is at the maximum IOPS level.

4K Latency
- Writes

 

This test allows us to observe how power consumption
characteristics vary as the demand for random 4K writes (in terms of IOPS) is
increased.  You can see that the demand for power increases gradually and in a
linear fashion.  We could use these two lines to calculate the sweetspot where
one would consume the least power per IO.  However, as the rate of increase in
power consumption is not as steep as the increase in IOPS we know intuitively
that the best power consumption to IO ratio is at the maximum IOPS level.

4K Mixed
Reads/Writes

 

This test allows us to see how power consumption
characteristics vary when performing 4K random reads and writes with different
combinations of read/write % and queue depth.  As would be expected, you can
see that as one increases the % of writes the power consumption increases.

We have then
taken the data to calculate the IOPS per mW for each combination, as follows -

The IOPS per mW results can then be compared to those for
the Toshiba THNSN960PCSZ (the drive with the best power consumption results,
that has thus far been subjected to our Enterprise Power Tests).

You can see
that the Toshiba is significantly more efficient particularly for read
activity. However, we must keep in mind that SAS drives are typically more
power hungry than SATA drives.

Sequential
Writes

(I apologise
for not presenting the results for Sequential Reads on this occasion)

This test allows us to see how power consumption
characteristics vary when sequential writes with different combinations of IO
Size and queue depth.  As might be expected, the power consumption increases as
the MB/s increases.

 

We have then used
this data to calculate the MB/s per mW as follows -

The MB/s per
mW results can then be compared to those for the Toshiba THNSN960PCSZ (the
drive with the best power consumption results, that has thus far been subjected
to our Enterprise Power Tests).

Again, you can see that the Toshiba is significantly more power
efficient.

Power Up
to Idle

This test
allows us to see the shape of the power demand as a drive is powered up. It
also allows us to see the peak level of current demanded to kick the drive into
life.

As you can
see power is drawn from both the 12v and 5v rails.

Seagate specifies that the ‘Max Start Current’ required is
0.7A from the 5v rail and 0.4A from the 12v rail.  Well if you look at the
Current (mA) Graph above you can see a very short peak on the 5v current (at
around 400mS) that has exceeded 1000mA (1A) and a more sustained peak on the 12v
current (between 2,200 and 2,800 mS) that edges above 400mA (0.4A).  So our incredibly
accurate Quarch equipment finds that Seagate’s specification for the Seagate
1200’s Max Start currents are a bit ambitious.  

Here is a
closer look at the first 150 mS.  You can see that the drive springs into life
when the supply has reached around 3,900mV on the 5v rail.

Idle

This test
allows us to view the power consumption characteristics when a drive is idling
(powered up but with no IO activity).

Here is a
picture of the raw data values that were recorded.

Here are the
statistics calculated for the recording.

The average
power used when idling was 2100mW from the 5v rail and 881mW from the 12v rail
giving a total of 2981mW (2.981W), which can be compared to Seagate’s
specification of 2.72W.

Data
Reliability

The 'Unrecoverable Bit Error Rate' (UBER),as defined by
JEDEC, the global leader in developing open standards for the microelectronic
industry, is a metric for data corruption rate equal to the number of data
errors per bit read after applying any specified error correction method. UBER
= number of data errors / number of bits read.  JDEC specifies that the maximum
error rate allowable for an Enterprise level SSS solution is one error in every
10^16 bits read.

Seagate specifies an UBER of 1 in 10^16 bits read
for the Seagate 1200.

The Seagate
1200 400GB is warranted to support up to 7300 Terabytes of writes over 5 years
(which equates to 10 Drive Writes per Day).

The Seagate
1200 includes sophisticated power failure support and end-to-end data
protection.

The Seagate
1200 benefits from the inherent failover support provided by SAS drives.

 

Now let's
head to the next page, to look at the Conclusions of this review.....

The Seagate 1200 is an excellent all round enterprise drive,
with particularly good intensive random write performance.

I am pleased to award the Seagate 1200 the Myce rating of ‘Excellent’.