Multiple Pure FlashArrays Behind VPLEX

In continuation of the previous blog post regarding supportability of Pure Storage behind VPLEX, I wanted to show an example of a customer running multiple Pure Storage FlashArrays behind a VPLEX Local environment.

As stated prior, when running multiple Pure Storage FlashArrays behind VPLEX there is a known issue with VPLEX where a “dual array name condition” could occur.  EMC has plans to address this however, there is a very simple best practice that can be followed to ensure this dual array name condition does not occur.

Here is an example mapping file creation best practice, as highlighted on my previous post:

  • Provision a set of volumes to VPLEX from the first PureStorage Array (PureStorage1)
  • Run a backend re-discovery on VPLEX to discover the newly added devices
  • Create a mapping file , adding the newly added volumes to the mapping file.
  • Name each volume based on it’s array name and volume ID

Generic storage-volumes

VPD83T3:624a93707bbe9dc5bb91417c000118d2 PureStorage1_volume_1

VPD83T3:624a93707bbe9dc5bb91417c000118d3 PureStorage1_volume_2

  • Provision a set of volumes to VPLEX from the second PureStorage array (PureStorage2)
  • Run the backend re-discovery on VPLEX to discover the newly added devices
  • Add those devices to the mapping file
  • Name each volume based on it’s array name and volume ID

Generic storage-volumes

VPD83T3:624a93707bbe9dc5bb91417c000118d2 PureStorage1_volume_1

VPD83T3:624a93707bbe9dc5bb91417c000118d3 PureStorage1_volume_2

VPD83T3:624a93707bbe9dc5bb91417c000118f8 PureStorage2_volume_1

VPD83T3:624a93707bbe9dc5bb91417c0001194f PureStorage2_volume_2

The below image is from a real customer running many Pure Storage FlashArrays behind a VPLEX Local cluster successfully without a dual array name issue.  For all third party arrays, it is standard practice to create mapping files.  This customer followed the mapping file best practice above without issue.

Screen Shot 2015-07-02 at 8.12.45 PM

In closing, multiple Pure Storage FlashArrays work just fine behind a VPLEX Local and/or VPLEX Metro.  Given this configuration has multiple Pure Storage FlashArrays, EMC still requires you to work with your EMC account team to create and submit an official RPQ.  EMC will confirm that you are free from this known issue and provide the conditional approval.

Hope this helps!

Supported: Pure Storage Behind VPLEX

Pure Storage has completed all qualification and all test cases that it is aware of that are required to support the FlashArray behind EMC VPLEX.  In fact, Pure received a confirmatory email from EMC stating that.

As part of the VPLEX qualification process, our Purity Operating Environment (4.0.20 or later) passed the requirements to be supported behind VPLEX meaning that the FlashArray can be supported provided it runs the same Purity Operating Environment even with underlying hardware changes.

Here is the current Support Statement provided by EMC:

Screen Shot 2015-06-25 at 4.08.36 PM

Below are some example scenarios for approved supportability without an EMC RPQ required:

Screen Shot 2015-07-11 at 3.29.58 PM

Screen Shot 2015-07-11 at 3.25.27 PM

Screen Shot 2015-07-11 at 3.36.38 PM

Screen Shot 2015-07-11 at 3.43.06 PMBelow are some example scenarios “conditionally approved” via the EMC RPQ process. Remember, proper mapping file creation is a simple work around to prevent any display issue

Screen Shot 2015-07-11 at 4.00.33 PM

Due to a known VPLEX issue called “dual array name”, in very rare cases VPLEX could recognize different arrays as being a single backend array, which could lead to confusion in provisioning and handling devices from both backend arrays. This is a display issue and not any functional failure as noted within the “conditions statement” provided by EMC. Note: Proper mapping file creation is a simple work around to prevent this confusion.

Known Issue: dual array-name identification

Cause of the issue: VPLEX is currently recognizing new backend storage arrays by the Node WWN, which is a 16 digit WWN unique to each storage frame. Due to a limitation in the code, VPLEX is only utilizing 6 digits of the 16 digits Node WWN number to identify the array, like in the below example:

NWWN: 52:4A:93:74:BB:F5:08:00

VPLEX: storage-array: PURE-FlashArray-9374bb

In very rare cases distinct Pure Storage arrays could be recognized as the same backend array, leading to confusion in storage provisioning if not properly discovered. In an effort to understand what the likelihood of this condition is, we quickly looked within Pure1 and determined that less than 1% of our total customer population would potentially encounter this known issue. Due to this extremely rare case, EMC is restricting the addition of Pure Storage as a supported platform on the VPLEX Simple Support Matrix, and requires an official RPQ for environments with multiple backend arrays to validate the backend arrays are uniquely discovered by VPLEX.

Despite the fact that single arrays per site have no restrictions and do not require an EMC RPQ as they are not susceptible to the dual array name issue, EMC is not adding those to the Simple Support Matrix and taking the position that no exceptions or limitations are to be added to the supported statement, though all other storage platforms have different limitation statements.

Even in the very rare case of hitting the dual array identification issue, properly discovering devices in VPLEX will allow working in this configuration with no restrictions.  As with any other 3rd party array, VPLEX will require the creation of a mapping file to properly name the configured and mapped devices. In the case of a dual array name identification issue, customers will have to provision volumes out of one array at a time, running the discovery on VPLEX and creation of the mapping file. This way the unique naming will allow customers to identify those devices and connect them to the correct Pure Storage array on the backend.  We have several customers running multiple Pure Storage FlashArrays behind VPLEX today without the dual array name issue.  This is specifically due to properly discovering these FlashArrays via the mapping file.

How to identify the Dual Array Identification issue?

Screen Shot 2015-06-25 at 4.53.48 PM 

How to create a mapping file in case of a dual array name issue?

Screen Shot 2015-06-25 at 5.28.05 PM

We don’t understand why EMC is unwilling to place Pure Storage on the VPLEX Support Matrix for configurations that don’t require an RPQ today. Nor do we understand for the multi-array configurations where a proper mapping file creation as a simple workaround, cannot be specified as a limitation like other third party arrays within the VPLEX Support Matrix.

This whole process has spawned a few additional thoughts:

  • If the fix to the dual array name issue is to use [array SN] for EMC and IBM arrays, why not use the same procedure for every new discovered device?
  • The fact that the [array SN] change was made for other arrays (including EMC arrays), why didn’t someone start thinking – why not change it generally on how VPLEX recognizes backend arrays?
  • Is it not easier to make this a general change rather than an exception every time they discover new backend storage?
  • If simple changes like this aren’t possible in VPLEX, how can you adapt to the rapid changes in storage right now and more importantly to the customers needs?
  • How does this benefit our joint customers who want a dual vendor strategy when they are met with the delays of an RPQ process for something that is a minor fix as addressed with other “now” supported storage arrays?
    • There are several open RPQ’s that are still pending approvals. Hopefully EMC can work towards getting these approved in short order for our joint customers.  I’m sure they would be happy to know they can proceed with their current projects requiring BOTH of our technologies.

In closing, I want to thank everyone involved with working towards this qualification process.  Thank you to Kumar, Tanya, Maharishi, and Haring from the EMC qualification team for working with our team during this qualification process.

The Foundations Of A Performance POC

One of the really cool aspects of Pure Storage is that customers and potential customers appreciate our ability to execute on a POC (Proof Of Concept) .  When I refer to existing customers, this is where they have other use cases they are exploring and we work to help “Validate The Value” with an additional POC.  Recently, I responded to a post on LinkedIn where a potential customer was looking for advice around how to test an AFA (All Flash Array).  I thought I would capture some of these foundational elements and share with the broader community.

You are likely asking yourself several questions…..

  • How do I test an All Flash Array?
  • Can I use the same methodologies as my traditional/legacy disk based array?
  • What tool(s) should I use?
  • What metrics are most valuable to capture?

While I do represent Pure Storage, this basic methodology I will discuss is vendor agnostic.  If you are one of our awesome Pure Partners, you may have captured a portion of this methodology from the Partner Training Webcast that Lou Lydiksen (Performance Engineering) and myself conducted on Dec 2nd.  You can find that here and we would love your feedback!

Let me start off with a question.  When should synthetic “performanceScreen Shot 2014-12-15 at 12.23.56 AM corners” testing be run?  The survey says.. Preferably never!  The reasoning is quite simple and can be answered with another question.  How many synthetic instances of your application do you have running in production?  None! In other words, you can likely take a Formula One race car on the ice to test its “performance corners”, but I suspect the results will offer little value relative to the typical racing environment.  The same applies to synthetic “performance corners” testing on any storage platform.  Baseline your testing to be more representative of your production environment(s) which provide a better expectation of the business value derived from the underlying technologies.

What is our recommendation around offering our customers the ability to see the true potential of Pure’s technology?

  1. Consider the no-risk “Love Your Storage Guarantee”.
  2. Move a “copy” of your Dev/Test/and or Production workloads onto the Pure Storage FlashArray. With non-disruptive hypervisor based mobility, this can offer up a quick way to validate application benefit, technology efficiency, and performance capabilities.
  3. If synthetic testing is leveraged, conduct a synthetically-modeled customer workload scaling test with application load generators approved by application vendors. Think SLOB, Hammerdb/ora, Oracle RAT, SQLIO, etc.
  4. If synthetic testing is leveraged, conduct a synthetically-modeled customer workload scaling test with generic load generators like Load DynamiX, vdbench, FIO, etc.
  5. Consider using the new Pure Storage vdbench kit 2.1 for the next best thing to performance testing.  This kit was designed with real-world characteristics which will allow customers to assess performance and data reduction of any AFA.  Real-world testing is our preferred and default recommendation, but this test kit provides the next best alternative.  Please reach out to your Pure Storage SE and/or Pure Storage Partner for the latest Pure Storage test kit.

What is our recommendation around synthetic “performance-corners” testing?

Pure Storage came up with a POC framework called ASAP which stands for:

Availability, Scalability, Affordability, and Performance

We have listed these in order of testing priority.  Given AFA’s continue to service mission-critical workloads, Availability cannot be compromised during planned and unplanned events.  More on this framework later.

One of our recommendations for Performance testing, would be to test with real-world data as that is the ONLY way to get the most accurate expectation in terms of performance, efficiency (data reduction), and resiliency. If you can avoid synthetic testing altogether and focus on actual real-world datastreams, this can shorten your testing effort while providing more realistic expectations of the technologies’ capabilities relative to your environment.  We want customers to validate that the AFA meets their requirements in the areas of cost, resiliency, and integration before performance, as performance varies greatly based on hardware configurations.  Value in areas of data reduction (infrastructure efficiency & cost), resiliency, and integration are constants of an architecture – unlike performance.

With regards to synthetic testing, we have moved away from recommending homogeneous fixed-block tests (1K,2K,4K,8K….), to tests with more real-world block size “mixes”.

Likewise, our recommendation is that 100% read and 100% write tests are of little value when compared to read/write mixed tests (what you see in real workloads).

Also, it is our recommendation that a dataset comprised of a mix of reducible data-subsets be created to run the performance tests against. I think we would all agree that testing a platform outside of its capabilities as a measure of a baseline might set false expectations. Hence why the best measure of a baseline is to leverage real datasets and datastreams that will have reducibility and characteristics that unfortunately you are challenged with reproducing through synthetic means.  Real-world datasets are data-reducible, the data reduction ratio is a critical element of an AFA, as it drives the economics and enables the transformational impact of an AFA to be applied broadly.  If you’re not testing data reduction capability of an array, you’re missing out on understanding a critical element of an AFA = incomplete evaluation.

Finally, we also believe that “performance corners” tests should attempt to emulate real-world datastreams that have, at the very least, the following characteristics:

– A read io size mix that is different than the write io size mix
– A read range that is different than the write range
– A set of “hotbands” within the ranges that are different for reads and writes.
– The dataset and the datastreams should use data that has both a deduplication component and a compression component that is modeled after the datasets and datastreams that the customer wishes to serve from the AFA
– We have settled on a vdbench datastream that reduces to about 5:1 on Purity 4.0.x today

We combine all of these into IO fingerprintramp tests so that we can show the relationship between latency, throughput, and concurrency for the dataset/datastreams combo.  Every dataset/datastream has its own Little’s Law fingerprint.

We will soon publish a whole methodology based upon these principles. We do have the VDBench scripts for this that we can produce now. Again, this is only if you are unable to test real working sets which we strongly urge as a best measure of the ANY technologies capabilities.

We believe that this will provide a much more realistic and valuable synthetic “Performance” testing environment for any storage array, but in particular, non-disruptive all-flash data-reduction storage arrays like a Pure Storage FlashArray. It is important to inject all sorts of failures through this process to highlight planned and unplanned events.

Screen Shot 2014-12-24 at 9.36.07 AM

As much as we would like to say technology never fails, we know that when it does it does so in a disruptive manner (generally not a graceful shutdown procedure). What a better way to baseline performance and resiliency other than pulling components while testing performance. You will find Pure Storage is very well-differentiated in this area.  More importantly, once you put mission critical business applications on flash there is a new expectation set (latency and/or velocity) and during planned/unplanned events this expectation should be maintained.

Hope this helps and thanks for evaluating Pure Storage.  We know you will Love Your Storage!

Does 1M IOPs Impress You?

When synthetic loads are created from IO generation benchmark tools, you shouldn’t be impressed.  Running these tools will tell you nothing meaningful for your application owners, end-users, and customers.  If you want to see how well these applications will work, run the applications themselves, not a tool designed to mimic them (which will not necessarily be representative of your environment).  If you can’t test your applications during AFA testing (highly recommended), don’t trust vendor-crafted benchmarks tools. Dissect them and understand how they relate (or not) to your actual applications.  Then, craft a better synthetic load (if that is your only option).

  • one with blended IO sizes (abandon fixed IO sizes)
  • reducible data sets and datastreams (these are data reduction arrays and a majority of customer datasets/datastreams are reducible)
  • mis-aligned IO (don’t allow vendors to align the test scripts to their fixed block sizes)
  • IO banding which incorporates temporal and spatial locality
  • appropriate queue depths

That way you’re at least closer to “Real World” through synthetic means.  Focus on what matters to the business and how flash can help make your company money, save your company money, increase productivity, differentiate you from your competition, or get your product(s) to market quicker.  Those are far more meaningful than some unrealistic and uniquely crafted IOPs test.  Yes, 1M IOPs is possible (even from an array that doesn’t scale out) but does it really matter to you?  Don’t be impressed.

image (1)

 

Pure Storage: New Python Automation Kit Availability

Below are links to a Python Automation Kit that we recently released as open source.

It is a full wrapping of the Purity OE REST v1.2 API. Any Purity features released through 4.0.8 are included in this release.

The kit should be useful to anyone with Python experience, and the links below include documentation, a PIP installable package, as well as the links to our opensource github repository.

Between this, the PowerShell work we are doing, and other 3rd party integrations, we believe the FlashArray is the easiest to automate AFA available!  Stay tuned, lots more coming….

Here are few key points:

  • The kit is designed to work with Python 2.6 or later. Python 2.6 has been around for at least 5 years and this or a later version of Python ships in major distributions such as Red Hat, SuSe, Ubuntu, etc…
  • The kit is released open source under the BSD-2 open source license: http://opensource.org/licenses/BSD-2-Clause
  • We will periodically update the open source code repository, package, and documentation as new capabilities of the Flash Array are surfaced in our REST API
  • We will be fostering an active Python discussion on our community.purestorage.com website in the Tools, Tips, and Tricks forum (http://community.purestorage.com/t5/Tools-Tips-Tricks/bd-p/tips-tricks)
  • Python is used broadly in Linux community and especially relevant for anyone interested in OpenStack, which is pervasive in OpenStack since many modules are built with it. Also commonly used in Big Data space
  • There are example code segments shown in the Quick Start guide on the documentation page below, along with a full API Glossary

For most users I would recommend a quick read through the documentation, then do a PIP install of the package and start playing with it.  We want your feedback!

Documentation link: http://pythonhosted.org//purestorage/

PIP installable image is found here: https://pypi.python.org/pypi/purestorage

Datasheet: http://www.purestorage.com/pdf/Pure_Storage_Datasheet_OpenStack.pdf

The open source code repository is on GitHub here: https://github.com/purestorage/rest-client

We appreciate any and all feedback.

Why Do I Need Flash?

It is not uncommon for someone to ask “Why do I need flash?  I really don’t have any performance problems”.  Sometimes there is a subtle reminder that:

We are an impatient society wanting instant infrastructure

I continue down the path of reminiscing by providing some of the following examples which generally draws some chuckles.

  • You had VHS so why did you need DVD? Screen Shot 2014-06-15 at 6.57.59 PMScreen Shot 2014-06-09 at 12.36.09 AM
  • You had a VCR so why did you need TiVO or DVR?
  • You had a flip phone so why did you need a smart phone?
  • You had physical servers so why did you need virtual servers?
  • You had a book of maps so why did you need GPS?
  • You had tapes so why did you need CD’s or MP3’s?
  • You had Blockbusters so why did you need Netflix?
  • You had a polaroid camera so why did you need a digital camera?
  • You had a laptop so why did you need a tablet?
  • You had newspapers/magazines why did you need a Kindle?

I think we would all agree we saw advantages to the advancements in technology.  The general result has been positive.  As a parent of “connected children” I might debate the smartphone :).

I have been asked this very question around “Why do I need flash?” and it is a fair question.  Flash offers multiple dimensions of transformation and performance is one of those dimensions.  I continue to ask more questions around the current application performance and the infrastructure required to deliver on said results.  As an example, a customer was achieving 10ms of average response time. Not bad right?

The Infrastructure?

4 cabinets consisting of controllers, disks, batteries (160U)

The total capacity?

200TB usable

Why do I need flash?

In many cases, it is an opportunity to drastically improve on performance efficiency, capacity efficiency, operational efficiency, environmentals, and data center footprint just to name a few.  In the above example we could consolidate 160U into <32U.  Pure Storage can obviously take this further through our other unique programs (no software licensing, forever flash, no customer education, no professional services).

Lets not stop here, what else?

If we can drop the latency on average from 10ms to <1ms what does that mean to the business?  What part of your bottom line would be positively impacted if your applications were 1/10th the latency allowing for double or triple the productivity? Turning a process from an multiday to within a workday, or a once a day to a twice a day process is always transformational. It changes the cadence of the entire business.

In closing, flash provides instant infrastructure and 360 degrees of transformation to the business.  Come check out Pure Storage and see how we rescue you from the Big Storage practices and provide immediate transformation.  We look forward to the opportunity.

[Update] FlashArray and Heartbleed bug

 

Description:

Heartbleed bug (CVE-2014-0160) is a vulnerability in OpenSSL library that can compromise secret keys used to encrypt the data, including names and password of the users. Certain versions of OpenSSL are effected by this bug. More information about HeartBleed bug is available at: http://heartbleed.com

FlashArray Vulnerability:

FlashArrays running Purity OE 3.3.x do not use the OpenSSL versions that are affected by Heartbleed vulnerability. Also, Purity OE 3.4.0 command-line interface (CLI) uses secure shell (SSH), which is not impacted by the bug.

FlashArrays running Purity use OpenSSL for web-based management interface and RESTful API. The OpenSSL version used in Purity 3.4.0 was vulnerable to the Heartbleed security bug. OpenSSL has been updated to a version that is not affected by the Heartbleed bug.  All customers running 3.4.0, we would recommend reaching our to your local account team and/or contact our uber awesome support team and have them upgrade Purity Operating Environment to 3.4.2 which removes any potential vulnerability to the Heartbleed bug.  This is a very quick and non-disruptive operation to your applications with zero performance degradation.

Note: Purity 3.3.X or any prior releases are not vulnerable to the Heartbleed bug.

Pure Storage’s CloudAssist servers were upgraded to a patched version of OpenSSL that is not vulnerable to Heartbleed. The FlashArray communicates to CloudAssist through SSH or SSH over an HTTP(S) proxy which uses OpenSSL. SSH is not affected by the Heartbleed vulnerability. SSH over HTTPS is also secure because CloudAssist uses a patched version of OpenSSL.

Hope this helps to show how responsive we are to industry security vulnerabilities.  We appreciate your business.

Keeping things Pure, Orange, Secure, and Awesome!

No Vendor PS BS – DIY Storage Capacity Adds

It is wildly known how Pure Storage has embedded SIMPLICITY into every aspect of our product.  This core design tenant has tremendous benefits to our partners and customers while forcing dramatic disruption to the legacy storage providers.  In our view, if a vendor creates professional services sku’s, we believe this highlights three things:

  1. Cost
  2. Complexity
  3. Competition/Conflict – Who delivers the services?  Partner? Storage Vendor?

We are a “channel friendly” and “channel only” model which means we fully embrace our channel ecosystem.  This is what I call the “This is how we roll” model.  It’s SIMPLE!

Continuing down the SIMPLE path, lets see what it takes to add some additional capacity to an existing Pure Storage FlashArray.  Everything I reference below is based on a real live production customer.

DIY Storage Capacity Adds | What Happens in the GUI?

Pre-amble:

When adding another shelf to an existing Pure FlashArray, you don’t always have the benefit of access to the GUI (unless you have figured out how to clone yourself). A common questions asked: what shall we expect once you plug the shelf in? Beyond technical curiosity, this question is asked by the storage admin because you are touching a production asset with potentially 100TB + in production. Here is a summary of the experience through the GUI and subsequent screen shots during intervals of the shelf addition:

  • Once plugged-in & powered on, ten seconds later, the third shelf, which was green, now looks yellow. In the screen shots shown here, the user did not logout and log back in.
  • The drives appear yellow at first. As they are acknowledged by the system, they turn green. As they turn green, the Total Capacity number in the GUI continues to grow.
  • This was an non-disruptive on-line upgrade. Note the I/O to the system as the shelf is added.
  • The entire process takes about 2.5 minutes. Yes, 4 cables and 2.5 minutes later, you have your full capacity.  SIMPLE?  Check!

Figure 1. A screen shot of the GUI just before the shelf is powered on. Highlighted here is the picture of the GUI on the left & the Total Capacity on the right.

Figure 2. The GUI shows Shelf02 (third shelf) going all yellow; this is Shelf03 (fourth shelf) being ingested by Purity. As noted above, we never logged out of the Pure Storage Dashboard, which would have shown all 4 shelves with the 4th shelf yellow.  Note that all drives are yellow; the Total Capacity has not grown higher (yet).

Figure 3. As the drives appear green in the GUI, the Total Capacity number begins to grow. Until this user logged out and back in, the shelf appeared as Shelf02 (third shelf).

Figure 4. More drives appear green; Total Capacity continues to grow.

Figure 5. Figures 5, 6 & 7 show more green and a higher Total Capacity number.

Figure 6.

Figure 7.

Figure 8. All drives are green and the 24.94TB Total Capacity is indicated.

Figure 9. The user has logged out and in; the GUI shows the system as a 4-shelf system all during production hours and without any disruption to the applications.

As you can see, the process for adding additional capacity is quite SIMPLE and any Pure Storage Partner can help you through this process if you prefer.  Yet another example of our Non-Disruptive Everything approach to enterprise storage.  We believe our product continues to drive out the legacy process, services, and support.  In the end, the customers win if we are able to drive out any extra cost or complexity.

Watch for my next blog post around “Executing on a Vision Part #2” which will take this Non-Disruptive Everything to another level around a real life use case doing generational upgrades.  More OrangeStorage Awesome-sauce coming your way!