White PapersIn a Continuous Access environment, the management appliance is often connected on the fabric between the source and destination arrays, where one array is considered to be local to the management appliance. That is, management commands do not need to be passed across the inter-switch links (ISLs). When the fabric is configured this way, and especially when you have low bandwidth ISLs, the transfer of the firmware image across the ISL can cause dropped jobs and degraded replication performance. To avoid problems with the controller firmware download in a Continuous Access environment, and to prevent the management appliance from timing out, the download should only be done from a local management appliance. In other words, you must perform the EVA firmware online upgrade using the instance of Command View EVA that is local to the EVA being upgraded. That is, do not perform code load by sending the firmware image over the ISL.
In a Continuous Access environment, online upgrades may be done while replicating data. Each replication group can be evaluated separately using the following table where the Continuous Access source and Continuous Access destination refer to the array doing the online download. For example, if the EVA being upgraded is the destination for one or more synchronous replication groups, then the synchronous replication groups must be suspended before the upgrade. All other replication groups are not required to be suspended when performing the EVA firmware online upgrade.
HP StorageWorks Enterprise Virtual Array (EVA) online controller firmware upgrades white paper Introduction......................................................................................................................................... 2 Definition of terms................................................................................................................................ 2 Understanding how the feature works .................................................................................................... 3 Considerations for systems using HP StorageWorks Continuous Access EVA............................................... 3 Host OS-specific guidelines................................................................................................................... 4 Windows........................................................................................................................................ 5 HP-UX............................................................................................................................................. 5 Linux............................................................................................................................................... 5 Solaris ............................................................................................................................................ 6 Tru64 UNIX..................................................................................................................................... 7 AIX................................................................................................................................................. 7 OpenVMS....................................................................................................................................... 7 EVA as XP external storage ............................................................................................................... 7 Application support for online controller firmware upgrades ..................................................................... 8 Process for performing an online controller firmware upgrade................................................................... 8 Finding an appropriate time for an online controller upgrade................................................................ 9 EVAPerf Array Performance Data Collection.................................................................................. 9 Array utilization data analysis...................................................................................................... 11 Approach to upgrading a busy array ........................................................................................... 17 Before the online controller firmware upgrade ................................................................................... 17 The online controller firmware upgrade ............................................................................................ 18 Verify the online controller firmware upgrade.................................................................................... 18 Online disk drive firmware upgrades ................................................................................................... 18 Customer experiences with online controller firmware upgrades.............................................................. 18 Telecommunications ....................................................................................................................... 18 Software development .................................................................................................................... 19 World leader in computer chip manufacturing................................................................................... 19 Untitled Document2 Introduction The HP StorageWorks Enterprise Virtual Array (EVA) online controller firmware upgrades are a misunderstood feature of the EVA. This paper provides detailed information that can help partners and customers decide if online controller firmware upgrades for the EVA are appropriate in their environment. It also provides supporting information and guidelines that will allow them to successfully execute online controller firmware upgrades. This paper is intended to complement the Updating Product Software Guide, which contains detailed information specific to a revision on product firmware. Note: The Upgrading Product Software Guide and firmware release notes should be consulted to get specific information on a particular upgrade, including upgrade paths, if an online upgrade is supported for a particular version, host requirements, driver requirements, host bus adapter (HBA) requirements, and so on. Definition of terms " Offline upgrade (shutdown upgrade) An offline upgrade is the most invasive type of upgrade for HP customers. All host I/Os to the storage system must be stopped. This typically requires that all applications accessing the storage system as well as all hosts are shut down before the controller firmware upgrade and are then restarted after the upgrade is complete. For customers, this typically involves scheduling time with the application owners, and then having staff on hand to shut down and restart the applications and the host. Although the time required to perform the actual EVA firmware upgrade is minimal, it is common for offline upgrades to require several hours before and after the actual controller firmware upgrade. " Online upgrade During an online upgrade, the EVA firmware is upgraded on the controllers while they are receiving host I/Os. The controllers resync simultaneously. Hosts as well as applications can remain running. It should be noted that this resync represents a single disruption to I/Os instead of multiple disruptions that are seen with rolling controller firmware upgrades. Host and application timeouts should be considered when doing online controller upgrades. " Rolling upgrade (staggered upgrade) During a rolling upgrade, the controller firmware is upgraded while the controllers are receiving host I/Os. The controllers reboot approximately 5 minutes apart with the controllers failing over LUNs between controllers. The host as well as applications can remain running. Contrary to popular belief, the failover of LUNs from one controller to another requires a brief pause in the servicing of I/Os as the control over the LUNs is passed between controllers. These pauses are repeated two to three times as the workload is failed over between the controllers and then back to the original configuration. In most situations, these repeated failovers are more disruptive to customers than the online controller firmware upgrades. Rolling controller firmware upgrades are not currently supported on the EVA. Untitled Document3 Understanding how the feature works The current EVA firmware versions provide customers the option of using online controller firmware upgrades. This has been used by many customers in a variety of environments to successfully upgrade controller firmware. This includes all major operating systems and major applications including, Microsoft Windows , HP-UX, Linux, Solaris, AIX, Tru64, OpenVMS, Oracle , Microsoft SQL Server, and Microsoft Exchange. The firmware upgrades on the array consists of the following array activities: " Firmware image is downloaded to the controllers. " Firmware image is validated. " Firmware is copied to flash memory. " In an HP StorageWorks Continuous Access system, the peer array is notified that a resync will occur. " Resync the controllers (note that the Fibre Channel lasers are kept on during most of the resync). Reset the program counter (PC). Fast boot (No HW Diagnostics). Memory is initialized. Pre-work for presenting units. Rebuild array configuration from the disk metadata. Present LUNs in stalled state, turns on the ports. Bring cache online. Finish outstanding operations. Un-stall LUNs. " Host I/Os that were not processed by the controllers during the resync are retried by the host/applications and completed. Host I/Os are processed while the firmware image is downloaded and validated as well as when Continuous Access systems are being notified. The controller resync in most upgrade situations can be kept to less than 40 seconds, which is well below the 60-second timeout allowed by many operating systems and applications. Considerations for systems using HP StorageWorks Continuous Access EVA In a Continuous Access environment, the management appliance is often connected on the fabric between the source and destination arrays, where one array is considered to be local to the management appliance. That is, management commands do not need to be passed across the inter-switch links (ISLs). When the fabric is configured this way, and especially when you have low bandwidth ISLs, the transfer of the firmware image across the ISL can cause dropped jobs and degraded replication performance. To avoid problems with the controller firmware download in a Continuous Access environment, and to prevent the management appliance from timing out, the download should only be done from a local management appliance. In other words, you must perform the EVA firmware online upgrade using the instance of Command View EVA that is local to the EVA being upgraded. That is, do not perform code load by sending the firmware image over the ISL. Untitled Document4 In a Continuous Access environment, online upgrades may be done while replicating data. Each replication group can be evaluated separately using the following table where the Continuous Access source and Continuous Access destination refer to the array doing the online download. For example, if the EVA being upgraded is the destination for one or more synchronous replication groups, then the synchronous replication groups must be suspended before the upgrade. All other replication groups are not required to be suspended when performing the EVA firmware online upgrade. Array role Synchronous replication Asynchronous replication Continuous Access source Replication does not need to be suspended. Replication does not need to be suspended. Continuous Access destination Replication should be suspended. This reduces the probability that the resync on the destination impacts workload on the host. Replication does not need to be suspended. For example, if an array is both the source and destination for synchronous replication, the replication groups for which the array is the destination should be suspended. The groups for which the array is a source can either be suspended or not suspended. Host OS-specific guidelines Online controller firmware upgrades are tested in several environments before EVA controller firmware is released to customers. Operating systems that are tested include all operating systems that are included in the list of supported operating systems for a given version of EVA firmware. The following operating systems have received extensive testing and are supported for online controller firmware upgrades using the guidelines described here. Note that this information was correct as of October 2006. For the latest information, refer to the Updating Product Software Guide. Host operating system Version EVA 3000/5000 EVA 4000/6000/8000 Linux Red-Hat Enterprise Linux (AS/ES/WS 3) Supported Supported Linux SUSE Linux SLES 9 SP 3 Supported Supported HP-UX 11.11, 11.23 Supported Supported Windows 2000 SP4 , 2003 SP1 Diskfs only * Diskraw- Supported Solaris 8, 9, 10 * Pending Supported AIX v5.2, v5.3 * Pending Supported Tru64 v5.1b-2, v5.1b3 Supported Supported OpenVMS 7.3, 8.2 Supported Supported Vmware ESX 2.5.3 * Pending Supported * Qualification to be completed by the end of January 2007 It should be noted that online controller firmware upgrade testing is conducted on systems configured according to the OS-specific guideline. Those documents should be consulted for the latest OS configuration settings. Untitled Document5 Note: At the time this paper was written, online controller firmware upgrades had only been tested in cluster environments for Trucluster/OpenVMS cluster configurations. In those environments, the EVA was configured as the cluster boot device. Other clustered environments had not been tested. Cluster testing has been added to future program requirements and will be conducted during the next program cycle. All operating systems are supported for online controller firmware upgrades when these guidelines are followed for both raw and file system devices. Windows The key setting for the Windows operating system is in registry at HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/Disk/TimeoutValue is set to 3c(hex) (or 60 seconds). Ensure that MPIO is installed and both paths are active and have access to the array. Boot devices are supported with online controller firmware upgrades. HP-UX Operating systems timeout values specifications: Sdisk timeout = 30 seconds (default) (LVM) lvol timeout = 0 seconds (default) To check these timeouts in HP-UX: (device within an LVM volume group) pvdisplay /dev/dsk/cxtxdx (lvol physical volume with an LVM group) lvdisplay /dev/vgxx/lvolx To change the timeout: Pvchange -t (seconds) /dev/dsk/cxtxdx Linux Online controller firmware upgrades are supported on systems running Linux. Configuration recommendations for Red Hat and SUSE are as follows. The QLogic Driver parameters : qdepth = 16 port_down_retry_count = 30 login_retry_count = 30 failover = 1 load_balancing = 1 excludemodel = 0x0 auto_restore = 0xA6 Untitled Document6 The QLogic Driver parameters : qdepth = 16 port_down_retry_count = 64 login_retry_count = 30 failover = 1 load_balancing = 1 excludemodel = 0x0 auto_restore = 0xA6 The Emulex Driver parameters : HPELXLPFC=y nodev_timeout=60 qdepth=30 discovery_threads=1 To check or set the parameters in Linux, there is a set_parm executable in the /opt/hp/specific> directory. When executed, the options to change timeout values are presented. Note that the timeout values must be increased for Emulex single path (without multipath support) and QLogic single path being used in the environment. This is not only important for online upgrades but for general data integrity. For details on how to apply the driver setting, refer to the quick connect guide. Online controller firmware upgrades are supported with Linux boot devices. Solaris Solaris supports online controller upgrades with the following driver timeouts: " Sun Drivers (qlc or emlxs) 60 seconds (default) " QLogic (qla2300) 60 seconds (default) " Emulex (lpfc) 60 seconds (manual setting) To check or change the driver timeouts: " Sun Drivers Add the following lines to /etc/system file set sd:sd_io_time = 60 set ssd:ssd_io_time = 60 " QLogic Edit the /kernel/drv/qla2300.conf file and change the hbax-link-down-timeout value to 60 as shown. hba0-link-down-timeout=60; " Emulex Edit the /kernel/drv/lpfc.conf file and change the linkdown-tmo value to 60 as shown. linkdown-tmo=60; For details on setting Solaris driver settings, refer to the Solaris connection guide. At the time this paper was written, cluster testing had not been performed on Solaris. Untitled Document7 Tru64 UNIX Tru64 UNIX online upgrades are supported. Tru64 UNIX has a 60-second timer for media command timeouts. In most cases, up to five retries can occur within the kernel s I/O stack, which results in a 5- to 6-minute window before the I/O is returned to the file system layer. No changes should be made to any Tru64 UNIX settings when performing an online firmware upgrade. Online controller firmware upgrades on Tru64 UNIX included testing in Trucluster four-node environments where the EVA was the cluster boot device. AIX AIX-supported systems using a StorageWorks EVA require the following disk settings for the native multipath drives. For detailed information, see http://www.hp.com/support/manuals. " PR_key_value: N/A Sets the key value for persistent reservations. Persistent reservations are not supported. " Algorithm: fail_over Sets the load balancing algorithm to fail_over. All I/O uses a single path. The remaining paths are in standby mode. The value round_robin is not supported. " hcheck_interval: 60 Sets the path health feature to check each device every 60 seconds. " hcheck_mode: nonactive Specifies the I/O paths monitored by the path health checking feature: nonactive Checks all I/O paths for Failed status, and checks standby paths for Used/Opened devices. " queue_depth :8 Sets the queue depth. " reserve_policy: single path Sets the reserve policy to standard SCSI-2 reservations. " rw_timeout: 60 Sets the read/write timeout to 60 seconds. For details on setting AIX driver settings, refer to the AIX connection guide. OpenVMS OpenVMS online upgrades are supported. Online controller firmware upgrades on OpenVMS included testing in 16-node OpenVMS cluster environments where the EVA was the cluster boot device. Note In clustered environments where the system disks or quorum disks are on the array, the cluster will stop processing requests while the array is resynching. The cluster will become available after the resync. In environments where both shadow disks are on the same EVA or where they are on different arrays in a 2-node cluster, activity will stop during the online controller firmware upgrade. EVA as XP external storage When a StorageWorks EVA is connected to a StorageWorks XP as external storage, the current procedure calls for the XP to be logically disconnected from the EVA before the upgrade and then to logically rediscover the EVA LUNs configured as external storage volumes on the XP when the upgrade is complete. Untitled Document8 Application support for online controller firmware upgrades Applications are typically insulated from the online controller firmware upgrade by the operating system and HBA driver software. For applications this means that when the application is running on supported operating systems that are correctly configured, the online controller upgrades will be processed successfully. Customers need to evaluate their applications to determine if any exist that have requirements that are more stringent relative to I/O timeouts than those of the operating system. If they exist, they should be evaluated to see if the timeouts involved with online controller firmware upgrades meet the requirements of those applications. Customers often ask if online controller firmware upgrades are supported in environments running several key applications including Microsoft Exchange, Microsoft SQL Server, and Oracle version 9i and 10g. These key applications have been tested by the Customer Focused Test team and have been tolerant of an online controller firmware upgrade. Several customers have reported successful online controller firmware upgrades with these key applications as well. Process for performing an online controller firmware upgrade All major array vendors including HP suggest that controller firmware upgrades be done at a time when the intensity of host I/Os is low or reduced when the upgrades are performed. For the EVA it is helpful to understand the ways in which workloads affect the success of the online upgrades. There are three primary areas: " First is the ability of the array to process the upgrade in parallel with host requests. When doing an online controller upgrade in parallel with host I/Os, those two processes compete for internal controller resources with host requests having a higher priority. Therefore, decreasing the host load, by finding a period of time where host demand is the smallest, allows a greater amount of resources to be used to execute the online controller firmware upgrade as quickly as possible. " Second is the array s ability to communicate with Command View EVA. The early steps of the controller firmware upgrade are done by a process that has a lower priority than that of host I/O. Under heavy workloads the process of doing the controller firmware upgrade can be extended beyond the timeouts expected by Command View EVA for the completion of the upgrade process. While the upgrades are typically successful, the timeouts may cause Command View EVA to lose connectivity to the array and ultimately lead to confusion in the upgrade process. " Third is the queuing effect on host requests. While the array is resynching following a firmware download, I/Os can continue to queue on the host. This creates a backlog of requests that must be processed when the array is again able to respond following the upgrade. While the array may resync in less than 60 seconds, a large backlog may require a great deal of time to clear and some host I/Os may experience I/O timeouts. As a result of these factors, for an online controller firmware upgrade to be successful it is recommended that an evaluation of the workload be done and a time for the upgrade be selected when the demand on the array is low. Untitled Document9 Finding an appropriate time for an online controller upgrade Finding an appropriate time is important to the success of the online controller upgrade. If you attempt to upgrade an array that is in the middle of processing a large I/O-intensive batch job, the upgrade will not be successful in that hosts will likely lose connectivity with the array. So finding an appropriate time for the upgrade is very important. HP has found that many customers believe that the arrays are busier than is actually true. One customer, for example, complained that the array was nearly always very busy and that there was never a convenient time to do online controller firmware upgrades. When the array s workload was measured, several time periods where the array was nearly idle were found. Do not take for granted that there is not a good time for online upgrades. EVAPerf Array Performance Data Collection EVAPerf is a good tool for collecting array performance data. This paper is not intended to provide detailed information on EVAPerf and how to format that data. There are, however, several sources for additional information on that topic available at www.hp.com. To check array performance, it is best to collect EVAPerf data over a period of time. The following command would collect all EVAPerf statistics: evaperf all csv cont x dur y ts2 sz array-wwid > foo.csv " csv Makes output Comma Separated. " cont x Where x is the interval. If nothing is entered, the default is 1. " dur y Where y is the duration. If nothing is entered, EVAPEF will run until CTRL-C is entered. " sz array-wwid [array-wwid] Only collect data from the arrays specified in the list, for example, -sz 5000-1FE1-5005-04C0. " -ts2 Use Microsoft Time Format. " > foo.csv redirects output to a file named foo.csv. The most important statistics for evaluating the state of the array for online controller firmware upgrades are the hps or host port statistics; the vd or virtual disk statistics; and the total array loading or array statistics as. The following EVAPerf commands could be used if a reduced data set is desired. evaperf as csv cont x dur y ts2 sz array-wwid > foo.csv More detailed information is obtained using the hps option. evaperf hps csv cont x dur y ts2 sz array-wwid - > foo.csv Example of evaperf with hps option on the EVA 5000 Untitled Document Figure 1. Additional detail is provided when the vd option is used. evaperf vd csv cont x dur y ts2 sz array-wwid > foo.csv Example of evaperf with vd option on the EVA 5000 Figure 2. Using the evaperf data with either the all, hps, or vd flags can be used to calculate the total demand on the array by summing the columns for a given point in time. Note While the vd or virtual disk option for evaperf is a bit harder to read when compared to the other options due to the increased amount of data returned, that additional data is useful in identifying virtual disks that have significant activity at a given point in time. Take for example an array with 72 virtual disks that are nearly idle with the exception of a single virtual disk. To do online upgrades, it may be desirable to shut down the single application that is generating the workload as opposed to doing an offline upgrade. 10 Untitled DocumentArray utilization data analysis When evaluating the performance of the EVA in relation to an online upgrade, it important to note that there are two factors that need to be monitored. In the case where the workload is predominantly small block I/Os, the ability of the array to service workloads is best measured by the number of I/Os processed in a given period of time, which is typically measured in IOPS or I/Os per second. Figure 3 shows an example of an HP StorageWorks 8000 Enterprise Virtual Array (EVA8000) doing an online controller firmware upgrade with an OLTP workload, with an I/O size of 8 KB. Figure 3. DOWNLOAD EVA8000, R1, OLTP 60/40, 8KB 40 PROCS, 32 LUNS, 95% OCCUPANCY, VCS6000 CR0D63, THROUGHPUT IN IO/s050010001500200025003000350040004500500055006000650070007500800085009000950010000TimeCTRL0_IO/sCTRL1_IO/sTOTAL_IO/s Note that in this example the total time that I/Os are not being processed is less than 30 seconds. 11 Untitled DocumentFigure 4 shows the throughput in MB/s of the array servicing the preceding workload. Figure 4. DOWNLOAD EVA8000, R1, OLTP 60/40, 8KB 40 PROCS, 32 LUNS, 95% OCCUPANCY, VCS6000 CR0D63, TRANSFER RATE IN MB/s0102030405060708090TimeCTRL0_MB/sCTRL1_MB/sTOTAL_MB/s Unlike small block transfers, when the workload on the array is dominated by large block I/Os, the workload capacity of the array is best monitored in terms of the throughput as measured in MB/s. Large block sequential workloads are typical of backup and restore operations. A good rule of thumb to determine if the array is servicing predominantly large block versus small block I/Os is to analyze the EVAPerf output and divide the total MB/s by the IOPS, which results in the average transfer size. When this result is less than 16 KB, the workload can be considered predominately small block I/Os. (When available, the EVA s transfer length histograms provide an even better indicator of the type of workload on the array.) After careful examination of several systems doing online upgrades, it is recommended that the demand on the array during an online upgrade be adjusted based on the number of installed disk drives. It is recommend during an online controller firmware upgrade that the demand on the array for the HP StorageWorks 4000/6000/8000 Enterprise Virtual Array (EVA4000/6000/8000) and the HP StorageWorks 3000/5000 Enterprise Virtual Array (EVA3000/5000) be limited to both the number of IOPS and the demand as measured in MB/s (see the chart below for recommendations). For example, an EVA8000 with 168 disk drives should have a demand less than both 70 MB/s and 7000 IOPS and an EVA5000 with 168 disk drives should have a demand less than both 40 MB/s and 4000 IOPS. This will ensure that a healthy array can resync well within the defined resync window. 12 Untitled Document Figure 5. Enterprise Virtual Array (EVA4000/6000/8000) MB/s as a function of the number of disks020406080100120122436485106 127 148 169 190 211 232Number of disks 13 Untitled Document Figure 6. Enterprise Virtual Array (EVA4000/6000/8000) IOPS as a function of the number of disks020004000600080001000012000122436485 106 127 148 169 190 211 232Number of disks 14 Untitled Document Figure 7. Enterprise Virtual Array (EVA3000/5000) MB/s as a function of the number of disks0102030405060122436485106127148169190211232Number of disks 15 Untitled Document Figure 8. Enterprise Virtual Array (EVA3000/5000) IOPS as a function of he number of disks0100020003000400050006000122436485106127148169190211232Number of disks It should be noted that in the lab, online controller firmware upgrades have been completed at much higher workloads. These guidelines were chosen to be very conservative to help ensure the highest probability of success. These guidelines may be increased in the future to reflect these additional test results. 16 Untitled Document17 Approach to upgrading a busy array In some systems there will not be a time period where the array utilization is low enough to comfortably do an online controller firmware upgrade. There are two approaches to resolve this issue. The first approach is to do an offline upgrade. The second approach is to use the EVAPerf data for the virtual disks to identify the active LUNs. When the active LUNs are identified, only those applications that are very active are halted. The rest of the systems may remain online and active during the controller firmware upgrade. This method can greatly reduce the amount of work required by customers to perform an online controller firmware upgrade by reducing the number of hosts and application that need to be quiesced. Example: Array: EVA5000 upgrading from V3.020 to V3.028. 168 disks Hosts: 20 hosts. HP-UX, Solaris, and Windows During the online upgrade window, one LUN has a lot of activity. Investigation reveals that a host using that LUN is doing backups. The identified host s backup application is taken offline reducing the array demand to a point where online controller upgrades can be done for the other hosts. This creates a significant cost and time savings for customers since they do not need to take their entire environment offline. This method of reducing array demand by taking only the active host offline was extremely effective at a major telecommunications company when that customer did online controller upgrades for nearly 100 arrays. Before the online controller firmware upgrade After a time period is selected for the online controller firmware upgrade, but before the upgrade itself, follow the instructions in the Upgrading Product Software Guide to prepare for the upgrade. Verify that online controller firmware upgrades are supported for this particular upgrade. Check other SAN infrastructure to ensure that that hosts, switches, HBAs, and other pieces of the SAN infrastructure are at the appropriate supported versions. Special note should be taken to validate the following: " Verify multipath software is configured correctly (see http://www.hp.com/support/manuals and this document) " Verify HBA driver levels (see http://www.hp.com/support/manuals) " Verify OS Patch levels (see http://www.hp.com/support/manuals) " Verify paths to external storage " Test failover between paths " Verify file systems are mounted " Reinitialize paths that are failed " Verify clustering is properly configured " Verify that multipath software is configured for clustering properly. Special care should be taken to ensure that the timeouts previously noted are set properly. " Ensure that proper system backups exist Untitled Document18 The online controller firmware upgrade " Immediately before the online controller firmware upgrade, verify that the array is operating in an optimal state (no warnings or critical failures) and resolve any issues indicated by the array. (Disk groups that are in a leveling active or pending state are acceptable.) " Use EVAPerf to verify that the demand on the array is at the expected levels. To check the demand on the array, collect the EVAPerf information with both the hps (Host Port) and the vd (virtual disk) flags. If all the preparation steps have been completed and the preceding indicators are positive, then proceed with the online controller firmware upgrade. For additional details on how to execute the controller firmware upgrade, see http://www.hp.com/support/manuals. Verify the online controller firmware upgrade " Use Command View EVA to check the overall status of the EVA to ensure that there are no problems. " Verify host connectivity to the EVA. " Restart any hosts or application that may have been stopped to reduce the workload to the recommended ranges. " Verify that host and applications are available. Online disk drive firmware upgrades Beginning with XCS version V6.000 for the EVA4000/6000/8000 product and VCS version V3.100 for the EVA3000/5000, HP has added the capability to do online, data in place, and disk drive firmware upgrades. Customer experiences with online controller firmware upgrades Telecommunications A major telecommunications company (referred to here as TelecomX ) is a very large EVA customer with hundreds of arrays. TelecomX decided that it needed to upgrade nearly 100 EVA5000 arrays from VCS V3.020 and V3.025 to V3.028. The magnitude of this upgrade made scheduling of offline upgrades complicated and painful to the business. The customer s management team requested that online upgrades be used to reduce the downtime of systems during the upgrade process. The support team started by initiating contact with the lab. The team explained its situation and worked with the lab to generate guidelines and processes that would make this effort successful. (Those guidelines are the basis of the procedures in this paper.) The team then monitored the arrays to find appropriate times to do the online controller firmware upgrade and found that even systems that were supposedly fully utilized generally had times where workloads were at a level where the online upgrades could be done successfully. When the team found an upgrade window, an upgrade was scheduled. At the time of the upgrade, the team monitored array throughput to validate that the workload was within the acceptable guidelines. If the demand on the array was at an acceptable level, the team proceeded with the upgrade. Untitled Document19 With nearly 100 arrays to upgrade, there were a few issues the most significant being that occasionally servers would lose connectivity to the array. While many of the faint of heart might have panicked, the TelecomX team very carefully and methodically investigated the problem. The investigation revealed that the third-party multipathing driver that was being used had been configured with paths to the array that no longer existed, so that when the driver detected the array resynching, it attempted a failover to a path that was nonexistent. In subsequent upgrades, the team added the verification of failover paths to the preparation process and the problem was eliminated. There were several arrays where upgrade windows could not be found where the workload fell off to the desired levels. In those cases, the team used two different approaches to upgrade those arrays. The first was to schedule an offline upgrade. The second was to identify the systems that were generating the higher workloads and then doing a limited shutdown of the systems to reduce the workload to an acceptable level. Again this reduced the overall downtime for TelecomX and allowed the highest levels of data availability. TelecomX uses Windows, HP-UX, and Solaris hosts. Applications include Microsoft Exchange, Microsoft SQL Server, Oracle, and several custom applications. Software development Another successful customer is a major software developer in the U.S. Pacific Northwest. In the first half of 2006, there were at least six cases where online controller firmware upgrades were performed in the software development datacenter. Five of the EVAs were used for internal applications and one EVA was used to host Exchange data. The most interesting case is the upgrade of the Exchange array. That controller firmware upgrade was performed at 4:00 PM during a business day after very careful review of that EVA to evaluate the demand on the array at the time of the upgrade. All the upgrades were successful. Note During the same time period, two EVAs were successfully expanded from 2C6D to 2C12D configurations while host I/Os were being processed. World leader in computer chip manufacturing A world leader in computer chip manufacturing is another longtime EVA customer with EVA arrays being used at several facilities. The company s experience with online upgrades is very different from the others previously described. The division that was recently visited had been using online upgrades as its preferred method of controller upgrades for a number of releases. The division had been using online upgrades from the early V3 versions of firmware. When asked about its experiences, the division indicated that online controller firmware upgrades work in its environment and they where part of all of their upgrade plans. This customer was one of the first to note the lack of online controller firmware upgrades from XCS V5.031 to V5.100. They were a big part of the reason that the lab developed the V5.032 bridge release. This computer chip manufacturer uses EVAs to host data for several systems including Oracle 9i and Oracle 10g. Untitled Document 2007 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. Oracle is a registered US trademark of Oracle Corporation, Redwood City, California. UNIX is a registered trademark of The Open Group. 4AA0-9102ENW, January 2007
