White PapersMany applications today, in both industry and in the public sector, require greater capability to read, write, and analyze large amounts of data with high throughput. Seismic processing in the oil and gas industry, credit analysis and prospect identification in the finance and banking industries, and image processing and digital video rendering in the media and entertainment industries are just a few examples. To compete in this environment, a processing system requires both high-performance processing capability and high-throughput I/O infrastructure in order to provide fast access to the large volumes of data required.
IBM develops systems with the goal of balanced performance, because it is of little value to have high-performance processing in a system if the I/O subsystem is not capable of getting data into the system to be processed. In the IBM BladeCenter chassis, the I/O architecture is designed to provide high I/O bandwidth to complement the high-performance processing capabilities built into the HS20 blade server.
bctigpfs_ds4800wp120706.doc Page 1 Sequential I/O Performance of GPFS on HS20 blades and IBM System Storage DS4800 December 2006 Authors: Bill Hartner Brent Jones James Wang Tamas Vilaghy Untitled Documentbctigpfs_ds4800wp120706.doc Page 2 1 Abstract The objective of this study was to demonstrate the sequential I/O capability of General Parallel File System (GPFS) for Linux on HS20 blades inside the IBM BladeCenter blade server with the System Storage" DS4800 storage subsystem using a 4 Gbps Fibre Channel interconnect. This study shows that the GPFS file system on HS20 Blades and the DS4800 storage subsystem using eight EXP710 expansion units is capable of about 1300 MB/sec file create, and about 1600 MB/sec file read, throughput using sequential I/O. For the purposes of this paper, Megabyte or MB is defined as 106 bytes when reporting results. This paper details the GPFS file system configuration used to achieve maximum sequential I/O for both file read and file create performance, as well as the SAN configuration. We also discuss differ-ent GPFS parameters and their effects on sequential I/O performance. 2 Introduction Many applications today, in both industry and in the public sector, require greater capability to read, write, and analyze large amounts of data with high throughput. Seismic processing in the oil and gas industry, credit analysis and prospect identification in the finance and banking industries, and image processing and digital video rendering in the media and entertainment industries are just a few ex-amples. To compete in this environment, a processing system requires both high-performance proc-essing capability and high-throughput I/O infrastructure in order to provide fast access to the large volumes of data required. IBM develops systems with the goal of balanced performance, because it is of little value to have high-performance processing in a system if the I/O subsystem is not capable of getting data into the system to be processed. In the IBM BladeCenter chassis, the I/O architecture is designed to provide high I/O bandwidth to complement the high-performance processing capabilities built into the HS20 blade server. 2.1 Study goals In this benchmark, eight BladeCenter chassis with HS20 blades were assembled to provide the base for evaluating the sequential I/O throughput using GPFS. Many client environments today that include a Storage Area Network ( SAN ) require a file system with the ability to do large block sequential I/Os at very high data rates. We configured the Blade-Center with 4 Gbps McData switch modules. All BladeCenter switch modules were used to attach a DS4800 with eight EXP710 drawers of disks using two external 4 Gbps McData SAN32M Fibre Channel switches. While it is possible to look at an I/O subsystem design to determine its theoretical I/O capabilities, such analyses do not always present an accurate picture of actual system performance. It is in-tended that the GPFS sequential I/O throughput data presented in this paper can be used to help answer questions such as: What is the GPFS sequential throughput of a single HS20 blade with a 4 Gbps Qlogic expansion card? What is the maximum aggregate GPFS sequential throughput with a single DS4800 storage server? How many blades are required to achieve this maximum? Determine how performance scales as additional blades are added to the I/O test, from one blade up to two blades, four blades, eight blades, 14 blades (one BladeCenter chassis), and then to 28 blades (two BladeCenter chassis). What are the configurations and tuning parameters used by GPFS to achieve this maximum? What are the DS4800 array configurations to achieve this maximum? Untitled Documentbctigpfs_ds4800wp120706.doc Page 3 3 GPFS file system The General Parallel File System (GPFS) is the IBM high-performance cluster file system which has been available on AIX 5L" clusters since 1998 and on Linux clusters since 2001. GPFS allows file data access from all nodes in the cluster by providing a global name space for files. Applications can efficiently access files using standard UNIX file system interfaces without modification. GPFS runs on 32-bit as well as 64-bit AIX 5L and Linux operating systems. The file system size can be dynamically increased or decreased by addition or deletion of logical disks. GPFS is available as: GPFS for AIX 5L on POWER" GPFS for Linux on IBM AMD processor-based servers and IBM eServer" xSeries servers GPFS for Linux on POWER The major features and intended benefits of GPFS is summarized Table 1. Table 1: GPFS features and benefits Feature Intended Benefit Highly Scalable File System Designed for a cluster environment, GPFS can support hundreds of nodes over 1000 disks comprising hun-dreds of terabytes of storage Although relatively small environments can benefit from GPFS, it allows expansion of file system throughput and capacity as service demands and data volumes increase. Parallel File System (Data Striping) Divides individual files into multiple blocks and stores these blocks across multiple storage nodes in parallel Improves performance and reliability by eliminating the bottlenecks that typically arise when an entire file resides on a single node or single disk. Automatic Journaling (Logging) of Metadata Logs all file system metadata (file system transaction records) to shared disks and can replay this data on demand Enable rapid file system recovery to a consistent state in the event of a node failure. Concurrent File Access and Block-level Locking Multiple clients (applications or uses) can access (differ-ent parts of) a single file simultaneously; sophisticated lock management prevents collisions/ conflicts to pro-vide data consistency Enable multiple users, programs or nodes to access a single file simultaneously accelerates process-ing for parallel applications that share data and eliminates the need for multiple copies of data, help-ing to reduce data storage and management costs. High Availability Automatically recovers from events that would normally interrupt data availability Support the ability of applications and users to con-tinue operation without interruption Client-side Data Caching Keeps recently written or read data blocks in client memory Improve performance by eliminating repetitive proc-essing of requests for frequently accessed files hides write latency by using write behind Large Blocksize Options Data blocks of 16KB, 64KB, 256KB, 512KB, 1024KB, 2048KB, or 4096KB can be used for different applica-tions. Large blocks allow more data to be written or read in a single I/O operation. Enable more efficient use of disk bandwidth and space, particularly important for RAID devices Access Pattern Recognition and Deep Pre-fetch Detects access patterns as sequential, random, fuzzy sequential, or strided, pre-fetches striped data in parallel and buffers it at the application node. Increase bandwidth by enabling disks to read and write simultaneously. Data Protection Both user data and metadata (file system transaction data) can be replicated in the GPFS installation Provides reliable access to data and metadata Untitled Documentbctigpfs_ds4800wp120706.doc Page 4 4 Test configuration This section describes the hardware configuration of the HS20 blades, Fibre Channel adapters, Fibre Channel switch modules, and the DS4800. The hardware used for this GPFS test, as shown in Figure 1, has the following components. Eight BladeCenter chassis; each BladeCenter has: " Two McData 4 Gbps 20 port switch modules, feature code 32R1835. Each switch module has 14 internal 2/4 Gbps ports and 6 external 1/2/4 Gbps auto-sensing ports. For this benchmark, only two external ports per switch module are connected to external IBM SAN32M switches. " Two Cisco Gigabit Switch Modules, feature code 13N2286. One switch module with all four ports Gigabit trunked with MTU 9000 is used for application access. All four ports of the other switch module were also Megabit trunked with MTU 1500 and were used as the sys-tem administration network. 100 HS20 blades, each blade has: " Two CPUs, 4GB memory, two internal hard disk drives, one QLogic 4 Gbit Standard Fibre Channel Expansion Card, feature code 26R0884 " Red Hat Enterprise Linux(RHEL) 3.0 Advanced Server(AS) Update 4 " GPFS 2.3.0 with PTF9 " RDAC version 09.01.B5.21 One CSM server with two CPUs, 4GB memory and six 36GB internal disks. The operating sys-tem is RHEL 3.0 AS with Update 4. Figure 1: Test configuration Untitled Documentbctigpfs_ds4800wp120706.doc Page 5 4.1 The BladeCenter The IBM BladeCenter is a 7U modular chassis that is capable of housing up to 14 blade servers. The BladeCenter chassis allows individual blades to share resources such as power, switch, manage-ment and blower modules. The front view of the BladeCenter chassis is shown in Figure 2. Figure 2: BladeCenter front view 4.2 BladeCenter I/O subsystem The midplane, as shown in Figure 3, has two similar sections that provide redundant functionality. The blade servers plug into the front of the midplane. Each blade server has two connectors; one connects to the upper section and one to the lower section of the midplane. Figure 3: Blade server to Fibre Channel connection view Untitled Documentbctigpfs_ds4800wp120706.doc Page 6 4.3 IBM TotalStorage SAN32M switches The IBM TotalStorage SAN32M-2 fabric switch has 32 ports. It is designed to help address the needs of medium-size and enterprise SAN environments. It can be used to create a wide range of high performance SAN solutions, from simple single-switch configurations to larger multi-switch con-figurations which support fabric connectivity and advanced business continuity capabilities. The SAN32M-2 can be used in a clustered server environment. Separate SAN switches enable two separate SAN fabrics, which are desirable as a means to help avoid single points of failure in the SAN. In this benchmark, a high-availability solution was created with two switches. The configura-tion, as shown in Figure 1, supports eight BladeCenter chassis, each with dual BladeCenter 4 Gbit Fibre Channel switches cross-connected to two SAN32M switches which are cross-connected to a dual-controller DS4800 storage system. The SAN32M-2 switch provides 4 Gbps performance on all ports when paired with storage system hardware that supports 4 Gbps throughput. Each port auto-negotiates to 4, 2 Gbps or 1 Gbps, full duplex. Up to 256 Gbps aggregate throughput is possible with 32-port configurations. Since both DS4800 and the BladeCenter Fibre Channel switch modules are 4 Gbps capable, each link supports 4 Gbps throughput. 4.4 IBM System Storage DS4800 system The IBM System Storage DS4800 disk storage system supports a high performance 4 Gbps Fibre Channel interface for increased host connectivity to deliver necessary bandwidth for high throughput applications. It has a dual controller storage system that can support up to 224 disks and provides a 4 Gbps Fibre Channel host interface that is backward compatible with 1 Gbps and 2 Gbps Fibre Channel host connections at well. The DS4800 is designed for data-intensive applications that demand connectivity provided by eight 4 Gbps host channels designed to provide about 1600 MB/s of sustained sequential read bandwidth, or about 1300 MB/s sequential write bandwidth for high-throughput applications through the eight channels directly attached to the host servers or connected to a Fibre Channel storage area network (SAN). 4 Gbps Fibre Channel systems are well suited for applications that need to quickly transfer large amounts of data such as remote replication across a SAN, database in memory; streaming video on demand; medical imaging; data mining and data warehousing; and large databases supporting online transaction processing (OLTP). For this test, a single DS4800 was configured with 4GB cache and eight EXP710 storage expansion units and 112 disks (8X14). Storage Manager V9.15 was used to communicate with the DS4800. Untitled Documentbctigpfs_ds4800wp120706.doc Page 7 Loop configuration To explore the DS4800 s full I/O capability, eight EXP710 s were used to support the required data throughput as shown on Figure 4. Each EXP710 has 14 x 73GB 15K rpm disk drives. Figure 4: DS4800 loop configuration In this configuration, the address of the eight EXP710 drawers from top to bottom is deliberately set as 10, 11, 24, 25, 30, 31, 44, and 45. EXP710 10 and 11 are wired through controller A port 4 to controller B port 1. EXP710 24 and 25 are wired through controller A port 3 to controller B port 2. EXP710 30 and 31 are wired through controller A port 2 and controller B port 3. EXP710 44 and 45 are wired through controller A port 1 and controller B port 4. The DS4800 was configured with two EXP710 per drive loop. For the four EXP710 performance tests, the RAID arrays were created on one EXP710 in each loop, the other four EXP710 remained part of the each loop, but were not accessed during the test. Untitled Documentbctigpfs_ds4800wp120706.doc Page 8 5 Test tools and test scenarios The following sections document both the test tools and the test configurations. 5.1 Test tools The sequential I/O performance tests used the gpfsperf file system benchmark program. The gpfsperf tool is shipped with GPFS and can be found in the /usr/lpp/mmfs/samples/perf directory. The tool is similar to iozone and ior benchmarks. Each configuration was tested with 1, 2, 4, 8, 14, and 28 blades. Each blade included in a test ran one instance of gpfsperf using one thread. Both sequential file create and sequential file read was tested with either a 20 GB file for four EXP710 tests or 32GB file for eight EXP710 tests. Each blade included in a test created or read a file located in a per blade subdirectory. The record size for gpfsperf was equal to the GPFS block size tested. Note: The processors have a 1MB L2 cache. In some cases, a gpfsperf record size of 512KB and 1024KB did not perform as well on a single node as smaller application I/O size. The smaller I/O size may have performed better because a smaller I/O size does not displace as much data from the L2 cache as large I/O does when copying from GPFS pagepool to gpfsperf buffer and allows for other data required for code execution to remain in the L2 cache. This has not been verified. 5.2 Test scenarios Eight scenarios were tested. The configurations for Test 1 through 4 included eight EXP710 and used RAID5 (4+P). Test 5 through 7 included eight EXP710 and used RAID5 (8+P). Test 8 included four EXP710 and used RAID5 (4+P). The following sections document the scenarios that we tested. 5.2.1 Test 1: RAID5 (4+P) drives selected by Storage Manager The Test 1 configuration included eight EXP710 enclosures. The GPFS file system was created on 20 RAID5 (4+P) arrays for GPFS data and two RAID1 (1+1) arrays for GPFS metadata arrays. The physical drives for each array were automatically selected by Storage Manager. The GPFS file sys-tem capacity was 5.3TB. This configuration provided enclosure protection for both RAID5 and RAID1 because no single array had more than one drive on any single enclosure. For this configuration, GPFS block sizes of 512KB and 1024KB were tested. 5.2.2 Test 2: RAID5 (4+P) drives selected manually, enclosure protection The Test 2 configuration included eight EXP710 enclosures. The GPFS file system was created on 20 RAID5 (4+P) for GPFS data and two RAID1 (1+1) for GPFS metadata arrays. The physical drives for each array were manually selected. The GPFS file system capacity was 5.3TB. This configuration provided enclosure protection for both RAID5 and RAID1 because no one array had more than one drive on any one enclosure. The objective of manual physical drive selection for each array was to isolate the physical drives for the controller A managed arrays to four drive side ports and the physical drives managed by control-ler B to the other four drive side ports and to balance the number of drives across each drive side Fi-bre Channel port. The number of physical drives accessed through each drive side port was uneven: Four of the drive side ports handled 13 data drives each, and the other four ports handled 12 data drives each for a total of 100 drives. For this configuration, GPFS block sizes of 256KB, 512KB, 1024KB, and 2048KB were tested. Untitled Documentbctigpfs_ds4800wp120706.doc Page 9 5.2.3 Test 3: RAID5 (4+P) drives selected manually, no enclosure protection The Test 3 configuration included eight EXP710 enclosures. The GPFS file system was created on 20 RAID5 (4+P) for GPFS data and two RAID1 (1+1) for GPFS metadata arrays. The physical drives for each array were manually selected. The GPFS file system capacity was 5.3TB, the same as Test 1 and Test 2. This configuration did not provide enclosure protection for all RAID5 because some RAID5 arrays had more than one drive in an enclosure. The objective of manual physical drive selection for each array was to further isolate the physical drives managed by each controller to a subset of drive side ports. One half of the arrays managed by controller-A were accessed through two drive side ports and the other one half through the other two ports. The same strategy was used for controller-B. In order to configure the RAID arrays this way, enclosure protection was not possible for all RAID5 arrays. The number of physical drives accessed through each drive side port was the same as Test 2: Four of the drive side ports handled 13 physical data drives each, and the other four ports handled 12 data drives each for a total of 100 drives. For this configuration, a GPFS block size of 1024KB was tested. 5.2.4 Test 4: RAID5 (4+P) drives selected manually, equal drives per port The Test 4 configuration included eight EXP710 enclosures. The GPFS file system was created on 16 RAID5 (4+P) for GPFS data and two RAID1 for GPFS metadata arrays. The physical drives for each array were manually selected. The file system capacity was 4.2TB This configuration did not provide enclosure protection for all RAID5 arrays because some arrays had more than one drive on an enclosure. The objective of manual physical drive selection for each array is the same as Test 3. With a reduced number of RAID5 (4+P), the number of drives accessed through each of the eight drive side Fibre Channel ports was balanced with ten drives accessed through each port. For this configuration, a GPFS block size of 1024KB was tested. 5.2.5 Test 5: RAID5 (8+P) drives selected by Storage Manager The Test 5 configuration included eight EXP710 enclosures. The GPFS file system was created on 12 RAID5 (8+P) for GPFS data and metadata. The physical drives for each RAID5 array were auto-matically selected by Storage Manager. The file system capacity was 6.3TB. There were no meta-data only drives. This configuration did not provide enclosure protection for all RAID5 arrays because some arrays had more than one drive on an enclosure. For this configuration, a GPFS block size of 2048KB was tested. 5.2.6 Test 6: RAID5 (8+P) drives selected manually, no enclosure protection The Test 6 configuration included eight EXP710 enclosures. The GPFS file system was created on 12 RAID5 (8+P) for GPFS data and metadata arrays. The physical drives for each array were manu-ally selected. The GPFS file system capacity was 6.3TB. There were no metadata drives. This configuration did not provide enclosure protection for all RAID5 arrays because some arrays had more than one drive on an enclosure. The objective of manual physical drive selection for each array was the same as Test 2 RAID5 (4+P), that is, to isolate the physical drives for the controller-A managed RAID to four drive side ports Untitled Documentbctigpfs_ds4800wp120706.doc Page 10 and the physical drives managed by controller-B to the other four drive side ports and to balance the number of drives across each drive side Fibre Channel port. The number of physical drives accessed through each drive side port was uneven: Four of the drive side ports handled 14 physical data drives each, and the other four ports handled 13 physical data drives each for a total of 108 drives. For this configuration, GPFS block sizes of 256KB, 512KB, 1024KB, 2048KB, and 4096KB were tested. 5.2.7 Test 7: RAID5 8+P drives selected manually, equal drives per port The Test 7 configuration included eight EXP710 enclosures. The GPFS file system was created on eight RAID5 (8+P) for GPFS data and two RAID1 (1+1) for GPFS metadata arrays. The physical drives for each array were manually selected. The file system capacity was 4.2TB. This configuration did not provide enclosure protection for all RAID5 arrays because some RAID5 ar-rays had more than one drive on an enclosure. The objective of manual physical drive selection for each array is the same as Test 6. With a reduced number of RAID5 (8+P), the number of drives accessed through each of the eight drive side Fibre Channel ports was balanced with nine drives accessed through each port for a total of 72 drives. For this configuration, a GPFS block size of 2048KB was tested. 5.2.8 Test 8: RAID5 (4+P) drives selected manually, 4 enclosures The previous seven tests all used eight EXP710 enclosures. The Test 8 configuration included only four EXP710 enclosures. The GPFS file system was created on 10 RAID5 (4+P) for GPFS data and the two RAID1 (1+1) for GPFS metadata arrays. The physical drives for each array were manually selected. The GPFS file system capacity was 2.6TB. This configuration did not provide enclosure protection for all RAID5 arrays because some RAID5 ar-rays had more than one drive on an enclosure. The objective of manual physical drive selection for each array was the same as for the previous tests. With a reduced number of RAID5 (4+P) arrays, the number of drives accessed through each of the eight drive side Fibre Channel ports was balanced with five drives accessed through each port for a total of 50 drives. For this configuration, GPFS block sizes of 256KB, 512KB, 1024KB, and 2048KB were tested. Untitled Documentbctigpfs_ds4800wp120706.doc Page 11 6 Tuning parameters This section documents the tuning parameters used for the large file sequential I/O performance measurements. 6.1 GPFS tuning pagepool = 512M The pagepool parameter is a per node parameter and specifies the size of the GPFS cache used to cache both file data and metadata on each node. The default pagepool size is 64M. A GPFS page-pool size of 512M was tested. A GPFS pagepool of 256M (or smaller) in size may have provided the same level of performance. maxMBpS = 1600 The maxMBpS parameter is a per node parameter and specifies an estimate of how many mega-bytes of data can be transferred per second into or out of a single node. The default maxMBpS value is 150. A GPFS maxMBpS of 1600 was tested. Since each HS20 blade has 2 x 4 Gbps FC capable of ~800 MB/s bandwidth, a maxMBpS value of 800 is required so each HS20 blade throughput would not be limited by GPFS. However, by doubling this value to 1600, single node performance improved for some test cases. maxblocksize = 4096K The maxblocksize parameter is a global parameter and specifies the maximum block size that can be used when creating a GPFS file system. The default maxblocksize value is 1024K. The max-blocksize was set to 4096K so GPFS file systems with a block size of 4096K could be tested. GPFS file system block size The GPFS block size is a per file system parameter and specifies the size of the data blocks. The GPFS block size is specified when creating the file system (mmcrfs B block-size). The default block size for a file system is 256KB. GPFS 2.3 supports block sizes of 16KB, 64KB, 256KB, 512KB, 1 MB, 2 MB, and 4 MB. A variety of block sizes were tested and specified later in this document. See the special considerations for matching RAID5 stripe size to GPFS block size in the DS4800 tuning section. GPFS block allocation map type The default is cluster for a GPFS cluster with 8 or fewer nodes, or a file system with eight or fewer disks, and scatter in all other cases. This setting determines which algorithm is used for allocating data blocks. When allocating blocks for a given file, GPFS follows the round-robin algorithm for se-lecting the next disk, and selects the location of the block on the disk based on the block allocation map type. When using the scatter type, the location of each block is chosen randomly. When using the cluster type, for a given file, GPFS attempts to allocate blocks in clusters on each disk with blocks adjacent to each other within each cluster. The scatter allocation approach is appropriate in most scenarios since it provides more consistent file system performance by averaging out perform-ance variations due to block location. For many disk subsystems, the location of data on the disk closer to the inner or the outer disk edge, has a substantial effect on performance. In some cases, dependent on the disk subsystem type, the cluster allocation approach may provide better disk per-formance. However, the benefits of clustered block allocation generally diminish when the number of nodes in the cluster or the number of disks in a file system increases and the free space in the file system becomes more fragmented. All results in this paper were performed on file systems created with -j scatter because there were always more than 8 nodes and always more than 8 disks in the GPFS cluster. Untitled Documentbctigpfs_ds4800wp120706.doc Page 12 QLogic driver tuning qla2xmaxqdepth = 3 The qla2xmaxqdepth parameter specifies the maximum number of outstanding I/O to each device. This parameter is used so the I/O queue depth of the Storage Device is not exceeded. The DS4800 I/O queue limit is 2048. The queue depth for each device was set to 3 by modifying the qla2xxx op-tions line in modprobe.conf. With a queue depth of three for each logical drive, and some test con-figurations having as many as 22 logical drives, that would allow up to 34 blades to have three I/O outstanding to each logical drive (2048 / 22 / 3 = 34). Up to 28 blades were tested. The value used for configurations that differ from the configuration tested will depend on (a) the queue depth of the storage server (b) the number of logical drives and (c) the number of nodes. 6.2 DS4800 tuning The DS4800 tuning is dependent on the RAID configuration. Configurations with only dataAndMetadata NSDs: For configurations that use only RAID5 for GPFS dataAndMetadata NSDs, the DS4800 cache block size was 16KB and all logical drives had both read and write cache disabled. Configurations with a mix of data and metaData NSDs: For configurations that use RAID5 for GPFS data NSDs and RAID1 for GPFS metadata NSDs, the DS4800 cache block size was 16KB and the RAID5 logical drives had both read and write cache disabled and the RAID1 logical drives had read, write, and mirror cache enabled with a read ahead multiplier of zero. RAID5 Segment Size = (GPFS Block Size/Number RAID5 data drives) The segment size of the RAID5 drives were configured so the GPFS block size was equal to the RAID5 stripe size. This is especially important for write operations to RAID5. The objective is to allow the DS4800 to calculate the RAID5 parity from the data contained in the I/O and avoid reading from disk to calculate parity (read-modify-write). For example, when testing a RAID5 (4+P) with a segment size of 128KB, the GPFS file system was created with a block size of 512KB (128KB * 4). Since GPFS does prefetch and write behind in block size chunks, the RAID5 parity can be calculated from the data contained in the I/O. RAID1 Metadata Only Segment Size = 512K The recommended segment size for RAID1 metadata only drives is 512K. However, for the tests that were conducted where metadata only drives were included in the configuration, the file systems with a 256KB or 512KB block size used a segment size of 16KB and the file systems with a 1024KB, 2048KB, or 4096KB block size used a segment size of 32KB. DS4800 Default Host Type = #5 Linux with ADT disabled The host type configured at the DS4800 was #5 Linux. ADT was disabled via a Storage Manager script since the RDAC driver was used. 6.3 Kernel tuning Large I/O No Kernel tuning was applied, however, a Linux Kernel Large I/O patch is recommended when using GPFS block size of 1MB or greater. By default, the Linux 2.6 Kernel shipped with the various Linux distributions has limited large I/O capability in terms of the maximum number of scatter gather entries in an I/O. The Linux kernel Large I/O patch increases the number of scatter gather entries per I/O re-quired for driving larger block I/O slabs for driving GPFS large I/O (1, 2, 4MB) down to the Linux Block I/O layer. For this testing effort, GPFS was not restarted more than one time after a node was Untitled Documentbctigpfs_ds4800wp120706.doc Page 13 rebooted. This avoided the fragmentation of the physical memory backing the GPFS I/O buffer pool (pagepool) and I/O as large as 4MB could be handled by the default number of scatter gather entries allowed in an I/O. Untitled Documentbctigpfs_ds4800wp120706.doc Page 14 7 Test results The following sections document the large file sequential I/O performance test results for the eight RAID configurations specified in Test Scenarios section. For some configurations, a series of GPFS block sizes were tested including 256KB, 512KB, 1024KB, 2048KB, and 4096KB. Note: Each table displays the create and read throughput values for the indicated number of blades in MB/s (106). Note: Single node sequential file read and create performance was limited by CPU. The gpfsperf benchmark program was executed with one thread. 7.1 Test 1: RAID5 (4+P) drives selected by Storage Manager The performance test results for Test 1 are shown in Table 2. Test 1 establishes a baseline performance for RAID5 (4+P) with 512KB and 1024KB block size for Storage Manager automatic selection of drives for each array. The Test 1 results will be compared to Test 2 results in the next section. Table 2: RAID5 (4+P) drives selected by Storage Manager GPFS Options Number of Blades Sequential File Create (MB/s) Number of Blades Sequential File Read (MB/s) Blocksize KB 1 2 4 8 14 28 1 2 4 8 14 28 512 602 763 772 778 779 780 747 981 1017 1026 1023 1027 1024 624 799 799 792 791 789 661 981 1033 1037 1036 1034 7.2 Test 2: RAID5 (4+P) drives selected manually, enclosure protec-tion The performance test results for Test 2 results are shown in Table 3. Table 3: RAID5 (4+P) drives selected manually, enclosure protection GPFS Options Number of Blades Sequential File Create (MB/s) Number of Blades Sequential File Read (MB/s) Blocksize KB 1 2 4 8 14 28 1 2 4 8 14 28 256 554 780 799 799 801 808 635 818 1026 1180 1202 1205 512 602 1145 1145 1140 1140 1142 799 1205 1431 1510 1512 1519 1024 624 1145 1145 1145 1142 1146 661 1321 1527 1535 1541 1544 2048 648 1126 1126 1130 1126 1128 673 1321 1527 1553 1556 1559 Test 2 observations: " Test 2 results for 512KB and 1024KB block size file create are 43 to 50 % better than Test 1 for two or more Blades. " Test 2 results for 512KB and 1024KB block size file read are 23 to 49 % better than Test 1 for two or more Blades. Untitled Documentbctigpfs_ds4800wp120706.doc Page 15 Figure 5: Test 1 vs. Test 2, file create 7.3 Test 3: RAID5 (4+P) drives selected manually, no enclo-sure protection The performance test results for Test 3 are shown in Table 4. Table 4: RAID5 (4+P) drives selected manually, no enclosure protection GPFS Options Number of Blades Sequential File Create (MB/s) Number of Blades Sequential File Read (MB/s) Blocksize KB 1 2 4 8 14 28 1 2 4 8 14 28 1024 624 1184 1205 1194 1200 1184 661 1321 1527 1535 1541 1544 Test 3 observations: " Test 3 with a configuration where each controller uses two drive side ports to access each RAID5 performed about the same as Test 2 with a configuration using more than two drive side ports to access each RAID5. Test-1 vs. Test-2 file create020040060080010001200140012481428Number of bladesTest-1-512KBTest-1-1024KBTest-2-512KBTest-2-1024KBUntitled Documentbctigpfs_ds4800wp120706.doc Page 16 Test 2 vs. Test 3 1024KB block size050010001500200012481428Number of bladesTest-2-CreateTest-3-CreateTest-2-ReadTest-3-Read Figure 6: Test 2 vs. Test 3, 1024KB GPFS block size 7.4 Test 4: RAID5 (4+P) drives selected manually, equal drives per port The performance test results for Test 4 are shown in Table 5. Table 5: RAID5 (4+P), drives selected manually, equal drives per port GPFS Options Number of Blades Sequential File Create (MB/s) Number of Blades Sequential File Read (MB/s) Blocksize KB 1 2 4 8 14 28 1 2 4 8 14 28 1024 613 1145 1174 1174 1172 1172 673 1296 1527 1589 1598 1600 Test 4 observations: " Test 4 with 16 RAID5 (4+P) performed about the same as Test 3 with 20 RAID5 (4+P). Untitled Documentbctigpfs_ds4800wp120706.doc Page 17 Figure 7: Test 3 (20 RAID5 (4+P)) vs. Test 4 (16 RAID5 (4+P)), 1024KB GPFS block size 7.5 Test 5: RAID5 (8+P) drives selected by Storage Manager The performance test results for Test 5 are shown in Table 6. Test 5 establishes a baseline performance for RAID5 (8+P) with a 2048KB block size for Storage Manager automatic selection of drives for each array. The Test 5 results will be compared to Test 6 results in the next section. Table 6: RAID5 (8+P) drives selected by Storage Manager GPFS Options Number of Blades Sequential File Create (MB/s) Number of Blades Sequential File Read (MB/s) Blocksize KB 1 2 4 8 14 28 1 2 4 8 14 28 2048 554 772 784 784 783 782 624 859 910 910 910 910 Test 3 (20) 4+P RAID5 vs. Test 4 (16) 4+P RAID5 1024K block size 02004006008001000120014001600180012481428Number of bladesTest-3-CreateTest-4-CreateTest-3-ReadTest-4-ReadUntitled Documentbctigpfs_ds4800wp120706.doc Page 18 7.6 Test 6: RAID5 (8+P) drives selected manually, no enclosure pro-tection The performance test results for Test 6 are shown in Table 7. Table 7: RAID5 (8+P) drives selected manually GPFS Options Number of Blades Sequential File Create (MB/s) Number of Blades Sequential File Read (MB/s) Blocksize 1 2 4 8 14 28 1 2 4 8 14 28 256 386 470 539 556 558 558 409 494 624 740 801 800 512 591 838 922 931 937 942 635 838 1002 1215 1269 1272 1024 624 1227 1237 1255 1258 1257 661 1272 1446 1518 1521 1527 2048 635 1272 1260 1260 1258 1258 673 1347 1510 1544 1547 1549 4096 648 1249 1260 1260 1258 1260 687 1374 1493 1502 1493 1498 Test 6 observations: " Test 6 results for 2048KB block size file create are 60 to 64 % better than Test 5 for two or more Blades. " Test 6 results for 2048KB block size file read are 56 to 70 % better than Test 5 for two or more Blades. Figure 8: Test 5 Storage Manager vs. Test 6 manual, 2048KB GPFS block size Test 5 Storage Manager vs. Test 6 Manual2048KB block size02004006008001000120014001600180012481428Number of bladesTest-5-CreateTest-6-CreateTest-5-ReadTest-6-ReadUntitled Documentbctigpfs_ds4800wp120706.doc Page 19 7.7 Test 7: RAID5 (8+P) drives selected manually, equal drives per port The performance test results for Test 7 are shown in Table 8. Table 8: RAID5 (8+P) drives selected manually, equal drives per port GPFS Options Number of Blades Sequential File Create (MB/s) Number of Blades Sequential File Read (MB/s) Blocksize KB 1 2 4 8 14 28 1 2 4 8 14 28 2048 648 1272 1272 1296 1293 1291 673 1321 1544 1579 1588 1590 Test 7 observations: " Test 7 with eight RAID5 (8+P) performed about the same as Test 6 with 12 RAID5 (8+P). Figure 9: Test 6: 12 RAID5 (8+P) vs. Test 7 eight RAID5 (4+P), 2048KB GPFS block size 7.8 Test 8: RAID5 (4+P) drives selected manually, four enclosures The performance test results for Test 8 are shown in Table 9. Table 9: RAID5 (4+P) drives selected manually, no enclosure protection GPFS Options Number of Blades Sequential File Create (MB/s) Number of Blades Sequential File Read (MB/s) Blocksize KB 1 2 4 8 14 28 1 2 4 8 14 28 256 351 397 407 409 409 408 364 447 554 618 626 626 512 564 671 692 704 709 707 564 692 867 1022 1058 1067 1024 564 894 1022 1066 1074 1070 631 1022 1209 1374 1424 1438 2048 631 1074 1086 1074 1077 1072 671 1342 1431 1455 1452 1455 Test 6 (12) 8+P RAID5 vs. Test 7 (8) 4+P RAID5 2048KB block size02004006008001000120014001600180012481428Number of bladesTest-6-CreateTest-7-CreateTest-6-ReadTest-7-ReadUntitled Documentbctigpfs_ds4800wp120706.doc Page 20 Test 8 observations: " Test 8 results for 2048KB block size and 10 RAID5 (4+P) performed within 10% of Test 2 with a 2048KB block size and 20 RAID5 (4+P). Figure 10: Test 2 (20 RAID5 (4+P)) vs. Test 8 (10 RAID5 (4+P)), 2048KB GPFS block size " The results for a large block size of 2048KB are 179 to 300% better than a small block size of 256KB. Test 2 (20) 4+P RAID5 vs. Test 8 (10) 4+P RAID52048KB block size 02004006008001000120014001600180012481428Number of bladesTest-2-CreateTest-8-CreateTest-2-ReadTest-8-ReadUntitled Documentbctigpfs_ds4800wp120706.doc Page 21 Figure 11: Test 8, 256KB block size vs. 2048KB GPFS block size Test 8 256KB vs. 2048KB block ize 0200400600800100012001400160012481428Number of blades256KB-Create2048KB-Create256KB-Read2048KB-ReadUntitled Documentbctigpfs_ds4800wp120706.doc Page 22 8 Conclusions A GPFS file system created on one DS4800 storage subsystem using eight EXP710 is capable of delivering up to 1590 MB/s sequential file read and 1296 MB/s sequential file create as demon-strated by Test 7. The best performance in these tests was achieved by manually selecting the drives for the arrays instead of allowing Storage Manger to select the drives. By manually selecting the drives for each array, one can achieve a better balance of I/O across the DS4800 drive side ports. Contention on the ports can be reduced by selecting disks for each array that are confined to a fewer number of ports. The peak performance was achieved using between four and eight blades. A GPFS file system created on one DS4800 storage subsystem using four EXP710 is capable of de-livering 1455 MB/s sequential file read and 1086 MB/s sequential file create as demonstrated by Test 8. The peak performance was achieved using between two and four blades. GPFS for Linux on a single HS20 blade is capable of 799 MB/s sequential file read and 648 MB/s sequential file create as demonstrated in Test 2. In both cases, the single threaded gpfsperf was CPU bound. It is not clear why sequential file create used more CPU resulting in lower throughput. Multiple threads were not tested. Creating the file system with larger block sizes can improve sequential I/O performance when there are fewer LUNs. Test 8, using four enclosures and ten RAID5 (4+P), showed that sequential I/O per-formance for 2048KB block size was 179 to 300 % better when compared to 256KB and 17-86% bet-ter when compared to 512KB block size. Untitled Documentbctigpfs_ds4800wp120706.doc Page 23 Appendix A.1 Storage layouts and array configurations Test 1: RAID5 (4+P) drives selected by Storage Manager A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 MD-A A 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp10 A1 B1 B2 A3 A4 A5 B5 B6 A7 A8 A9 B9 B10 exp11 A1 B1 B2 B3 A4 A5 B5 B6 B7 A8 A9 B9 B10 exp30 A1 A2 A3 B3 B4 A5 A6 A7 B7 B8 A9 A10 MD-A exp31 B1 A2 A3 B3 B4 B5 A6 A7 B7 B8 B9 A10 MD-A B1 B2 B3 B4 B5 B6 B7 B8 B9 B10 MD-B B 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp24 A1 A2 B2 B3 A4 A5 A6 B6 B7 A8 A9 A10 B10 exp25 A1 A2 B2 B3 B4 A5 A6 B6 B7 B8 A9 A10 B10 exp44 B1 A2 A3 A4 B4 B5 A6 A7 A8 B8 B9 A10 MD-B exp45 B1 B2 A3 A4 B4 B5 B6 A7 A8 B8 B9 B10 MD-B Figure 12: 20 RAID5 4+P data, two RAID1 metadata arrays, 8 hot spares, 80 data drives Test 2: RAID5 (4+P) drives selected manually, enclosure protection A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 MD-A A 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp10 A1 B2 A2 B4 A4 B5 A5 B7 A7 B8 A9 B10 A10 Spare exp11 A1 B2 A2 B3 A4 B5 A6 B7 A7 B8 A9 B10 MD-A Spare exp30 A1 B2 A3 B3 A4 B5 A6 B6 A7 B8 A9 B10 A10 Spare exp31 A1 B1 A3 B3 A4 B5 A6 B6 A8 B8 A9 B9 MD-A Spare B1 B2 B3 B4 B5 B6 B7 B8 B9 B10 MD-B B 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp24 B1 A1 B3 A3 B4 A5 B6 A6 B8 A8 B9 A9 MD-B Spare exp25 B1 A2 B3 A3 B4 A5 B6 A6 B7 A8 B9 A10 B10 Spare exp44 B1 A2 B2 A3 B4 A5 B6 A7 B7 A8 B9 A10 MD-B Spare exp45 B1 A2 B2 A4 B4 A5 B5 A7 B7 A8 B9 A10 B10 Spare Figure 13: 20 RAID5 4+P data, two RAID1 metadata arrays, 8 hot spares, 80 data drives Untitled Documentbctigpfs_ds4800wp120706.doc Page 24 Test 3: RAID5 (4+P) drives selected manually, no enclosure protection A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 MD-A A 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp10 A1 A2 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A9 Spare exp11 A1 A2 A3 A4 A3 A4 A5 A6 A7 A8 A9 A10 Spare MD-A exp30 A1 A2 A3 A4 A5 A6 A5 A6 A7 A8 A9 A10 Spare A10 exp31 A1 A2 A3 A4 A5 A6 A7 A8 A7 A8 A9 A10 MD-A Spare B1 B2 B3 B4 B5 B6 B7 B8 B9 B10 MD-B B 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp24 B1 B2 B1 B2 B3 B4 B5 B6 B7 B8 B9 B10 B9 Spare exp25 B1 B2 B3 B4 B3 B4 B5 B6 B7 B8 B9 B10 Spare MD-B exp44 B1 B2 B3 B4 B5 B6 B5 B6 B7 B8 B9 B10 Spare B10 exp45 B1 B2 B3 B4 B5 B6 B7 B8 B7 B8 B9 B10 MD-B Spare Figure 13: 20 RAID5 4+P data, two RAID1 metadata arrays, 8 hot spares, 80 data drives Test 4: RAID5 (4+P) drives selected manually, equal drives per port A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 MD-A A 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp10 A1 A2 A1 A2 A3 A4 A5 A6 A7 A8 Spare exp11 A1 A2 A3 A4 A3 A4 A5 A6 A7 A8 Spare MD-A exp30 A1 A2 A3 A4 A5 A6 A5 A6 A7 A8 Spare exp31 A1 A2 A3 A4 A5 A6 A7 A8 A7 A8 MD-A Spare B1 B2 B3 B4 B5 B6 B7 B8 B9 B10 MD-B B 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp24 B1 B2 B1 B2 B3 B4 B5 B6 B7 B8 Spare exp25 B1 B2 B3 B4 B3 B4 B5 B6 B7 B8 Spare MD-B exp44 B1 B2 B3 B4 B5 B6 B5 B6 B7 B8 Spare exp45 B1 B2 B3 B4 B5 B6 B7 B8 B7 B8 MD-B Spare Figure 14: 20 RAID5 4+P data, two RAID1 metadata arrays, 8 hot spares, 80 data drives Untitled Documentbctigpfs_ds4800wp120706.doc Page 25 Test 5: RAID5 (8+P) drives selected by Storage Manager A1 A2 A3 A4 A5 A6 A 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp10 A1 A1 A1 A1 A1 A1 A1 A1 A1 B1 B1 B1 B1 B1 exp11 B1 B1 B1 B1 A2 A2 A2 A2 A2 A2 A2 A2 A2 B2 exp30 A4 A4 A4 A4 A4 A4 A4 B4 B4 B4 B4 B4 B4 B4 exp31 B4 B4 A5 A5 A5 A5 A5 A5 A5 A5 A5 B5 B5 B5 B1 B2 B3 B4 B5 B6 B 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp24 B2 B2 B2 B2 B2 B2 B2 B2 A3 A3 A3 A3 A3 A3 exp25 A3 A3 A3 B3 B3 B3 B3 B3 B3 B3 B3 B3 A4 A4 exp44 B5 B5 B5 B5 B5 B5 A6 A6 A6 A6 A6 A6 A6 A6 exp45 A6 B6 B6 B6 B6 B6 B6 B6 B6 B6 spare spare spare spare Figure 15: 12 RAID5 8+P data and metadata arrays, four hot spares, 96 data drives Test 6: RAID5 (8+P) drives selected manually, no enclosure protection A1 A2 A3 A4 A5 A6 A 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp10 A1 A1 A1 A2 A2 A3 A3 A4 A4 A5 A5 A5 A6 A6 exp11 A1 A1 A2 A2 A2 A3 A3 A4 A4 A5 A5 A6 A6 Spare exp30 A1 A1 A2 A2 A3 A3 A3 A4 A4 A5 A5 A6 A6 A6 exp31 A1 A1 A2 A2 A3 A3 A4 A4 A4 A5 A5 A6 A6 Spare B1 B2 B3 B4 B5 B6 B 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp24 B1 B1 B1 B2 B2 B3 B3 B4 B4 B5 B5 B5 B6 B6 exp25 B1 B1 B2 B2 B2 B3 B3 B4 B4 B5 B5 B6 B6 Spare exp44 B1 B1 B2 B2 B3 B3 B3 B4 B4 B5 B5 B6 B6 B6 exp45 B1 B1 B2 B2 B3 B3 B4 B4 B4 B5 B5 B6 B6 Spare Figure 16: 12 RAID5 8+P data and metadata arrays, four hot spares, 96 data drives Untitled Documentbctigpfs_ds4800wp120706.doc Page 26 Test 7: RAID5 (8+P) drives selected manually, equal drives per port A1 A2 A3 A4 MD-A A 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp10 A1 A3 A1 A3 A1 A3 A1 A3 A1 MD-A exp11 A1 A3 A1 A3 A1 A3 A1 A3 A3 exp30 A2 A4 A2 A4 A2 A4 A2 A4 A2 MD-A exp31 A2 A4 A2 A4 A2 A4 A2 A4 A4 B1 B2 B3 B4 MD-B B 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp24 B1 B3 B1 B3 B1 B3 B1 B3 B1 MB-B exp25 B1 B3 B1 B3 B1 B3 B1 B3 B3 exp44 B2 B4 B2 B4 B2 B4 B2 B4 B2 MD-B exp45 B2 B4 B2 B4 B2 B4 B2 B4 B4 Figure 17: 8 RAID5 8+P data and two metadata arrays, 64 data drives Test 8: RAID5 (4+P) drives selected manually, four enclosures A1 A2 A3 A4 A5 MD-A A 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp10 A1 A3 A1 A3 A1 A3 A1 A3 A1 A3 A5 A5 A5 MD-A exp11 exp30 A2 A4 A2 A4 A2 A4 A2 A4 A2 A4 A5 A5 Spare MD-A exp31 B1 B2 B3 B4 B5 MD-B A 1 2 3 4 5 6 7 8 9 10 11 12 13 14 exp24 B1 B3 B1 B3 B1 B3 B1 B3 B1 B3 B5 B5 Spare MD-B exp25 exp44 B2 B4 B2 B4 B2 B4 B2 B4 B2 B4 B5 B5 B5 MD-B exp45 Figure18: Ten RAID5 4+P data and metadata arrays, two hot spares, 40 data drives Untitled Documentbctigpfs_ds4800wp120706.doc Page 27 A.2 References " IBM eServer xSeries and BladeCenter Server Management, SG24-6495 http://www.redbooks.ibm.com " GPFS InfoCenter http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.gpfs.doc/gpfsbooks.html " "Implementing 4 Gb/s Fibre Channel Technology" http://www.engenio.com/pdf/whitepapers/4Gb_technology_white_paper.pdf " 'IBM System Storage DS4800" http://www.ibm.com/servers/storage/disk/ds4000/ds4800/index.html " GPFS Homepage http://www.ibm.com/servers/eserver/clusters/software/gpfs.html " GPFS: A Parallel File System, SG24-5165 http://publib-b.boulder.ibm.com/Redbooks.nsf/RedbookAbstracts/sg245165.html?Open A.3 Acknowledgements Thanks to the following people for their contribution to this project: " Tom Henebery pSeries Benchmark Center, IBM " Bill Holland BladeCenter I/O architect, IBM " Jean-Michael Carr - Engenio " Kurt Sulzbach - Infrastructure team, Industry Solutions and PoC Centers, IBM Untitled Documentbctigpfs_ds4800wp120706.doc Page 28 IBM Corporation 2006 IBM Corporation Systems and Technology Group Route 100 Somers, New York 10589 Produced in the United States of America December 2006 All Rights Reserved This document was developed for products and/or services offered in the United States. IBM may not offer the products, features, or services discussed in this document in other countries. The information provided in this document is distrib-uted "AS IS" without any warranty, either express or implied. IBM EXPRESSLY DISCLAIMS any warran-ties of merchantability, fitness for a particular purpose OR NONINFRINGEMENT. IBM shall have no respon-sibility to update this information. IBM products are warranted according to the terms and conditions of the agreements (e.g., IBM Customer Agreement, State-ment of Limited Warranty, International Program License Agreement, etc.) under which they are pro-vided. IBM is not responsible for the performance or interoperability of any non-IBM products discussed herein. The information may be subject to change without notice. Consult your local IBM business contact for information on the products, features and services available in your area. All statements regarding IBM future directions and intent are subject to change or withdrawal without notice and represent goals and objectives only. IBM, the IBM logo, AIX 5L, BladeCenter, eServer, System Storage, TotalStorage and xSeries are trade-marks or registered trademarks of International Busi-ness Machines Corporation in the United States or other countries or both. A full list of U.S. trademarks owned by IBM may be found at: http://www.ibm.com/legal/copytrade.shtml. UNIX is a registered trademark of The Open Group in the United States, other countries or both. Linux is a trademark of Linus Torvalds in the United States, other countries or both. Other company, product, and service names may be trademarks or service marks of others. IBM hardware products are manufactured from new parts, or new and used parts. In some cases, the hardware product may not be new and may have been previously installed. Regardless, our warranty terms apply. Photographs show engineering and design models. Changes may be incorporated in production models. Copying or downloading the images contained in this document is expressly prohibited without the written consent of IBM. This equipment is subject to FCC rules. It will comply with the appropriate FCC rules before final delivery to the buyer. Information concerning non-IBM products was ob-tained from the suppliers of these products or other public sources. Questions on the capabilities of the non-IBM products should be addressed with those suppliers. All performance information was determined in a controlled environment. Actual results may vary. Performance information is provided AS IS and no warranties or guarantees are expressed or implied by IBM. Buyers should consult other sources of informa-tion, including system benchmarks, to evaluate the performance of a system they are considering buying. When referring to storage capacity, TB, GB and MB refer to 1,000,000,000,000, 1,000,000,000 and 1,000,000 bytes respectively. Accessible capacity depends on formatting and RAID implementation choices and may be less than total aggregate capacity of unformatted disk. The IBM home page on the Internet can be found at: http://www.ibm.com. The IBM BladeCenter home page on the Internet can be found at: http://www.ibm.com/systems/bladecenter. CLW03002-USEN-00
