|
GPFS 2.3 Tuning Points [영문] 자료입니다.
GPFS 2.3 Tuning Points
2006. 04.28
Total Availability Services
GTS, IBM
목 차
1.1.4. Memory Usage Summary. 6
1.4.1. Using Direct I/O on a file in GPFS FileSystem.. 7
1.4.2. Restriping a GPFS Filesystems. 8
- The GPFS pagepool is used to cache user data and file system metadata. The pagepool mechanism allows GPFS to implement read as well as write requests asynchronously. Increasing the size of pagepool increases the amount of data or metadata that GPFS may cache without requiring synchronous I/O. The default value for pagepool is 64 MB. The maximum GPFS pagepool size is 8 GB. The minimum allowed value is 4 MB. On Linux systems, the maximum pagepool size is one half the physical memory in the machine. The amount of memory available for GPFS pagepool on a particular node may be restricted by the operating system and other software running on the node. The optimal size of the pagepool depends on the needs of the application and effective caching of its re-accessed data. For systems where applications access large files, reuse data, benefit from GPFS prefetching of data, or have a random I/O pattern, increasing the value for pagepool may prove beneficial. However, if the value is set too large, GPFS will not start. See the General Parallel File System: Problem Determination Guide and search on The GPFS daemon will not come up.
- used for read-ahead and write-behind and for reuse the cached data
- To change the pagepool to 100 MB: mmchconfig pagepool=100M
- #mfsadm dump pgalloc
- The total number of different files that can be cached at one time. Every entry in the file cache requires some pageable memory to hold the content of the file’s inode plus control data structures. This is in addition to any of the file’s data and indirect blocks that might be cached in the page pool. The total amount of memory required for inodes and control data structures can be calculated as: maxFilesToCache × 2.5 KB where 2.5 KB = 2 KB + 512 bytes for an inode Valid values of maxFilesToCache range from 0 to 1,000,000. For systems where applications use a large number of files, of any size, increasing the value for maxFilesToCache may prove beneficial. This is particularly true for systems where a large number of small files are accessed. The value should be large enough to handle the number of concurrently open files plus allow caching of recently used files. The default value is 1000.
- #mmfsadm dump all > $(hostname -s).dumpall (확인 가능)
- #mmfsadm dump fs > $(hostname -s).fsout
- This parameter sets aside additional pageable memory to cache attributes of files that are not currently in the regular file cache. This is useful to improve the performance of both the system and GPFS stat( ) calls for applications with a working set that does not fit in the regular file cache. The memory occupied by the stat cache can be calculated as: maxStatCache × 176 bytes Valid values of maxStatCache range from 0 to 1,000,000. For systems where applications test the existence of files, or the properties of files, without actually opening them (as backup applications do), increasing the value for maxStatCache may prove beneficial. The default value is: 4 × maxFilesToCache
- The total amount of memory GPFS uses to cache file data and metadata is arrived at by adding pagepool to the amount of memory required to hold inodes and control data structures (maxFilesToCache × 2.5 KB), and the memory for the stat cache (maxStatCache × 176 bytes) together. The combined amount of memory to hold inodes, control data structures, and the stat cache is limited to 50% of the physical memory on a node running GPFS. With an inode size of 512 bytes, the default 4-to-1 ratio of maxStatCache to maxFilesToCache would result in a maximum 250,000 stat cache entries and 65,000 file cache entries. During configuration, you can specify the maxFilesToCache, maxStatCache, and pagepool parameters that control how much cache is dedicated to GPFS. These values can be changed later, so experiment with larger values to find the optimum cache size that improves GPFS performance without affecting other applications. The mmchconfig command can be used to change the values of maxFilesToCache, maxStatCache, and pagepool. The pagepool parameter is the only one of these parameters that may be changed while the GPFS daemon is running. A pagepool change occurs immediately when using the -i option on the mmchconfig command. Changes to the other values are effective only after the daemon is restarted.
- memory used to cache file attributes (required by the stat() function); this speeds up applications which work with the status of the files but do not necessary open those files (like backup applications).
- The inode cache contains copies of inodes for open files and for some recently used files which are no longer open.
- The stat cache contains enough information to respond to inquiries about the file and open it, but not enough information to read from it orwrite to it.
- The stat cache entries are kept for Recently accessed files or Directories recently accessed by a number of stat() calls
- pagepool is used for the storage of data and metadata in support of I/O operations. With some access patterns, increasing the amount of pagepool storage may increase I/O performance for file systems with the following operating characteristics:
: Heavy use of writes that can be overlapped with application execution
: Heavy reuse of files and sequential reads of a size such that prefetch will benefit the application
- The maxMBpS option determines the maximum amount of I/O in Mb that can be submitted by GPFS per second. If the default value is not adjusted accordingly it will affect GPFS performance. Note that setting this number too high can have an adverse effect on performance of the system since overrunning the capabilities of the I/O bus or network adapter tends to drastically degrade throughput. This number is normally set after empirical study to determine your nodes I/O bandwidth limits. The default value is 150 MB/s. To change maxMBpS to 500 MB/s:
- mmchconfig maxMBpS=500
- The ipqmaxlen network option should be considered when configuring for GPFS. The ipqmaxlen parameter controls the number of incoming packets that can exist on the IP interrupt queue. Since both GPFS and IBM Virtual Shared Disk use IP, the default of 100 is often insufficient. The suggested setting is 512. To set the value to 512 after the next reboot, issue the command: no -r -o ipqmaxlen=512
- This value will persist throughout subsequent reboots.
- The Direct I/O caching policy can be set for files in a GPFS file system by specifying the -D option on the mmchattr command.
- This caching policy bypasses file cache and transfers data directly from disk into the user space buffer, as opposed to using the normal cache policy of placing pages in kernel memory.
- Applications with poor cache hit rates or very large I/Os may benefit from the use of Direct I/O.
- For example, Oracle Database uses Direct I/O
- Direct I/O bypasses the GPFS caching mechanisms.
- Direct I/O may also be specified by supplying the O_DIRECT file access mode on the open() of the file.
------------------------------------------------------------------------------------------
#mmlsattr /home/test1
replication factors
metadata(max) data(max) file [flags]
------------- --------- ---------------
1 ( 1) 1 ( 1) /home/test1
#mmchattr -D yes /home/test1
mmlsattr /home/test1
replication factors
metadata(max) data(max) file [flags]
------------- --------- ---------------
1 ( 1) 1 ( 1) /home/test1 [directio]
----------------------------------------------------------------------------------------------------------------------------------------
- Writing data into a GPFS file system correctly stripes the file. However,if you have added disks to a GPFS file system that is seldom updated,you can restripe the file system to achieve maximum performance using the mmrestripefs command.
- Re-striping offers the opportunity to specify useful options in addition to rebalancing (the -b option).
- Re-replicate (-r option) ensures that all data and metadata is properly replicated. If you use replication, this option is useful to protect against additional failures after losing a disk.