Re:오라클9i Real Application Cluster에 사용할 공유디스크에 대해(linux에서)

민천사님. 다음을 참조하시면 도움이 되실듯 합니다. 그리고 disk 를 share 할때 별도의 network card 를 이용하면 두 node 간의 IPC message 송수신의 tuning 에 도움이 됩니다. Subject: Setting up Real Application Cluster (RAC) environment on Linux - Single node Type: BULLETIN Status: PUBLISHED Content Type: TEXT/PLAIN Creation Date: 01-DEC-2001 Last Revision Date: 10-MAR-2003 DISCLAIMER ---------- The following steps represent way to install and configure Oracle9i Real Application Clusters on Linux - Single node. The use of a single computer to run two or more instances against a RAC database is unsupported and should be used for testing or educational purposes only. Please see the following note for setting up a 2 (or more) node RAC database on Linux: Note 184821.1 Step-By-Step Installation of RAC on Linux PURPOSE ------- This document is intended for people who would wish to test Real Application Cluster (RAC) technology on a common hardware. Basically this article is based on the same assumptions and limitations discussed in Note 136077.1 (Setting up Oracle Parallel Server environment on Linux - Single node [1]), where we argumented the setup for Oracle Parallel Server (OPS v817). This article will adopt the key concepts of the above article and extend them to Oracle RAC 9.0.1 on Linux port. In particular here we will simplify furthermore the steps necessary to setup a 'RAC-in-a-Box' environment quickly and easily. SCOPE & APPLICATION ------------------- The intended audience, are technical people with a good experience on the Linux operating environment and who are also introduced to OPS/RAC technology. Here we will detail the steps necessary to setup the environment and part of the theory behind these steps. TITLE ----- Setting up Real Application Cluster (RAC) on Linux - Single node CONTENTS -------- 1) Introduction 1.1) DLM, GCS, GES : [2] 2) Architecture 2.1) Two-node / Two-instance RAC Archtecture 2.2) Single-Node / Two-instance RAC Architecture 3) Kernel and System Configuration 3.1) Bill of Materials [3] 3.2) - Prepare your disk for RAW partitions - simplyfied 3.2.1) Cluster Manger RAW Partition can be on FS 3.2.2) GSD Partition (optional) 3.3) Prepare your kernel for RAC single node 3.3.1) KERNEL PARAMETERS relevant to RAC single-node 3.3.2) Hack softdog.c 3.3.3) Kernel memory requirements 4) install Oracle software 5) Setup Oracle Cluster Manager components 5.1) load the softdog module in your kernel at startup 5.2) Create the NM File 5.3) Configure Oracle Cluster Manger (OCM) 5.4) Troubleshooting Oracle Cluster Manger (OCM) 5.4.1) Check filesystem rights 5.4.2) OCM does not start 5.4.3) OCM did not restart 5.4.4) is OCM running ? 5.4.5) how to stop OCM 6) Clustered database setup 6.1) Create RAC database 6.1.1) Database on File System , easyer to setup on FS 6.1.2) create RAC related catalogs 6.1.3) Create extra RLog Threads 6.1.4) Create extra UNDO tablespaces 6.2) setup for use SPFILE 6.3)Configure SPFILE for RAC 7) Startup both the instances in parallel mode 7.1) Start OCM 7.2) setup to unix environment 7.3) startup the instances 8) Switching between RAC and NON-RAC 8.1) Activating RAC feature 8.2) De-activacting RAC feature 8.3) Check if RAC is enabled or not 9) Conclusions References 1) Introduction ------------ 1.1) DLM, GCS, GES : [2] ------------------- The term Distributed Lock Manager (DLM) is obsolete. The background processes that were formerly aggregated under the DLM in previous cluster software products still exist, but the DLM no longer manages resources in Real Application Clusters. The Global Cache Service (GCS) and Global Enqueue Service (GES) now handle the management functions that had been handled by the DLM in earlier Oracle cluster software releases. Briefly, Global Cache Services (GCS) synchronization layer that allows lock of resources among all the instances partecipating to the RAC database in a distributed fashion. This layer is part of the RDBMS kernel and mantains an inter-instance consistent view of the database of all the locks and enqueues (GES). Most of these tasks are carried out by RAC specific background processes, if compared with [1] Sect 1, we will notice that the bkg. process names are changed and some of the old process roles has been merged in other : LMON - (Lock Monitor Process) -> Global Enqueue Service Monitor The background Global Enqueue Service Monitor (LMON) monitors the entire cluster to manage global resources. LMON manages instance and process expirations and the associated recovery for the Global Cache and Global Enqueue Services. In particular, LMON handles the part of recovery associated with global resources. LMON-provided services are also known as cluster group services (CGS). LMS - The Global Cache Service Processes (LMSn) handle remote Global Cache Service messages. Current Real Application Clusters software provides for up to 10 Global Cache Service Processes. The number of LMSn processes varies depending on the amount of messaging traffic among nodes in the cluster. The LMSn processes handle the acquisition interrupt and blocking interrupt requests from the remote instances for Global Cache Service resources. For cross-instance consistent read requests, LMSn creates a consistent read version of the block and sends it to the requesting instance. LMSn also controls the flow of messages to remote instances. LMD - The Global Enqueue Service Daemon (LMD) is the lock agent process that manages enqueue manager service requests for Global Cache Service enqueues to control access to global enqueues and resources . The LMD process also handles deadlock detection and remote enqueue requests . Remote enqueue requests are requests originating from another instance . LCK - The LCK process has been simplified . Much of the LCK functionality in previous products is now handled by Global Cache Service Processes (LMSn). In Real Application Clusters, there is just one remaining LCK process . It is used for non-Cache Fusion operations such as instance requests and cross-instance call operations. BSP - No longer exist in 9i - Blocks shipping is carried out by GCS layers provided by LMSn We point you to [1] Sect.1 for further informations regarding OSD layers in linux (WD,CM and NM) that are still valid here . 2) Architecture ------------ 2.1) Two-node / Two-instance RAC Archtecture --------------------------------------- Shared Disks, Database on RAW partitions / \ / \ Node 1 Node 2 --------------- --------------- | SGA Instance A| |SGA Instance B | |---------------| |---------------| | |pmon|__ | | |pmon|___ | | |smon |lgwr|| | |smon |lgwr|| | ---|dbwr| | | ---|dbwr| | | ---- | | ---- | | ------------| |------------ | | |lmon lms0 | |lmon lms0 | | lipc | | lmd | nipc | lmd | | lipc ------- GCS/GES <=================> GCS/GES ------- | | |------------| |------------| | | | | | CM | | CM | | | +------> <===================> <------+ | | |___________ | | ___________| | | | | | NM | | NM | | | +------> <===================> <------+ | | |___________ | |____________| | | ------> WDT | | WDT <------ | | | | | | --------------- --------------- |////kernel/////| |////kernel/////| --------------- --------------- '-' lipc : local inter-process communication '=' nipc : networked inter-process communication (via tcp/ip) Note : Typically the shared disks box is wired via SCSI or Optical Fiber to the main boards, this introduce compatibility issues related to the drivers used and to the controllers used. Please refer to HW certification matrices for that. 2.2) Single-Node / Two-instance RAC Architecture ------------------------------------------- Local Disk, Database on FILE SYSTEM | single Node | --------------------------------------------- | SGA Instance A| |SGA Instance B | |--------------- ---------------| | |pmon|__ |pmon|___ | | |smon |lgwr| |smon |lgwr|| | ---|dbwr| ---|dbwr| | | ---- (localhost) ---- | | __________ nipc __________ | | | GCS/GES | <=========> | GCS/GES | | | | lmon | | lmon | | | | lmd lms0 | | lmd lms0 | | | |__________| |__________| | | ____|________________________|_____ | | | | | | | | | +--> CM svcs <--+ | | | |____|________________________|_____| | | | | | | | | | +--> NM svcs <--+ | | | |____|________________________|_____| | | | | | | | | | --> WDT svcs <-- lipc | | --------------------------------------------- |//////////////// kernel ////////////////////| --------------------------------------------- '-' lipc : local inter-process communication '=' nipc : networked inter-process communication (via tcp/ip) Note : this kind of configuration is not so exposed to compatibility issues related to various SCSI controller and drivers that may limit the implementations . Note : we will use normal ext2 file system, see below for further details (3.2) . 3) Kernel and System Configuration -------------------------------- we tested on Dell latitude pIII 500Mhz - 12Gb IDE - 256MB ram - CDROM) 3.1) Bill of Materials [3] ------------------- Intel x86 X-windowed and networked box SuSE 7.2,Red Hat 7.1 distributions Related Linux kernel source 2.4 rpms (kernel-2.4.XX-YY.i386.rpm) Oracle 9.0.1.0. for Linux Intel 32bit Operating System Linux : kernel 2.4 Operating System Patches: Patch 4 or later Raw should be included in above distr. System Libraries : GNU Lib C 2.2,make-3.79,binutils-2.10 Disk Space Requirements : ~2GB for an average install, Over 2.5GB for Default installation . 3.2) - Prepare your disk for RAW partitions - (simplyfied) -------------------------------------------------------- In [1] we showed that every database file was positioned on a dedicated RawDevice (RD) , and this common in a multinode RAC environments to override dangerous unsyncronized File System caches . In a single-node configuration we can remove the RD constraint because FS caches coerency issues does not arise us when the FS caches involved is just one . 3.2.1) Cluster Manger RAW Partition can be on FS ----------------------------------------- Node Monitor needs a raw partition to use as syncronization/voting aera among nodes partecipating the cluster . This partition can be a file in force of the single node hipotesys and the size should be at least : 4 + [(the number of nodes in the cluster) * 4] KB [1] We will use 1M for our scopes (see 5.2) 3.2.2) GSD Partition (optional) ------------------------ If you plan to use 'SRVCTL' utility [4] then you will need a RawDevice partition of about 100M to let GSD (Global Services Daemon) work [5]. Infact this daemon checks (with an iostat primitive) that the file descriptor points really to a RD . infact in : $ORACLE_HOME/srvm/config there is : 'srvConfig.loc' that contains : '/u01/ora/oradata/RAC_Cluster_Configuration_File' that should point to your RD . if you issue for example : % gsd % srvconfig -version will raise -> "oracle.ops.mgmt.rawdevice.RawDeviceException: PRKR-1020 : reading of version information failed, /u01/ora/oradata/RAC_Cluster_Configuration_File is not a raw device file, No such file or directory" To avoid this you must provide a true Raw Partition . 3.3) Prepare your kernel for RAC single node ---------------------------------------- In [1] we discussed in this section how to build the kernel and which parameters define in order to allow OPS work . Here we will remove some constraints in order to let your RAC environment be more usable for the scopes of this document . - we assume that you have kernel sources in place and you are familiar with linux kernel setup (see [1] Sect-3.3 for hints) 3.3.1) KERNEL PARAMETERS relevant to RAC single-node --------------------------------------------- 'Character Devices' -> 'Watchdog Cards' : CONFIG_WATCHDOG 'Watchdog Timer Support' = 'y' CONFIG_SOFT_WATCHDOG 'Software Watchdog' = 'm' CONFIG_WATCHDOG_NOWAYOUT 'Disable watchdog shutdown on close' = 'n' (*) ----------------- *Important Note* : ----------------- RAC multinode needs 'y' [1] , infacts, the default watchdog behaviour (which you get if you say N here) is to stop the timer if the process managing it closes the file /dev/watchdog. It's always remotely possible that this process might get killed. If you say Y here, the watchdog cannot be stopped once it has been started . In our single-node environment is not unfrequent that 'watchdogd' hangs when OCM crash. In these cases we have to stop and restart, for this reason we use 'n' in order to avoid us to be forced to bounce the entire box when 'watchdogd' stucks . 3.3.2) Hack softdog.c -------------- The software watchdog 'softdog' that we use in our environment have a pourpose in life : reboot the node when one of the registered processes forget to 'ping' watchdogd . This is reasonable multi-node RAC, because prevents that the entire cluster hangs due to fact that a single remote key processes (LMON..) does not respond to posts because is dead or is spinning . It is preferrable with linux port, to loose the node and reconfigure the cluster and let the system proceede . This happens for example, in case of an instance failure. The processes belonging to that RAC instance typically will not de-register correctly in case of a crash and softdog will trigger the node reboot.In our case, this is not so good, because we have more than one instance on the same node and we expect that we can continue with the surviving ones. For this reason we will modify the source code of the softdog, 'softdog.c', in order to skip the reboot (*) Currently, we observed that in certain types of sudden crashes we loose also OCM and conseguently the other instances goes down , this is normal and is tolerated within the limits of this special environment . (become root, we use kernel label '2.4.7', adapt to your one) % cd /usr/src/linux-2.4.7/drivers/char % cp softdog.c softdog.c.orig #-- make a backup copy modify softdog.c in this way, adding the line at the beginning of the file : #define ONLY_TESTING % cd /usr/src/linux-2.4.7 % make modules wait until all modules are built (typically unless you performed a 'clean' all obj should be already in place and only our softdog will be rebuilt . Check that the module was rebuilt : % ls -l /usr/src/linux-2.4.7/drivers/char/softdog.o check if softdog is already loaded, shutdown all his clients and unload it : % lsmod Module Size Used by ... softdog 1504 1 <--- in this case YES autofs 10944 1 (autoclean) serial_cb 1632 1 cb_enabler 2688 2 [serial_cb] ... % rmmod softdog substite the newly one : % cd /lib/modules/2.4.7/kernel/drivers/char % cp softdog.o softdog.o.orig #-- back it up % cp /usr/src/linux-2.4.7/drivers/char/softdog.o . now you can 'insmod' it again ----------------- *Important Note* : ----------------- RAC on Linux does not allow the system to continue upon a key process failure and the softdog.c hack has to be intended finalyzed to scopes and limitations of this article . >>Modifying the sofdog code when using RAC is not supported << 3.3.3) Kernel memory requirements -------------------------- Oracle 9i naturally is more memory consumer than previously releases and you have to instruct linux kernel to allow bigger memory mappings. The followings are a suitable configuration, merge it in your /etc/rc.d/rc.local : #--- Oracle 9i needed : rc.local PART 1/2 cd /proc/sys/kernel cat sem export SEMMSL=200 export SEMMNS=512 export SEMOPM=200 export SEMMNI=200 echo $SEMMSL $SEMMNS $SEMOPM $SEMMNI > sem cat shmmax shmmni shmmni shmseg export SHMMAX=2147483648 export SHMMIN=1 export SHMMNI=100 export SHMSEG=4096 echo $SHMMAX > shmmax echo $SHMMIN > shmmin echo $SHMMNI > shmmni echo $SHMSEG > shmseg cat shmmax shmmni shmmni shmseg #--- End of rc.local PART 1/2 4) install Oracle software ----------------------- Simply install the software as usual taking in care of explicitly select : 'Oracle Real Application Clusters 9.0.1.0.0' from the software list in addition to the other options you will need . This will install : Oracle Cluster Manager 2.0.0.10.0 (in $ORACLE_HOME/oracm) Oracle RAC Configuration Tool 5) Setup Oracle Cluster Manager components --------------------------------------- 5.1) load the softdog module in your kernel at startup ------------------------------------------------- appending to your system startup file the : #--- Oracle 9i needed : rc.local PART 2/2 # (this command must be done by root) /sbin/insmod softdog soft_margin=60 #--- End of rc.local PART 2/2 5.2) Create the NM File ------------------ we will use a 1M file that is enough % dd if=/dev/zero of=/u01/ora/oradata/RAC_Node_Monitor_file \ bs=1024 count=1024 1024+0 records in 1024+0 records out 5.3) Configure Oracle Cluster Manger (OCM) ------------------------------------- like for OPS, in RAC for linux the OSD layer is developped by Oracle and is located under $ORACLE_HOME/oracm . Here you will find : $ORACLE_HOME/oracm/ admin/ -> CM configuration files resides : nmcfg.ora -> Node Monitor conf. file ocmargs.ora -> OCM startup parameters bin/ -> the modules : ocmstart.sh -> OCM startup script oracm -> CM binary oranm -> NM binary watchdogd -> WD binary rawio rdevtest -> troubleshooting tools wdttest1 wdttest2 log/ -> where the logs will go : cm.log cm.out -> CM log stream and stdout nm.log nm.out -> NM log stream and stdout ocmstart.ts -> ocmstart.sh lock file wdd.log -> WD out configure the NM : edit 'nmcfg.ora' and substitute the 'DefinedNodes' with your `hostname` , and 'CmDiskFile' with the file defined above (5.2) DefinedNodes=cclinux CmDiskFile=/u01/ora/oradata/RAC_Node_Monitor_file Note : the node name should be returned by 'hostname' and resolved in /etc/hosts (if you're using DHCP you may set the /etc/hosts alias for 127.0.0.1 to the output of `hostname`). This will prevent unnecessary IP roundtrips . configure OCM : edit 'ocmargs.ora' and check the following defaults watchdogd -g dba oranm oracm norestart 1800 if you take a look to 'ocmstart.sh' you will understand the meaning of each OCM parameters 5.4) Troubleshooting Oracle Cluster Manger (OCM) ------------------------------------------- may happen that OCM won't start, the reasons should be investigated analyzing the log files of NM,CM,WD . 5.4.1) Check filesystem rights ----------------------- When starting OCM , 'ocmstart.sh' may return something like : oranm: cnmrequest.c:138: ConnectListener: Assertion `0' failed the 'nm.log' shows : ... WARNING | ConnectListener(): failed to remove /tmp/.oranmsock errno =1 MESSAGE | failed to bind at /tmp/.oranmsock errno=98 ... infact in this case : % ls -l /tmp/.ora*sock* srwxrwxrwx 1 root bin 0 Dec 3 11:23 /tmp/.oracmsock srwxr-xr-x 1 root bin 0 Dec 3 11:23 /tmp/.oranmsock (stop all OCM and remove them , they will be re-created) If you start OCM with 'oracle' account, then verify that you have some FS rights to do that . /tmp/.watchdog/* - directory where resides the sockets used by 'oracm','oranm', 'lmon' <-> 'watchdogd' in the form cl_sock_<PID>_XXXX /tmp/.oranmsock - socket for NM /tmp/.oracmsock - socket for CM /u01/ora/product/9.0.1/oracm/log/ocmstart.ts - Lock semaphore 5.4.2) Oracle does not start --------------------- typically when facing errors like this during the startup SQL> startup CM:CmThreadAttach: socket not connected (error=111) you should check that all modules of OCM are running, if not, or only a part of OCM is running , stop all OCM modules and restart them again 5.4.3) OCM did not restart ------------------- if you experience errors like : ocmstart.sh: Error: Restart is too frequent Info: Check the system configuration and fix the problem. Info: After you fixed the problem, remove the timestamp file Info: "/u01/ora/product/9.0.1/oracm/log/ocmstart.ts" and OCM is not running , you need to remove the semaphore file indicated . % rm /u01/ora/product/9.0.1/oracm/log/ocmstart.ts 5.4.4) is OCM running ? ----------------- a quick way to check if OCM is running is to check if you can see a similar processes tree: watchdogd -g dba oranm \_ oranm \_ oranm \_ oranm \_ oranm \_ oranm \_ oranm oracm \_ oracm \_ oracm 5.4.5) how to stop OCM --------------- basically you have to kill all the above OCM related processes, after you determe that there are not any RAC instance is running . % <kill the 'oranm' 'init' parented ones firstly> % <kill the 'oracm' 'init' parented ones then> % <kill the 'watchdogd'> 6) Clustered database setup ----------------------- At this point our system is ready to start a couple of instances on the same database, but before that, we have to create the database. A database that is intended to be opened in cluster mode is quite the same that one in exlusive mode, except for some main issues . - SPFILE.ora : In a clustered database we will have an specific section for each instance that will mount the database . With 9i, we introduced the SPFILE (Server Parameter FILE) that store all the instance parameters. SPFILE's command syntax is CLUSTER-Awared and we can specify all our needs in a single file . With 9i, in a multi-nodes RAC configuration, SPFILE can be efficiently pointed to a shared RD partition simplyfiing the setup and avoiding parameter misconfiguration . - Redo Logs : In a clustered database each instance must have his own Redo Logs, so we will talk of 'Redo Log Threads' (RLT). RLT are numbered from 1 to n and normally we have only one thread of Redo Logs and this is the RLT #1. In RAC each starting instance will use exclusively the RLT number 'n' where 'n' is equal to the 'thread=n' parameter specific for that istance . - Rollback Segments : Each instance must acquire at least one private RBS, wich one is controlled with instance specific 'rollback_segment=(....)' parameter. This is the pre-9i way to define the undo space for OPS , 9i introduced the concept of Self Managed UNDO Space, through the concept of UNDO tablespace built in the database. In a RAC environment, you should setup an UNDO tablespace for every instance you plan to join the database. 6.1) Create RAC database ------------------- DataBase Configuration Assistan (DBCA) can be used to build the scripts that define our database . DBCA is a RAC-Awared utility, in therms that it can drive you through the creation of a clustered database but to do that require it an extra setup not covered by this document (see GSD,srvconfig, srvctl .. [4],[5]) . We prefer to create a normal database as we are used, then 'transform' it in a 'clusterizable' one . In this way we think that we will understand better some concepts that can be helpfull if using more sophisticated tools like DBCA in the future . 6.1.1) Database on File System , easyer to setup on FS --------------------------------------------- In force of above hipotesys (3.2) we will use normal File System datafiles (*) simplyfing the environment we are going to setup DataBase 'NINEI' ---------------- Create it using your preferred script/tool ... PATH |Description ----------------------------------------|--------------- $ORACLE_BASE/oradata/NINEI/ control01.ctl control02.ctl <- Controlfiles +) redo11.log +) redo12.log <- Thread#1 RLogs +) redo21.log +) redo22.log <- Thread#2 RLogs system01.dbf temp01.dbf tools01.dbf users01.dbf undotbs01.dbf <- UNDO TS Instance 1 (20M) +) ora_undotbs__y0hrj0og.dbf <- UNDO TS Instance 2 (20M) (autogenerated) $ORACLE_BASE/oradata/ RAC_Node_Monitor_file <- Node Monitor Note : the '+' marked files are created by the next steps . Note : as you can see is quite like a normal test database and the respective sizes are also nothing special . In following sections we will assume that you start from a standard database created with DBCA using OFA statements. ----------------- *Important Note* : ----------------- DO NOT USE File System based RAC databases in a multi-node environment unless it is explicity certified and supported (i.e. Tru64 RAC or OpenVMS RAC ..) 6.1.2) create RAC related catalogs --------------------------- % cd $ORACLE_HOME/rdbms/admin % sqlplus "/ as sysdba" @catparr.sql 6.1.3) Create extra RLog Threads ------------------------- once the database is started exclusive the first time, we can create the RLT for both the instances : %sqlplus "/ as sysdba" ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 11 '/u01/ora/oradata/NINEI/redo11.log size' 10M ; ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 12 '/u01/ora/oradata/NINEI/redo12.log size' 10M ; ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 21 '/u01/ora/oradata/NINEI/redo21.log size' 10M ; ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 22 '/u01/ora/oradata/NINEI/redo22.log size' 10M ; ALTER DATABASE ENABLE THREAD 2; (optionally use the 'reuse' clause) 6.1.4) Create extra UNDO tablespaces ----------------------------- every 9i RAC instance needs his own UNDO tablespace when using undo_management=AUTO : %sqlplus "/ as sysdba" create UNDO tablespace UNDOTBS_B datafile size 20M; 6.2) setup for use SPFILE -------------------- Assuming that you created a database named 'NINEI' and an instance named 'NINEI' for normal non-clustered operations, we are going to create the SPFILE that will used by all the instances when mounting clustered database . normally is (or check that it is) : % cd $ORACLE_HOME/dbs % ls -l initNINEI.ora initNINEI.ora -> /u01/ora/admin/NINEI/pfile/initNINEI.ora where 'initNINEI.ora' contains all parameters, now we create a new SPFILE to use with RAC starting from this file % sqlplus "/ as sysdba" create spfile='/u01/ora/admin/NINEI/pfile/spfileNINEI.ora.RAC' from pfile='u01/ora/admin/NINEI/pfile/initNINEI.ora'; now save a backup copy of you initNINEI.ora and edit . modify it in order to contain only the following line : spfile=/u01/ora/admin/NINEI/pfile/spfileNINEI.ora.RAC pointing to our new SPFILE . Verify that shutting down the instance and then starting up all is working with SPFILE . 6.3)Configure SPFILE for RAC ------------------------ the following steps assume that we will start the clustered database from two instances named 'RAC_A' and 'RAC_B', errors like : 'ORA-32010: cannot find entry to delete in SPFILE ' are expected and can be ignored . Note also that the order to run these statements is relevant .. Key Point | RAC Parameter ------|----------------------------------------------------- (1) alter system reset cluster_database scope=spfile sid='*'; alter system set cluster_database=TRUE scope=spfile sid='RAC_A'; alter system set cluster_database=TRUE scope=spfile sid='RAC_B'; (2) alter system reset instance_name scope=spfile sid='*'; alter system set instance_name=RAC_A scope=spfile sid='RAC_A'; alter system set instance_name=RAC_B scope=spfile sid='RAC_B'; (3) alter system reset undo_tablespace scope=spfile sid='*'; alter system set undo_tablespace=UNDOTBS scope=spfile sid='RAC_A'; alter system set undo_tablespace=UNDOTBS_B scope=spfile sid='RAC_B' (4) alter system reset instance_number scope=spfile sid='*'; alter system set instance_number=1 scope=spfile sid='RAC_A'; alter system set instance_number=2 scope=spfile sid='RAC_B'; (5) alter system reset thread scope=spfile sid='*'; alter system set thread=1 scope=spfile sid='RAC_A'; alter system set thread=2 scope=spfile sid='RAC_B'; (6) alter system reset cluster_database_instances scope=spfile sid='*'; alter system set cluster_database_instances=2 scope=spfile sid='*'; (7) [optional if not in ArchiveLogMode] alter system reset log_archive_dest_1 scope=spfile sid='*'; alter system reset log_archive_dest scope=spfile sid='*'; alter system set log_archive_dest_1='location=/u01/ora/oradata/NINEI/logA' scope=spfile sid='RAC_A'; alter system set log_archive_dest_1='location=/u01/ora/oradata/NINEI/logB' scope=spfile sid='RAC_B'; alter system reset log_archive_format scope=spfile sid='*'; alter system set log_archive_format='log_%t_%s' scope=spfile sid='*'; (8) [optional if not using TAF features] alter system reset service_names scope=spfile sid='*'; alter system set service_names='NINEI,SERVICE_A' scope=spfile sid='RAC_A'; alter system set service_names='NINEI,SERVICE_B' scope=spfile sid='RAC_B'; Key Point Considerations : ------------------------ (1) We explicity set 'CLUSTER_DATABASE=TRUE' for each instance and not with the 'sid=*' qualifier in order to allow us to start the 'NINEI' database in exlusive mode if needed . It will be enough to switch to the orginal 'NINEI' instance environment used for the creation . (2) Usefull for service registration and OEM . (3) Every instance have his own UNDO tablespace (4),(5) Obvious (6) In 9i RAC, this parameter is used to dimensionate the inner kernel structures that lay under GCS/GES, it's used for DLM Fast Reconfiguration algorithms (Lazy Remastering). It should be setted to the number of potential instances joining the clustered database . (7) RAC instances normally keep different archive log destinations (8) used for TAF configuration & tests 7) Startup both the instances in shared mode ----------------------------------------- Assuming that your oracle is linked with RAC enabled, we perform the following task from the 'oracle' unix account . 7.1) Start OCM --------- login with you 'oracle' account check that softdog is in place (5.1) % /usr/sbin/lsmod | grep softdog start OCM with the new provided script % cd $ORACLE_HOME/oracm/bin % ./ocmstart.sh 7.2) setup unix environment ------------------------- in order to have the ORACLE_HOME pointing to the 9.0.1.0.0 code path and two distict ORACLE_SID : Instance RAC_A Instance RAC_B -------------- -------------- ORACLE_SID=RAC_A ORACLE_SID=RAC_B ORACLE_BASE=/u01/ora ORACLE_BASE=/u01/ora ORACLE_HOME=/u01/ora/product/9.0.1 ORACLE_HOME=/u01/ora/product/9.0.1 ORACLE_TERM=vt100 ORACLE_TERM=vt100 PATH=$PATH:$ORACLE_HOME/bin: PATH=$PATH:$ORACLE_HOME/bin: PATH=$PATH:$ORACLE_HOME/oracm/bin PATH=$PATH:$ORACLE_HOME/oracm/bin LD_LIBRARY_PATH=$ORACLE_HOME/lib LD_LIBRARY_PATH=$ORACLE_HOME/lib export ORACLE_BASE export ORACLE_BASE export ORACLE_HOME export ORACLE_HOME export ORACLE_TERM export ORACLE_TERM export ORACLE_SID export ORACLE_SID export PATH export PATH export LD_LIBRARY_PATH export LD_LIBRARY_PATH 7.3) startup the instances --------------------- The usual symbolic links to init files should already be created in $ORACLE_HOME/dbs . initRAC_A.ora -> /u01/ora/admin/NINEI/pfile/initNINEI.ora initRAC_B.ora -> /u01/ora/admin/NINEI/pfile/initNINEI.ora Note : we point to the same init.ora that contains the pointing to the same shared SPFILE . <RAC_A> <RAC_B> % sqlplus "/ as sysdba" % sqlplus "/ as sysdba" startup startup ... ... if all worked ok, both the istances will startup, monitor 'alertRAC_A.log' , 'alertRAC_B.log' for errors the process view should show the follwing process runnings : ora_pmon_RAC_A ora_pmon_RAC_B ora_diag_RAC_A ora_diag_RAC_B ora_lmon_RAC_A ora_lmon_RAC_B ora_lmd0_RAC_A ora_lmd0_RAC_B ora_lms0_RAC_A ora_lms0_RAC_B ora_lms1_RAC_A ora_lms1_RAC_B ora_dbw0_RAC_A ora_dbw0_RAC_B ora_lgwr_RAC_A ora_lgwr_RAC_B ora_ckpt_RAC_A ora_ckpt_RAC_B ora_smon_RAC_A ora_smon_RAC_B ora_reco_RAC_A ora_reco_RAC_B ora_arc0_RAC_A ora_arc0_RAC_B ora_lck0_RAC_A ora_lck0_RAC_B 8) Switching between RAC and NON-RAC --------------------------------- You can switch back your oracle to a NON-RAC mode relinking the executable. If you use the default 'NINEI' SID name you can point to the same SPFILE . 8.1) Activating RAC feature ---------------------- If you want to open your database in SHARED MODE (CLUSTER_DATABASE=TRUE) you MUST have oracle RAC-ENABLED % cd $ORACLE_HOME/rdbms/lib % make -f ins_rdbms.mk rac_on % make -f ins_rdbms.mk ioracle you can however open a database with CLUSTER_DATABASE=FALSE using a RAC ENABLED oracle . 8.2) De-activacting RAC feature -------------------------- If you want to open any database without OCM running you MUST have oracle RAC-DISABLED . % cd $ORACLE_HOME/rdbms/lib % make -f ins_rdbms.mk rac_off % make -f ins_rdbms.mk ioracle 8.3) Check if RAC is enabled or not ------------------------------ In linux (only) you can use the follwing statement in order to chech if an Oracle RAC specific library is linked against the current executable % ldd $ORACLE_HOME/bin/oracle | grep libwddapi.so IF <1 row> IS RETURNED THEN your oracle IS RAC-ENABLED ELSE your oracle IS NOT RAC-ENABLED 9) Conclusions ----------- The solution exposed here looses most of the key features of a RAC solution that depends on multiple nodes to scale-up application performances and to be Highly Available eliminating single points of failures such instances and nodes . This document should be intended principally for didactic scopes and used by peoples that are testing the RAC technology and need to setup a quick solution for simulation, test and developement purposes . The 'RAC-in-a-box' solution can be suggested also for : - estimate scaling factor curve for an application with RAC - scale up on a single node with multiple SGAs in case of very high latch contention issues - more than 2 clustered instances, we tested with up to 6 RAC instances on a laptop -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- RELATED DOCUMENTS ----------------- [1] Note:136077.1 Setting up Oracle Parallel Server environment on Linux - Single node [2] Oracle9i RAC - Concepts (9.0.1) [3] Oracle9i Release Notes (9.0.1) for Linux Intel [4] Note:159371.1 SRVCTL utility in Oracle9I RAC [5] Oracle9i RAC - Installation and Configuration . Copyright (c) 1995,2000 Oracle Corporation. All Rights Reserved. Legal Notices and Terms of Use.