Friday, November 13, 2009

How to remove the source node name from cloned RAC Oracle Home inventory

Problem: when running opatch lsinventory for the cloned RAC Oracle Home, the source node name was in the inventory:

Rac system comprising of multiple nodes
Local node = it-pwl-ods01
Remote node = it-pwl-ods02
Remote node = pind42

The remote node pind42 prevents patching and upgrading the Oracle Home except using -local switch for opatch apply -local and runInstaller -local.

Update following file in central inventory:
/opt/oracle/oraInventory/ContentsXML/inventory.xml

Remove the node name pind42 entry.

For local inventory in each cloned Oracle Home:
/opt/oracle/product/10.2.0/GOLD/inventory/ContentsXML/oraclehomeproperties.xml

Update above file and remove node name of pind42.

Check again by running opatch lsinventory.

Hot Clone EBS 11i on 10gR2 RAC and ASM

Rapid Clone Prerequisites

Software Minimum Version Current Version Location
Oracle Universal Installer 2.2.0.19 2.2.0.19 All source system nodes
Perl 5.0.0.5 5.8.8 Source and target database nodes
JRE 1.1.8 1.4.2 Source database node
JDK 1.3.1 1.5.0_10 Target middle-tier applications nodes
Zip 2.3 2.31 All source nodes
OS utilities: make, ld, cc, ar Db tier: ld is in version of 217.
Apps tier: link ld to ld215 Need to be in the PATH when running adcfgclone.pl
AD Minipack 11i.AD.I.6 (6502082) All source nodes
Autoconfig Template Patch 11i.TXK.S (6372396) All source nodes
Raplid Clone Patches 11i Rapidclone Consolidated Fix (6718351) All source nodes

10gR2 (10.2.0.4) CRS and ASM were installed.


On the Source System: PROD

1. run adpreclone.pl on all six production nodes

1.1 database tier: pind41 and pind42
$ORACLE_HOME/appsutil/scripts/

cd /opt/oracle/product/10.2.0/PROD/appsutil/scripts/PROD1_pind41
perl adpreclone.pl dbTier

cd /opt/oracle/product/10.2.0/PROD/appsutil/scripts/PROD2_pind42
perl adpreclone.pl dbTier

1.2 concurrent manager and admin server tier: pind43 and pind44

$COMMON_TOP/admin/scripts/

cd /data/applprod/common/admin/scripts/PROD_pind43
perl adpreclone.pl appsTier

cd /data/applprod/common/admin/scripts/PROD_pind44
perl adpreclone.pl appsTier

1.3 web, forms and mwa server tier: pind23 and pind24

$COMMON_TOP/admin/scripts/

cd /data/wfmprod/common/admin/scripts/PROD_pind23
perl adpreclone.pl appsTier

cd /data/wfmprod/common/admin/scripts/PROD_pind24
perl adpreclone.pl appsTier

2. create backups on the application tier

Run backup scripts on pind23 and pind43:

pind23: /home/wfmprod/scripts/backup and put backups to /data/wfmprod/backup
pind43:/home/applprod/scripts/backup and put backups to /data/applprod/backup

3. create tar file for the Oracle database binary on just one database node (pind41).

cd $ORACLE_HOME/..
tar -cpvzf /backup/PROD/racdb_techstack.tgz PROD

4. run rman backup

Use each morning’s daily rman backup. Need to run backup archivelogs to get archive logs generated after daily rman backup run. Rman backup code is on pind42

Backup files in /backup/PROD/archivelogs and datafiles.

5. move above backups to destination nodes:


on pind41:
cd /backup/PROD
scp racdb_techstack.tgz it-pwl-ods01:/backup/PROD

[/backup/PROD/datafiles] (only need today’s backup)
oracle@pind41(PROD1/DBTier):$ scp PROD_LVL0_20090505_gnke7tbk_s535_p1 PROD_LVL0_20090505_gmke7tbk_s534_p1 PROD_LVL0_20090505_gpke7tbl_s537_p1 PROD_LVL0_20090505_goke7tbk_s536_p1 it-pwl-ods01:/backup/PROD/datafiles


cd /backup/PROD/archivelogs (only need today’s backup)
oracle@pind41(PROD1/DBTier):$
scp * it-pwl-ods01:/backup/PROD/archivelogs

[/backup/PROD/controlfiles]
oracle@pind41(PROD1/DBTier):$ scp * it-pwl-ods01:/backup/PROD/controlfiles


[/pcp/applprod/backup]
applprod@pind43(PROD_806_BALANCE/CCM:Admin):$ scp -r * applmgr@it-pwl-oas01:/pcp/applmgr/backup

[/data/applprod/backup]
wfmprod@pind23(PROD_806_BALANCE/Web:Forms:MWA):$ scp –r * applmgr@it-pwl-ows01:/data/applmgr/backup

wfmprod@pind23(PROD_806_BALANCE/Web:Forms:MWA):$ scp –r * applmgr@it-pwl-ows02:/data/applmgr/backup


On the Destination System: GOLD

6. destination system preparation
6.0 destination should have crs and asm installed on db nodes, /pcp ocfs2 shared disks among ccm and database tier nodes.
6.1 clean the previous old instance on all tiers in refreshing old instance

6.2 for new system, check following OS level requirements
6.2.1 check hugepage
./home/oracle/scripts/hugepages_settings.sh
Recommended setting: vm.nr_hugepages = 2115 (when SGA size is 4GB).

To do for setting hugepage:

Vi /etc/sysctl.conf

Add following at the bottom:

#Custom Setting
#Huge pages for database
Vm.nr_hugepages=2115

Run following command to make the change in effect:
sysctl –p

oracle@it-pwl-ods02(GOLD2/DBTier):$ grep Huge /proc/meminfo
HugePages_Total: 2115
HugePages_Free: 17
HugePages_Rsvd: 14
Hugepagesize: 2048 kB

Run grep, if total is not 2115, then reboot servers.

6.2.2 update /etc/hosts on all servers to include database tiers’ public, private, and virtual ip address, cocurrent manager tier and web tiers. Or use DNS.

6.2.3 central inventory
Create /etc/oratab and /etc/oraInst.loc for new installation on all nodes. Make them chmod 666 oratab oraInst.loc and chown oracle:oinstall oratab oraInst.loc

On apps tiers
applmgr.it-pwl-oas01>cat oraInst.loc
inventory_loc=/opt/oracle/oraInventory
inst_group=oinstall

on database tier
GOLD1.oracle.it-pwl-ods01>cat /etc/oraInst.loc
inventory_loc=/opt/oracle/oraInventory
inst_group=oinstall

6.2.4 ownership of following directories
On database tier:
chown oracle:oinstall /opt/oracle /pcp /backup
On apps tier:
chown applmgr:oinstall /opt/oracle /data
6.2.5 create sub directories
On apps tier
mkdir –p /pcp/applmgr/common/temp
mkdir –p /opt/oracle/oraInventory
mkdir –p /data/applmgr/11i
mkdir –p /data/applmgr/common
mkdir –p /data/applmgr/product
on database tier
mkdir –p /backup/GOLD
mkdir –p /backup/PROD
mkdir –p /opt/oracle/product/10.2.0/GOLD

7. Clone the RAC database
7.1 make sure the ASM listener name is not in the format of LISTENER_. Make sure +ASM and listener are up on it-pwl-ods01 and ods02.

7.2 untar database binary backup on it-pwl-ods01

cd /opt/oracle/product/10.2.0
tar -xvzf /backup/PROD/racdb_techstack.tgz

mv PROD GOLD

7.3 create a pairsfile.txt for new primary target system node (it-pwl-ods01) and put the file in /appsutils/clone

cat pairsfile.txt

s_undo_tablespace=APPS_UNDOTS1
s_dbClusterInst=2
s_db_oh=/opt/oracle/product/10.2.0/GOLD


7.4 create context file for primary target system node: it-pwl-ods01

cd /opt/oracle/product/10.2.0/GOLD/appsutil/clone/bin
run_adclonectx.sh
perl adclonectx.pl \
contextfile=/opt/oracle/product/10.2.0/GOLD/appsutil/PROD1_pind41.xml \
template=/opt/oracle/product/10.2.0/GOLD/appsutil/template/adxdbctx.tmp \
pairsfile=/opt/oracle/product/10.2.0/GOLD/appsutil/clone/pairsfile.txt


Provide the values required for creation of the new Database Context file.

Do you want to use a virtual hostname for the target node (y/n) [n] ?:n

Target hostname [pind41]:it-pwl-ods01

It is recommended that your inputs are validated by the program.
However you might choose not to validate your inputs under following \
circumstances:

-If cloning a context on source system for a remote system.
-If cloning a context on a machine where the ports are taken and you do not \
want to shutdown the services at this point.
-If cloning a context but the database it needs to connect is not available.

Do you want the inputs to be validated (y/n) [n] ?:n

Target instance is a Real Application Cluster (RAC) instance (y/n) [y]: y

Current node is the first node in an N Node RAC Cluster (y/n)[n]:y

Target System database name [prod]:GOLD

Do you want to preserve the port values from the source system on the target \
system (y/n) [y] ?:n

Clone Context uses the same port pool mechanism as the Rapid Install

Enter the port pool number [0-99]: 80 (avoid 80 as tomcat uses 8080).

Database port is 1601

Provide information for the Node 1 (current node):

Host name [pind41]:it-pwl-ods01

Virtual Host name [null]:it-pwl-ods01-vip

Instance number [1]: 1

Private interconnect name [pind41-priv]:it-pwl-ods01-priv

Provide information for the Node 2:

Host name [pind42]:it-pwl-ods02

Virtual Host name [null]:it-pwl-ods02-vip

Instance number [2]: 2

Private interconnect name [pind42]:it-pwl-ods02-priv

Target system quorum disk location required for cluster manager and node monitor \
[/dev/raw/raw2]:/dev/raw/raw2

Target system cluster manager service port [9998]:

Oracle OS User [oracle]:

Oracle OS Group [oinstall]:

Target system utl_file accessible directories list [/usr/tmp]: /pcp/applmgr/common/temp, /usr/tmp

Number of DATA_TOP's on the target system [3]:1

Target system DATA_TOP 1:+DSKGRP1

Do you want to preserve the Display set to :0.0 (y/n) [y] ?:n

Target system Display [it-pwl-oas01:0.0]:it-pwl-oas01.visioncorp.com:1.0


Perl executable location is set to:
/usr/bin/perl

New context path and file name [/opt/oracle/product/10.2.0/GOLD/appsutil/GOLD1_it-pwl-ods01.xml]:

Creating the new Database Context file from :
/opt/oracle/product/10.2.0/GOLD/appsutil/template/adxdbctx.tmp


7.5 Run the adclone.pl utility as follows:
Check KSH_VERSION is set and ld, ar, cc, make are set. Make sure ld link to ld215 (install a package on dbtier?, ld217). After rapidclone patchset, do not need to set KSH_VERION.
cd /opt/oracle/product/10.2.0/GOLD/appsutil/clone/bin.
Run_adclone.sh
export KSH_VERSION='@(#)PD KSH v5.2.14 99/07/13.2'
perl adclone.pl
java=/opt/oracle/product/10.2.0/GOLD/appsutil/clone/jre
component=dbTechStack
mode=apply
stage=/opt/oracle/product/10.2.0/GOLD/appsutil/clone
method=CUSTOM
dbctxtg=/opt/oracle/product/10.2.0/GOLD/appsutil/GOLD1_it-pwl-ods01.xml
showProgress contextValidated=true

Note: Asks for apps password even though the database hasn’t been created yet.

7.6 rman duplicate to create cloned database

7.6.1 create password files

GOLD1.oracle.it-pwl-ods01>orapwd file=orapwGOLD1 password=syspasswd entries=10
(run on two nodes).

7.6.2 create initGOLD1.ora.dup

Cp initPROD1.ora to initGOLD1.ora

Make changes in initGOLD1.ora
Replace PROD with GOLD
Replace applprod with applmgr
Replace pind41 with it-pwl-ods01,pind42 with it-pwl-ods02
Comment out cluster_database,control_files,
Change sga_target from 24G to 4G
Change log_archive_dest_1 to ‘LOCATION=/backup/GOLD/arch’
Change the arch log format backup from GOLD to PROD.
Update temporary tnsnames.ora with local and remote listeners name in initGOLD1.ora, create entry for PROD1.

7.6.3 create temporary tnsnames.ora

set $TNS_ADMIN=/opt/oracle/product/10.2.0/GOLD/network/admin
Tnsnames.ora:

[/opt/oracle/product/10.2.0/GOLD/network/admin]
oracle@it-pwl-ods01(GOLD1/DBTier):$ cat tnsnames.ora
#
# $Header: ad8itns.ora 115.4 2004/04/01 12:57:26 rsanders ship $
#
# ###############################################################
#
# This file is automatically generated by AutoConfig. It will be read and
# overwritten. If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink document
# 165195.1 for assistance.
#
# ###############################################################

#

#
# Net8 definition for the database
#
PROD1=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=pind41-vip.visioncorp.com)(PORT=1531))
(CONNECT_DATA=
(SERVICE_NAME=PROD)
(INSTANCE_NAME=PROD1)
)
)


GOLD1 = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=it-pwl-ods01)(PORT=1601))
(CONNECT_DATA=(SID=GOLD1))
)

GOLD1_LOCAL =
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=it-pwl-ods01)(PORT=1601))
)

GOLD_REMOTE =
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=it-pwl-ods01)(PORT=1601))
)

#
# Intermedia
#
extproc_connection_data =
(DESCRIPTION=
(ADDRESS_LIST =
(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROCGOLD1))
)
(CONNECT_DATA=
(SID=PLSExtProc)
(PRESENTATION = RO)
) )

IFILE=/opt/oracle/product/10.2.0/GOLD/network/admin/GOLD1_it-pwl-ods01/GOLD1_it-pwl-ods01_ifile.ora


[/opt/oracle/product/10.2.0/GOLD/network/admin]

7.6.4 check ASM and listener

Start asm if it is not up.
crs.oracle.it-pwl-ods01>srvctl start asm -n it-pwl-ods01
crs.oracle.it-pwl-ods01>srvctl start asm -n it-pwl-ods02

change asm listener name.

7.6.5 check backups

cd /home/oracle/scripts/clone/GOLD

make sure rman backups (datafile and archive logs) in right location.
It-pwl-ods01:/backup/PROD/archivelogs and datafiles


Check connection to PROD1. set until time in the rman run block accordingly. Make sure archive logs are backed up to this point of time and the archive log backup pieces are moved over.

7.6.6 run rman duplicate

startup new instance in nomount stage.

SQL> startup nomount;

Rman target sys/syspw@PROD1 auxiliary /
RMAN> @dupPRODtoGOLD.rman

(duplicate rman will restore archivelogs from backupsets to /backup/GOLD/arch (set in initGOLD1.ora) and apply the logs, after applying, delete them.)

[~/scripts/clone/GOLD]
oracle@it-pwl-ods01(GOLD1/DBTier):$ cat dupPRODtoGOLD.rman
CONFIGURE CHANNEL DEVICE TYPE disk CLEAR;
CONFIGURE DEFAULT DEVICE TYPE TO disk;
CONFIGURE DEVICE TYPE disk PARALLELISM 2;
run {
set until time "to_date('05-MAY-2009 11:00:00','DD-MON-YYYY HH24:mi:ss')";
set autolocate on;
DUPLICATE TARGET DATABASE TO "GOLD" pfile="/opt/oracle/product/10.2.0/GOLD/dbs/initGOLD1.ora";
}
[~/scripts/clone/GOLD]

7.6.7 post database duplicate steps

7.6.7.1 Update initGOLD1.ora

Update Control_file location, uncommented it.
Uncomment cluster_database.
Update *.log_archive_format from PROD to GOLD

Remember to update local and remote listener to match with previous definition.

7.6.7.2 Bounce database. Might need to turn off archivelog.

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area 4294967296 bytes
Fixed Size 2089432 bytes
Variable Size 1392512552 bytes
Database Buffers 2885681152 bytes
Redo Buffers 14684160 bytes
Database mounted.
SQL> alter database noarchivelog;

Database altered.


SQL> alter database open;

7.6.7.3 In case need to turn on FRA:
Change init.ora
*.db_flashback_retention_target=1440
*.db_recovery_file_dest='/backup'
*.db_recovery_file_dest_size=500G
*.recovery_parallelism=2
#*.log_archive_dest_1='LOCATION=/backup/GOLD/arch'
#*.log_archive_format = 'GOLD_%r_%t_%s.arc'

SQL> startup mount;
SQL> alter database archivelog;
SQL> alter database flashback on;
SQL> alter database open;

7.6.7.4 Use spfile:

SQL> create spfile='+DSKGRP1' from pfile;

Switch to asm instance to find the location of spfile.

Create initGOLD1.ora and initGOLD2.ora with same a pointer to spfile.
GOLD1.oracle.it-pwl-ods01>cat initGOLD1.ora
spfile='+DSKGRP1/GOLD/PARAMETERFILE/spfile.372.686313427'



7.6.7.5 Clean up source system node topology (fnd_node etc):

SQL> connect apps/apps
Connected.

SQL> exec fnd_conc_clone.setup_clean;

PL/SQL procedure successfully completed.

-- must run autoconfig after this
--
EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;


7.6.8 create temporary listener.ora and start temporary listener.

Cd $TNS_ADMIN
(/opt/oracle/product/10.2.0/GOLD/network/admin)
Config temp listener and start it

Listener.ora

# $Header: adlsnr10RAC.ora 115.3 2006/05/02 14:40:32 sbandla noship $
#
# ###############################################################
#
# This file is automatically generated by AutoConfig. It will be read and
# overwritten. If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink document
# 165195.1 for assistance.
#
# ###############################################################

#

#
# Definition for RAC Database listener
#

LISTENER_it-pwl-ods01 =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = it-pwl-ods01-vip.visioncorp.com)(PORT = 1601)(IP = FIRST)))
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = it-pwl-ods01)(PORT = 1601)(IP = FIRST)))
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROCGOLD1)))
)
)

SID_LIST_LISTENER_it-pwl-ods01 =
(SID_LIST =
(SID_DESC = (ORACLE_HOME = /opt/oracle/product/10.2.0/GOLD)(SID_NAME = GOLD1))
(SID_DESC = (SID_NAME = PLSExtProc)(ORACLE_HOME = /opt/oracle/product/10.2.0/GOLD)(PROGRAM = extproc))
)

STARTUP_WAIT_TIME_LISTENER_it-pwl-ods01 = 0
CONNECT_TIMEOUT_LISTENER_it-pwl-ods01 = 10
TRACE_LEVEL_LISTENER_it-pwl-ods01 = OFF

LOG_DIRECTORY_LISTENER_it-pwl-ods01 = /opt/oracle/product/10.2.0/GOLD/network/admin
LOG_FILE_LISTENER_it-pwl-ods01 = GOLD1
TRACE_DIRECTORY_LISTENER_it-pwl-ods01 = /opt/oracle/product/10.2.0/GOLD/network/admin
TRACE_FILE_LISTENER_it-pwl-ods01 = GOLD1
ADMIN_RESTRICTIONS_LISTENER_it-pwl-ods01 = OFF


IFILE=/opt/oracle/product/10.2.0/GOLD/network/admin/GOLD1_it-pwl-ods01/listener_ifile.ora

7.6.9 run adautocfg

7.6.9.1 modify adlsnr10RAC.ora template in /opt/oracle/product/10.2.0/GOLD/appsutil/template to change s_dbhost in the listener name to s_dbSid: LISTENER_%s_dbhost% to LISTENER_%s_dbSid% in case more than one apps database on the nodes.

7.6.9.2 Run adautocfg.sh in /opt/oracle/product/10.2.0/GOLD/appsutil/scripts/GOLD1_it-pwl-ods01

7.6.9.3 Source environment
Bounce listener and database using scripts in /opt/oracle/product/10.2.0/GOLD/appsutil/scripts/GOLD1_it-pwl-ods01

8. create database instance on secondary node

untar database binary code

On second node: it-pwl-ods02:

cd /opt/oracle/product/10.2.0
tar -xvzf /backup/PROD/racdb_techstack.tgz

mv PROD GOLD

create pairsfile.txt
create a pairsfile.txt for new secondary target system node (it-pwl-ods02) and put the file in /appsutils/clone

s_undo_tablespace=APPS_UNDOTS2
s_dbClusterInst=2
s_db_oh=/opt/oracle/product/10.2.0/GOLD
run adclonectx.pl to create contextfile.
Navigate to /appsutil/clone/bin. Run the adclonectx.pl utility as follows:
perl adclonectx.pl \
contextfile=/opt/oracle/product/10.2.0/GOLD/appsutil/PROD1_pind41.xml \
template=/opt/oracle/product/10.2.0/GOLD/appsutil/template/adxdbctx.tmp \
pairsfile=/opt/oracle/product/10.2.0/GOLD/appsutil/clone/pairsfile.txt

oracle@it-pwl-ods02(GOLD2/DBTier):$ run_adclonectx.sh
../jre/bin/java -classpath ../jlib/classes12.jar:../jlib/xmlparserv2.zip:../jre/lib/rt.jar:../jlib/java: oracle.apps.ad.context.CloneContext -e /opt/oracle/product/10.2.0/GOLD/appsutil/PROD1_pind41.xml -tmpl /opt/oracle/product/10.2.0/GOLD/appsutil/template/adxdbctx.tmp -pairsfile /opt/oracle/product/10.2.0/GOLD/appsutil/clone/pairsfile.txt -perl "/usr/bin/perl" -perl5lib "/usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi:/usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi:/usr/lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi:/usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi:/usr/lib/perl5/site_perl/5.8.8:/usr/lib/perl5/site_perl/5.8.7:/usr/lib/perl5/site_perl/5.8.6:/usr/lib/perl5/site_perl/5.8.5:/usr/lib/perl5/site_perl:/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi:/usr/lib64/perl5/vendor_perl/5.8.7/x86_64-linux-thread-multi:/usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi:/usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi:/usr/lib/perl5/vendor_perl/5.8.8:/usr/lib/perl5/vendor_perl/5.8.7:/usr/lib/perl5/vendor_perl/5.8.6:/usr/lib/perl5/vendor_perl/5.8.5:/usr/lib/perl5/vendor_perl:/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi:/usr/lib/perl5/5.8.8:.:/opt/oracle/product/10.2.0/GOLD/appsutil/perl"

Provide the values required for creation of the new Database Context file.

Do you want to use a virtual hostname for the target node (y/n) [n] ?:n

Target hostname [pind41]:it-pwl-ods02

It is recommended that your inputs are validated by the program.
However you might choose not to validate your inputs under following circumstances:
-If cloning a context on source system for a remote system.
-If cloning a context on a machine where the ports are taken and you do not want to shutdown the services at this point.
-If cloning a context but the database it needs to connect is not available.

Do you want the inputs to be validated (y/n) [n] ?:n

Target instance is a Real Application Cluster (RAC) instance (y/n) [y]:y

Current node is the first node in an N Node RAC Cluster (y/n)[n]:n

Please provide the details to connect to one of live RAC nodes


Host name of the live RAC node:it-pwl-ods01

Domain name of the live RAC node:visioncorp.com

Database SID of the live RAC node:GOLD1

Listener port number of the live RAC node:1601

APPS password is required to fetch the information of the other nodes from the database
Enter APPS Password [APPS]:timn3h
Current Node:
Host Name : it-pwl-ods02

SID : GOLD2

Instance Name : GOLD2

Instance Number : 2

Instance Thread : 2

Undo Table Space: APPS_UNDOTS2

Listener Port : 1601

Target system quorum disk location required for cluster manager and node monitor [/dev/raw/raw2]:

Target system cluster manager service port [9998]:

Oracle OS User [oracle]:

Oracle OS Group [oinstall]:

Target system utl_file accessible directories list [/usr/tmp]:/pcp/applmgr/common/temp,/usr/tmp

Number of DATA_TOP's on the target system [3]:1

Target system DATA_TOP 1:+DSKGRP1

Do you want to preserve the Display set to pind43.visioncorp.com:1.0 (y/n) [y] ?:n

Target system Display [it-pwl-ods02:0.0]:it-pwl-oas02.visioncorp.com:1.0

Perl executable location is set to:
/usr/bin/perl

New context path and file name [/opt/oracle/product/10.2.0/GOLD/appsutil/GOLD2_it-pwl-ods02.xml]:

Creating the new Database Context file from :
/opt/oracle/product/10.2.0/GOLD/appsutil/template/adxdbctx.tmp

The new database context file has been created :
/opt/oracle/product/10.2.0/GOLD/appsutil/GOLD2_it-pwl-ods02.xml

Log file located at /tmp/CloneContext_05110903.log

8.1 Modify adlsnrlORAC.ora
cd $ORACLE_HOME/appsutil/template
vi adlsnrlORAC.ora
Modify dbhost = dbSid (Not all, will provide example)
8.2 run adclone

oracle@it-pwl-ods02(GOLD1/DBTier):$ cat run_adclone.sh

export KSH_VERSION='@(#)PD KSH v5.2.14 99/07/13.2'
perl adclone.pl java=/opt/oracle/product/10.2.0/GOLD/appsutil/clone/jre component=dbTechStack mode=apply stage=/opt/oracle/product/10.2.0/GOLD/appsutil/clone method=CUSTOM dbctxtg=/opt/oracle/product/10.2.0/GOLD/appsutil/GOLD2_it-pwl-ods02.xml showProgress contextValidated=true
[/opt/oracle/product/10.2.0/GOLD/appsutil/clone/bin]

8.3 startup instance on second node

Before running autocfg, cp initGOLD1.ora, rename initGOLD2.ora and startup database on it-pwl-ods02.

8.4 create temporary listener

set $TNS_ADMIN=/opt/oracle/product/10.2.0/network/admin

Config temporary listener.ora and start it.

# $Header: adlsnr10RAC.ora 115.3 2006/05/02 14:40:32 sbandla noship $
#
# ###############################################################
#
# This file is automatically generated by AutoConfig. It will be read and
# overwritten. If you were instructed to edit this file, or if you are not
# able to use the settings created by AutoConfig, refer to Metalink document
# 165195.1 for assistance.
#
# ###############################################################

#

#
# Definition for RAC Database listener
#

LISTENER_it-pwl-ods02 =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = it-pwl-ods02.visioncorp.com)(PORT = 1601)(IP = FIRST)))
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = it-pwl-ods02)(PORT = 1601)(IP = FIRST)))
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROCGOLD2)))
)
)

SID_LIST_LISTENER_it-pwl-ods02 =
(SID_LIST =
(SID_DESC = (ORACLE_HOME = /opt/oracle/product/10.2.0/GOLD)(SID_NAME = GOLD2))
(SID_DESC = (SID_NAME = PLSExtProc)(ORACLE_HOME = /opt/oracle/product/10.2.0/GOLD)(PROGRAM = extproc))
)

STARTUP_WAIT_TIME_LISTENER_it-pwl-ods02 = 0
CONNECT_TIMEOUT_LISTENER_it-pwl-ods02 = 10
TRACE_LEVEL_LISTENER_it-pwl-ods02 = OFF

LOG_DIRECTORY_LISTENER_it-pwl-ods02 = /opt/oracle/product/10.2.0/GOLD/network/admin
LOG_FILE_LISTENER_it-pwl-ods02 = GOLD2
TRACE_DIRECTORY_LISTENER_it-pwl-ods02 = /opt/oracle/product/10.2.0/GOLD/network/admin
TRACE_FILE_LISTENER_it-pwl-ods02 = GOLD2
ADMIN_RESTRICTIONS_LISTENER_it-pwl-ods02 = OFF


IFILE=/opt/oracle/product/10.2.0/GOLD/network/admin/GOLD2_it-pwl-ods02/listener_ifile.ora

8.5 run adautocfg


Run adautocfg.sh in /opt/oracle/product/10.2.0/GOLD/appsutil/scripts/GOLD2_it-pwl-ods02

Source environment
Bounce listener and database using scripts in /opt/oracle/product/10.2.0/GOLD/appsutil/scripts/GOLD2_it-pwl-ods02

run adautocfg.sh on primary and bounce listener and database

Run adautocfg.sh again on node1: it-pwl-ods01 and bounce listener and database.

8.6 Database postclone steps:
Clean up system level profile options, WF data and other source system specific data which remain in the newly cloned database:

On it-pwl-ods02: /home/oracle/scripts/clone/GOLD/post_clone1_gold.sql as system
And post_clone2_gold.sql as apps

8.6.1 update libraries

cd $ORACLE_HOME/appsutil/install/
adupdlib.sql is called from adcrdb.sql during the standard clone procedure. For RAC, we use manual clone and adcrdb.sql is not used. Thus, run adupdlib.sql here.

Sqlplus “/ as sysdba” @adupdlib.sql so

(so is the library extention).

Select * from dba_libraries where file_spec is not null;

8.6.2 update directories in database

sqlplus “/ as sysdba”

Select * from dba_directories;

Use following command to update the directory path:

drop directory POWL_OUTBOUND_EDI;
create directory POWL_OUTBOUND_EDI
as '/pcp/applmgr/common/xx_custom/outbound/edi';
Grant ALL on Directory POWL_OUTBOUND_EDI to APPS;
Grant ALL on Directory POWL_OUTBOUND_EDI to XX_CUSTOM;

Some directories only used by sys or system, not need for the grants.

8.6.3 check first before running following commands:

sqlplus “/ as sysdba”

--alter tablespace temp add tempfile size 2000m autoextend on maxsize 6000m;
--alter database rename global_name to PTCH.VISIONCORP.COM;
--alter database add logfile thread 2 group 16;
--alter database add logfile thread 2 group 17;
--alter database enable thread 2;

-- Added to create 1GB online redo logs - default is 1MB.
--alter database drop logfile group 2;
--alter database add logfile thread 2 group 3 ('+DSKGRP1') size 1048576k;
--alter database add logfile thread 2 group 4 ('+DSKGRP1') size 1048576k;
--alter database add logfile thread 2 group 5 ('+DSKGRP1') size 1048576k;
--alter database add logfile thread 2 group 6 ('+DSKGRP1') size 1048576k;
--alter database add logfile thread 2 group 7 ('+DSKGRP1') size 1048576k;
--alter database add logfile thread 2 group 8 ('+DSKGRP1') size 1048576k;

8.6.4 update session_cookie_name due to load balancing

sqlplus apps/apps

update icx_parameters set session_cookie_name=’gold’;

8.6.5 update profile option values

select profile_option_value from fnd_profile_option_values;

8.6.6 update workflow related data

select text_alue from wf_notification_attributes;

update wf_systems set display_name = ‘GOLD.VISIONCORP.COM’;
update wf_agents set address=replace(address,’PRD34,US.ORACLE.COM’,’GOLD.VISIONCORP.COM’);

select web_host_name from fnd_form_functions;

8.6.7 concurrent requests

delete from fnd_concurrent_requests where status_code=’C’;

select logfile_name,logfile_node_name,outfile_node_name from fnd_concurrent_requests;

update fnd_concurrent_requests set phase_code = ‘C’, status_code = ‘X’ where status_code = ‘T’;

update fnd_concurrent_requests set phase_code = ‘C’ where phase_code=’P’;

update fnd_concurrent_requests set status_code = ‘C’ where status_code = ‘P’

update fnd_concurrent_requests set hold_flag=’Y’ where phase_code = ‘P’ and status_code in (‘Q’,’I’);


select node_name from fnd_concurrent_queues;
update fnd_concurrent_queues set running_pocesses = 0, max_processes = 0;
update fnd_concurrent_queues set control_code = ‘X’ where concurrent_queue_name = ‘FNDICM’;


9. On Concurrent manager and administration servers (it-pwl-oas01 and oas02):


9.1 untar application binary backup

on it-pwl-oas01 and oas02

cd /data/applmgr/11i

tar –xzf /pcp/applmgr/backup/APPLTOP/11.5.tar.gz

cd /data/applmgr

unzip /pcp/applmgr/backup/COMMON/common.zip

cd /data/applmgr

tar -xzf /pcp/applmgr/backup/ORATOP/product.tar.gz

9.2 check central inventory
/etc/oraInst.loc
/opt/oracle/oraInventory (ownership: applmgr:oinstall).

check ld linked to ld215

check ulimit –a

stack size and open files

/etc/security/limits.conf

9.3 run adcfgclone in $COMMON_TOP/clone/bin
[applmgr@it-pwl-oas01 bin]$ perl adcfgclone.pl appsTier
Enter the APPS password [APPS]:
apps

First Creating a new context file for the cloned system.
The program is going to ask you for information about the new system:


Provide the values required for creation of the new APPL_TOP Context file.

Do you want to use a virtual hostname for the target node (y/n) [n] ?:n

Target system database SID [PROD]:GOLD

Target system database server node [pind41]:it-pwl-ods01

Target system database domain name [visioncorp.com]:visioncorp.com

Does the target system have more than one application tier server node (y/n) [y] ?:y

Does the target system application tier utilize multiple domain names (y/n) [n] ?:n

Target system concurrent processing node [pind43]:it-pwl-oas01

Target system administration node [pind43]:it-pwl-oas01

Target system forms server node [pind23]:it-pwl-ows01

Target system web server node [pind23]:it-pwl-ows01

Is the target system APPL_TOP divided into multiple mount points (y/n) [n] ?:n

Target system APPL_TOP mount point [/data/applprod/11i]:/data/applmgr/11i

Target system COMMON_TOP directory [/data/applprod/common]:/data/applmgr/common

Target system 8.0.6 ORACLE_HOME directory [/data/applprod/product/8.0.6]:/data/applmgr/product/8.0.6

Target system iAS ORACLE_HOME directory [/data/applprod/product/iAS]:/data/applmgr/product/iAS

Do you want to preserve the Display set to pind43.visioncorp.com:1.0 (y/n) [y] ?:n

Target system Display [it-pwl-oas01:0.0]:it-pwl-oas01.visioncorp.com:1.0

Do you want to preserve the port values from the source system on the target system (y/n) [y] ?:n

Clone Context uses the same port pool mechanism as the Rapid Install
Once you choose a port pool, Clone Context will validate the port availability.

Enter the port pool number [0-99]:
80

Checking the port pool 80
done: Port Pool 80 is free

Complete port information available at /data/applmgr/11i/admin/out/GOLD_it-pwl-oas01/portpool.lst

Verifying Database Connection...

Database Connection on Port 1601: Successful

UTL_FILE_DIR on database tier consists of the following directories.

1. /pcp/applmgr/common/temp
2. /pcp/applmgr/common/xx_custom/outbound/gl
3. /pcp/applmgr/common/xx_custom/inbound/gl
4. /pcp/applmgr/common/EDI/inholding
5. /pcp/applmgr/common/EDI/outholding
6. /pcp/applmgr/common/xx_custom/outbound/acsis
7. /opt/oracle/product/10.2.0/GOLD/appsutil/outbound/GOLD1_it-pwl-ods01
8. /pcp/applmgr/common/xx_custom/outbound/ceridian
9. /usr/tmp
Choose a value which will be set as APPLPTMP value on the target node [1]: 1

9.4 Stop ccm
cd $STOP
adstpall apps/apps

9.5 Source environment
Cd $TNS_ADMIN to create ifile.
8.0.6 and iAS homes.

9.6 run adcfgclone on node2
applmgr.it-pwl-oas02>perl adcfgclone.pl appsTier
Enter the APPS password [APPS]:
timn3h

First Creating a new context file for the cloned system.
The program is going to ask you for information about the new system:


Provide the values required for creation of the new APPL_TOP Context file.

Do you want to use a virtual hostname for the target node (y/n) [n] ?:n

Target system database SID [PROD]:GOLD

Target system database server node [pind41]:it-pwl-ods02

Target system database domain name [visioncorp.com]:visioncorp.com

Does the target system have more than one application tier server node (y/n) [y] ?:y

Does the target system application tier utilize multiple domain names (y/n) [n] ?:n

Target system concurrent processing node [pind43]:it-pwl-oas02

Target system administration node [pind43]:it-pwl-oas02

Target system forms server node [pind23]:it-pwl-ows02

Target system web server node [pind23]:it-pwl-ows02

Is the target system APPL_TOP divided into multiple mount points (y/n) [n] ?:n

Target system APPL_TOP mount point [/data/applprod/11i]:/data/applmgr/11i

Target system COMMON_TOP directory [/data/applprod/common]:/data/applmgr/common

Target system 8.0.6 ORACLE_HOME directory [/data/applprod/product/8.0.6]:/data/applmgr/product/8.0.6

Target system iAS ORACLE_HOME directory [/data/applprod/product/iAS]:/data/applmgr/product/iAS

Do you want to preserve the Display set to pind43.visioncorp.com:1.0 (y/n) [y] ?:n

Target system Display [it-pwl-oas02:0.0]:it-pwl-oas02.visioncorp.com:1.0

Do you want to preserve the port values from the source system on the target system (y/n) [y] ?:n

Clone Context uses the same port pool mechanism as the Rapid Install
Once you choose a port pool, Clone Context will validate the port availability.

Enter the port pool number [0-99]:
80

Checking the port pool 80
done: Port Pool 80 is free

Complete port information available at /data/applmgr/11i/admin/out/GOLD_it-pwl-oas02/portpool.lst

Verifying Database Connection...

Database Connection on Port 1601: Successful

UTL_FILE_DIR on database tier consists of the following directories.

1. /pcp/applmgr/common/temp
2. /pcp/applmgr/common/xx_custom/outbound/gl
3. /pcp/applmgr/common/xx_custom/inbound/gl
4. /pcp/applmgr/common/EDI/inholding
5. /pcp/applmgr/common/EDI/outholding
6. /pcp/applmgr/common/xx_custom/outbound/acsis
7. /opt/oracle/product/10.2.0/GOLD/appsutil/outbound/GOLD2_it-pwl-ods02
8. /pcp/applmgr/common/xx_custom/outbound/ceridian
9. /usr/tmp
Choose a value which will be set as APPLPTMP value on the target node [1]:1

10. On Web and Forms servers (it-pwl-ows01 and ows02):


10.1 untar application binary backup

on it-pwl-ows01 and ows02
on it-pwl-ows01:

mkdir –p /data/applmgr/backup

on pind23
cd /data/applprod/backup
scp -r * applmgr@it-pwl-ows01:/data/applmgr/backup


cd /data/applmgr/11i

tar –xzf /data/applmgr/backup/APPLTOP/11.5.tar.gz

cd /data/applmgr

unzip /data/applmgr/backup/COMMON_TOP/common.zip

cd /data/applmgr/

tar -xzf /data/applmgr/backup/ORATOP/product.tar.gz


10.2 check central inventory
/etc/oraInst.loc
/etc/oratab
/opt/oracle/oraInventory (ownership: applmgr:oinstall).

check ld linked to ld215 (On web/forms tier)

check ulimit –a

stack size and open files

/etc/security/limits.conf

Ping database tier’s vip and public ip address and hostname from web and ccm tiers. (/etc/hosts or DNS)
Otherwise, database connection check error when running adcfgclone.pl

Chown applmgr:oinstall /opt/oracle
Mkdir /opt/oracle/oraInventory
Cp /etc/oraInst.loc and /etc/oratab and chown applmgr:oinstall


cd /data/applmgr/common/clone/bin
10.3 run adcfgclone
[applmgr@it-pwl-ows01 bin]$ perl adcfgclone.pl appsTier
Enter the APPS password [APPS]:
apps

First Creating a new context file for the cloned system.
The program is going to ask you for information about the new system:


Provide the values required for creation of the new APPL_TOP Context file.

Do you want to use a virtual hostname for the target node (y/n) [n] ?:n

Target system database SID [PROD]:GOLD

Target system database server node [pind41]:it-pwl-ods01

Target system database domain name [visioncorp.com]:visioncorp.com

Does the target system have more than one application tier server node (y/n) [y] ?:y

Does the target system application tier utilize multiple domain names (y/n) [n] ?:n

Target system concurrent processing node [pind43]:it-pwl-oas01

Target system administration node [pind43]:it-pwl-oas01

Target system forms server node [pind23]:it-pwl-ows01

Target system web server node [pind23]:it-pwl-ows01

Is the target system APPL_TOP divided into multiple mount points (y/n) [n] ?:n

Target system APPL_TOP mount point [/data/wfmprod/11i]:/data/applmgr/11i

Target system COMMON_TOP directory [/data/wfmprod/common]:/data/applmgr/common

Target system 8.0.6 ORACLE_HOME directory [/data/wfmprod/product/8.0.6]:/data/applmgr/product/8.0.6

Target system iAS ORACLE_HOME directory [/data/wfmprod/product/iAS]:/data/applmgr/product/iAS

Do you want to preserve the Display set to pind43.visioncorp.com:1.0 (y/n) [y] ?:n

Target system Display [it-pwl-ows01:0.0]:it-pwl-oas01.visioncorp.com:1.0

Do you want to preserve the port values from the source system on the target system (y/n) [y] ?:n

Clone Context uses the same port pool mechanism as the Rapid Install
Once you choose a port pool, Clone Context will validate the port availability.

Enter the port pool number [0-99]:
80

Checking the port pool 80
done: Port Pool 80 is free

Complete port information available at /data/applmgr/11i/admin/out/GOLD_it-pwl-ows01/portpool.lst

Verifying Database Connection...

Database Connection on Port 1601: Successful

UTL_FILE_DIR on database tier consists of the following directories.

1. /pcp/applmgr/common/temp
2. /pcp/applmgr/common/xx_custom/outbound/gl
3. /pcp/applmgr/common/xx_custom/inbound/gl
4. /pcp/applmgr/common/EDI/inholding
5. /pcp/applmgr/common/EDI/outholding
6. /pcp/applmgr/common/xx_custom/outbound/acsis
7. /opt/oracle/product/10.2.0/GOLD/appsutil/outbound/GOLD1_it-pwl-ods01
8. /pcp/applmgr/common/xx_custom/outbound/ceridian
9. /usr/tmp
Choose a value which will be set as APPLPTMP value on the target node [1]:9

Creating the new APPL_TOP Context file from :
/data/applmgr/11i/ad/11.5.0/admin/template/adxmlctx.tmp

The new APPL_TOP context file has been created :
/data/applmgr/11i/admin/GOLD_it-pwl-ows01.xml

Log file located at /tmp/CloneContext_05130112.log

Running Rapid Clone with command:
perl /data/applmgr/common/clone/bin/adclone.pl java=/data/applmgr/common /clone/bin/../jre mode=apply stage=/data/applmgr/common/clone/bin/.. component=a ppsTier method=CUSTOM appctxtg=/data/applmgr/11i/admin/GOLD_it-pwl-ows01.xml sh owProgress contextValidated=true


/etc/oraInst.loc points to an Inventory that does not exist: /opt/oracle/oraInve ntory
Inventory Directory created...
Beginning application tier Apply - Wed May 13 13:15:57 2009
Log file located at /data/applmgr/11i/admin/GOLD_it-pwl-ows01/log/ApplyAppsTier_ 05131315.log


Completed Apply...
Wed May 13 13:25:09 2009

Beginning APPSIAS_GOLD registration to central inventory...

ORACLE_HOME NAME : APPSIAS_GOLD
ORACLE_HOME PATH : /data/applmgr/product/iAS
Using Inventory location in /etc/oraInst.loc
Log file located at /opt/oracle/oraInventory/logs/OracleHomeCloner_05130125.log
ORACLE_HOME /data/applmgr/product/iAS was registered successfully.

Starting application Services for GOLD:
Running:
/data/applmgr/common/admin/scripts/GOLD_it-pwl-ows01/adstrtal.sh APPS/

You are running adstrtal.sh version 115.19

Executing service control script:
/data/applmgr/common/admin/scripts/GOLD_it-pwl-ows01/adapcctl.sh start
script returned:
****************************************************

adapcctl.sh version 115.55

Apache Web Server Listener is not running.
Starting Apache Web Server Listener (dedicated HTTP) ...

05/13/09-13:25:16 :: Removing gantt cache directory

05/13/09-13:25:17 :: Successfully removed the gantt cache directory:/data/applmgr/common/html/cabo/images/cache/gantt
Apache Web Server Listener (PLSQL) is not running.
Starting Apache Web Server Listener (dedicated PLSQL) ...

adapcctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


Executing service control script:
/data/applmgr/common/admin/scripts/GOLD_it-pwl-ows01/adalnctl.sh start
script returned:
****************************************************

adalnctl.sh version 115.17

Checking for FNDFS executable.
Starting listener process APPS_GOLD.

adalnctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


Executing service control script:
/data/applmgr/common/admin/scripts/GOLD_it-pwl-ows01/adfrmctl.sh start
script returned:
****************************************************

You are running adfrmctl.sh version 115.38

Starting forms server for GOLD on port 9080.

adfrmctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


Executing service control script:
/data/applmgr/common/admin/scripts/GOLD_it-pwl-ows01/adfmcctl.sh start
script returned:
****************************************************

You are running adfmcctl.sh version 115.20

Starting forms load balancing client for GOLD.

adfmcctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


Executing service control script:
/data/applmgr/common/admin/scripts/GOLD_it-pwl-ows01/adfmsctl.sh start
script returned:
****************************************************

You are running adfmsctl.sh version 115.16

starting forms metrics server for GOLD.

adfmsctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


Executing service control script:
/data/applmgr/common/admin/scripts/GOLD_it-pwl-ows01/jtffmctl.sh start
script returned:
****************************************************

You are running jtffmctl.sh version 115.16

Validating Fulfillment patch level via /data/applmgr/common/java
Fulfillment patch level validated.
Starting Fulfillment Server for GOLD on port 9380 ...

jtffmctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


All enabled services for this node are started.
Check logfile /data/applmgr/common/admin/log/GOLD_it-pwl-ows01/05131325.log for details

Exiting with status 0
[/data/applmgr/common/clone/bin]
applmgr@it-pwl-ows01(web/forms):$

10.4 Shutdown web/forms
cd
. ./.profile
cd $STOP
adstpall apps/apps

10.5 run adcfgclone web node 2: it-pwl-ows02.
Check it-pwl-ows02 using step 10.2
cd /data/applmgr/common/clone/bin

applmgr@it-pwl-ows02(/CCM:Admin):$ perl adcfgclone.pl appsTier
Enter the APPS password [APPS]:
apps

First Creating a new context file for the cloned system.
The program is going to ask you for information about the new system:


Provide the values required for creation of the new APPL_TOP Context file.

Do you want to use a virtual hostname for the target node (y/n) [n] ?:n

Target system database SID [PROD]:GOLD

Target system database server node [pind41]:it-pwl-ods01

Target system database domain name [visioncorp.com]:visioncorp.com

Does the target system have more than one application tier server node (y/n) [y] ?:y

Does the target system application tier utilize multiple domain names (y/n) [n] ?:n

Target system concurrent processing node [pind43]:it-pwl-oas02

Target system administration node [pind43]:it-pwl-oas02

Target system forms server node [pind23]:it-pwl-ows02

Target system web server node [pind23]:it-pwl-ows02

Is the target system APPL_TOP divided into multiple mount points (y/n) [n] ?:n

Target system APPL_TOP mount point [/data/wfmprod/11i]:/data/applmgr/11i

Target system COMMON_TOP directory [/data/wfmprod/common]:/data/applmgr/common

Target system 8.0.6 ORACLE_HOME directory [/data/wfmprod/product/8.0.6]:/data/applmgr/product/8.0.6

Target system iAS ORACLE_HOME directory [/data/wfmprod/product/iAS]:/data/applmgr/product/iAS

Do you want to preserve the Display set to pind43.visioncorp.com:1.0 (y/n) [y] ?:n

Target system Display [it-pwl-ows02:0.0]:it-pwl-oas02.visioncorp.com:1.0

Do you want to preserve the port values from the source system on the target system (y/n) [y] ?:n

Clone Context uses the same port pool mechanism as the Rapid Install
Once you choose a port pool, Clone Context will validate the port availability.

Enter the port pool number [0-99]:
80

Checking the port pool 80
done: Port Pool 80 is free

Complete port information available at /data/applmgr/11i/admin/out/GOLD_it-pwl-ows02/portpool.lst

Verifying Database Connection...

Database Connection on Port 1601: Successful

UTL_FILE_DIR on database tier consists of the following directories.

1. /pcp/applmgr/common/temp
2. /pcp/applmgr/common/xx_custom/outbound/gl
3. /pcp/applmgr/common/xx_custom/inbound/gl
4. /pcp/applmgr/common/EDI/inholding
5. /pcp/applmgr/common/EDI/outholding
6. /pcp/applmgr/common/xx_custom/outbound/acsis
7. /opt/oracle/product/10.2.0/GOLD/appsutil/outbound/GOLD2_it-pwl-ods02
8. /pcp/applmgr/common/xx_custom/outbound/ceridian
9. /usr/tmp
Choose a value which will be set as APPLPTMP value on the target node [1]:1

10.6 Shutdown web/forms
cd
. ./.profile
cd $STOP
adapstpall apps/apps

10.7 update contextfile

10.7.1 web tier JDBC and listener load balancing
Update contextfile: s_tools_two_task from GOLD to GOLD_806_BALANCE
And s_weboh_twotask from GOLD to GOLD_BALANCE.

10.7.2 workflow

s_javamailer_reply_to from workflow.prod@visioncorp.com to workflow.gold@visioncorp.com
s_javamailer_imap_user from visioncorp\oraclework to visioncorp\oraclegold
s_smtphost from it-pwl-ows01 to pemco21


Need to create account on exchange server: PEMCO21 with account name: oraclegold and password: powelloracle (for all instances) with email address of workflow.gold@visioncorp.com

use webmail to check the account:

http://email.visioncorp.com
username: oraclegold
password:powelloracle

then OAM->workflow manager->service components->workflow notification mailer->edit->update password

then restart the workflow mailer.

10.7.3 Discoverer (10g, on separate node)
s_disco_machine from it-pwl-ows01.visioncorp.com to pind30.visioncorp.com
s_disco_port from 8080 to 7778

Need to create tnsnames.ora entry on pind30.

10.7.4 web tier load balancing

s_webentryhost=it-pwl-ows01
s_webentrydomain=visioncorp.com
s_webentryurlprotocol=http
s_active_webport=8080
s_login_page=http://it-pwl-ows01.visioncorp.com:8080/OA_HTML/US/ICXINDEX_GOLD_it-pwl-oas01.htm

10.7.5 update context_name_ifile.ora for concurrent processing tiers

GOLD_806_BALANCE=
(DESCRIPTION_LIST=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=it-pwl-ods01-vip.visioncorp.com)(PORT=1601))
(CONNECT_DATA=
(SERVICE_NAME=GOLD)
(INSTANCE_NAME=GOLD1)
)
)
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=it-pwl-ods02-vip.visioncorp.com)(PORT=1601))
(CONNECT_DATA=
(SERVICE_NAME=GOLD)
(INSTANCE_NAME=GOLD2)
)
)
)

Update concurrent manager tier context file: s_cp_twotask from GOLD to GOLD_806_BALANCE in environment->tools_home

s_cp_twotask=GOLD_806_BALANCE



Environment->applsys: s_applcsf: update data to pcp

s_applcsf=/pcp/applmgr/common/admin
s_applptmp=/pcp/applmgr/common/temp
s_appltmp=/pcp/applmgr/common/temp

On web tier:

s_applcsf=/data/applmgr/common/admin
s_applptmp=usr/tmp
s_appltmp=data/applmgr/common/temp

Need to create directories (EDI etc) under /pcp/applmgr/common for new instance.


10.7.6 Modify template for MWATOP on web tier.
Cd $FND_TOP/admin/template
mkdir custom
Cp jserv_ux_ias1022.properties to custom
Change mwa entry to %s_mwatop%

10.7.7 database tier

Update s_dbService from GOLD to GOLD1 (changed back)

S_instLocalListener from LISTENER_GOLD1 to GOLD1_LOCAL
S_instRemoetListener from LISTENERS_GOLD to GOLD_REMOTE

Change default listener name
Cd /opt/oracle/product/10.2.0/GOLD/appsutil/template
Mkdir custom
Cp adlsnr10RAC.ora custom
Cd custom
Update _%s_dbhost% to _%s_dbSid%

10.8 clean clone setup

[~/scripts/clone]
oracle@it-pwl-ods02(GOLD2/DBTier):$ cat fnd_clone_clean.sql
-- must run autoconfig after this
--
EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;
EXIT;

10.9 Run autoconfig on all nodes
cd $ORACLE_HOME/appsutil/scripts/GOLD1_it-pwl-ods01

Run adautocfg.sh ods01, ods02, ods01, oas01, oas02, ows01, ows02, oas01,oas02,ows01

10.10 checking
Select node_name,server_id from fnd_nodes

Make sure all server_id got populated.

Bounced listener (for using listener_ and database for init.ora parameter _immediate_commit_propagation)

Grant debug connect session to apps;

11 apps post clone steps
11.1 create custom.env on all apps node in $APPL_TOP
11.2 start xvnc on oas01 and oas02. copy startvnc.sh from prod ccm tier and run it.
11.3 tomcat for unitask: port 8080 used. In /data/applmgr/11i/xxlpr/11.5.0/apache-tomcat-4.1.36/conf/server.xml file, update to 8081.
11.4 workflow mailer: need to create email account visioncorp\oraclegold
11.5 unitask license for new implementation. Check with George Burgos.
11.6 PIC UI license: run following query and send dbid info to Michael Kendrick


1 select host_name, global_name, dbid
2* from v$database, v$instance, global_name
SQL> /

HOST_NAME GLOBAL_NAME DBID
-------------------- -------------------- ----------
pind42.visioncorp.com PROD.VISIONCORP.COM 94528928

11.7 Update icx_parameters
Login as system
SQL> update icx_parameter set session_cookie_domain = “.visioncorp.com” where session_cookie_name = ‘gold’
11.8 Update gsmstart.sh for MRP
cd $STOP
vi gsmstart.sh
modify TWO_TASK = ‘GOLD’ – not load balanced.

Or modify template $FND_TOP/admin/template/gsmstart.sh.
Cd $FND_TOP/admin/template
Mkdir custom
Cp gsmstart.sh custom
Cd custom
Cp gsmstart.sh gsmstat.sh.orig
Vi gsmstart.sh, modify s_cp_twotask to s_dbSid.

11.9 change passwords
cd scripts/clone/GOLD

FNDCPASS apps/appsPROD 0 Y system/systemPROD SYSTEM APPLSYS newappspw
FNDCPASS apps/newappspw 0 Y system/systemPROD ALLORACLE newpass
FNDCPASS apps/newappspw 0 Y system/systemPROD ORACLE xx_custom newpass
FNDCPASS apps/newappspw 0 Y system/systemPROD USER SYSADMIN newsysadminpw

modify chg.sh
execute chg.sh
11.10 start mwa
11.11 scramble HR data (script takes awhile) (from ccm tier)
cd $HOME/scripts/scramble
sqlplus apps/apps @powl_scramble_data

due to html tag issue, replace "<" with "leftarrow" and ">" with "rightarrow"

EBS trouble shooting--check database and apps logs

1. database alert.log

$ORACLE_HOME/admin/$CONTEXT_NAME/bdump

For database related problems, check this log first.

2. listener.log

$ORACLE_HOME/network/admin

For security and connection issues, check listener.log.

3. concurrent request log and out

$APPLCSF/$APPLLOG
$APPLCSF/$APPLOUT

For concurrent reqest errors etc.

4. Apache log

$ORA_TOP/iAS/Apache/Apache/logs

For login issue to EBS, need to check access and error logs first. Check error_pls log for starting forms related problem.

5. JVM log

$ORA_TOP/iAS/Apache/Jserv/logs

For forms, java etc related errors. In jvm directory, check for jvm garbage collection etc.

6. Third party apps: Unitask Print Director

$APPL_TOP/xxlpr/11.5.0/apache-tomcat-4.1.36/logs

How to change sysman password for OEM Grid Control and DBControl

1. login to OEM Grid Control repository or database without grid control (using db control)

sqlplus “/ as sysdba”
alter user sysman identified by ‘newpassword’;

2. shutdown oms

emctl stop oms

3. update emoms.properties at $OMS_ORACLE_HOME/sysman/config

oracle.sysman.eml.mntr.emdRepPwd=’newpassword’
oracle.sysman.eml.mntr.emdRepPwdSeed=5219492391084290160 #do not change this
oracle.sysman.eml.mntr.emdRepPwdEncrypted=FALSE #changed from TRUE

4. startup oms

emctl start oms

5. check emoms.properties at $OMS_ORACLE_HOME/sysman/config

Now the emdRepPwd will be encrypted. emdRepPwdEncrypted will be changed to TRUE.

6. for repository host em agent
cd $AGENT_HOME/sysman/emd
edit targets.xml and modify the oracle_emrep target definition
UserName value to be sysman, ENCRYPTED=”FALSE”
Password value to be newpasswd, ENCRYPTED=”FALSE”

Bounce the agent. The username and password will be encrypted and ENCRYPTED will be changed to “TRUE”.

7. for dbcontrol, you can change sysman without changing the config file.

emctl start|stop dbconsole

Unique characteristics of Oracle ASM instance

1. do not have controlfile and datafiles, do not have online redo logs
2. do have init.ora and a passwordfile
3. on the ASM instance node, connect to asm instance using connect “/ as sysdba” at SQL> or sqlplus “/ as sysdba” on OS command line. Username and password in front of / and after / are ignored.
SQL> connect anyusername/anypassword as sysdba ##you can login
4. for connecting remotely, create passwordfile and set following in init.ora

remote_login_passwordfile=exclusive

5. create a password file

$ORACLE_HOME/bin/orapwd file=orapw+ASM1 password=yourpw entries=10

6. ASM instance can not be in open status as there are not datafiles. Can be in mount (although there is no controlfile) and nomount status. When in mount status, database can use the diskgroup. The mount status actually means mount disk groups.

Log rotation for alertSID.log and LISTENER_SID.log

1. Alert.log

cp alertSID.log alertSID.log.`date "+%m%d%y"`
> alertSID.log

or
cat /dev/null >alertSID.log

For the old version of databases, if you delete the alertSID.log, due to the file descriptor issue, you have to bounce the database so that a new alert.log will be created. See Note 122401.1 deleting the alert.log when the database is up. See Note 296354.1 How do you reduce the size of an alert.log that is too large to edit while the DB is running for not using mv command for alert.log. Above approach is better than the one mentioned in Note 74966.1: Renaming or deleting the alert.log while an Oracle instance is up and running.


2. listener.log

cp listener.log listener.log.`date "+%m%d%y"`
> listener.log

I tried this approach for 10gR2 and 11gR2 DB listener. The old way is too much trouble. See Note 739530.1: How to delete or refresh/recycle a SQL*Net Listener log file while the Listener is active.


Put above commands in a shell script (need to set the path for alert.log and listener.log), run once a month will be enough.

Steps to recovery or recreate applications context file if it is corrupted or deleted accidentally

Here are the options to recover or recreate applications (db) contextfile

1. download from database
The contextfiles are saved into a database table: FND_OAM_CONTEXT_FILES. In case the contextfile in the file system is corrupted or lost, login to OAM->Site Map->Administration->System Configuration->AutoConfig->View. Will prompt to open or save it.

For EBS Rel 12, you can user adclonectx.pl retrieve to download the contextfile from database. See Appendix 3.

2. use adbldxml.pl to rebuild contextfile see appendix 1.

3. copy a contextfile from other instance and run adclonectx.pl to create a new contextfile. See appendix 2.

APPENDIX
A1. cd $AD_TOP/bin, then perl adbldxml.pl -help

A2. cd $COMMON_TOP/clone/bin; then perl adclonectx.pl help

A3. Using AutoConfig to Manage System Configurations in Oracle E-Business Suite Release 12 [ID 387859.1]

4) How do I recover the Applications context file if it is lost or deleted accidentally?
Answer:The Applications context file can be retrieved by running the adclonectx.pl script.
To retrieve the applications tier context file,
Execute the following command on the applications tier.
perl /clone/bin/adclonectx.pl retrieve
On being prompted for the context file to be retrieved, select the option of retrieving the applications tier context file that has been lost and retrieve it to the default location specified by the script.
The above command can be used only when the is still intact. In case the has also been lost accidentally, the applications tier context file may be retrieved as follows:
Execute the following command on the database tier:
perl /appsutil/clone/bin/adclonectx.pl retrieve
On being prompted for the context file to be retrieved, select the option of retrieving the applications tier context file that has been lost.
While confirming the location for the context file, set it to any existing directory with write permission.
Once the context file has been generated in the specified location, move it to the location specified for the context file in the context variable 's_contextfile'.
To retrieve the database tier context file,
Execute the following command on the database tier:
perl /appsutil/clone/bin/adclonectx.pl retrieve
On being prompted for the context file to be retrieved, select the database tier context file and retrieve it to the default location specified by the script.

Shared Application File System for EBS on NFS -- pros and cons

1. What is the shared application file system?
Oracle EBS 11i has following mid-tier application servers: web server, forms server, MWA server, concurrent manager and report server, and administration server in our environment. The application code base is scattered on different servers. For example, web, forms and MWA servers code is on web/form tier and concurrent manager and report server, and adminstration server code is on ccm/admin node. For the shared application file system, all servers code is installed on a single storage media. This storage location is then mounted onto different physical nodes. We can configure each node to run specific application servers.

2. Advantages to implement the shared application file system
2.1 enhance DBA's productive and shorten downtime for maintenance
DBAs only need to run adpatch for applying patches and adadmin for system maintenance once on the shared application file system rather than four times now. Furthermore, we can utilize the distributed AD to shorten adpatch and adadmin sessions more.

2.2 reduce storage and tape usage and shorten tape backkup time
As all the application codebase is on the shared media, the codebase duplication will be greatly reduced. The tape backup for the application code only needs done once from any physical node rather than four times now.

2.3 code migration will be easier
Currently, we have to keep forms/reports code in synch for the customized code among two web/forms and two ccm nodes respectively. With the shared application file system, we only need to migrate the code into the shared location and all servers will see the new changes.

2.4 shorten cloning
Currently, we have to run preclone procedure on web/forms and ccm/admin node. Then we have to take a backup on the respectively node. Then we need to move all code to destination nodes. Run untar, adcfgclone and other commands on all the servers. With the shared application file system, we only need to run once.

2.5 easy route to virtual servers
As all the application code is on one location, we can put application server tiers on virtual servers.

2.6 easy to expand capacity
It is much easy to add a new node to expand capacity for the shared application file system with small or no downtime.

3. Disadvantages for the shared application file system
Only concern is that as the codebase is on the shared storage media, in case the shared storage media fails, the whole system gone. As we are not using the local disk for the application codebase now, storage media failure will create a problem too.

4. Concerns
4.1 NFS mount is slower for running rm and tar commands. However, we only need to run these commands during cloning or server migration. For the cloning part, as we only need to run from one node rather than from four nodes, the slowness is not a problem.

4.2 Cloning before PROD got shared application file system implementation
We will keep the original mounting points. In case we need to clone DEV/INT/GOLD, we will follow the current procedure. After PROD implementation, we will update the clone procedure.

5. pre-requisites
5.1 need new shared mounting point: /mnt/appldev, /mnt/applmgr, /mnt/applprod etc with a disk space size of 75GB for each environment.
5.2 create appldev, applmgr, applprod user on current web/forms tier
5.3 create /opt/oracle/appldev, applmgr, applprod on all apps nodes.

6. plan
After the pre-requisites met, we can begin the implementation. For PTCH, I spent about 13 hours. We could shorten the downtime more. Also, we do not need to shutdown database. We also have easy fall back plan in case the shared application file system approach does not work.

We will start from DEV and then GOLD. Let us test it on GOLD for about 2 months before migrating into PROD. As we do not have any code changes, we do not need a full cycle for testing.

Some background: web/forms/MWA tier: two nodes; concurrent manager/admin server: tow nodes; RAC database: two nodes.

Backup and Recovery (0) -- recommended setting for RMAN and init.ora

1. Settings for the database in init.ora

1.1 it is better to use flash_recovery_area for archive logs, controlfile backup and rman backup files

ALTER SYSTEM RESET log_archive_dest_1 SCOPE=SPFILE SID='*';ALTER SYSTEM RESET log_archive_format SCOPE=SPFILE SID='*';
alter system set db_recovery_file_dest = /flash scope=spfile sid='*';alter system set db_recovery_file_dest_size= 1073741824000 scope=spfile sid='*'

alter system set control_file_record_keep_time = 30 #in case not using rman catalog

1.2 it is better to enable archive log even on DEV and UAT environment. For DEV environment, the chances that the developers make a mistake are far higher. For UAT, we need to mimic production setting. Need to shutdown database and startup mount before running following command:

alter database archivelog;

archive log list;

1.3 it is better to enable flash back database feature as application patching does not have an easy rollback procedure. Startup mount then:

alter database flashback on;

1.4 check if recyclebin=on via show parameter recyclebin;

1.5 enable block change tracking in case we need to use incremental RMAN backup.

alter database enable block change tracking using file ‘/path/change_tracking.f’;

2. settings for RMAN
CONFIGURE CONTROLFILE AUTOBACKUP ON; #default OFFCONFIGURE CHANNEL 1 DEVICE TYPE DISK CONNECT 'sys/syspw@PROD1';CONFIGURE CHANNEL 2 DEVICE TYPE DISK CONNECT 'sys/syspw@PROD1';CONFIGURE CHANNEL 3 DEVICE TYPE DISK CONNECT 'sys/syspw@PROD2';CONFIGURE CHANNEL 4 DEVICE TYPE DISK CONNECT 'sys/syspw@PROD2';CONFIGURE DEVICE TYPE DISK PARALLELISM 4 BACKUP TYPE TO compressed BACKUPSET; #default parallelism 1, no compressed

It is better to turn on controlfile autobackup so that any structure changes for database are saved. Turn on compressed backupset to save disk space. Need to configure channels in RAC environment. For a single node, ignore. Also use spfile so that it got backed up too when controlfile autobacup is on.

3. rman backup script

After above configuration, rman backup script will be very simple:

RMAN> run {2> backup archivelog all delete input;3> }

RMAN> run {
2> backup database plus archivelog all delete input;
}

Backup and Recovery (1) -- recover controlfile and spfile (pfile)

1. recover controlfile
Controlfile is vital to an Oracle database. It records structure of the database: datafile name and location, online redo log name and location, characterset, database name, rman settings, rman backup info etc.

It is recommended to have three copies of controlfile on different mounting points.

It is recommended to turn on rman controlfile autobackup.

1.1 rman approach
Rman> startup nomount;
Rman> set dbid ;
Rman> restore controlfile from autobackup;

1.2 rebuild controlfile from saved textfile
SQL> alter database backup controlfile to trace [“a new location”];
Edit the trace file and run from SQL>.

1.3 in case trace is not available and no rman backup of controlfile. Then last resort: rebuild it manually.
Need to collect datafile location and name, characterset, database name and online redolog name and location. Create a trace file from other database and build the new controlfile trace file for the missing controlfile database.

2. recover spfile
need to turn on rman controlfile autobackup.
Rman> startup force nomount; ##use dummy init.ora
Rman> set dbid <>;
Rman> restore spfile from autobackup;

3. recover pfile
3.1 restore from spfile
SQL> create pfile from spfile;

3.2 restore from rman backup of spfile
RMAN> restore spfile to pfile '/tmp/initPROD.ora' from autobackup;

3.3 above approaches failed and no textfile backup, then need to rebuild init.ora from alert.log. When database got startup, the none default settings in init.ora were got written in alert.log. So do not delete alert.log.

4. Experience

We needed to change some memory setting for our OBIEE database. The database was created with spfile and we had not turned on rman backup yet. My fellow DBA set the SGA_TARGET to a huge number. After shutdown the database, he could start it as no enough memory on the node. He got stuck. Fortunately, I had a backup on my desktop. Otherwise, we need to go through the alert.log and rebuild the init.ora.

Dr. Z's monitoring scripts for EBS

1. check for runway forms processes: runaway_processes.ksh
#!/bin/ksh

##running morethan one hour
proc=`ps -efgrep wfmprodgrep -v xxlpregrep -v "00:*:*"`;

##running morethan one day
proc1=`ps -efgrep wfmprodgrep -v xxlprgrep "1-00:*:*"grep -v grep`;

if ! [ -z $proc ]; then
echo $proc mail -s "Long running processes on `hostname -a`" linn.hu@visioncorp.com
fi

if ! [ -z $proc1 ]; then
echo $proc1 mail -s "Long running processes on `hostname -a`" linn.hu@visioncorp.com
fi

2. check changed files on apps tier: check_changed_files.sh
find /data/applprod -mtime -1grep -v loggrep -v ganttgrep -v tempmail -s "`hostname –a`: changed files in last 24 hours" linn.hu@visioncorp.com

3. check changed files on db tier: check_changed_files.sh
find /opt/oracle/product/10.2.0/PROD -mtime 1grep -v auditgrep -v dumpmail -s "`hostname –a`: changed files in last 24 hours" linn.hu@visioncorp.com

4. check MRP/SCP error: check_mrp.sh
#!/bin/bash
#
. $HOME/.profile
SCRIPTS=$HOME/scripts
cd $SCRIPTS
> $SCRIPTS/check_mrp.log
sqlplus -s / as sysdba @check_mrp >> check_mrp.log

if [ $(grep -c -i 'ORA-01034: ORACLE not available' check_blocking_locks.log) -eq 1 ]
then
exit
fi

if [ $(grep -c -i 'no rows selected' check_mrp.log) -eq 1 ]
then
exit
else
mail -s "PROD: MRP/SCP Memory Based Snapshorts have errors" linn.hu@visioncorp.com < check_mrp.log
fi

exit

check_mrp.sql
set linesize 200
col user_concurrent_program_name format a30
col completion_text format a50

select r.request_id,p.user_concurrent_program_name,r.actual_start_date,r.completion_text
from apps.fnd_concurrent_requests r, apps.fnd_concurrent_programs_tl p
where r.concurrent_program_id = P.CONCURRENT_PROGRAM_ID
and r.status_code = 'E'
and p.user_concurrent_program_name like '%Snapshot%'
and r.actual_start_date between trunc(sysdate-1/24) and trunc(sysdate)
order by 2,3 desc
/

exit

5. check errors in alert.log: check_alertlog.sh
#!/bin/bash
#
. $HOME/.profile
SCRIPTS=$HOME/scripts/
cd $SCRIPTS
> $SCRIPTS/check_alertlog.log

diff $ORACLE_HOME/admin/PROD1_pind41/bdump/alert_PROD1.log /tmp/alert_PROD1.log.old grep -C4 -i "ORA-" >>check_alertlog.log

if [ -s check_alertlog.log ]
then
mail -s "PROD1: ORA- errors in alert.log" linn.hu@visioncorp.com < check_alertlog.log
fi

cp $ORACLE_HOME/admin/PROD1_pind41/bdump/alert_PROD1.log /tmp/alert_PROD1.log.old

exit

Sunday, November 8, 2009

init.ora parameters reset for 10gR2 db

DB_FILE_MULTIBLOCK_READ_COUNT

Per note 841444.1 How To Set DB_FILE_MULTIBLOCK_READ_COUNT in 10g
Oracle Database 10g Release 2 automatically selects the appropriate value for this parameter depending on the operating system optimal I/O size and the size of the buffer cache. After unset it, db_file_multiblock_read_count=128 for our system.


AQ_TM_PROCESSES

Per Note 305662.1 Queue Monitor Process: Architecture and Known Issues
For version 10.1 onwards it is no longer necessary to set AQ_TM_PROCESSES when Oracle Streams AQ or Streams is used.

alter system reset aq_tm_processes scope=spfile sid='*';


_pga_max_size

Per Note 844542.1 PGA_AGGREGATE_TARGET Assigned Memory Is Left Unconsumed When Set High

It was needed to fix a bug in 9i to 10gR1.


LOG_CHECKPOINT_TIMEOUT
LOG_CHECKPOINT_INTERVAL
FAST_START_IO_TARGET

Per Note 472821.1 Which Checkpoint Parameters to Set in Oracle 10g?

Just set FAST_START_MTTR_TARGET instead.

Create easily recalled or memorizable and robustic and strong passwords

Until the scanner for finger prints, eye iris and other biometrics gets widespread implementation, we need passwords. However, creating and maintaining the password has been challenging. It is recommended to avoid dictionary words, reverse of them, username, birthday and some other easily guessed or cracked password. However, strong and robust passwords are hard to be memorized or recalled. Those passwords have been written down in notebook or even on post note and hanged from monitor by users. The user’s action defeats the purpose for strong passwords.

Here is my approach: a formula and a sentence. You can design a formula or hash function so that you can recall easily. Then create a sentence or open a book and find a sentence that you like. Memorizing this formula and the sentence is much easily to memorize the password. The password will be created based on the formula and the sentence.

Here is an example:

Formula: {first letter of username}{a number}{first letter of the sentence including punctuation mark}

Sentence: What a beautiful world!

Username: system

Password: s3Wabw!

Try to memorize “s3Wabw!” might be hard. Memorize the formula and the sentence is much easier.

Also, the password is strong with number, upper/lower case letter and punctuation mark. As we have 10 digits and 52 upper/lower letters and assume we can use 8 punctuation marks, it is hard to crack above password as we have 70^7=8,235,430,000,000 choices of passwords.

Customize Linux and Oracle sqlplus prompt and Linux oracle and apps account environment setup

There are a few dot files in the OS account home directory (/home/applmgr or /home/oracle). The first one is .bash_profile which calls .bashrc which calls .profile_GOLD for some customized setting. .profile_GOLD calls user’s customization.

1. .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH
unset USERNAME

2. .bashrc
In this file, I set the command history related settings. It is a good idea to have the command history timestamp.

# .bashrc

# User specific aliases and functions

# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
set -o vi
export HISTCONTROL=ignoredups
export HISTSIZE=9999
export HISTFILESIZE=999999
export HISTTIMEFORMAT="%F %T "
export EDITOR=vi
source $HOME/.profile_GOLD

3. .profile_PTCH
It is for Oracle applications. Use APPSPTCH_pind31.env rather APPSORA.env. For shared appl_top, these two files are different.

#.profile_PTCH
. /mnt/applptch/ptchappl/APPSPTCH_pind31.env

. /share/oracle/.me.ccmtier

For the database tier, use following:
#.profile_PTCH
. /opt/oracle/product/10.2.0/PTCH/PTCH2_pind22.env

. /share/oracle/.me.dbtier

4. .me.ccmtier
My preferred alias setting for reducing typing and os prompt for showing current working direct and time, login os user and hostname, instance name and apps servers.

#.me.ccmtier
##ALIAS
alias tns="cd $TNS_ADMIN"
alias itns="cd ${IAS_ORACLE_HOME}/network/admin/${CONTEXT_NAME}"
alias apacheconf="cd ${IAS_ORACLE_HOME}/Apache/Apache/conf"
alias apachelog="cd $IAS_ORACLE_HOME/Apache/Apache/logs"
alias jservetc="cd $IAS_ORACLE_HOME/Apache/Jserv/etc"
alias jservlog="cd $IAS_ORACLE_HOME/Apache/Jserv/logs"
alias oh="cd $ORACLE_HOME"
alias iasoh="cd $IAS_ORACLE_HOME"
alias admin="cd $COMMON_TOP/admin/scripts/$CONTEXT_NAME"
alias commontop="cd $COMMON_TOP"
alias appltop="cd $APPL_TOP"
alias contextfile="cd $APPL_TOP/admin"
alias custop="cd $XX_CUSTOM_TOP"
alias mwa="cd $MWA_TOP/bin"
alias tomcat="cd $XXLPR_TOP/bin"

##PROMPT
export PS1="[\w] :[time \t]\n\u@\h(${TWO_TASK}/CCM:Admin):$ "

##Send X display to Linn's Desktop, run xming config first.
#export DISPLAY=172.18.33.73:0.0

5. .me.webtier
This one is for web tier. Difference is for os prompt.

#.me.webtier
##ALIAS
alias tns="cd $TNS_ADMIN"
alias itns="cd ${IAS_ORACLE_HOME}/network/admin/${CONTEXT_NAME}"
alias apacheconf="cd ${IAS_ORACLE_HOME}/Apache/Apache/conf"
alias apachelog="cd $IAS_ORACLE_HOME/Apache/Apache/logs"
alias jservetc="cd $IAS_ORACLE_HOME/Apache/Jserv/etc"
alias jservlog="cd $IAS_ORACLE_HOME/Apache/Jserv/logs"
alias oh="cd $ORACLE_HOME"
alias iasoh="cd $IAS_ORACLE_HOME"
alias admin="cd $COMMON_TOP/admin/scripts/$CONTEXT_NAME"
alias commontop="cd $COMMON_TOP"
alias appltop="cd $APPL_TOP"
alias contextfile="cd $APPL_TOP/admin"
alias custop="cd $XX_CUSTOM_TOP"
alias mwa="cd $MWA_TOP/bin"
alias tomcat="cd $XXLPR_TOP/apache-tomcat-4.1.36/bin"

##PROMPT
export PS1="[\w] :[time \t]\n\u@\h(${TWO_TASK}/Web:Forms:MWA):$ "

##Send X display to Linn's Desktop, run xming config first.
#export DISPLAY=172.18.33.73:0.0


6. .me.dbtier

#.me.dbtier

##ALIAS
alias tns="cd $TNS_ADMIN"
alias bdump="cd $ORACLE_HOME/admin/$CONTEXT_NAME/bdump"
alias udump="cd $ORACLE_HOME/admin/$CONTEXT_NAME/udump"
alias oh="cd $ORACLE_HOME"
alias dbs="cd $ORACLE_HOME/dbs"
alias admin="cd $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME"
alias contextfile="cd $ORACLE_HOME/appsutil"

##PROMPT
export PS1="[\w] :[time \t]\n\u@\h(${ORACLE_SID}/DBTier):$ "

##Send X display to Linn's Desktop, run xming config first.
#export DISPLAY=172.18.33.73:0.0

7. examples

[/opt/oracle/product/10.2.0/PTCH]
oracle@pind21(PTCH1/DBTier):$

[/mnt/applptch/ptchappl]
applptch@pind31(PTCH_806_BALANCE/CCM:Admin):$

[/mnt/applptch/ptchappl]
applptch@pind33(PTCH_806_BALANCE/Web:Forms:MWA):$

8. customize database sqlplus prompt

add following to the bottom of glogin.sql in $ORACLE_HOME/sqlplus/admin (client or server side) to customized sqlplus prompt. It is very convenient at client side as we might connect to different database.

set lines 132
set pages 100
set termout off
col dbname new_value prompt_dbname
select instance_name dbname from v$instance;
set sqlprompt "_USER@&&prompt_dbname>"
set termout on
set time on

Steps to install companion CD 10gR2 over 10.2.0.4 Oracle Home

I got a new job in the beginning of this year. The configuration of the E-Business 11.5.10.2 is quite complex. There are two web/forms nodes, two concurrent manager and administration server nodes. Database is on the two nodes RAC. When I just joined the company as a DBA, the database is on 32-bit system. You know the performance is a problem.

I found another problem for our system. The Companion CD for 10gR2 was never installed. As we have applied the 10gR2 patchset 3 (10.2.0.4), we can not run installer from the Companion CD. Here are the steps to get this done. Run step 1 to 5 on one node.

shutdown apps, database and listener.

start OUI from $ORACLE_HOME/oui/bin/runInstaller –ignoreSysPrereqs (which is version 10.2.0.4) (see Note 741643.1 for not using OUI from companion CD at 10.2.0.1)

(pick middle option: Oracle Database 10g Products 10.2.0.1)

companion CD binary code is at /share/oracle/install/lnx64_10gr2/companion/stage/products.xml

After installed companion CD 10.2.0.1, need to apply 10gR2 patchset 3 (10.2.0.4) again (see Note 757129.1).

Start OUI at:
/share/oracle/install/lnx64_10gr2/10204/Disk1/runInstaller -ignoreSysPrereqs

Post installation tasks (on one node only) (need about 1.5 hours)
comment out cluster_database in init.ora
startup database: startup upgrade;
spool /tmp/patch.log
@?/rdbms/admin/catupgrd.sql
Spool off
Review patch.log, check for error, warning, fail etc.
Shutdown immediate
Uncomment cluster_database in init.ora
Startup
@?/rdbms/admin/utlrp.sql
Select comp_name,version,status from sys.dba_registry, make sure all comp are valid.

shutdown database and apply the one-off patches (check if these patches got reversed. Opatch lsinventory, otherwise, skip step 7)
$PATCHHOME/rdbms/p6139856, p6471770, p6971433, p7331323, p7441176 and p8479537.

Why does cloned EBS instance deliver HTML pages faster than PROD does

This question was also from an Expert Panel session at Oracle Open World (10/11-15/2009). The experts were beating the brushes and I answered the question.

The cloned instance does not copy the Web Application Server cache at $COMMON_TOP/_pages. Thus, for the first access to a JSP pages, the system needs to compile it into HTML and put in _oa__html (HTML pages) and _oa__java (JAVA class). The future access to this page will be faster. In the production environment, the cache is never refreshed. After a few rounds of patching or apply consolidated updates, there are a lot of junks in the cache. Thus, production instance delivers the HTML pages slower than cloned instances even if the cloned instance on the same set of machines as the production instance.

Here is the way to refresh EBS web server cache:

1. rename _oa__html and _oa__java.
cd $COMMON_TOP/_pages
mv _oa__html _oa__html.old
mv _oa__java _oa_java.old

mkdir _oa__html _oa__java

2. recreate the cache

perl -x $JTF_TOP/admin/scripts/ojspCompile.pl --compile –quiet

3. If during the clone procedure, the cache got copied over, you might get login problem due to missing java class etc. You also need to refresh the cache. See Note 466739.1 AppsLocalLogin.jsp Fails With java.lang.NoSuchMethodError oracle.apps.fnd.sso.HttpLanguageMap.getOracleFromHttp

Monday, November 2, 2009

How to recreate missing or corrupted Central Inventory for Oracle Homes?

Last Monday (10/26/2009), our OBIEE went alive. At the same time, we are in the process to upgrade OEM Grid Control from 10.2.0.4 to 10.2.0.5 to take advantage the new features. My fellow co-worker logged into the Windows server 2003 and began the upgrade of the agent. To his surprise, he found that the Oracle Central Inventory was not there to be found. The default location should be “C:\Program Files\Oracle\Inventory”. There is an Inventory for 32 bit in another drive. However, the OBIEE database BIPRD is 64bit.

Following Note 556834.1 Steps to recreate central inventory (oraInventory) in RDBMS Homes. He was able to recreate the central inventory and went on to finish the agent upgrade.

Here are the steps:

1. Check windows registry to locate central inventory location, all Oracle Homes on the system.
START->RUN-regedit->KHEY_LOCAL_MACHINE/SOFTWARE/ORACLE

Find the value of the key “inst_loc” this is the location Oracle Central Inventory
Find the value of ORACLE_HOME and ORACLE_HOME_NAME
We have three Oracle Homes on the system: OEM Agent Home, Oracle 11gR1 Client and Oracle 11gR1 RDBMS.
2. Run following command to recreate Oracle Central Inventory (one line)
%ORACLE_HOME%/oui/bin/setup.exe –silent –ignoreSysPrereqs –attachHome ORACLE_HOME=”!Oracle_Home!” ORACLE_HOME_NAME=”!Oracle_Home_Name!

The "!Oracle_Home!" and "!Oracle_Home_Name!" were identified from the registry.

Need to run above command from three different Oracle Homes.

3. After each setup.exe run, run opatch to check the inventory

%ORACLE_HOME%/OPatch/opatch lsinventory


In case your system is on Unix/Linux, you can try this:
1. determine the location of Oracle Central Inventory
cat /etc/oraInst.loc (AIX and Linux) or /var/opt/oracle/oraInst.loc (Solaris etc)

inventory_loc= "!orainventory location!"
inst_group= oinstall

2. rename the central inventory if it is corrupted.
3. Determine the Oracle Home on the machine and pick a new name for each Oracle Home
4. run following command (one line)
$ORACLE_HOME/oui/bin/runInstaller –silent –ignoreSysPrereqs –attachHome ORACLE_HOME=”!Oracle_Home!” ORACLE_HOME_NAME=”!Oracle_Home_Name!

If you have more than one Oracle Home on the box, run above command from all Oracle Homes. and were identified in step 3.

5. check the inventory
$ORACLE_HOME/OPatch/opatch lsinventory

6. delete old Central Inventory if it is corrupted.

Note 556834.1 did not mention need to run setup.exe as runInstaller does not exist on Windows. Also, it did not mention need to get Oracle Home and Oracle Home Name from the Windows registry. It did mention that recreating global/central inventory is only supported from 10.2.0.1.0 or later and not downtime is needed.

How to prevent order confirmation mails sent to customers from cloned instances which are used as DEV or UAT environment?

I was attending an Expert Panel Session about EBS performance at Oracle Open World (10/11-15/2009) and one DBA raised this question. The expert panel could not answer this question directly. I believe other DBAs might have similar problem. Here, I list three ways to solve this problem.

1. Disable sendmail in EBS
Login as sysadmin and pick Alert Manager responsibility. Then pick options under System. Finally, uncheck Unix sendmail In Use check box.

After disabling sendmail from EBS, no emails from Oracle Alert and Oracle Workflow Notification mailer will be sent out.

However, for developers or testers for checking Oracle Alerts and Workflow Notification mailer, they need to receive the mails. Following two ways will meet this requirement.

2. provide an override or test email address from workflow notification mailer
2.1 via OAM
Login to OAM, navigate to Workflow Manager (upper right dropdown box), at the bottom left Related Links section, click Service Components. Then click Workflow Notification Mailer at the bottom. At the upper right of the page, click Set Override Address which will give you the ability to send all emails to a test address. See Email Servers section, Outbound Email Account for Test Address.

2.2 via sql script (from Note 459932.1 How to set a different "test email address" for the workflow notification without connecting to OAM)

2.2.1.Run the following query to check the present value for the test address:

select p.parameter_id, p.parameter_name, v.parameter_value,v.parameter_description,v.default_parameter_value
from fnd_svc_comp_param_vals_v v,fnd_svc_comp_params_b p, fnd_svc_components c where c.component_type = 'WF_MAILER' and v.component_id = c.component_id and v.parameter_id = p.parameter_id and p.parameter_name ='TEST_ADDRESS';

2.2.2 Do the following to change the value for the test address from the back end without logging into the Oracle Application Manager :

2.2.2.1.Run the following script $FND_TOP/sql/afsvcpup.sql from sqlplus as apps user

2.2.2.2.For prompt Enter Component Id: enter id corresponds to Workflow Notification Mailer

2.2.2.3.For prompt Enter the Comp Parameter Id to update enter id corresponds to parameter you want to change

2.2.2.4.For prompt Enter a value for the parameter enter your parameter value