CM terminated with status 139 (Segmentation fault) – Oracle Apps 11i

Problem Statement:

In one of the instance, I faced an issue regarding concurrent manager. None of the concurrent mangers were coming up. When I checked the Internal Monitor (Which is responsible for bring up Internal Manager) it was showing the status deactivated.

Click on Internal Monitor in CM Administration Screen -> Process -> Internal Manager logs

I see following error

======================================================================== Starting PQP10MS6_0729@PQP10MS6 Internal Concurrent Manager — shell process ID 14385 logfile=/slot02/appmgr/PQP10MS6comn/admin/log/PQP10MS6/PQP10MS6_0729.mgr PRINTER=noprint mailto=appmgr02 restart=N diag=N sleep=60 (default) pmon=20 (default) quesiz=1 (default) +—————————————————————————+ Application Object Library: Concurrent Processing version 11.5 Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved. Internal Concurrent Manager started : 29-JUL-2008 22:34:31 +—————————————————————————+

Spawned Process 14472 Process monitor session started : 29-JUL-2008 22:34:31 Spawned Process 14473 Spawned Process 14474 Starting INVTMRPM Concurrent Manager : 29-JUL-2008 22:34:31

Spawned Process 14475 sh: line 71: 14390 Segmentation fault (core dumped) FNDLIBR FND CPMGR “FNDCPMBR sysmgr=\”\” $maxreq $sleep $pmon $quesiz $diag logfile=$logfile $target” <<STOP $sysmanager STOP The PQP10MS6_0729@PQP10MS6 internal concurrent manager has terminated with status 139 – giving up.

Cause:

stub files are incorrect and are referring to iAS stubs. Check metalink note ID 343249.1 for more details.

Solution:

Apply patch 3830807: PATCH 3293983 NEEDS TO INCLUDE ALL THE MAKEFILES UNDER THE 8.0.6 ORACLE_HOME

Once patch is applied relink all executables from adadmin -> 2. Maintain Applications Files menu -> 1. Relink Applications programs

Bounce all services. This will resolve the issue.

Hope this helps !!

References:

Metalink note ID 343249.1

Error Viewing CM log files – “File server could not verify its initialization parameters” – Oracle Apps 11i

Some times we face following error while viewing CM log files from Concurrent Manager Administer screen.

‘File server could not verify its initialization parameters’

Cause:

The issue is because of the syntax error in listener.ora file on appmgr side. If you check the parameter APPLFSTT in listener.ora on appmgr side (cd $TNS_ADMIN), you will see there is a space between the value.

Example:

APPLFSTT=PQP10MS6_806_BALANCE; PQP10MS6_FO; PQP10MS6_BALANCE; PQP10MS6

FNDFS cannot interpret the space between the value

Workaround:

Remove the space between the values and bounce apps listener.

Permanent Fix:

As per metalink note ID 304568.1, apply patch 4244610 to the environment.

Hope this helps !!

Important tables for ADPATCH

Here are some of the important tables used by and updated by ADPATCH utility.

AD_APPL_TOPS

This table holds the various APPL-TOP’s in the Oracle Applications installation that have ever been patched.

AD_APPLIED_PATCHES

AD_APPLIED_PATCHES holds information about the “distinct” Oracle Applications patches that have been applied. If 2 patches happen to have the same name but are different in content (eg. “merged” patches), then they are considered distinct and this table will therefore hold 2 records.

AD_BUGS

AD_BUGS holds information about the various Oracle Applications bugs whose fixes have been applied (ie. patched) in the Oracle Applications installation.

AD_PATCH_DRIVERS

This table holds information about the patch drivers that comprise a patch.

AD_FILE_VERSIONS

This table holds the various versions of Oracle Applications files (real files, not “pseudo-files”), that have ever been patched or executed in the Oracle Applications installation.

AD_FILES

AD_FILES is the “files repository”. It contains information about the various files that have been patched in the Oracle Applications installation.
Some entries are “pseudo-files” and not real files, (eg. directories) in which case some of the columns are not applicable and would then hold the value “DUMMY”

AD_PATCH_DRIVER_LANGS

NLS patches (or more specifically, NLS patch drivers) pertain to a language or multiple languages. This table holds that language (or multiple languages).

AD_PATCH_DRIVER_MINIPKS

This table holds information about the various Mini Packs contained in a patch (driver)

AD_PATCH_RUN_BUG_ACTIONS holds the various actions present in “applied” bug (fix). If Autopatch determined not to apply a bug (fix), then this table will not hold any records for that “unapplied” bug fix.

AD_PATCH_RUN_BUG_ACTIONS

Even though a patch may have been applied on an Oracle Applications installation, some actions in some of its included bugs (fixes) may not have got executed if the “Autopatch” utility determined that it was not necessary to execute those actions. In such cases, EXECUTED_FLAG is set to N.

AD_PATCH_RUN_BUGS

This table holds information about the bugs fixed in a specific run of Autopatch.
AD_PATCH_RUN_BUGS holds information about the various bugs fixed in a specific run of Autopatch.

Even though a patch may have been applied on an Oracle Applications installation, some bugs (fixes) contained in it may not get applied due to some reason. In such cases, the REASON_NOT_APPLIED column holds the reason.

AD_PATCH_RUNS

AD_PATCH_RUNS holds information about the various invocations of Autopatch for applying Oracle Applications patches to a specific release of an Oracle Applications installation.

If multiple drivers are run in one invocation of Autopatch, they result in multiple records in this table. These multiple records will all have the same SESSION_ID (because they arose from one Autopatch invocation), but different TASK_NUMBER’s. The TASK_NUMBER’s in this case will be numbered sequentially as 1, 2, 3, etc.
Note that when the database driver of a Maintenance Pack is applied, it bumps up the release version by creating a new record in AD_RELEASES, which is then pointed to by the UPDATED_TO_RELEASE_ID column of the old record.

AD_RELEASES

AD_RELEASES holds the various Oracle Applications releases that an installation of Oracle Applications has gone through in its entire life cycle.

It should be noted that START_DATE_ACTIVE, END_DATE_ACTIVE and BASE_RELEASE_FLAG are loosely-maintained informational columns and are not accurately maintained, and therefore should not be relied upon heavily.

AD_PATCH_COMMON_ACTIONS

This table holds distinct information about the various actions that are (often repeatedly) performed by Autopatch as part of applying patches.

Hope this helps !!

References:

Metalink ETRM DIRECT ACCESS DOCUMENT 150230.1

Patching history of a file – adfhrept.sql

Oracle Apps 11i and E-Business Suite R12 provides a wonderful utility to get the history of any file applied through patches. When a patch it applied higher version file get implemented in application. This utility will give complete history about a file starting from the initial release till now and which patch has introduced which release at what date.

The file name is adfhrept.sql and is present under $AD_TOP/patch/115/sql

Following is the usage of file.

adfhrept.sql <filename>\

<latest file version only? (Y/N)> \
<start date(mm/dd/rr or ALL)> \
<end date (mm/dd/rr or ALL)> <patchtype/ALL> <language/ALL> \
<appltop/ALL>

<limit to forms server? (Y/N)> \
<limit to web server?(Y/N)> \
<limit to node server? (Y/N)> \
<limit to admin server?(Y/N)> \
<only patches that change DB? (Y/N)>

For example:

-bash-2.05b$ sqlplus apps/apps @adfhrept.sql adphst.odf N 12/01/00 12/31/08 ALL ALL ALL N N N N N

SQL*Plus: Release 8.0.6.0.0 – Production on Mon Jun 30 12:56:21 2008

(c) Copyright 1999 Oracle Corporation.  All rights reserved.

Connected to:
Oracle8i Enterprise Edition Release 8.1.7.4.0 – Production
With the Partitioning option
JServer Release 8.1.7.4.0 – Production

Writing data to report file adfilerep.xml…

Done writing data to report file adfilerep.xml

Note:
To view the XML report from browser
Copy file adfilerep.xml to OA_HTML top directory

Disconnected from Oracle8i Enterprise Edition Release 8.1.7.4.0 – Production
With the Partitioning option
JServer Release 8.1.7.4.0 – Production

This sql will create an XML file at the same location from where you are running this utility. You need to mode the XML file to OA_HTML where its corresponding XSLT file is present. You can then access this XML file using the URL: http://(hostname):(Apache port)/OA_HTML/adfilerep.xml

The output will looks like this.

Hope this helps !!

Reference:

Metalink Note ID: 162498.1

AutoPatch modes, arguements and options

Here are the various options for applying the patch in Oracle apps 11i and R12. ADpatch comes with lots of option that can be used, especially when we are applying the patch in production.

Modes of ADPATCH

If we talk about the mode of applying patch, there are 3 modes

1) Pre-Install Mode

Pre-install mode is used to update AD utilities before an upgrade and to apply family consolidated upgrade packs.
AutoPatch Pre-AutoInstall mode allows you to apply patches when your installation is missing database information and/or filesystem information that AutoPatch requires to run in normal mode.

Examples of when you would run AutoPatch in Pre-AutoInstall mode (and cannot run it in normal mode) include:

  • Prior to installing Oracle Applications for the first time
  • Prior to upgrading Oracle Applications to the latest release.
  • During an upgrade (to apply a bug fix that stopped your upgrade)

Applying patch in pre-install mode performs following tasks:

  • Version checking
  • File copy actions
  • Relink FND and AD executables
  • Save patch history information to file system

AutoPatch in pre-install mode will NOT:

  • Run SQL of EXEC command
  • Generate files
  • Read product driver files
  • Apply maintenance pack

To apply patch in pre-install mode, run  adpatch preinstall=y

2) Test Mode

AutoPatch provides a test mode in which it tells you everything it would have done in applying a patch, but doesn’t actually apply the patch.

To run AutoPatch in Test Mode, you must include ‘apply=no’ on the AutoPatch command line. For example:

$ adpatch apply=no

Instead of performing an action, AutoPatch indicates that it is not performing the action because “Apply=No”. In general, AutoPatch lists each file it would have copied, generated, relinked, or executed. This shows you exactly what actions it would have performed.

AutoPatch test mode works the same as normal mode, with the following exceptions:

  • It does not copy any files from your patch directory into your installation area.
  • It does not copy any files from your APPL_TOP to JAVA_TOP or OAH_TOP.
  • It does not archive any object modules into your product libraries.
  • It does not generate any forms or reports.
  • It does not relink any executables.
  • It does not run any ‘sql’ or ‘exec’ commands.
  • It does not update the release version in the database.
  • It does not update the patch history file.

AutoPatch asks you the same initial questions in test mode as in normal mode. It performs the following actions to determine what it would have done if run in normal mode:

  • Reads and validates the patch driver file.
  • Reads product file driver files.
  • Extracts object modules from your product libraries (so it can perform version checking on the object modules it extracts).
  • Performs version checking.
  • Looks in the database to determine what ‘sql’ and ‘exec’ comands it would have run.

Its a good practice to run the patch in test mode and analyze the things before applying the patch in normal mode.

3) Non-Interactive Mode

Starting in Release 11.5, you can run AutoPatch non-interactively.

Creating a defaults file

Before you can run AutoPatch non-interactively, you must first create an AutoPatch defaults file for your current environment.

Here is a simple way to create an AutoPatch defaults file for your current environment:

1. Specify defaultsfile=<New Defaults File Name> on the AutoPatch command line. The defaults file must be located under $APPL_TOP/admin/<SID>.

For example:

adpatch defaultsfile=$APPL_TOP/admin/testdb1/my_def.txt

2. Run AutoPatch up to the point where it asks you for the directory where your Oracle Applications patch has been unloaded. Then type ‘abort’ at this prompt.

3. Verify that your defaults file exists.

Once you have an AutoPatch defaults file for your current environment, you can run AutoPatch non-interactively.

Applying a single patch driver file non-interactively

Before applying any Oracle Applications patch, either interactively or non-interactively, you should read the README file (usually called readme.txt) supplied with the patch. You should also read the documentation supplied with the patch (if any).

It is possible to apply just a single patch driver file non-interactively using AutoPatch. Here is an example:

Assume the following:

  • defaults file is $APPL_TOP/admin/testdb1/def.txt
  • Applying copy driver for patch 123456, which is located in directory $APPL_TOP/patch/123456.
  • Using three parallel workers
  • AutoPatch log file name is cpy123456.log

The AutoPatch command line would be:

adpatch defaultsfile=$APPL_TOP/admin/testdb1/def.txt \
logfile=cpy123456.log \
patchtop=$APPL_TOP/patch/123456 \
driver=c123456.drv \
workers=3 \
interactive=no

If we dont give any of the mode as mentioned above and apply the patch simply using adpatch command then its a normal mode of patch application.

Having seen the modes of patch application, now we will see various arguements for applying patch.

1) defaultsfile
Purpose: This option is used when we are running the patch in non interactive mode. In that case we create defaults file and provide that file as an option for running patch in non-interactive mode.
Default: none. No default file read or written.

2) logfile
Purpose: This is the name of adpatch log file which it will write during patch application.
Default: none. Adpatch prompts for this value.

3) workers
Purpose: Specifies the number of workers to run. This value depends on number of CPU and other factors.
Default: none. Adpatch prompts for this value.

4) patchtop
Purpose: Top-level directory for the current patch. This is the directory after unzipping the patch. This directory will a patch number.
Default: none. Adpatch prompts for this value.

5) driver
Purpose: Name of the patch driver file. This comes with the patch and is present in patch directory.
Default – none. Adpatch prompts for this value.

6) restart
Purpose: To restart an existing session. Only valid when interactive=no is also specified
Default: No

7) localworkers
Purpose: Used in Distributed AD to specify the number of workers to be run on the current machine. If you have multi node instance (example RAC and shared APPL_TOP), then you can utilize this paramter to run the patch parallely in multiple nodes. You can start few workers on node 1, few on node 2 and so on. The way this can be done is that, you can start adpatch on one node with localworker=<some value less then total workers>. Then run adctrl on other node in distributed mode and start some mode workers. This will speed up the process and utilized the resources effectively.
Default: Value specified for workers.

8) printdebug
Purpose: To display extra debugging information.
Default: No.

Now lets consider some common options that can be used with adpatch options=<value>

1) checkfile
Purpose: To skip running exec, SQL, and exectier commands if they are recorded as already run. Indicates that Autopatch should run the command *only* if a certain file is newer than the version of it that was last run. The idea behind it is to reduce the duration of an Autopatch session by skipping actions that don’t really need to be performed. When used in the right manner, it can dramatically improve Autopatch performance, especially for big patches and/or long running actions.
Default: checkfile (use ‘nocheckfile’ to skip)

2) compiledb
Purpose: To compile invalid objects in the database after running actions in the database driver.
Default: compiledb (use ‘nocompiledb’ to skip)

3) compilejsp
Purpose: To compile out-of-date JSP files, if the patch has copy actions for at least one JSP file.
Default: compilejsp (use’nocompilejsp’ to skip)

4) copyportion
Purpose: To run commands found in a copy driver. This will copy the higher version files from patch to product top.
Default: copyportion (Use ‘nocopyportion’ to skip. Use it only when mentioned in readme of patch)

5) databaseportion
Purpose: To run commands found in a database driver. This portion includes applying the files (like sql, pls etc) to database.
Default: databaseportion (use ‘nodatabaseportion’ to skip. Use it only when mentioned in readme of patch)

6) generateportion
Purpose: To run commands found in a generate driver. This portion will generate new executable files from the copied code of patch. For example if will generate new forms files (fmx) from new .fmb files.
Default: generateportion (use ‘nogenerateporation’ to skip)

7) integrity
Purpose: To perform patch integrity checking. Tells adpatch whether to perform patch integrity checking, which verifies that the version of each file referenced in a copy action matches the version present in the patch.
Default: nointegrity (By default the integrity is not checked)

8) maintainmrc
Purpose: To maintain the MRC schema after running actions found in the database driver.
Default: maintainmrc (use ‘nomaintainmrc’ to skip)

9) autoconfig
Purpose: Tells adpatch to run Autoconfig after patch installation.
Default: autoconfig (use ‘noautoconfig’ to skip)

10) parallel
Purpose: To run actions that update the database or actions (like SQL) that generate files in parallel (like genform).
Default: parallel (use ‘noparallel’ to skip)

11) prereq
Purpose: Tells adpatch whether to perform prerequisite patch checking prior to running patch driver files that contain actions normally found in the copy driver.
Default: prereq (use ‘noprereq’ to skip)

12) validate
Purpose: To connect to all registered Oracle Applications schemas at the start of the patch. Adpatch validates the passwords for each schema.
Default: novalidate (use ‘validate’ to validate schema passwords)

Following flags can be passed to adpatch

1) hidepw
Purpose: This argument is used to hide the passwords in log files
Default: nohidepw

2) trace
Purpose: Tells the adpatch utility whether to log all database operations to a trace file
Default: notrace

3) logging
Purpose: Tells the adpatch utility whether to create indexes using the logging or nologging mode.
Default: logging

Hope this helps !!

Rolling Back Previous Autoconfig Run

In case your autoconfig run has created some issues in your environment and you want to rollback autoconfig run, you can do so by running restore.sh script.

For every autoconfig run, autoconfig will create a directory “MMDDhhmm” under APPL_TOP/admin/TWO_TASK/out. Inside these directories, there will be several files which autoconfig has taken a backup off before making changes. Also there will be a script restore.sh. This script is going to copy these backup files to the original location and this has the effect to rolling back autoconfig run.

(appmgr02) 06190950 – -bash $ pwd
/SLOTS/slot02/appmgr/PQDC2MS1appl/admin/PQDC2MS1/out/06190950
(appmgr02) 06190950 – -bash $ restore.sh

Hope this helps !!

Running Autoconfig in Test Mode

Before running the autoconfig in production to propogate your changes, its usually a good idea to run autoconfig in test mode. This will reduce the risk and you can understand the impact better. AD tool provides a script to run autoconfig in test mode. This script will create a report which will list all files & profile options that is going to change when you run AutoConfig.

Below is the sample run for the same.

(appmgr02) bin – -bash $ adchkcfg.sh
Enter the full path to the Applications Context file:
/slot02/appmgr/PQDC2MS1appl/admin/PQDC2MS1.xml
Enter the APPS user password:

AutoConfig is running in test mode and building diffs…

AutoConfig will consider the custom templates if present.
Using APPL_TOP location     : /slot02/appmgr/PQDC2MS1appl
Classpath                   : /local/java/jdk1.4.2_04/jre/lib/rt.jar:/local/java/jdk1.4.2_04/lib/dt.jar:/local/java/jdk1.4.2_04/lib/tools.jar:/slot02/appmgr/PQDC2MS1comn/java/appsborg2.zip:/slot02/appmgr/PQDC2MS1comn/java

Using Context file          : /slot02/appmgr/PQDC2MS1appl/admin/PQDC2MS1/out/06190701/PQDC2MS1.xml

Context Value Management will now update the test Context file
Updating test Context file…COMPLETED

[ Test mode ]
No uploading of Context File and its templates to database.

Testing templates from all of the product tops…
Testing AD_TOP……..COMPLETED
Testing FND_TOP…….COMPLETED
Testing ICX_TOP…….COMPLETED
Testing IEO_TOP…….COMPLETED
Testing ABM_TOP…….COMPLETED
Testing EMS_TOP…….COMPLETED
Testing ECX_TOP…….COMPLETED
Testing BIS_TOP…….COMPLETED
Testing GL_TOP……..COMPLETED
Testing AMS_TOP…….COMPLETED
Testing CCT_TOP…….COMPLETED
Testing WSH_TOP…….COMPLETED
Testing CLN_TOP…….COMPLETED
Testing OKE_TOP…….COMPLETED
Testing OKL_TOP…….COMPLETED
Testing OKS_TOP…….COMPLETED
Testing CSF_TOP…….COMPLETED
Testing XNC_TOP…….COMPLETED
Testing IGS_TOP…….COMPLETED
Testing IBY_TOP…….COMPLETED
Testing PA_TOP……..COMPLETED
Testing JTF_TOP…….COMPLETED
Testing MWA_TOP…….COMPLETED
Testing CN_TOP……..COMPLETED
Testing CSI_TOP…….COMPLETED
Testing WIP_TOP…….COMPLETED
Testing CSE_TOP…….COMPLETED
Testing EAM_TOP…….COMPLETED
Testing IMT_TOP…….COMPLETED
Testing FTE_TOP…….COMPLETED
Testing ONT_TOP…….COMPLETED
Testing AR_TOP……..COMPLETED
Testing AHL_TOP…….COMPLETED
Testing OZF_TOP…….COMPLETED
Testing IES_TOP…….COMPLETED
Testing CSD_TOP…….COMPLETED
Testing IGC_TOP…….COMPLETED

Differences text report is located at: /SLOTS/slot02/appmgr/PQDC2MS1appl/admin/PQDC2MS1/out/06190702/cfgcheck.txt

Generating Profile Option differences report…COMPLETED
Generating File System differences report……COMPLETED
Differences html report is located at: /slot02/appmgr/PQDC2MS1appl/admin/PQDC2MS1/out/06190702/cfgcheck.html

Differences Zip report is located at: /slot02/appmgr/PQDC2MS1appl/admin/PQDC2MS1/out/06190702/ADXcfgcheck.zip

AutoConfig completed successfully.
The log file for this session is located at: /slot02/appmgr/PQDC2MS1appl/admin/PQDC2MS1/log/06190701/adconfig.log

Once autoconfig is run on test mode, it will generate a zip file (ADXcfgcheck.zip) as shown above. You can copy the zip file to $OA_HTML location and unzip the same. You can then view the report using following URL

http://hostname:port/OA_HTML/cfgcheck.html

Hope this helps !!