All About Apache Issues ...

0)General Information to Troubleshoot Apache Issues.

************************************************************************************************************************

Sometimes there will be problem in our Apache Server where although the status shown is not running, there was still httpd running on the background.

What you can do is shutdown the Apache listener process and clear all the httpd background process.

$ ps -ef | grep Apache | wc -l

28

$ cd $COMMON_TOP/admin/scripts/$CONTEXT_NAME

$ ./adapcctl.sh status

Apache Web Server Listener is not running.

Apache Web Server Listener (PLSQL) is not running.

adapcctl.sh: exiting with status 1

$ ./adapcctl.sh stop

adapcctl.sh version 115.55

Apache Web Server Listener is not running.

Apache Web Server Listener (PLSQL) is not running.

adapcctl.sh: exiting with status 0

$ ps -ef | grep Apache | wc -l

28

$ kill -9 `ps -ef | grep Apache | grep applprod | grep -v grep | awk '{print $2}'`

$ ps -ef | grep Apache | wc -l

1

$ ./adapcctl.sh start

adapcctl.sh version 115.55

Apache Web Server Listener is not running.

Starting Apache Web Server Listener (dedicated HTTP in SSL mode) ...

Apache Web Server Listener (PLSQL) is not running.

Starting Apache Web Server Listener (dedicated PLSQL) ...

adapcctl.sh: exiting with status 0

$ ps -ef | grep Apache | wc -l

22

$ ps -ef | grep Apache | wc -l

28

***************************************************************************************************************************************

1)If you mistype any directive in Apache (httpd.conf or any conf file), Apache will not start & it will not write even in error_log. In these case it will report any error message to file

$COMMON_TOP/admin/log/$CONTEXT_NAME/adapcctl.txt

So first check in the above file

***************************************************************************************************************************************

2)Found *.pid file in Apache_Top/Apache/logs/ even after bounce of an Apache web server (When Apache starts it creates .pid file in logs directory and when you shutdown apache it cleans this .pid file). So if this file is already there then Apache will not start.

Fix: Simply move this file

***************************************************************************************************************************************

3)Could not bind port (Port already in Use) or Sometimes apache and concurrent

Manager will interchange their port no.’s while starting while results error and both of them will not up

Fix: To overcome this up the services in this manner

1)Listener

2)Apache web server

3)Concurrent manager

4)adstrtal.sh

***************************************************************************************************************************************

4)Error 500 - Internal server error

Fix: 1) Clear Cache (Application Server Cache)and start the services

***************************************************************************************************************************************

5)If there are no error messages in error_log or error_log_pls

then there might be issues in starting Jserv component of web server (which caters Java requests by mod_jserv)

***************************************************************************************************************************************

6)Mobile is enabled in zone.properties disable it if you are not using it.

***************************************************************************************************************************************

7)If you don't know port number of your web server you can check it in file $IAS_ORACLE_HOME/Apache/Apache/conf/httpd.conf with directive Listen & Port

***************************************************************************************************************************************

8)Once you are able to access http://hostname.domainname:port and if you are getting any error message that means your web server (Core Apache) is working but there are various components called as modules and there may be issues with those modules which we will check these modules . few modules of our interest are

-mod_jserv

-mod_pls

-mod_oprocmgr

-mod_ossl (If you have ssl enabled on web server)

-mod_osso (If single sign on server is enabled on web server)

***************************************************************************************************************************************

9)Once in a while adapcctl.sh start will not work then ultimate you have to go and

invoke apachectl script file in Apache top/apache/bin to start Apache webserver.

***************************************************************************************************************************************

10)If we remove _pages from COMMON_TOP it will be created for the next time only when we bounce and up the Apache web server.

***************************************************************************************************************************************

11)Jsp pages are complied with the file ojspcompile.pl executable. It is located at $JSP_TOP/bin/ and in R12 it is there in

$FND_TOP/patch/115/bin/

***************************************************************************************************************************************

12)Error : Your session is no longer valid while trying to loging to applications.

The issue is resolved by changing the profile option session_cookie_domain

to null from domain.com

***************************************************************************************************************************************

13)HTTP-403 forbidden message after entering the username and password in the "Apps Logon Links" page. [Not authorised to view this page]

Check for Apps password in $APACHE_TOP/modplsql/cfg/wdbsvr.app ,Comment out custom_auth entry in $APACHE_TOP/modplsql/cfg/wdbsvr.app and bounce apache.

***************************************************************************************************************************************

14)Error message: You have insufficient privileges for the current operation.

Verify and update SESSION_COOKIE_DOMAIN in ICX_parameters if it is wrong.

Example :

update icx_parameters set SESSION_COOKIE_DOMAIN='NEW_VALUE' where SESSION_COOKIE_DOMAIN='OLD_VALUE'

***************************************************************************************************************************************

15)AppsLocalLogin?.jsp page error outs

Edit the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties

Set wrapper.bin.parameters=-DLONG_RUNNING_JVM=false

add the line wrapper.bin.parameters=-DCACHEMODE=LOCAL

Restarted Apache for these changes to take effect and retested the issue

16) Login to the applications URL fails with

500 Internal server Error

java.lang.NoClassDefFoundError at

oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:126) at

oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:170) at com.evermind[Oracle

Containers for J2EE 10g (10.1.3.0.0)

].server.http.HttpApplication.loadServlet(HttpApplication.java:2231) at com.evermind[Oracle

Containers for J2EE 10g (10.1.3.0.0)

].server.http.HttpApplication.findServlet(HttpApplication.java:4617) at com.evermind[Oracle

Containers for J2EE 10g (10.1.3.0.0)

].server.http.HttpApplication.findServlet(HttpApplication.java:4541) at com.evermind[Oracle

Containers for J2EE 10g (10.1.3.0.0)

Performed the below steps, but it did not resolved the issue

1) Bounced Apache server, cleared cache and validated ports

2) Bounced all Middle tier services and killed defunct processes

3) Bounced Unix Server

Solution

=======

Java object Caching Configuration parameter was Enabled in oc4j.properties

$INST_TOP/ora/10.1.3/j2ee/oacore/config

Disabled LONG_RUNNING_JVM=false from True

Bounced Apache server and could login to the applications URL without issues

ISSUES AND SOLUTIONS

ISSUE 1:

Description

When Apache script adapcctl.sh is issued.It shows that Apache is started.On checking the status Apache is stopped

Cause

The error log file in the folder

cd $APACHE_TOP/Apache/logs

Showed

fopen: No such file or directory httpd: could not log pid to file /var/opt/_/Apache/logs/httpd_pls.pid

SOLUTION 1

Created the missing directories /_/Apache/logs under /var/opt folder

Bounced Apache

ISSUE 2:

Description

oacore was not comming up in the configurator node.

opmnctl status

ias-component process-type pid status

OC4J oacore N/A Down

OC4J oacore N/A Down

OC4J oacore N/A Down

OC4J oacore N/A Down

Because of this configurator cannot be invoked.

Cause

Check the oacore error log.

Example:

Below Log file gives root cause. //mtlog/_/logs/ora/10.1.3/opmn/oacore_default_group_2/oacorestd.err

14:11:56 Error initializing server: //inst/apps/_/ora/10.1.3/j2ee/oacore/config/server.xml,

Fatal error at line 70 offset 1 in file://inst/apps/_/ora/10.1.3/j2ee/oacore/config/server.xml: .:

XML-20100: (Fatal Error) Expected 'EOF'.

SOLUTION 2

Edit the /sid/inst/apps/SID_hostname/ora/10.1.3/j2ee/oacore/configserver.xml file and remove the extra line in the file. Restart and check oacore will be starting. This sudden change in server.xml seems to be because of a bug.

Bug - 6702510

ISSUE 3

AppsLocalLogin?.jsp page was displaying errors

Cause

Extract from the Jserv log file =========================== [27/02/2008 16:57:14:447 CST] weboamLocal/oracle.apps.fnd.oam. servlet.ui.OAMServlet:

Sucessfully initialize oaosu.OAMServlet [27/02/2008 16:57:14:447 CST] Servlet Zone rootauohstrzt11 initia lization complete [27/02/2008 16:57:15:101 CST]

Creating noun for: oracle.jsp.JspServlet [27/02/2008 16:57:15:102 CST] Creating noun for: AppsLocalLogin?.jsp

[27/02/2008 16:57:15:120 CST] oracle.jsp.JspServlet: init [27/02/2008 16:57:28:066 CST] Creating noun for: oracle.jsp.JspServlet

[27/02/2008 16:57:28:067 CST] Creating noun for: AppsLocalLogin?.jsp [27/02/2008 16:57:28:078 CST] oracle.jsp.JspServlet: init

[27/02/2008 16:57:29:753 CST] JspServlet?: unable to dispatch to requested page: Exception:java.lang.NoClassDefFoundError

SOLUTION 3:

Edited the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties

a) Set wrapper.bin.parameters=-DLONG_RUNNING_JVM=false

b) Add the line wrapper.bin.parameters=-DCACHEMODE=LOCAL

c) Restarted Apache for these changes to take effect and retested the issue

ISSUE 4

New Message when starting Apache using adapcctl.sh start

Cause

It takes 7/10 minutes to clear this cache directory if present

apache Web Server Listener is not running.

Starting Apache Web Server Listener (dedicated HTTP) ...

Removing gantt cache directory

Successfully removed the gantt cache directory:/tcge1i/applmgr/common/html/cabo/images/cache/gantt

Apache Web Server Listener (PLSQL) is not running.

Starting Apache Web Server Listener (dedicated PLSQL) ...

SOLUTION 4

Wait for atleast 7 to 10 minutes and this cache directory will be cleared and Apache will be started successfully

ISSUE 5

Apache server is up and running.

But adapcctl.sh status shows

adapcctl.sh version 115.54

Apache Web Server Listener is not running. Apache Web Server Listener (PLSQL) is not running.

adapcctl.sh: exiting with status 1

Cause

Incorrect Lock file path given in httpd.conf

LockFile? $APACHE_TOP/Apache/log/PUPONI_httpd.lock

and

Incorrect path given in adapcctl.sh script.

SOLUTION 5:

Modified the path of lock file in $APACHE_TOP/Apache/conf/httpd.conf file.

LockFile? /var/opt/SID_/httpd.lock

and

Modified the path of adapcctl.sh file by commenting the wrong path

ISSUE 6

After selecting the E-Business home page got internal server error Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Cause

Found that all the properties files as below

jserv.properties jserv_restrict.properties zone.properties zone_restrict.properties

were pointing to a dbc file name _.dbc

which was not physically present at the path

/sid/applmgr/11510/fnd/11.5.0/secure/_

$ cd //applmgr/11510/fnd/11.5.0/secure/_

$ ls .dbc

SOLUTION 6:

copied the file

twwyai.dbc

which was physically present at path

//applmgr/11510/fnd/11.5.0/secure/_

to the dbc file name which was being pointed by all the properties files

$ cp .dbc _>sid>.dbc

$ ls

_>sid>.dbc .dbc

This resolved the issue.

ISSUE 7

Instance is very slow / inaccessible with a warning on login page:

Warning

Low-level logging is currently enabled. Your application will not perform as well while Low-level logging is on.

Cause

Logging was enabled through profile values:

FND: Debug Log Enabled -- Yes

FND: Debug Log Level -- Statement

FND: Debug Log Module -- %

SLA: Enable Diagnostics -- Yes

FND Validation Level -- Error

SOLUTION 7

1. Change these values to:

FND: Debug Log Enabled -- No

FND: Debug Log Level -- ( Blank )

FND: Debug Log Module -- ( Blank )

SLA: Enable Diagnostics -- No

FND Validation Level -- None

2. Bounce Apache on all the MTs and clear Apache cache.

ISSUE 8

Login to the applications URL fails with

500 Internal Server Error java.lang.NoClassDefFoundError at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect?.java:126) at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect?.java:170) at com.evermind[Oracle Containers for J2EE? 10g (10.1.3.0.0) ].server.http.HttpApplication.loadServlet(HttpApplication?.java:2231) at com.evermind

Performed the below steps, but it did not resolved the issue

1) Bounced Apache server, cleared cache and validated ports

2) Bounced all Middle tier services and killed defunct processes

3) Bounced Unix Server

Cause

No Cause Entered

SOLUTION 8

Java object Caching Configuration parameter was Enabled in oc4j.properties $INST_TOP/ora/10.1.3/j2ee/oacore/config

Disabled LONG_RUNNING_JVM=false from True

Bounced Apache server and could login to the applications URL without issues

Java Caching is a mechanism used to improve performance by storing data in the Middle Tier JVM memory, instead of retrieving the data from the database every time it is requested.

ISSUE 9

When trying to access the URL (Either Big IP or the node level), it hangs for a long time and getting redirected to a different URL and gives a blank page. For example, When try to access the ipayments URL which is hosted on node2, it hangs for a long time and its getting redirected towards the Sales URL which is hosted on node3, and displays a blank page.

Cause

All the host reference and the URL directives are pointing towards node3 and sales URL respectively, in the below files. Hence every attempt tries to launch the sales URL and get fails as the apache configuration files and parameters are pointing towards the correct one.

jserv.conf forms.properties zone.properties jserv.properties

SOLUTION 9

Stop Apache. Edit the files jserv.conf, forms.properties, zone.properties, jserv.properties, jserv.conf to replace the host reference and the URL directives to the correct one (In our case, its node3 . Clear the cache and restart the Apache.

ISSUE 10

Apache login issues occurs in multi MT environment. Unable to login to oracle applications.

Cause

Issue due to distributed cache.

SOLUTION 10

Set is_distributed to false in javacache.xml Bounce Apache.

ISSUE 11

Apache server is down. Apache logfile did not provide useful information.

Started the Apache server, but the Apache Web listener did not started.

1) Verified no log files (Apache,Jserv) crossed 2GB limit. error_log, access.log, mod_jserv

2) Veified that the port is listening (netstat -an |grep ) Apache port(httpd.conf), pls listener(httpd_pls.conf)

3) Verified that the Apache and Jserv configuration files are not changed recently httpd.conf & Jserv.conf

4) Shutdown all the Middle tier services and check for ipcs (semaphores) did not resolve the outage.

$ ipcs on application middle tier as Application user ap

Cause

No Cause Entered

SOLUTION 11

Rebooted the server and started the services. All services came back normal.

ISSUE 12

Unable to login to the home page URL. Login fails with the error:

Error Page

You have encountered an unexpected error. Please contact the system administrator for assistance.

Click here for exception details

Checks:

Upon clicking the link, can observe the error:

oracle.apps.fnd.framework.OAException: Application: FND, Message Name: SQL_PL SQL_ERROR. Tokens: ROUTINE = createSession(int)(userId=6,sessionMode='null','2D8 FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412?'); REASON = java.s ql.SQLException: ORA-01422: exact fetch returns more than requested number of ro ws ORA-06512: at "APPS.FND_SESSION_MANAGEMENT", line 295 ORA-06512: at "APPS.FND_SESSION_MANAGEMENT", line 455 ORA-06512: at line 1 ; ERRNO = 1422;

Cause

FND_NODES table was updated with the same server_id for all Nodes.

SOLUTION 12

Update the table FND_NODES with correct server_ids for all nodes as in $FND_TOP/secure/sid_host/sid.dbf

Bounced apache on all apache nodes.

and Run select node_name, server_id from fnd_nodes;

to confirm the values.

Below is the example for the solution:

Found three hosts using same server_id:

SQL> select node_name, server_id from fnd_nodes; NODE_NAME SERVER_ID

AUOHSACPS06 2D8FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412

AUOHSACPS03 2D8FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412

AUOHSACPS02 2D8FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412

reset server_id with original server_id from each hosts .dbc file @ $FND_TOP/secure:

SQL> select node_name, server_id from fnd_nodes; NODE_NAME SERVER_ID

AUOHSACPS06 2D8FF21843830F82E040558CDBC6195C27690802237516122831445560236127

AUOHSACPS03 2D90005D5A13C774E040558CDBC61D9436546148971863330582284123371818

AUOHSACPS02 2D8FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412

ISSUE 13

Users getting '404 page not found errors' when tyring to access the URL

(OR)

Getting Exception in thread "main" java.lang.ClassFormatError: Truncated class file

(OR)

adstrtal.sh or adstpall.sh is throwing below error message

You are running adstrtal.sh version 115.16

Exception in thread "main" java.lang.ClassFormatError: oracle/apps/ad/context/AppsContext (Truncated class file) at java.lang.ClassLoader.defineClass0(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader?.java:539) at java.security.SecureClassLoader.defineClass(SecureClassLoader?.java:123) at java.net.URLClassLoader.defineClass(URLClassLoader?.java:251) at java.net.URLClassLoader.access$100(URLClassLoader?.java:55) at java.net.URLClassLoader$1.run(URLClassLoader?.java:194) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader?.java:187) at java.lang.ClassLoader.loadClass(ClassLoader?.java:289) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:274) at java.lang.ClassLoader.loadClass(ClassLoader?.java:235) at java.lang.ClassLoader.loadClassInternal(ClassLoader?.java:302) at oracle.apps.ad.autoconfig.ServiceControl.(ServiceControl?.java:319) at oracle.apps.ad.autoconfig.ServiceControl.main(ServiceControl?.java:662) Check logfile //applmgr/common/admin/log/_/08061104.log for details

Exiting with status 1

(OR)

[06/08/2007 11:03:12:181] (ERROR) an error returned handling request via protocol "ajpv12" [06/08/2007 11:03:12:181] (EMERGENCY) ajp12: can not connect to host140.85.188.36:20021 [06/08/2007 11:03:13:184] (EMERGENCY) ajp12: can not connect to host 140.85.188.36:20021 [06/08/2007 11:03:15:038] (EMERGENCY) ajp12: auth did not receive challenge size [06/08/2007 11:03:15:038] (EMERGENCY) ajp12: auth fail

Cause

Extract from the JVM log file: //product/iAS/Apache/Jserv/logs/jvm/OACoreGroup.0.stderr

Exception in thread "main" java.lang.ClassFormatError: Truncated class file at org.apache.jserv.JServServletManager.load_init(JServServletManager?.java:765) at org.apache.jserv.JServServletManager.loadServlet(JServServletManager?.java:659) at org.apache.jserv.JServServletManager.loadStartupServlets(JServServletManager?.java:789) at org.apache.jserv.JServServletManager.init(JServServletManager?.java:447) at org.apache.jserv.JServ.start(JServ.java:625)

Verified that there are no log files that exceed 2GB limit (OS limit).

Clearing the Apache and Jserv cache did not resolve the issue. Unmount / mount product specific file system did not resolve

SOLUTION 13:

Try each of these solutions until the problem is resolved, in order to have the least impact on the system.

1. If you can determine the specific class file from the log files, use the 'touch' command.

e.g. touch classfile

If necessary, restart Apache and clear server cache.

2. Shut down all MT services, exit the Apps owner and ask the System Admin to unmount and remount the /sid/applmgr file system (mount point above $JAVA_TOP). Ask the System Admin to kill any processes that may be holding the file system open. Then log in to the Apps owner and restart MT services.

3. Shutdown all the MT services and ask System Admin to reboot the server if the actions above don't resolve the outage.

ISSUE 14:

Got login error - Internal Server Error after bounced all services on . find following message from Jserv log file:

$ tail -8 mod_jserv.log

[07/05/2007 18:41:06:915] (ERROR) ajp12: Servlet Error: java.lang.NoClassDefFoundError: null

[07/05/2007 18:41:06:916] (ERROR) an error returned handling request via protocol "ajpv12"

[07/05/2007 18:41:06:916] (ERROR) balance: 11906 internal servlet error in serverauohsnops03.oracleoutsourcing.com:20720

[07/05/2007 18:41:06:916] (ERROR) an error returned handling request via protocol "balance"

[07/05/2007 20:07:52:732] (ERROR) ajp12: Servlet Error: java.lang.NoClassDefFoundError: null

No runaway, no lock on port 20720.

SOLUTION 14:

Shutdown all services on MT. Removed all log file (standard error...) from Jserv/jvm.

Started All services with successful login.

ISSUE 15

Customer reported Not able to login with “Internal Server Error” on .

We could not start some of services during bounced All Services with following error message: adapcctl.sh version 115.47

Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.

Apache Web Server Listener is not running (dedicated HTTP) ...

Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.

Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.

Apache Web Server Listener is already stopped.

Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.

adapcctl.sh: exiting with status 2

Unable to login the application URL, facing the error while login:

Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, ap@domain.comand inform them of the time the error occurred, and anything you might have done that may have caused the error.

Cause:

SQL> select END_DATE from fnd_user where user_name='GUEST';

END_DATE

---------

05-OCT-10

SOLUTION 15

We found out the apps and applsys were not in sync.

Just manual change the apps password from sqlplus.

Restarted all services and login successfully.

or

Update the FND_USER table with END_DATE=null and bounced apache

SQL> update fnd_user set END_DATE=null where user_name='GUEST';

1 row updated.

SQL> commit;

Commit complete.

SQL> select END_DATE from fnd_user where user_name='GUEST';

END_DATE

-----------

ISSUE 16

Unable to login to oracle applications:

error message: You have insufficient privileges for the current operation.

Checks

Unable to login to oracle applications:

error message: You have insufficient privileges for the current operation.

SOLUTION 16

Verify and update SESSION_COOKIE_DOMAIN in ICX_parameters if it is wrong.

Example :

update icx_parameters set SESSION_COOKIE_DOMAIN='NEW_VALUE' where SESSION_COOKIE_DOMAIN='OLD_VALUE';

ISSUE 17

Apache home page is not coming up even after bounced

Found in jvm log asking for java version

SOLUTION 17

//product/iAS/Apache/Apache/bin/java.sh: /usr/java/jdk1.3.1_15/bin/java: not found

Check the version in /usr/java and update that version in java.sh file and Bounced Apache.

ISSUE 18

Unable to access applications

cause

All applications are up and running.

"Solaris Error: 12: Not enough space" errors are encounterd in database listener logfile.

SOLUTION 18:

Check if the OS is SUN Solaris.

Check if /tmp filesystem is 100% full.

Check Swap space utilization.

Check the application process that is hagging resources. (Check if any sql sessions are hagging resources)

Clear the process in issue after getting required approval.

NOTE:

The above is applicable only for environments on Solaris OS.

ISSUE 19

Not able to login to APPS. JSP Error:

Unable to create an user session or ORA-01403 : no data found has been detected in FND.SIGNON.AUDIT_USER

Checks

Check number of rows in dual table.

Select * from dual;

DUAL table should always contain only one row.

The table named DUAL is a small table in the data dictionary that Oracle and

user-written programs can reference to guarantee a known result.

This table has one column called DUMMY and one row containing the value X.

SOLUTION 19

DUAL table should always contain only one row and one column.

Insert a row with value X into dual table if the dual table contains 0 rows.

If the dual table contains more than one row then delete the rows by keeping one row.

The rows in dual table not equal to zero.

ISSUE 20

Not able to launch forms from PHP

Checks

Login to PHP.

Select system administrator responsibility and click on any menu option.

The page shows "done" without launching forms (jinit)

Check the apps version.

If the apps version is 11.5.10 then check the Check the file version of $FND_TOP/admin/template/appsweb.cfg

If the file version is 115.148, then verify that "!WindowsDPI= " exists in $FORMS60_WEB_CONFIG_FILE

The windowsDPI value should be 96. If the value for WindowsDPI is not set in the

$FORMS60_WEB_CONFIG_FILE then update PROFILE_OPTION_VALUE.

SOLUTION 20

select PROFILE_OPTION_VALUE from fnd_profile_option_values where PROFILE_OPTION_ID=3769;

Update the tar with the output of the above command.

UPDATE fnd_profile_option_values

set PROFILE_OPTION_VALUE='https://.domainname.com/dev60cgi/f60cgi?windowsDPI=96' where PROFILE_OPTION_ID=3769;

This should update only one row.

commit;

ISSUE 21

Signature for "https URL redirecting to http URL"

Issue:

https URL redirecting to http URL and the page cannot be displayed error occurs.

SOLUTION 21

Check the httpd.conf and url_fw.conf, in url_fw.conf the Rewrite rule was

RewriteRule ^/$/OA_HTML/AppsLocalLogin.jsp.

Changed into

RewriteRule https://: /OA_HTML/AppsLocalLogin.jsp.

***************************************************************************************************************************************

What is Oracle RAC One Node ?

Oracle introduced a new option called RAC One Node with the release of 11gR2 in late 2009. This option is available with Enterprise edition only. Basically, it provides a cold failover solution for Oracle databases. It’s a single instance of Oracle RAC running on one node of the cluster while the 2nd node is in a cold standby mode. If the instance fails for some reason, then RAC One Node detects it and first tries to restart the instance on the same node. The instance is relocated to the 2nd node in case there is a failure or fault in 1st node and the instance cannot be restarted on the same node. The benefit of this feature is that it automates the instance relocation without any downtime and does not need a manual intervention. It uses a technology called Omotion, which facilitates the instance migration/relocation. RAC One is Oracle’s answer or solution to OS clustering solution like Veritas Storage Foundation, Sun Solaris cluster, IBM HACMP, and HP Service guard etc.

Purpose

Its Oracle’s attempt to tie customers to a single vendor by eliminating the need to buy 3rd party OS cluster solutions. First, it introduced Oracle Clusterware with 10g and stopped the need to rely on 3rd party cluster software and now it intends to conquer the rest who are still using HACMP, Sun Solaris cluster etc. for cold failover.

Benefits

1. The Oracle RAC One node provides the following benefits:
2. Built-in cluster failover for high availability
3. Rolling patches for single instance database
4. Proactive migration / failover of the instance
5. Live migration of instances across servers
6. Online upgrade to RAC

 

The rolling upgrade is really useful. Upgrade to the OS, and Database can be done without any downtime unless upgrade requires some scripts to be run against the database. With RAC One Node, the DBA’s and Sys admins can be proactive and migrate/failover the instance to another node to perform any critical maintenance activity.

What it's not suited for

According to me the RAC one node is not a viable or recommended solution in the following scenarios:

1. To load balance unlike regular RAC
2. A true high availability solution
3. As a DR solution; Data guard best suits the bill
4. For mission critical applications

Cost

It is definitely not FREE. Oracle has priced RAC one at par with Active Data Guard. The RAC One node is priced separately and costs $10,000 per processor as against $23,000 for regular RAC. The licensing cost is required for ONE node only (in a 2-node setup). RAC one node is eligible for 10-day rule, allowing a customer to migrate to another without the need to buy additional license up to 10-days in a calendar year. People arguing against paying a license fee for resources they are not using will still lament.

Conclusion

I am still not very convinced on the usefulness of RAC one node. I think customers invest in RAC for their mission critical applications and achieving high availability and load balancing at the same time. Those who don’t go for RAC rely on Data Guard and now with 11g, on Active Data Guard. So don’t see a huge requirement for RAC One except seamless failover within a data center. The licensing is a bit disappointing; they are making clients pay $10 K. Moreover RAC is free with Standard edition though one doesn’t get enterprise features and limited to 4 CPU sockets only. So, thinking RAC One will be popular among customers who are currently using standard edition and want to switch to enterprise will be wrong.

How to reclaiming Unused LOB Space ??

SELECT table_name, column_name, segment_name, a.bytes
FROM dba_segments a JOIN dba_lobs b
USING (owner, segment_name)
WHERE b.table_name = 'NITI_TEST_LOB';

TABLE_NAME             COLUMN_NAME    SEGMENT_NAME                      BYTES
------------                    -------------- ---------------------------- ----------
NITI_TEST_LOB             IMAGE          SYS_LOB0000148988C00003$$    50,331,648


SELECT NVL((SUM(DBMS_LOB.GETLENGTH(image))),0) AS BYTES
FROM niti_test_lob;

     BYTES
----------
16,706,000

Use following command to manually shrink the LOB segment to free up the space immediately:


ALTER TABLE niti_test_lob MODIFY LOB (image) (SHRINK SPACE);

Table altered.


SELECT table_name, column_name, segment_name, a.bytes
FROM dba_segments a JOIN dba_lobs b
USING (owner, segment_name)
WHERE b.table_name = 'NITI_TEST_LOB';

TABLE_NAME     COLUMN_NAME             SEGMENT_NAME                      BYTES
------------             --------------                    ----------------------------           ----------
NITI_TEST_LOB     IMAGE                  SYS_LOB0000148988C00003$$        65,536


SELECT NVL((SUM(DBMS_LOB.GETLENGTH(image))),0) AS BYTES
FROM niti_test_lob;

    BYTES
----------
         0

The alter table <table_name> shrink space statement was introduced in Oracle 10g R1.
The ability to extend the SHRINK SPACE command to LOBs was introduced in Oracle 10g R2

How to run APList.sql ?

This test displays ALL of the Data for ALL of the tables used to process an invoice in AP.

FILE NAME:   APList.sql
PRODUCT:    501 -Oracle Payables
SUBCOMPONENT:  PAYMENT
PRODUCT VERSIONS:  11.5, 11.0, 10.7
PLATFORM:    All Platforms
PARAMETERS:   Invoice/Check Number
                             Supplier Name
                             Invoice ID
                             Print Null Columns Y or N - Default N
                             Details Layout Portrait (P) or Landscape (L) - Default P
                            GL Details Y or N - Default N

Included Files:

 CoreApiHtml.sql
 APListh.sql (HTML version)
 CoreApiTxt.sql
 APListt.sql (Text version)

Execution Environment:

 SQL*Plus

Access Privileges:

 Requires APPS user access

Usage:
 sqlplus apps/password @APListh.sql (HTML version)
 sqlplus apps/password @APListt.sql (Text version)

Instructions:
The files should be unzipped to a common directory.
For HTML output (recommended), run the file APListh.sql in SQL*Plus in the APPS schema. 
Follow the prompts and enter the requested information. The test will create an output file in html
format named APListh_<invoice_ID>_diag.html. Open the file in your browser and review the output.

Note:  The HTML version of this test is incompatible NLS Character Set UTF8 for RDBMS version 8.1.7.3 or lower.

Please run the Text version of this test, if you have UTF8 and RDMBS version 8.1.7.3 or lower.
For Text output, run the file APListh.sql in SQL*Plus in the APPS schema. 
Follow the prompts and enter the requested information. The test will create an output file in Text
format named APListt_<invoice_id>_<mmddhhmm>_diag.lst. Open the file in a Text editor and review the output.



The test has 3 parameters that allow some control of the appearance of the data and what data is returned.

Parameter 1:  Print Null Columns (Y or N) - Default N
This parameter is defaulted to No (N).  That is the recommended setting for this parameter.
If set to Y, the test would display all columns in each table, including columns with NULL values.

Parameter 2: Details Layout Portrait (P) or Landscape (L) - Default P (HTML Only)
This parameter is defaulted to P.  That is the recommended setting for this parameter.
Setting this option to P, will display all table details in a portrait format.
Setting this option to L, will display all table details in a Landscape format.

Parameter 3: GL Details Y or N - Default N
This parameter is defaulted to N.  That is the recommended setting for this parameter.
Setting this option to Y, will display any transaction data transferred to the GL.
WARNING:  SETTING THIS OPTION TO Y WILL RESULT IN LONGER RUN TIME FOR THE TEST

Errors in running APList : unable to open file "CoreApiHtmlx.sql"

Error:
unable to open file "CoreApiHtmlx.sql"
unable to open file "AddOnApiHtml.sql"
begin -- begin1
*
ERROR at line 1:
ORA-06550: line 13, column 1:
PLS-00201: identifier 'SHOW_HEADER' must be declared
ORA-06550: line 13, column 1:
PL/SQL: Statement ignored
ORA-06550: line 29, column 14:
PLS-00201: identifier 'V2T' must be declared
ORA-06550: line 29, column 14:
PL/SQL: Item ignored
ORA-06550: line 30, column 14:
PLS-00201: identifier 'V2T' must be declared
ORA-06550: line 30, column 14:
PL/SQL: Item ignored
ORA-06550: line 54, column 3:
PLS-00201: identifier 'INSERT_HTML' must be declared
ORA-06550: line 54, column 3:
PL/SQL: Statement ignored
ORA-06550: line 58, column 3:
PLS-00201: identifier 'INSERT_HTML' must be declared
ORA-06550: line 58, column 3:
PL/SQL: Statement ignored
ORA-06550: line 61, column 45:
PLS-00201: identifier 'V2T' must be declared
ORA-06550: line 61, column 1:
PL/SQL: Item ignored
ORA-06550: line 86, column 6:
PL/SQL: ORA-00942: table or view does not exist
ORA-06550: line 84, column 1:
PL/SQL: SQL Statement ignored
ORA-06550: line 107, column 3:
PLS-00201: identifier 'ERRORPRINT' must be declared


Solution:

APlist.sql calls another file - CoreApiHtml.sql that comes with APlist script. Make sure that this file is in the same directory as APList.sql, it contains all the procedures that APList calls.

 

 

 

Oracle RAC 11g Training with ASM

Oracle RAC Training with ASM

For Oracle RAC Training with ASM Check the following URL:

http://www.youtube.com/watch?v=y9Wgo0XpuUs&feature=colike

FRM-92101: There was a failure in the Forms Server during stratup.

While trying to run forms forms for the first time hitting error “FRM-92101: There was a failure in the Forms Server during stratup.”

Solution :

1 )    Download openmotif21-2.1.30-11.EL5.i386.rpm from http://oss.oracle.com/projects/compat-oracle/files/Enterprise_Linux/

2 )      Install the rpm in server.

$ rpm -ivh openmotif21-2.1.30-11.EL5.i386.rpm

3 ) Run forms again now it will work fine.

R12 : httpd: error while loading shared libraries: libdb.so.2

Problem :

After the installation faced issues bringing up the HTTP server.

Processes in Instance: r12test_myhost.myhost.mydomain.com-------------------+--------------------+---------+---------ias-component | process-type | pid | status-------------------+--------------------+---------+---------OC4J | oafm | 11510 | AliveOC4J | forms | 11508 | AliveOC4J | oacore | 11509 | AliveHTTP_Server | HTTP_Server | 0 | NONEASG | ASG | N/A | Down

06/10/12-00:11:48 :: adopmnctl.sh: exiting with status 0

adapcctl.txt showed the error message
06/10/12-00:04:03 :: adapcctl.sh: starting OPMN managed OHS instance
opmnctl: starting opmn managed processes...================================================================================opmn id=myhost.mydomain.com:6205no processes or applications matched this request
06/10/09-00:04:06 :: adapcctl.sh: exiting with status 150
================================================================================

$LOG_HOME/ora/10.1.3/opmn/HTTP_Server~1 was giving the error

--------12/06/10 00:25:49 Start process--------/app/inst/apps/r12test_myhost/ora/10.1.3/Apache/Apache/bin/apachectl startssl: execing httpd/app/apps/tech_st/10.1.3/Apache/Apache/bin/httpd: error while loading shared libraries: libdb.so.2: cannot open shared object file: No such file or directory

Solution:

Download and Install patch "6078836" from MOS:

[root 6078836]# pwd

/u01/sw_home/6078836

[root 6078836]# ls

libdb.so.2  README.txt

[root 6078836]# ls -l /usr/lib/libdb.so.2

ls: /usr/lib/libdb.so.2: No such file or directory

[root 6078836]# cp libdb.so.2 /usr/lib/

[root 6078836]# ls -l /usr/lib/libdb.so.2

-rw-r--r-- 1 root root 5825 Nov 27 11:38 /usr/lib/libdb.so.2

[root 6078836]# 

R12 MultiNode Installation with VMWare

Setup Requirement : Node with atleast 1 GB of Ram and 80GB of HardDisk , OS Dump , R12 Dump and VMWare Software

Download a Copy of VMWare WorkStation from http://www.vmware.com/download/ws/ for Linux
You will need an evaluation key
http://www.vmware.com/vmwarestore/newstore/wkst_eval_login.jsp , make sure you have the key before you start the Installation.

I downloaded an Image of Redhat Linux RHEL4-U4-i386-ES


Linux Node Setup Information:
1. Install OS Image on to the Box
2. Name the host , say r12lab01.r12.com and assign a static IP
3. Make sure you have all the latest RPM's in place. For more info refer R12 OS Pre-requisite Notes.
4. Open a VNC Server as Root
5. Download the VMWare RPM Package "VMware-workstation-5.5.3-34685.i386.rpm"
6. Install the RPM Package
7. Configure VMWare using "http://www.vmware.com/support/ws5/doc/ws_install_linux.html"
8. Start VMware Workstation ( /usr/bin/vmware )
9. Enter your serial number.
10. Create a virtual machine using the New Virtual Machine Wizard.
( ref : http://www.vmware.com/support/ws5/doc/newguest_ws.html )
11. Install a guest operating system in the new virtual machine.
You need the installation media CD-ROM for your guest operating system
12. Name the New Virtual Host as r12lab02.r12.com and assign a static IP
13. Make sure that you can communicate between first and the second node.
14. Power Off the Node and Start Fresh , make sure you have no OS Issues.
15. Keep the VNC Session Active. ( If you loose it , your will not have the new node )

R12 Setup Information:
1. Download Dump of R12 from http://edelivery.oracle.com
2. Stage them on the Hard Disk( I did not include the space requirement of the Dump above)
3. Run a Single Node Installation
4. Make sure that the Installation Completed Normal and all the Service are Up and Running.
5. Perform a Complete Health Check.

Adding a Node to the Existing Setup:
1. Power on the Second Node
2. Mount the R12 Mount Points on the Second Node.
3. cd /home1/aptest/test/apps/apps_st/comn/clone/bin/
4. perl adclonectx.pl addnode contextfile=TEST_r12lab01.xml
Target hostname (virtual or normal) [r12lab02]:
5. Make sure that the above process generate the Secondary Node's Context File
6. Make the Service Enabled that you want to run on the Second Node.
Initially have web service configured and later modify according to your need.
7. Move into AD_TOP/bin and run adconfig.sh , context_file=TEST_r12lab01.xml
8. Make sure that Autoconfig Complete's Normal.
9. Source the Environment File
10. Start the Service on the Seconday Node
11. Navigate to OAM and verify that you have the Second Node.

SYSAUX tablespace is growing like anything

SYSAUX tablespace is growing like anything
##################################

AWR is identified to be consuming more space.

SQL> select * from v$sysaux_occupants ;

SQL> SELECT occupant_name, occupant_desc, space_usage_kbytes FROM v$sysaux_occupants ORDER BY space_usage_kbytes DESC ;


OCCUPANT_NAME                   OCCUPANT_DESC                                          SPACE_USAGE_KBYTES

SM/AWR                  Server Manageability - Automatic Workload Repository               19906624
SM/OPTSTAT              Server Manageability - Optimizer Statistics History                  191808
EM                      Enterprise Manager Repository                                         59648


Solution
#########

1.) See the snapshots interval and retention period time

SQL> select * from DBA_HIST_WR_CONTROL;

Check whether the retention period is set too high. In such case we need to modify  the retention period.

If it is set to high value, then consider reducing the retention period to avoid growing of space.

By default, it is set to 7 days.

The new retention time is specified in minutes.

SQL> execute DBMS_WORKLOAD_REPOSITORY.MODIFY_SNAPSHOT_SETTINGS(retention => 4320);

Here 4320 is 3*24*60 ( Convert the days to minutes). For 3 days, it 4320 minutes.



2). Next thing is to check the amount of snapshots have been generated and available.


If this is too high, then try to drop the unwanted snapshots to reclaim the space.


SQL> exec dbms_workload_repository.drop_snapshot_range(low_snap_id,high_snap_id);

low_snap_id : The low snapshot id of snapshots to drop.

high_snap_id : The high snapshot id of snapshots to drop.



3) Then check whether the statistics_level parameter is set to typical.


SQL> show parameter statistics


NAME                                         TYPE                        VALUE

statistics_level string ALL   ----->  Should be set TYPICAL


If it is set to ALL, please change it to TYPICAL. Because statistics_level=ALL will gather lot of additional information in AWR repository which would consume more space. Most of the cases, if the statistics_level is set to TYPICAL then the growth would be stopped.

SQL> alter system set statistics_level=typical;

What is Split Brain and Node Eviction in RAC ???

What is Voting Disk ??


Oracle Clusterware uses the voting disk to determine which instances are members of a cluster. The voting disk must reside on a shared disk. Basically all nodes in the RAC cluster register their heart-beat information on thes voting disks. The number decides the number of active nodes in the RAC cluster. These are also used for checking the availability of instances in RAC and remove the unavailable nodes out of the cluster. It helps in preventing split-brain condition and keeps database information intact. The split brain syndrome and its affects and how it has been managed in oracle is mentioned below.
For high availability, Oracle recommends that you have a minimum of three voting disks. If you configure a single voting disk, then you should use external mirroring to provide redundancy. You can have up to 32 voting disks in your cluster. What I could understand about the odd value of the number of voting disks is that a noe should see maximun number of voting disk to continue to function, so with 2, if it can see only 1, its not the maximum value but a half value of voting disk. I am still trying to search more on this concept.

Split Brain  in RAC


In a Oracle RAC environment all the instances/servers communicate with each other using high-speed interconnects on the private network. This private network interface or interconnect are redundant and are only used for inter-instance oracle data block transfers. Now talking about split-brain concept with respect to oracle rac systems, it occurs when the instance members in a RAC fail to ping/connect to each other via this private interconnect, but the servers are all pysically up and running and the database instance on each of these servers is also running. These individual nodes are running fine and can conceptually accept user connections and work independently. So basically due to lack of commincation the instance thinks that the other instance that it is not able to connect is down and it needs to do something about the situation. The problem is if we leave these instance running, the sane block might get read, updated in these individual instances and there would be data integrity issue, as the blocks changed in one instance, will not be locked and could be over-written by another instance. Oracle has efficiently implemented check for the split brain syndrome.


Node Eviction in RAC


In RAC if any node becomes inactive, or if other nodes are unable to ping/connect to a node in the RAC, then the node which first detects that one of the node is not accessible, it will evict that node from the RAC group. e.g. there are 4 nodes in a rac instance, and node 3 becomes unavailable, and node 1 tries to connect to node 3 and finds it not responding, then node 1 will evict node 3 out of the RAC groups and will leave only Node1, Node2 & Node4 in the RAC group to continue functioning.
The split brain concepts can become more complicated in large RAC setups. For example there are 10 RAC nodes in a cluster. And say 4 nodes are not able to communicate with the other 6. So there are 2 groups formed in this 10 node RAC cluster ( one group of 4 nodes and other of 6 nodes). Now the nodes will quickly try to affirm their membership by locking controlfile, then the node that lock the controlfile will try to check the votes of the other nodes. The group with the most number of active nodes gets the preference and the others are evicted. Moreover, I have seen this node eviction issue with only 1 node getting evicted and the rest function fine, so I cannot really testify that if thats how it work by experience, but this is the theory behind it.

When we see that the node is evicted, usually oracle rac will reboot that node and try to do a cluster reconfiguration to include back the evicted node.

You will see oracle error: ORA-29740, when there is a node eviction in RAC. There are many reasons for a node eviction like heart beat not received by the controlfile, unable to communicate with the clusterware etc.

Using find Command in ASM

Purpose

Displays the absolute paths of all occurrences of the specified name pattern (with wildcards) in a specified directory and its subdirectories.

Syntax and Description

find [-t type] dir_name name_pattern

This command searches the specified directory and all subdirectories under it in the directory tree for the supplied name_pattern. The value that you use forname_pattern can be a directory name or a filename, and can include wildcard characters. See "Wildcard Characters".

In the output of the command, directory names are suffixed with the slash character (/) to distinguish them from filenames.

You use the -t flag to find all the files of a particular type (specified as type). For example, you can search for control files by specifying type asCONTROLFILE. Valid values for type are the following:

CONTROLFILE

DATAFILE

ONLINELOG

ARCHIVELOG

TEMPFILE

BACKUPSET

DATAFILE

PARAMETERFILE

DATAGUARDCONFIG

FLASHBACK

CHANGETRACKING

DUMPSET

AUTOBACKUP

XTRANSPORT

These are the type values from the type column of the V$ASM_FILE view.

Examples

The following example searches the dgroup1 disk group for files that begin with 'UNDO'.

ASMCMD> find +dgroup1 undo*

+dgroup1/sample/DATAFILE/UNDOTBS1.258.555341963

+dgroup1/sample/DATAFILE/UNDOTBS1.272.557429239

The following example returns the absolute path of all the control files in the +dgroup1/sample directory.

ASMCMD> find -t CONTROLFILE +dgroup1/sample *

+dgroup1/sample/CONTROLFILE/Current.260.555342185

+dgroup1/sample/CONTROLFILE/Current.261.555342183

FRM-92101: There was a Failure in the Forms Server During Startup

Symptoms
Intermittently E-Business suite forms sessions are failing with the error

Error

FRM-92101 : There was a failure in the Forms Server during startup. This could happen due to invalid configuration.

Steps to Reproduce

The issue can be reproduced at will with the following steps:

1:- Log into the E-Business suite.
2:- Choose a responsibility and click a function which loads forms (any form)

Business Impact

The issue has the following business impact:
Due to this issue, forms sessions are being disconnected for up to 100 users.
Changes
Implemented a shared application tier file system (shared APPL_TOP)
Cause
The OPMN process pings the HTTP server process to check that it is still alive. If OPMN does not get a response from the HTTP server in the timeframe configured by the E-Business suite, it presumes that the HTTP server is down and restarts it, affecting all users connected to that application server.

In the $LOG_HOME/ora/10.1.3/opmn/opmn.log we can see that OPMN is restarting the HTTP server because it has become unresponsive.

10/02/16 12:08:22 [libopmnohs] Process Ping Failed:
HTTP_Server~HTTP_Server~HTTP_Server~1 (241450421:15863) [The connection
receive timed out]
10/02/16 12:09:13 [libopmnohs] Process Ping Failed:
HTTP_Server~HTTP_Server~HTTP_Server~1 (241450421:15863) [The connection
receive timed out]
10/02/16 12:09:13 [libopmnohs] Process Unreachable:
HTTP_Server~HTTP_Server~HTTP_Server~1 (241450421:15863)
10/02/16 12:09:15 [pm-process] Stopping Process:
HTTP_Server~HTTP_Server~HTTP_Server~1 (241450421:15863)
10/02/16 12:09:23 [pm-process] Starting Process:
HTTP_Server~HTTP_Server~HTTP_Server~1 (241450421:0)

The reason that OPMN is restarting the HTTP server is that a shared APPL_TOP is being used, but the recommended settings for the Network-attached storage (NAS) file system have not been implemented.

As the NAS mount gets busier, the HTTP server process becomes swamped and does not respond to OPMN. OPMN then restarts the process, causing all forms users connected to that Application server to fail with the FRM-92100 error.

Self service session users are much less likely to notice the issue as most self service application screens are stateless HTML, so no context is lost when the HTTP server is bounced.
Solution
To implement the solution, please execute the following steps:

1:- Stop ALL middle tier services
2:- Update the $CONTEXT_FILE and change "s_lock_pid_dir" to a directory on a local disk.

For example

/d01

3:- Run autoconfig

4:- Ensure that the NAS device is configured with the correct mount options.

On Linux, it should be using "nolock" option.
On Solaris, it should be using "llock" option.

5:- Start ALL middle tier services
6:- Retest the issue.
7:- Migrate the solution as appropriate to other environments.


References

BUG:5407140 - [LIBOPMNOHS] PROCESS PING FAILED: HTTP_SERVER~HTTP_SERVER~HTTP_SERVER~1
BUG:9279182 - USER SESSION CRASHES WITH FRM-92102 AND SEGMENTATION FAULT
NOTE:384248.1 - Sharing The Application Tier File System in Oracle E-Business Suite Release 12
NOTE:802704.1 - Poor Performance of APPS 11i or 12 When Using Shared APPL_TOP with NAS