0)General Information to Troubleshoot Apache Issues.
******************************
Sometimes there will be problem in our Apache Server where although the status shown is not running, there was still httpd running on the background.
What you can do is shutdown the Apache listener process and clear all the httpd background process.
$ ps -ef | grep Apache | wc -l
28
$ cd $COMMON_TOP/admin/scripts/$
$ ./adapcctl.sh status
Apache Web Server Listener is not running.
Apache Web Server Listener (PLSQL) is not running.
adapcctl.sh: exiting with status 1
$ ./adapcctl.sh stop
adapcctl.sh version 115.55
Apache Web Server Listener is not running.
Apache Web Server Listener (PLSQL) is not running.
adapcctl.sh: exiting with status 0
$ ps -ef | grep Apache | wc -l
28
$ kill -9 `ps -ef | grep Apache | grep applprod | grep -v grep | awk '{print $2}'`
$ ps -ef | grep Apache | wc -l
1
$ ./adapcctl.sh start
adapcctl.sh version 115.55
Apache Web Server Listener is not running.
Starting Apache Web Server Listener (dedicated HTTP in SSL mode) ...
Apache Web Server Listener (PLSQL) is not running.
Starting Apache Web Server Listener (dedicated PLSQL) ...
adapcctl.sh: exiting with status 0
$ ps -ef | grep Apache | wc -l
22
$ ps -ef | grep Apache | wc -l
28
******************************
1)If you mistype any directive in Apache (httpd.conf or any conf file), Apache will not start & it will not write even in error_log. In these case it will report any error message to file
$COMMON_TOP/admin/log/$
So first check in the above file
******************************
2)Found *.pid file in Apache_Top/Apache/logs/ even after bounce of an Apache web server (When Apache starts it creates .pid file in logs directory and when you shutdown apache it cleans this .pid file). So if this file is already there then Apache will not start.
Fix: Simply move this file
******************************
3)Could not bind port (Port already in Use) or Sometimes apache and concurrent
Manager will interchange their port no.’s while starting while results error and both of them will not up
Fix: To overcome this up the services in this manner
1)Listener
2)Apache web server
3)Concurrent manager
4)adstrtal.sh
******************************
4)Error 500 - Internal server error
Fix: 1) Clear Cache (Application Server Cache)and start the services
******************************
5)If there are no error messages in error_log or error_log_pls
then there might be issues in starting Jserv component of web server (which caters Java requests by mod_jserv)
******************************
6)Mobile is enabled in zone.properties disable it if you are not using it.
******************************
7)If you don't know port number of your web server you can check it in file $IAS_ORACLE_HOME/Apache/
******************************
8)Once you are able to access http://hostname.domainname:
-mod_jserv
-mod_pls
-mod_oprocmgr
-mod_ossl (If you have ssl enabled on web server)
-mod_osso (If single sign on server is enabled on web server)
******************************
9)Once in a while adapcctl.sh start will not work then ultimate you have to go and
invoke apachectl script file in Apache top/apache/bin to start Apache webserver.
******************************
10)If we remove _pages from COMMON_TOP it will be created for the next time only when we bounce and up the Apache web server.
******************************
11)Jsp pages are complied with the file ojspcompile.pl executable. It is located at $JSP_TOP/bin/ and in R12 it is there in
$FND_TOP/patch/115/bin/
******************************
12)Error : Your session is no longer valid while trying to loging to applications.
The issue is resolved by changing the profile option session_cookie_domain
to null from domain.com
******************************
13)HTTP-403 forbidden message after entering the username and password in the "Apps Logon Links" page. [Not authorised to view this page]
Check for Apps password in $APACHE_TOP/modplsql/cfg/
******************************
14)Error message: You have insufficient privileges for the current operation.
Verify and update SESSION_COOKIE_DOMAIN in ICX_parameters if it is wrong.
Example :
update icx_parameters set SESSION_COOKIE_DOMAIN='NEW_
******************************
15)AppsLocalLogin?.jsp page error outs
Edit the $IAS_ORACLE_HOME/Apache/Jserv/
Set wrapper.bin.parameters=-DLONG_
add the line wrapper.bin.parameters=-
Restarted Apache for these changes to take effect and retested the issue
16) Login to the applications URL fails with
500 Internal server Error
java.lang.NoClassDefFoundError at
oracle.apps.fnd.sso.
oracle.apps.fnd.sso.
Containers for J2EE 10g (10.1.3.0.0)
].server.http.HttpApplication.
Containers for J2EE 10g (10.1.3.0.0)
].server.http.HttpApplication.
Containers for J2EE 10g (10.1.3.0.0)
].server.http.HttpApplication.
Containers for J2EE 10g (10.1.3.0.0)
Performed the below steps, but it did not resolved the issue
1) Bounced Apache server, cleared cache and validated ports
2) Bounced all Middle tier services and killed defunct processes
3) Bounced Unix Server
Solution
=======
Java object Caching Configuration parameter was Enabled in oc4j.properties
$INST_TOP/ora/10.1.3/j2ee/
Disabled LONG_RUNNING_JVM=false from True
Bounced Apache server and could login to the applications URL without issues
ISSUES AND SOLUTIONS
ISSUE 1:
Description
When Apache script adapcctl.sh is issued.It shows that Apache is started.On checking the status Apache is stopped
Cause
The error log file in the folder
cd $APACHE_TOP/Apache/logs
Showed
fopen: No such file or directory httpd: could not log pid to file /var/opt/_/Apache/logs/httpd_
SOLUTION 1
Created the missing directories /_/Apache/logs under /var/opt folder
Bounced Apache
ISSUE 2:
Description
oacore was not comming up in the configurator node.
opmnctl status
ias-component process-type pid status
OC4J oacore N/A Down
OC4J oacore N/A Down
OC4J oacore N/A Down
OC4J oacore N/A Down
Because of this configurator cannot be invoked.
Cause
Check the oacore error log.
Example:
Below Log file gives root cause. //mtlog/_/logs/ora/10.1.3/
14:11:56 Error initializing server: //inst/apps/_/ora/10.1.3/j2ee/
Fatal error at line 70 offset 1 in file://inst/apps/_/ora/10.1.3/
XML-20100: (Fatal Error) Expected 'EOF'.
SOLUTION 2
Edit the /sid/inst/apps/SID_hostname/
Bug - 6702510
ISSUE 3
AppsLocalLogin?.jsp page was displaying errors
Cause
Extract from the Jserv log file =========================== [27/02/2008 16:57:14:447 CST] weboamLocal/oracle.apps.fnd.
Sucessfully initialize oaosu.OAMServlet [27/02/2008 16:57:14:447 CST] Servlet Zone rootauohstrzt11 initia lization complete [27/02/2008 16:57:15:101 CST]
Creating noun for: oracle.jsp.JspServlet [27/02/2008 16:57:15:102 CST] Creating noun for: AppsLocalLogin?.jsp
[27/02/2008 16:57:15:120 CST] oracle.jsp.JspServlet: init [27/02/2008 16:57:28:066 CST] Creating noun for: oracle.jsp.JspServlet
[27/02/2008 16:57:28:067 CST] Creating noun for: AppsLocalLogin?.jsp [27/02/2008 16:57:28:078 CST] oracle.jsp.JspServlet: init
[27/02/2008 16:57:29:753 CST] JspServlet?: unable to dispatch to requested page: Exception:java.lang.
SOLUTION 3:
Edited the $IAS_ORACLE_HOME/Apache/Jserv/
a) Set wrapper.bin.parameters=-DLONG_
b) Add the line wrapper.bin.parameters=-
c) Restarted Apache for these changes to take effect and retested the issue
ISSUE 4
New Message when starting Apache using adapcctl.sh start
Cause
It takes 7/10 minutes to clear this cache directory if present
apache Web Server Listener is not running.
Starting Apache Web Server Listener (dedicated HTTP) ...
Removing gantt cache directory
Successfully removed the gantt cache directory:/tcge1i/applmgr/
Apache Web Server Listener (PLSQL) is not running.
Starting Apache Web Server Listener (dedicated PLSQL) ...
SOLUTION 4
Wait for atleast 7 to 10 minutes and this cache directory will be cleared and Apache will be started successfully
ISSUE 5
Apache server is up and running.
But adapcctl.sh status shows
adapcctl.sh version 115.54
Apache Web Server Listener is not running. Apache Web Server Listener (PLSQL) is not running.
adapcctl.sh: exiting with status 1
Cause
Incorrect Lock file path given in httpd.conf
LockFile? $APACHE_TOP/Apache/log/PUPONI_
and
Incorrect path given in adapcctl.sh script.
SOLUTION 5:
Modified the path of lock file in $APACHE_TOP/Apache/conf/httpd.
LockFile? /var/opt/SID_/httpd.lock
and
Modified the path of adapcctl.sh file by commenting the wrong path
ISSUE 6
After selecting the E-Business home page got internal server error Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Cause
Found that all the properties files as below
jserv.properties jserv_restrict.properties zone.properties zone_restrict.properties
were pointing to a dbc file name _.dbc
which was not physically present at the path
/sid/applmgr/11510/fnd/11.5.0/
$ cd //applmgr/11510/fnd/11.5.0/
$ ls .dbc
SOLUTION 6:
copied the file
twwyai.dbc
which was physically present at path
//applmgr/11510/fnd/11.5.0/
to the dbc file name which was being pointed by all the properties files
$ cp .dbc _>sid>.dbc
$ ls
_>sid>.dbc .dbc
This resolved the issue.
ISSUE 7
Instance is very slow / inaccessible with a warning on login page:
Warning
Low-level logging is currently enabled. Your application will not perform as well while Low-level logging is on.
Cause
Logging was enabled through profile values:
FND: Debug Log Enabled -- Yes
FND: Debug Log Level -- Statement
FND: Debug Log Module -- %
SLA: Enable Diagnostics -- Yes
FND Validation Level -- Error
SOLUTION 7
1. Change these values to:
FND: Debug Log Enabled -- No
FND: Debug Log Level -- ( Blank )
FND: Debug Log Module -- ( Blank )
SLA: Enable Diagnostics -- No
FND Validation Level -- None
2. Bounce Apache on all the MTs and clear Apache cache.
ISSUE 8
Login to the applications URL fails with
500 Internal Server Error java.lang.NoClassDefFoundError at oracle.apps.fnd.sso.
Performed the below steps, but it did not resolved the issue
1) Bounced Apache server, cleared cache and validated ports
2) Bounced all Middle tier services and killed defunct processes
3) Bounced Unix Server
Cause
No Cause Entered
SOLUTION 8
Java object Caching Configuration parameter was Enabled in oc4j.properties $INST_TOP/ora/10.1.3/j2ee/
Disabled LONG_RUNNING_JVM=false from True
Bounced Apache server and could login to the applications URL without issues
Java Caching is a mechanism used to improve performance by storing data in the Middle Tier JVM memory, instead of retrieving the data from the database every time it is requested.
ISSUE 9
When trying to access the URL (Either Big IP or the node level), it hangs for a long time and getting redirected to a different URL and gives a blank page. For example, When try to access the ipayments URL which is hosted on node2, it hangs for a long time and its getting redirected towards the Sales URL which is hosted on node3, and displays a blank page.
Cause
All the host reference and the URL directives are pointing towards node3 and sales URL respectively, in the below files. Hence every attempt tries to launch the sales URL and get fails as the apache configuration files and parameters are pointing towards the correct one.
jserv.conf forms.properties zone.properties jserv.properties
SOLUTION 9
Stop Apache. Edit the files jserv.conf, forms.properties, zone.properties, jserv.properties, jserv.conf to replace the host reference and the URL directives to the correct one (In our case, its node3 . Clear the cache and restart the Apache.
ISSUE 10
Apache login issues occurs in multi MT environment. Unable to login to oracle applications.
Cause
Issue due to distributed cache.
SOLUTION 10
Set is_distributed to false in javacache.xml Bounce Apache.
ISSUE 11
Apache server is down. Apache logfile did not provide useful information.
Started the Apache server, but the Apache Web listener did not started.
1) Verified no log files (Apache,Jserv) crossed 2GB limit. error_log, access.log, mod_jserv
2) Veified that the port is listening (netstat -an |grep ) Apache port(httpd.conf), pls listener(httpd_pls.conf)
3) Verified that the Apache and Jserv configuration files are not changed recently httpd.conf & Jserv.conf
4) Shutdown all the Middle tier services and check for ipcs (semaphores) did not resolve the outage.
$ ipcs on application middle tier as Application user ap
Cause
No Cause Entered
SOLUTION 11
Rebooted the server and started the services. All services came back normal.
ISSUE 12
Unable to login to the home page URL. Login fails with the error:
Error Page
You have encountered an unexpected error. Please contact the system administrator for assistance.
Click here for exception details
Checks:
Upon clicking the link, can observe the error:
oracle.apps.fnd.framework.
Cause
FND_NODES table was updated with the same server_id for all Nodes.
SOLUTION 12
Update the table FND_NODES with correct server_ids for all nodes as in $FND_TOP/secure/sid_host/sid.
Bounced apache on all apache nodes.
and Run select node_name, server_id from fnd_nodes;
to confirm the values.
Below is the example for the solution:
Found three hosts using same server_id:
SQL> select node_name, server_id from fnd_nodes; NODE_NAME SERVER_ID
AUOHSACPS06 2D8FCCC57459B66EE040558CDBC611
AUOHSACPS03 2D8FCCC57459B66EE040558CDBC611
AUOHSACPS02 2D8FCCC57459B66EE040558CDBC611
reset server_id with original server_id from each hosts .dbc file @ $FND_TOP/secure:
SQL> select node_name, server_id from fnd_nodes; NODE_NAME SERVER_ID
AUOHSACPS06 2D8FF21843830F82E040558CDBC619
AUOHSACPS03 2D90005D5A13C774E040558CDBC61D
AUOHSACPS02 2D8FCCC57459B66EE040558CDBC611
ISSUE 13
Users getting '404 page not found errors' when tyring to access the URL
(OR)
Getting Exception in thread "main" java.lang.ClassFormatError: Truncated class file
(OR)
adstrtal.sh or adstpall.sh is throwing below error message
You are running adstrtal.sh version 115.16
Exception in thread "main" java.lang.ClassFormatError: oracle/apps/ad/context/
Exiting with status 1
(OR)
[06/08/2007 11:03:12:181] (ERROR) an error returned handling request via protocol "ajpv12" [06/08/2007 11:03:12:181] (EMERGENCY) ajp12: can not connect to host140.85.188.36:20021 [06/08/2007 11:03:13:184] (EMERGENCY) ajp12: can not connect to host 140.85.188.36:20021 [06/08/2007 11:03:15:038] (EMERGENCY) ajp12: auth did not receive challenge size [06/08/2007 11:03:15:038] (EMERGENCY) ajp12: auth fail
Cause
Extract from the JVM log file: //product/iAS/Apache/Jserv/
Exception in thread "main" java.lang.ClassFormatError: Truncated class file at org.apache.jserv.
Verified that there are no log files that exceed 2GB limit (OS limit).
Clearing the Apache and Jserv cache did not resolve the issue. Unmount / mount product specific file system did not resolve
SOLUTION 13:
Try each of these solutions until the problem is resolved, in order to have the least impact on the system.
1. If you can determine the specific class file from the log files, use the 'touch' command.
e.g. touch classfile
If necessary, restart Apache and clear server cache.
2. Shut down all MT services, exit the Apps owner and ask the System Admin to unmount and remount the /sid/applmgr file system (mount point above $JAVA_TOP). Ask the System Admin to kill any processes that may be holding the file system open. Then log in to the Apps owner and restart MT services.
3. Shutdown all the MT services and ask System Admin to reboot the server if the actions above don't resolve the outage.
ISSUE 14:
Got login error - Internal Server Error after bounced all services on . find following message from Jserv log file:
$ tail -8 mod_jserv.log
[07/05/2007 18:41:06:915] (ERROR) ajp12: Servlet Error: java.lang.
[07/05/2007 18:41:06:916] (ERROR) an error returned handling request via protocol "ajpv12"
[07/05/2007 18:41:06:916] (ERROR) balance: 11906 internal servlet error in serverauohsnops03.oracleoutsourcing.
[07/05/2007 18:41:06:916] (ERROR) an error returned handling request via protocol "balance"
[07/05/2007 20:07:52:732] (ERROR) ajp12: Servlet Error: java.lang.
No runaway, no lock on port 20720.
SOLUTION 14:
Shutdown all services on MT. Removed all log file (standard error...) from Jserv/jvm.
Started All services with successful login.
ISSUE 15
Customer reported Not able to login with “Internal Server Error” on .
We could not start some of services during bounced All Services with following error message: adapcctl.sh version 115.47
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Apache Web Server Listener is not running (dedicated HTTP) ...
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Apache Web Server Listener is already stopped.
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
adapcctl.sh: exiting with status 2
Unable to login the application URL, facing the error while login:
Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, ap@domain.comand inform them of the time the error occurred, and anything you might have done that may have caused the error.
Cause:
SQL> select END_DATE from fnd_user where user_name='GUEST';
END_DATE
---------
05-OCT-10
SOLUTION 15
We found out the apps and applsys were not in sync.
Just manual change the apps password from sqlplus.
Restarted all services and login successfully.
or
Update the FND_USER table with END_DATE=null and bounced apache
SQL> update fnd_user set END_DATE=null where user_name='GUEST';
1 row updated.
SQL> commit;
Commit complete.
SQL> select END_DATE from fnd_user where user_name='GUEST';
END_DATE
-----------
ISSUE 16
Unable to login to oracle applications:
error message: You have insufficient privileges for the current operation.
Checks
Unable to login to oracle applications:
error message: You have insufficient privileges for the current operation.
SOLUTION 16
Verify and update SESSION_COOKIE_DOMAIN in ICX_parameters if it is wrong.
Example :
update icx_parameters set SESSION_COOKIE_DOMAIN='NEW_
ISSUE 17
Apache home page is not coming up even after bounced
Found in jvm log asking for java version
SOLUTION 17
//product/iAS/Apache/Apache/
Check the version in /usr/java and update that version in java.sh file and Bounced Apache.
ISSUE 18
Unable to access applications
cause
All applications are up and running.
"Solaris Error: 12: Not enough space" errors are encounterd in database listener logfile.
SOLUTION 18:
Check if the OS is SUN Solaris.
Check if /tmp filesystem is 100% full.
Check Swap space utilization.
Check the application process that is hagging resources. (Check if any sql sessions are hagging resources)
Clear the process in issue after getting required approval.
NOTE:
The above is applicable only for environments on Solaris OS.
ISSUE 19
Not able to login to APPS. JSP Error:
Unable to create an user session or ORA-01403 : no data found has been detected in FND.SIGNON.AUDIT_USER
Checks
Check number of rows in dual table.
Select * from dual;
DUAL table should always contain only one row.
The table named DUAL is a small table in the data dictionary that Oracle and
user-written programs can reference to guarantee a known result.
This table has one column called DUMMY and one row containing the value X.
SOLUTION 19
DUAL table should always contain only one row and one column.
Insert a row with value X into dual table if the dual table contains 0 rows.
If the dual table contains more than one row then delete the rows by keeping one row.
The rows in dual table not equal to zero.
ISSUE 20
Not able to launch forms from PHP
Checks
Login to PHP.
Select system administrator responsibility and click on any menu option.
The page shows "done" without launching forms (jinit)
Check the apps version.
If the apps version is 11.5.10 then check the Check the file version of $FND_TOP/admin/template/
If the file version is 115.148, then verify that "!WindowsDPI= " exists in $FORMS60_WEB_CONFIG_FILE
The windowsDPI value should be 96. If the value for WindowsDPI is not set in the
$FORMS60_WEB_CONFIG_FILE then update PROFILE_OPTION_VALUE.
SOLUTION 20
select PROFILE_OPTION_VALUE from fnd_profile_option_values where PROFILE_OPTION_ID=3769;
Update the tar with the output of the above command.
UPDATE fnd_profile_option_values
set PROFILE_OPTION_VALUE='https://
This should update only one row.
commit;
ISSUE 21
Signature for "https URL redirecting to http URL"
Issue:
https URL redirecting to http URL and the page cannot be displayed error occurs.
SOLUTION 21
Check the httpd.conf and url_fw.conf, in url_fw.conf the Rewrite rule was
RewriteRule ^/$/OA_HTML/AppsLocalLogin.
Changed into
RewriteRule https://: /OA_HTML/AppsLocalLogin.jsp.
******************************
