Too many files open causing HTTP in Oracle App Server to hang.

Latest response

Hello all,
This is my first post, so please forgive any mistakes in protocol.
We have OAS (Oracel App Server) 10g running on RHEL 5.9. Recently the OAS's HTTP server hung and had to be restarted. The logs indicated "Too many open file", but didn't give a number that was reached.
OAS is running under user oracle. The limits are all set to default (i.e. no modifications to /etc/security/limits.conf).
ulimit -Hn and ulimit -Sn for user oracle both return 1024.
I figured the user oracle must have tried to open more than 1024 files and couldn't so the HTTP server hung.
Unfortunately another member of the team restarted OAS to resolve the problem before I could verify how many files oracle had open at the time.
The confusion I have is, now when I check for the number of open files for user oracle, I get 3393.
[root@xxxxx]# lsof -u oracle |wc
3393 30895 444349

How can user oracle have more then 1024 files open?

I ran the following command to print the number of open files per PID for user oracle.
[root@xxxxx]# lsof -u oracle |awk '{print $2":"$1}' |sort |uniq -c | sort -n
1 PID:COMMAND
15 7554:opmn
22 7613:rotatelog
24 30019:perl
27 7616:rotatelog
27 7617:rotatelog
28 7615:rotatelog
39 7555:opmn
73 7577:java
94 7614:httpd
103 7576:httpd
104 7618:httpd
106 7627:httpd
106 8284:httpd
107 30307:httpd
107 5357:httpd
107 5360:httpd
107 5363:httpd
107 5370:httpd
107 5373:httpd
107 7625:httpd
107 7631:httpd
107 7634:httpd
107 8267:httpd
107 8278:httpd
107 8281:httpd
107 8293:httpd
108 7620:httpd
108 7824:httpd
108 8274:httpd
108 8286:httpd
108 8290:httpd
174 18986:java
255 7578:java
718 7579:java

I'm wondering if the open files limit of 1024 is a "per user" limit or actually a "per user/per PID" limit.
Although the total number of files opened by user oracle is well above 1024, the most open by any PID owned by oracle is 718 for PID 7579.
Please advise.
Thank you

MA

Responses