Multiple sessions of the same user. When one of them gets to the point that it can no longer run new programs, none of them can, not even a new login of that user. Other users can still run new programs just fine, including new logins.
Normally user limits are in limits.conf, but its documentation says "please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session."
I'm nowhere close to running out of ram (44GB available), but I can't figure out what else to look at. What limits exist that would have a global effect on all sessions using the same UID, but not other UIDs?
Edited on 6/12/16 at 8:45p to add:
While writing the below I realized that the problem could be X11 related. This user account on this box is used nearly exclusively for GUI applications. Is there a good text based program I can try to run from bash that will use lots of resources and give good error messages? The box does not get to the point where it cannot even run ls.
Unfortunately, the GUI programs this problem normally affects (Chrome and Firefox) do not do a good job of leaving error messages behind. Chrome tabs will start showing up blank or with the completely useless "Aw, Snap!" error. Firefox simply will refuse to start. The only even partially helpful error messages I managed to obtain came from trying to start Firefox from bash:
[pascal@firefox ~]$ firefox --display=:0 --safe-mode
Assertion failure: ((bool)(__builtin_expect(!!(!NS_FAILED_impl(rv)), 1))) && thread (Should successfully create image decoding threads), at /builddir/build/BUILD/firefox-45.2.0/firefox-45.2.0esr/image/DecodePool.cpp:359
#01: ???[/usr/lib64/firefox/libxul.so +0x10f2165]
#02: ???[/usr/lib64/firefox/libxul.so +0xa2dd2c]
#03: ???[/usr/lib64/firefox/libxul.so +0xa2ee29]
#04: ???[/usr/lib64/firefox/libxul.so +0xa2f4c1]
#05: ???[/usr/lib64/firefox/libxul.so +0xa3095d]
#06: ???[/usr/lib64/firefox/libxul.so +0xa52d44]
#07: ???[/usr/lib64/firefox/libxul.so +0xa4c051]
#08: ???[/usr/lib64/firefox/libxul.so +0x1096257]
#09: ???[/usr/lib64/firefox/libxul.so +0x1096342]
#10: ???[/usr/lib64/firefox/libxul.so +0x1dba68f]
#11: ???[/usr/lib64/firefox/libxul.so +0x1dba805]
#12: ???[/usr/lib64/firefox/libxul.so +0x1dba8b9]
#13: ???[/usr/lib64/firefox/libxul.so +0x1e3e6be]
#14: ???[/usr/lib64/firefox/libxul.so +0x1e48d1f]
#15: ???[/usr/lib64/firefox/libxul.so +0x1e48ddd]
#16: ???[/usr/lib64/firefox/libxul.so +0x20bf7bc]
#17: ???[/usr/lib64/firefox/libxul.so +0x20bfae6]
#18: ???[/usr/lib64/firefox/libxul.so +0x20bfe5b]
#19: ???[/usr/lib64/firefox/libxul.so +0x21087cd]
#20: ???[/usr/lib64/firefox/libxul.so +0x2108cd2]
#21: ???[/usr/lib64/firefox/libxul.so +0x210aef4]
#22: ???[/usr/lib64/firefox/libxul.so +0x22578b1]
#23: ???[/usr/lib64/firefox/libxul.so +0x228ba43]
#24: ???[/usr/lib64/firefox/libxul.so +0x228be1d]
#25: XRE_main[/usr/lib64/firefox/libxul.so +0x228c073]
#26: ???[/usr/lib64/firefox/firefox +0x4c1d]
#27: ???[/usr/lib64/firefox/firefox +0x436d]
#28: __libc_start_main[/lib64/libc.so.6 +0x21b15]
#29: ???[/usr/lib64/firefox/firefox +0x449d]
#30: ??? (???:???)
Segmentation fault
[pascal@firefox ~]$ firefox --display=:0 --safe-mode -g
1465632860286DeferredSave.extensions.jsonWARNWrite failed: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860287addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860288addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860289addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860289addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860290addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860358DeferredSave.addons.jsonWARNWrite failed: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
1465632860359addons.repositoryERRORSaveDBToDisk failed: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: [email protected]:173:18 < [email protected]:292:9 < [email protected]:315:40 < [email protected]:933:23 < [email protected]:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < [email protected]:770:1 < [email protected]:284:9
Segmentation fault
[pascal@firefox ~]$
[pascal@localhost ~]$ ulimit -aH
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 579483
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 579483
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[pascal@localhost ~]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 579483
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4096
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[pascal@localhost ~]$ set /proc/*/task/*/cwd/.; echo $#
306
[pascal@localhost ~]$ prlimit
RESOURCE DESCRIPTION SOFT HARD UNITS
AS address space limit unlimited unlimited bytes
CORE max core file size 0 unlimited blocks
CPU CPU time unlimited unlimited seconds
DATA max data size unlimited unlimited bytes
FSIZE max file size unlimited unlimited blocks
LOCKS max number of file locks held unlimited unlimited
MEMLOCK max locked-in-memory address space 65536 65536 bytes
MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes
NICE max nice prio allowed to raise 0 0
NOFILE max number of open files 32768 65536
NPROC max number of processes 4096 579483
RSS max resident set size unlimited unlimited pages
RTPRIO max real-time priority 0 0
RTTIME timeout for real-time tasks unlimited unlimited microsecs
SIGPENDING max number of pending signals 579483 579483
STACK max stack size 8388608 unlimited bytes
Edited on 6/13/16 at 10:24p to add:
Not a GUI problem. When I tried to su to the user today, that doesn't even work. Root is fine. I can ls, vi, create a new user, su to that user, everything works fine for that user, I exit and try to su to the problem user and no go. Bash kinda loaded the first time, but even exit didn't work. I had to reconnect to get back to root.
[root@firefox ~]# su - pascal
Last login: Sat Jun 11 03:08:47 CDT 2016 on pts/1
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: Resource temporarily unavailable
-bash-4.2$ ls
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: Resource temporarily unavailable
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: Resource temporarily unavailable
-bash-4.2$ exit
logout
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: Resource temporarily unavailable
-bash-4.2$
[root@firefox ~]# ls -l /
total 126
lrwxrwxrwx. 1 root root 7 Jan 28 23:53 bin -> usr/bin
---- snip ----
drwxr-xr-x. 19 root root 23 May 27 18:03 var
[root@firefox ~]# vi /etc/rc.local
[root@firefox ~]# useradd test
[root@firefox ~]# su - test
[test@firefox ~]$ cd
[test@firefox ~]$ ls -l
total 0
[test@firefox ~]$ ls -l /
total 126
lrwxrwxrwx. 1 root root 7 Jan 28 23:53 bin -> usr/bin
---- snip ----
drwxr-xr-x. 19 root root 23 May 27 18:03 var
[test@firefox ~]$ vi /etc/rc.local
[test@firefox ~]$ exit
logout
[root@firefox ~]# su - pascal
Last login: Mon Jun 13 22:12:12 CDT 2016 on pts/1
su: failed to execute /bin/bash: Resource temporarily unavailable
[root@firefox ~]#
nproc was the problem:
[root@localhost ~]# ps -eLf | grep pascal | wc -l
4068
[root@localhost ~]# cat /etc/security/limits.d/20-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.
* soft nproc 4096
root soft nproc unlimited
[root@localhost ~]#
man limits.conf states:
Also, please note that all limit settings are set per login. They are
not global, nor are they permanent; existing only for the duration of
the session. One exception is the maxlogin option, this one is system
wide. But there is a race, concurrent logins at the same time will not
always be detected as such but only counted as one.
It appears to me that nproc is only enforced per login but counts globally. So a login with nproc 8192 and 5000 threads would have no problems, but a simultaneous login of the same UID with nproc 4096 and 50 threads would not be able to create more because the global count (5050) is above its nproc setting.
[root@localhost ~]# ps -eLf | grep pascal | grep google/chrome | wc -l
3792
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments