View previous topic :: View next topic |
Author |
Message |
TheFlyingBear
Joined: 08 Dec 2010 Posts: 15 Location: France
|
Posted: Fri Dec 17, 2010 3:56 pm Post subject: Icecast 2.3.2-kh28 - open files |
|
|
And a new one... (I'm really sorry)
I have this problem on already running icecast (never noticed it before my switch to 2.3.32-kh38) or a freshly installed and booted system...
Shortly after icecast's start, I have this in my error.log:
Code: |
[2010-12-17 16:20:23] WARN connection/accept_client accept() failed with error 24: Too many open files |
But, at the same moment, my /proc/sys/fs/file-nr says:
If I stop icecast, and then restart it (sometimes 3 or 4 times), the problem disapears
I have this problem on Linux servers running:
- Gentoo Linux with kernel 2.6.30-gentoo-r8 ;
- Debian (stable) Linux with kernel 2.6.26-2
My opened files limit is 786432, and my icecast connections limit is 65536.
Any idea? _________________ Guillaume |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
karlH Code Warrior
![Code Warrior Code Warrior](templates/subRebel/images/ranks/rank_dev.gif)
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Fri Dec 17, 2010 4:19 pm Post subject: |
|
|
The too many open files limit is the process limit so you are reaching the limit as shown in the /proc/<pid>/limits file. Maybe you can send me the level 4 error log showing what is going on?
karl. |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
TheFlyingBear
Joined: 08 Dec 2010 Posts: 15 Location: France
|
Posted: Fri Dec 17, 2010 4:31 pm Post subject: |
|
|
I only have level 3 error log. I'll change this for the next time.
I had a look in /proc/`pidof icecast`/limits, and here is what I found (but I don't have the problem at the moment):
Quote: |
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited ms
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 393216 393216 processes
Max open files 786432 786432 files
Max locked memory 32768 32768 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 106496 106496 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us |
The strange thing is that I can't reproduce this problem each time I start or restart Icecast. _________________ Guillaume |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
karlH Code Warrior
![Code Warrior Code Warrior](templates/subRebel/images/ranks/rank_dev.gif)
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Fri Dec 17, 2010 4:39 pm Post subject: |
|
|
you can increase the log level and kill -HUP to change the running copy without dropping any connections.
karl. |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
TheFlyingBear
Joined: 08 Dec 2010 Posts: 15 Location: France
|
Posted: Sun Dec 19, 2010 7:09 pm Post subject: |
|
|
Hello,
I'm currently having this problem, on icecast start, after a reboot:
/proc/sys/fs/file-nr:
/proc/sys/fs/file-max:
sysctl fs.file-max:
Quote: |
fs.file-max = 786432 |
/proc/net/sockstat:
Quote: |
sockets: used 1078
TCP: inuse 1066 orphan 35 tw 19 alloc 1066 mem 3839
UDP: inuse 2 mem 1
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0 |
But:
icecast error log:
Quote: |
[2010-12-19 20:02:49] WARN connection/accept_client accept() failed with error 24: Too many open files |
and /proc/`pidof icecast`/limits:
Quote: |
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited ms
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 106496 106496 processes
Max open files 1024 1024 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 106496 106496 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us |
I'm about to send you the error.log by email. _________________ Guillaume |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
karlH Code Warrior
![Code Warrior Code Warrior](templates/subRebel/images/ranks/rank_dev.gif)
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Sun Dec 19, 2010 9:11 pm Post subject: |
|
|
Looks like the listener count and sources streaming reaches the 1024 open files limit causing the failures you are seeing. Assuming those number of listener/sources is expected then I would increase the open file limit via /etc/security/limits.conf)
eg
icecast - nofile 10000
ulimit -a will show the current shell limits and icecast logs the limit at the start, I'm not aware of a way to increase the limit externally.
karl. |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
TheFlyingBear
Joined: 08 Dec 2010 Posts: 15 Location: France
|
Posted: Mon Dec 20, 2010 7:39 am Post subject: |
|
|
I already have this line in my /etc/security/limits.conf:
As you can see, the value is already high...
The other strange things, is, that if I restart icecast, the problem disappears, and then things runs nicely. _________________ Guillaume |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
karlH Code Warrior
![Code Warrior Code Warrior](templates/subRebel/images/ranks/rank_dev.gif)
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Mon Dec 20, 2010 12:23 pm Post subject: |
|
|
you should check the ulimit -a list before starting icecast, the
Max open files 1024 1024 files
says the actual limit for the process. Maybe you haven't logged back in again or maybe you start icecast from something with a 1024 FD limit already imposed.
karl. |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
TheFlyingBear
Joined: 08 Dec 2010 Posts: 15 Location: France
|
Posted: Mon Dec 20, 2010 4:52 pm Post subject: |
|
|
The limits are set at boot time via sysctl, and then, the icecast is started by init.
My init script is really simple: a test to know if y ask for a start or a stop, and the usual: start-stop-daemon --background --start --make-pidfile --pidfile /var/run/icecast.pid --exec /usr/bin/icecast -- -c /etc/icecast2/icecast.xml
I can't find where these limits can be changed from 786432 to 1024 ![Shocked](images/smiles/icon_eek.gif) _________________ Guillaume |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
karlH Code Warrior
![Code Warrior Code Warrior](templates/subRebel/images/ranks/rank_dev.gif)
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Mon Dec 20, 2010 5:36 pm Post subject: |
|
|
you mentioned the init is starting icecast so icecast is inheriting the init limits, so check /proc/`pidof init`/limits, that probably has 1024 set. I suspect you have not restarted init since changing the limits.conf file.
Seeing that you are starting from a root script then you should be able to increase the limit just before starting icecast with ulimit -n 786432
karl. |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
TheFlyingBear
Joined: 08 Dec 2010 Posts: 15 Location: France
|
Posted: Tue Dec 21, 2010 9:15 am Post subject: |
|
|
Yes you're right: the open file limit of init is set at 1024, but, I did restart many times since changing the limits.conf.
I have other servers (not running icecast, but handling many connections too) that have the same limit setup (with the limit at 1024 for init), and I don't have this problem.
I'm going to change my icecast init script so that it can increase itself the open file limit. _________________ Guillaume |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
karlH Code Warrior
![Code Warrior Code Warrior](templates/subRebel/images/ranks/rank_dev.gif)
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Tue Dec 21, 2010 1:57 pm Post subject: |
|
|
you can only ever increase the limit as root and many icecasts are not started as root. Icecast has traditionally inherited the limits given to it, and as the only vaguely related setting is clients then we would only guess at some figure larger than that.
karl. |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
TheFlyingBear
Joined: 08 Dec 2010 Posts: 15 Location: France
|
Posted: Tue Dec 21, 2010 4:13 pm Post subject: |
|
|
I changed my init script so that they can set this limit if it's not already done.
It should solve the issue. _________________ Guillaume |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
TheFlyingBear
Joined: 08 Dec 2010 Posts: 15 Location: France
|
Posted: Thu Dec 23, 2010 4:44 pm Post subject: |
|
|
The problem is solved with the new init script.
We can close the case. _________________ Guillaume |
|
Back to top |
|
![](templates/subRebel/images/spacer.gif) |
|