Icecast Streaming Media Server Forum Index Icecast Streaming Media Server
Icecast is a Xiph Foundation Project
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Icecast 2.3.2-kh28 - open files

 
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Bug Reports
View previous topic :: View next topic  
Author Message
TheFlyingBear



Joined: 08 Dec 2010
Posts: 15
Location: France

PostPosted: Fri Dec 17, 2010 3:56 pm    Post subject: Icecast 2.3.2-kh28 - open files Reply with quote

And a new one... (I'm really sorry)

I have this problem on already running icecast (never noticed it before my switch to 2.3.32-kh38) or a freshly installed and booted system...
Shortly after icecast's start, I have this in my error.log:
Code:
[2010-12-17  16:20:23] WARN connection/accept_client accept() failed with error 24: Too many open files

But, at the same moment, my /proc/sys/fs/file-nr says:
Code:
1216   0   786432


If I stop icecast, and then restart it (sometimes 3 or 4 times), the problem disapears Confused

I have this problem on Linux servers running:
- Gentoo Linux with kernel 2.6.30-gentoo-r8 ;
- Debian (stable) Linux with kernel 2.6.26-2

My opened files limit is 786432, and my icecast connections limit is 65536.

Any idea?
_________________
Guillaume
Back to top
View user's profile Send private message MSN Messenger
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Fri Dec 17, 2010 4:19 pm    Post subject: Reply with quote

The too many open files limit is the process limit so you are reaching the limit as shown in the /proc/<pid>/limits file. Maybe you can send me the level 4 error log showing what is going on?

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
TheFlyingBear



Joined: 08 Dec 2010
Posts: 15
Location: France

PostPosted: Fri Dec 17, 2010 4:31 pm    Post subject: Reply with quote

I only have level 3 error log. I'll change this for the next time.
I had a look in /proc/`pidof icecast`/limits, and here is what I found (but I don't have the problem at the moment):
Quote:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited ms
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 393216 393216 processes
Max open files 786432 786432 files
Max locked memory 32768 32768 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 106496 106496 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us


The strange thing is that I can't reproduce this problem each time I start or restart Icecast.
_________________
Guillaume
Back to top
View user's profile Send private message MSN Messenger
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Fri Dec 17, 2010 4:39 pm    Post subject: Reply with quote

you can increase the log level and kill -HUP to change the running copy without dropping any connections.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
TheFlyingBear



Joined: 08 Dec 2010
Posts: 15
Location: France

PostPosted: Sun Dec 19, 2010 7:09 pm    Post subject: Reply with quote

Hello,

I'm currently having this problem, on icecast start, after a reboot:
/proc/sys/fs/file-nr:
Quote:
1312 0 786432

/proc/sys/fs/file-max:
Quote:
786432

sysctl fs.file-max:
Quote:
fs.file-max = 786432

/proc/net/sockstat:
Quote:
sockets: used 1078
TCP: inuse 1066 orphan 35 tw 19 alloc 1066 mem 3839
UDP: inuse 2 mem 1
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0


But:
icecast error log:
Quote:
[2010-12-19 20:02:49] WARN connection/accept_client accept() failed with error 24: Too many open files

and /proc/`pidof icecast`/limits:
Quote:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited ms
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 106496 106496 processes
Max open files 1024 1024 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 106496 106496 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us


I'm about to send you the error.log by email.
_________________
Guillaume
Back to top
View user's profile Send private message MSN Messenger
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Sun Dec 19, 2010 9:11 pm    Post subject: Reply with quote

Looks like the listener count and sources streaming reaches the 1024 open files limit causing the failures you are seeing. Assuming those number of listener/sources is expected then I would increase the open file limit via /etc/security/limits.conf)

eg

icecast - nofile 10000

ulimit -a will show the current shell limits and icecast logs the limit at the start, I'm not aware of a way to increase the limit externally.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
TheFlyingBear



Joined: 08 Dec 2010
Posts: 15
Location: France

PostPosted: Mon Dec 20, 2010 7:39 am    Post subject: Reply with quote

I already have this line in my /etc/security/limits.conf:
Quote:
* - nofile 786432


As you can see, the value is already high...

The other strange things, is, that if I restart icecast, the problem disappears, and then things runs nicely.
_________________
Guillaume
Back to top
View user's profile Send private message MSN Messenger
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Mon Dec 20, 2010 12:23 pm    Post subject: Reply with quote

you should check the ulimit -a list before starting icecast, the

Max open files 1024 1024 files

says the actual limit for the process. Maybe you haven't logged back in again or maybe you start icecast from something with a 1024 FD limit already imposed.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
TheFlyingBear



Joined: 08 Dec 2010
Posts: 15
Location: France

PostPosted: Mon Dec 20, 2010 4:52 pm    Post subject: Reply with quote

The limits are set at boot time via sysctl, and then, the icecast is started by init.
My init script is really simple: a test to know if y ask for a start or a stop, and the usual: start-stop-daemon --background --start --make-pidfile --pidfile /var/run/icecast.pid --exec /usr/bin/icecast -- -c /etc/icecast2/icecast.xml

I can't find where these limits can be changed from 786432 to 1024 Shocked
_________________
Guillaume
Back to top
View user's profile Send private message MSN Messenger
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Mon Dec 20, 2010 5:36 pm    Post subject: Reply with quote

you mentioned the init is starting icecast so icecast is inheriting the init limits, so check /proc/`pidof init`/limits, that probably has 1024 set. I suspect you have not restarted init since changing the limits.conf file.

Seeing that you are starting from a root script then you should be able to increase the limit just before starting icecast with ulimit -n 786432

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
TheFlyingBear



Joined: 08 Dec 2010
Posts: 15
Location: France

PostPosted: Tue Dec 21, 2010 9:15 am    Post subject: Reply with quote

Yes you're right: the open file limit of init is set at 1024, but, I did restart many times since changing the limits.conf.
I have other servers (not running icecast, but handling many connections too) that have the same limit setup (with the limit at 1024 for init), and I don't have this problem.
I'm going to change my icecast init script so that it can increase itself the open file limit.
_________________
Guillaume
Back to top
View user's profile Send private message MSN Messenger
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Tue Dec 21, 2010 1:57 pm    Post subject: Reply with quote

you can only ever increase the limit as root and many icecasts are not started as root. Icecast has traditionally inherited the limits given to it, and as the only vaguely related setting is clients then we would only guess at some figure larger than that.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
TheFlyingBear



Joined: 08 Dec 2010
Posts: 15
Location: France

PostPosted: Tue Dec 21, 2010 4:13 pm    Post subject: Reply with quote

I changed my init script so that they can set this limit if it's not already done.
It should solve the issue.
_________________
Guillaume
Back to top
View user's profile Send private message MSN Messenger
TheFlyingBear



Joined: 08 Dec 2010
Posts: 15
Location: France

PostPosted: Thu Dec 23, 2010 4:44 pm    Post subject: Reply with quote

The problem is solved with the new init script.
We can close the case.
_________________
Guillaume
Back to top
View user's profile Send private message MSN Messenger
Display posts from previous:   
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Bug Reports All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2002 phpBB Group
subRebel style by ktauber