View previous topic :: View next topic |
Author |
Message |
als
Joined: 14 Oct 2008 Posts: 18
|
Posted: Tue Oct 14, 2008 11:03 am Post subject: Icecast 2.3.2 1000+ memory usage explodes some day |
|
|
Icecast serves up to 1800 listeners with 6 streams on 512 Meg virtual server. Memory usage merely grows until 15-17% of RAM, but some time suddenly grows up. This time it was 80%. Previously it occupied all available RAM & swap (it happened on Sep04). "explosion" started at 00:50 AM, when listener count goes down significantly.
Log files doesn't show any special. (sorry, can't hold debug level for nearly a month ;-(
limits section of .config file :
Code: |
<limits>
<clients>3000</clients>
<sources>8</sources>
<threadpool>30</threadpool>
<queue-size>8524288</queue-size>
<client-timeout>10</client-timeout>
<header-timeout>15</header-timeout>
<source-timeout>15</source-timeout>
<burst-on-connect>0</burst-on-connect>
<burst-size>65535</burst-size>
</limits>
|
Some possible hint: It seems that there was a kind if global (within my country) internet connectivity problems at night. |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Tue Oct 14, 2008 3:53 pm Post subject: |
|
|
Is this a win32 build? It could be down to some unused resources that are allocated by some of the libs, I heard of issues on some win32 setups because of this. If so, you can try another build I did which is from a post 2.3.2 cut from trunk.
www.icecast.pwp.blueyonder.co.uk/icecast2_win32_2.3.2-3_setup.exe
karl. |
|
Back to top |
|
|
als
Joined: 14 Oct 2008 Posts: 18
|
Posted: Wed Oct 15, 2008 8:58 am Post subject: No it's linux |
|
|
It's debian virtual server
~/icecast-2.3.2# uname -r
2.6.24-18-xen
Built from source.
I can e-mail you zipped log's, if it helps.
BTW, are there any guidelines for selecting "threadpool" size for known RAM size & listener count? |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Wed Oct 15, 2008 12:56 pm Post subject: |
|
|
sure, along with the xml so that I know which parts of the server are being used. You could try a post 2.3.2 build, as the issue that I mentioned is a general thing but only reported as a problem on win32. Nothing "unstable" has been added since the release, a snapshot is available in people.xiph.org/~brendan/snapshots/icecast
wrt a guide to threadpool values, nothing, as it is ignored in 2.3.2
karl. |
|
Back to top |
|
|
als
Joined: 14 Oct 2008 Posts: 18
|
Posted: Wed Oct 15, 2008 2:27 pm Post subject: |
|
|
Sent You log's (mem.log contains memory usage statistics);
Installed icecast-2.3.2-20081015, will report if smth happen (hope not earlier than in 30 days ).
>threadpool values, nothing, as it is ignored in 2.3.2
Ignored ;-( ....I was so proud that, reducing it from 50 to 30 - limited memory usage from 200% RAM to 80%.... |
|
Back to top |
|
|
epe Asesor espaņol
Joined: 27 Aug 2005 Posts: 132 Location: Quito - Ecuador
|
Posted: Mon Nov 17, 2008 4:02 pm Post subject: |
|
|
karlH wrote: |
sure, along with the xml so that I know which parts of the server are being used. You could try a post 2.3.2 build, as the issue that I mentioned is a general thing but only reported as a problem on win32. Nothing "unstable" has been added since the release, a snapshot is available in people.xiph.org/~brendan/snapshots/icecast
karl. |
Hi Karl
We have used icecast for 4 years now.... however we have noticed that since icecast-2.3.2 compiled for CentOS-3, 4 and 5 we have to restart icecast every week or so, otherwise it starts eating up all the RAM and swapping. We have noticed that the problem is more noticeable with centos-3 and 4.. but it might be a problem of how much traffic they handle and other factors.
Today I restarted one server (a centos-3), here is some info:
Code: |
[root@srv6 root]# ps auwwx|egrep icecast
icecast 8831 1.0 8.0 274896 81560 ? S Nov13 61:21 icecast -b -c /etc/icecast.xml
root 4068 0.0 0.0 3696 508 ttyp0 S 10:57 0:00 egrep icecast
[root@srv6 root]#
[root@srv6 root]# free -m
total used free shared buffers cached
Mem: 987 950 37 0 79 593
-/+ buffers/cache: 277 710
Swap: 1027 214 813
[root@srv6 root]#
[root@srv6 root]# killall icecast
[root@srv6 root]# icecast -b -c /etc/icecast.xml
Starting icecast2
Detaching from the console
[root@srv6 root]# Changed root successfully to "/usr/share/icecast".
Changed groupid to 1236.
Changed userid to 1236.
[root@srv6 root]# free -m
total used free shared buffers cached
Mem: 987 872 114 0 79 594
-/+ buffers/cache: 199 788
Swap: 1027 52 975
[root@srv6 root]#
|
It is a production server but if you think we must send some other info let me know.
BTW we compiled it with rpmbuild (using the provided src.rpm)
Most of the affected servers has been working with icecast for years.. and the problem started happening after we updated to 2.3.2. What we notice is that customers start complaining that radios start dropping and some other effects (webpages not showing, etc)... and when we restart icecast, the swap usage goes normal and the ram usage too.
thanks
epe _________________ --
http://www.NuestroServer.com
USA: +1 305 359 4495, Espaņa: +34 911 877 602
Ecuador: +593 2 600 4454
Se habla espaņol |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Mon Nov 17, 2008 4:23 pm Post subject: |
|
|
Can you check the logs for source disconnections that happen some time before the crash (or heavy usage become apparent). The investigation I'm doing for als seems to be down to that (a source timeout typically) but is not easily triggered. In that case the leak starts around reconnection and slowly increases.
karl. |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Wed Jan 14, 2009 5:12 pm Post subject: |
|
|
After a number of tests, I think we have solved that memory leak bug which was down to some memory corruption. If people experiencing this can try the current trunk code then that can be verified as fixed. If people need a windows build then let me know.
karl. |
|
Back to top |
|
|
epe Asesor espaņol
Joined: 27 Aug 2005 Posts: 132 Location: Quito - Ecuador
|
Posted: Tue Jan 20, 2009 7:04 pm Post subject: |
|
|
I created and installed an RPM from the svn source, and icecast has been running for 2 days now and yes, the RAM usage has been stable (~36MB similar to yesterday's usage).
If eveything goes ok by today I will recompile my other servers with this version.
One question: When will this fixes be available to the general public? Will it be named 2.3.2-1 or 2.3.3 ? _________________ --
http://www.NuestroServer.com
USA: +1 305 359 4495, Espaņa: +34 911 877 602
Ecuador: +593 2 600 4454
Se habla espaņol |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Tue Jan 20, 2009 8:34 pm Post subject: |
|
|
I would say it would be a 2.3.3. Thanks for checking and letting me know, but do let us know if the problem shows up again.
karl. |
|
Back to top |
|
|
epe Asesor espaņol
Joined: 27 Aug 2005 Posts: 132 Location: Quito - Ecuador
|
Posted: Wed Mar 04, 2009 3:33 am Post subject: |
|
|
ok, it's been more than a month since I started using this patched version and the servers have been working flawlessly with my home-made rpm of icecast-2.3.3
thanks!
epe _________________ --
http://www.NuestroServer.com
USA: +1 305 359 4495, Espaņa: +34 911 877 602
Ecuador: +593 2 600 4454
Se habla espaņol |
|
Back to top |
|
|
xmarchais
Joined: 22 Jul 2009 Posts: 1 Location: Paris
|
Posted: Wed Jul 22, 2009 9:03 am Post subject: |
|
|
Hi,
We've got same problem as epe and als, memory consumption and need to restart praticly every day. installed version is : Icecast 2.3.2-2 from debian source.
Using svn source should be resolve this problem ?
Regards, |
|
Back to top |
|
|
mobhaku
Joined: 18 May 2006 Posts: 8
|
Posted: Thu Apr 29, 2010 9:47 am Post subject: |
|
|
Hi Karl,
karlH wrote: |
After a number of tests, I think we have solved that memory leak bug which was down to some memory corruption. If people experiencing this can try the current trunk code then that can be verified as fixed. If people need a windows build then let me know.
karl. |
Currently I'm facing a memory-issue with the windows build of the official 2.3.2 build. We have a dynamic number of mountpoints, but can easily grow to more than 300. The number of clients can go up to 2000 in total. I really would like to test if this "new" version is better, is it somewhere available?
Another question is regarding the thread-pool:
With such source and client figures, what is a good value for the tread-pool?
Thanks in advance and best regards,
Harreld. |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Fri Apr 30, 2010 3:20 am Post subject: |
|
|
I'll have to do a more recent win32 build of trunk or you can use my branch work. As for threadpool, you can use whatever setting you like as the setting is ignored. If you really have 300 streams active at the same time then you may be getting hit hard by threading in which case you can use my branch work as that uses a fixed N worker thread setup.
karl.
Last edited by karlH on Sat May 01, 2010 7:50 pm; edited 1 time in total |
|
Back to top |
|
|
mobhaku
Joined: 18 May 2006 Posts: 8
|
Posted: Sat May 01, 2010 12:29 pm Post subject: |
|
|
Ok, sounds good.
How hard is it to build a Win32 version?
And... how safe is it to take your branch work in production?
I guess that for Linux I can fetch it from SVN and then build it? |
|
Back to top |
|
|
|