Icecast Streaming Media Server Forum Index Icecast Streaming Media Server
Icecast is a Xiph Foundation Project
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Icecast 2.3.2 1000+ memory usage explodes some day
Goto page 1, 2  Next
 
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Bug Reports
View previous topic :: View next topic  
Author Message
als



Joined: 14 Oct 2008
Posts: 18

PostPosted: Tue Oct 14, 2008 11:03 am    Post subject: Icecast 2.3.2 1000+ memory usage explodes some day Reply with quote

Icecast serves up to 1800 listeners with 6 streams on 512 Meg virtual server. Memory usage merely grows until 15-17% of RAM, but some time suddenly grows up. This time it was 80%. Previously it occupied all available RAM & swap (it happened on Sep04). "explosion" started at 00:50 AM, when listener count goes down significantly.
Log files doesn't show any special. (sorry, can't hold debug level for nearly a month ;-(
limits section of .config file :
Code:
    <limits>                                                                                                                     
<clients>3000</clients>                                                                                                   
<sources>8</sources>                                                                                                     
<threadpool>30</threadpool>                                                                                               
        <queue-size>8524288</queue-size>                                                                                         
        <client-timeout>10</client-timeout>                                                                                       
        <header-timeout>15</header-timeout>                                                                                       
        <source-timeout>15</source-timeout>                                                                                       
        <burst-on-connect>0</burst-on-connect>                                                                                   
        <burst-size>65535</burst-size>                                                                                           
    </limits>   


Some possible hint: It seems that there was a kind if global (within my country) internet connectivity problems at night.
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Tue Oct 14, 2008 3:53 pm    Post subject: Reply with quote

Is this a win32 build? It could be down to some unused resources that are allocated by some of the libs, I heard of issues on some win32 setups because of this. If so, you can try another build I did which is from a post 2.3.2 cut from trunk.

www.icecast.pwp.blueyonder.co.uk/icecast2_win32_2.3.2-3_setup.exe

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
als



Joined: 14 Oct 2008
Posts: 18

PostPosted: Wed Oct 15, 2008 8:58 am    Post subject: No it's linux Reply with quote

It's debian virtual server
~/icecast-2.3.2# uname -r
2.6.24-18-xen

Built from source.
I can e-mail you zipped log's, if it helps.

BTW, are there any guidelines for selecting "threadpool" size for known RAM size & listener count?
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Wed Oct 15, 2008 12:56 pm    Post subject: Reply with quote

sure, along with the xml so that I know which parts of the server are being used. You could try a post 2.3.2 build, as the issue that I mentioned is a general thing but only reported as a problem on win32. Nothing "unstable" has been added since the release, a snapshot is available in people.xiph.org/~brendan/snapshots/icecast

wrt a guide to threadpool values, nothing, as it is ignored in 2.3.2

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
als



Joined: 14 Oct 2008
Posts: 18

PostPosted: Wed Oct 15, 2008 2:27 pm    Post subject: Reply with quote

Sent You log's (mem.log contains memory usage statistics);

Installed icecast-2.3.2-20081015, will report if smth happen (hope not earlier than in 30 days Very Happy ).

>threadpool values, nothing, as it is ignored in 2.3.2
Ignored ;-( ....I was so proud that, reducing it from 50 to 30 - limited memory usage from 200% RAM to 80%....
Back to top
View user's profile Send private message
epe
Asesor espaņol
Asesor espaņol


Joined: 27 Aug 2005
Posts: 132
Location: Quito - Ecuador

PostPosted: Mon Nov 17, 2008 4:02 pm    Post subject: Reply with quote

karlH wrote:
sure, along with the xml so that I know which parts of the server are being used. You could try a post 2.3.2 build, as the issue that I mentioned is a general thing but only reported as a problem on win32. Nothing "unstable" has been added since the release, a snapshot is available in people.xiph.org/~brendan/snapshots/icecast

karl.


Hi Karl

We have used icecast for 4 years now.... however we have noticed that since icecast-2.3.2 compiled for CentOS-3, 4 and 5 we have to restart icecast every week or so, otherwise it starts eating up all the RAM and swapping. We have noticed that the problem is more noticeable with centos-3 and 4.. but it might be a problem of how much traffic they handle and other factors.

Today I restarted one server (a centos-3), here is some info:
Code:

[root@srv6 root]# ps auwwx|egrep icecast
icecast   8831  1.0  8.0 274896 81560 ?      S    Nov13  61:21 icecast -b -c /etc/icecast.xml
root      4068  0.0  0.0  3696  508 ttyp0    S    10:57   0:00 egrep icecast
[root@srv6 root]#
[root@srv6 root]# free -m
             total       used       free     shared    buffers     cached
Mem:           987        950         37          0         79        593
-/+ buffers/cache:        277        710
Swap:         1027        214        813
[root@srv6 root]#
[root@srv6 root]# killall icecast
[root@srv6 root]# icecast -b -c /etc/icecast.xml
Starting icecast2
Detaching from the console
[root@srv6 root]# Changed root successfully to "/usr/share/icecast".
Changed groupid to 1236.
Changed userid to 1236.

[root@srv6 root]# free -m
             total       used       free     shared    buffers     cached
Mem:           987        872        114          0         79        594
-/+ buffers/cache:        199        788
Swap:         1027         52        975
[root@srv6 root]#


It is a production server but if you think we must send some other info let me know.

BTW we compiled it with rpmbuild (using the provided src.rpm)

Most of the affected servers has been working with icecast for years.. and the problem started happening after we updated to 2.3.2. What we notice is that customers start complaining that radios start dropping and some other effects (webpages not showing, etc)... and when we restart icecast, the swap usage goes normal and the ram usage too.

thanks
epe
_________________
--
http://www.NuestroServer.com
USA: +1 305 359 4495, Espaņa: +34 911 877 602
Ecuador: +593 2 600 4454
Se habla espaņol
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Mon Nov 17, 2008 4:23 pm    Post subject: Reply with quote

Can you check the logs for source disconnections that happen some time before the crash (or heavy usage become apparent). The investigation I'm doing for als seems to be down to that (a source timeout typically) but is not easily triggered. In that case the leak starts around reconnection and slowly increases.


karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Wed Jan 14, 2009 5:12 pm    Post subject: Reply with quote

After a number of tests, I think we have solved that memory leak bug which was down to some memory corruption. If people experiencing this can try the current trunk code then that can be verified as fixed. If people need a windows build then let me know.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
epe
Asesor espaņol
Asesor espaņol


Joined: 27 Aug 2005
Posts: 132
Location: Quito - Ecuador

PostPosted: Tue Jan 20, 2009 7:04 pm    Post subject: Reply with quote

I created and installed an RPM from the svn source, and icecast has been running for 2 days now and yes, the RAM usage has been stable (~36MB similar to yesterday's usage).

If eveything goes ok by today I will recompile my other servers with this version.

One question: When will this fixes be available to the general public? Will it be named 2.3.2-1 or 2.3.3 ?
_________________
--
http://www.NuestroServer.com
USA: +1 305 359 4495, Espaņa: +34 911 877 602
Ecuador: +593 2 600 4454
Se habla espaņol
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Tue Jan 20, 2009 8:34 pm    Post subject: Reply with quote

I would say it would be a 2.3.3. Thanks for checking and letting me know, but do let us know if the problem shows up again.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
epe
Asesor espaņol
Asesor espaņol


Joined: 27 Aug 2005
Posts: 132
Location: Quito - Ecuador

PostPosted: Wed Mar 04, 2009 3:33 am    Post subject: Reply with quote

ok, it's been more than a month since I started using this patched version and the servers have been working flawlessly with my home-made rpm of icecast-2.3.3

thanks!
epe
_________________
--
http://www.NuestroServer.com
USA: +1 305 359 4495, Espaņa: +34 911 877 602
Ecuador: +593 2 600 4454
Se habla espaņol
Back to top
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger
xmarchais



Joined: 22 Jul 2009
Posts: 1
Location: Paris

PostPosted: Wed Jul 22, 2009 9:03 am    Post subject: Reply with quote

Hi,

We've got same problem as epe and als, memory consumption and need to restart praticly every day. installed version is : Icecast 2.3.2-2 from debian source.

Using svn source should be resolve this problem ?

Regards,
Back to top
View user's profile Send private message
mobhaku



Joined: 18 May 2006
Posts: 8

PostPosted: Thu Apr 29, 2010 9:47 am    Post subject: Reply with quote

Hi Karl,

karlH wrote:
After a number of tests, I think we have solved that memory leak bug which was down to some memory corruption. If people experiencing this can try the current trunk code then that can be verified as fixed. If people need a windows build then let me know.

karl.


Currently I'm facing a memory-issue with the windows build of the official 2.3.2 build. We have a dynamic number of mountpoints, but can easily grow to more than 300. The number of clients can go up to 2000 in total. I really would like to test if this "new" version is better, is it somewhere available?

Another question is regarding the thread-pool:
With such source and client figures, what is a good value for the tread-pool?
Thanks in advance and best regards,

Harreld.
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Fri Apr 30, 2010 3:20 am    Post subject: Reply with quote

I'll have to do a more recent win32 build of trunk or you can use my branch work. As for threadpool, you can use whatever setting you like as the setting is ignored. If you really have 300 streams active at the same time then you may be getting hit hard by threading in which case you can use my branch work as that uses a fixed N worker thread setup.

karl.


Last edited by karlH on Sat May 01, 2010 7:50 pm; edited 1 time in total
Back to top
View user's profile Send private message Send e-mail Visit poster's website
mobhaku



Joined: 18 May 2006
Posts: 8

PostPosted: Sat May 01, 2010 12:29 pm    Post subject: Reply with quote

Ok, sounds good.
How hard is it to build a Win32 version?
And... how safe is it to take your branch work in production? Smile
I guess that for Linux I can fetch it from SVN and then build it?
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Bug Reports All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2002 phpBB Group
subRebel style by ktauber