| View previous topic :: View next topic |
| Author |
Message |
Nonapeptide
Joined: 08 Sep 2012
Posts: 40
Location: Scottsdale, Arizona
|
 Posted: Sat Sep 15, 2012 1:38 am Post subject: Windows 2.3.3-KH2 hits 220 / 225 listeners then chokes |
 |
|
I have a Windows Server 2008 R2 server running the 2.3.3-KH2 branch of IceCast. It's got a Xeon E5430, 2GB of RAM (over half of which is currently free), and a 10Gb interface. I've got about 40 mount points that are streaming at 48Kbps.
When I get about 220 to 225 people listening across the entire server, suddenly the IceCast admin page doesn't load properly. This is how the page should look: http://i.imgur.com/TzAr4.png <-- notice that there are currently 220 listeners. When I see that number, I know what's about to happen...
Once we hit a certain level around 225, this is how it looks: http://i.imgur.com/H8lUp.png <-- That's somewhat deceiving because the page looks all black, I have to click and drag to see select the text and highlight it to view it. Other times, the server response with a white page and no text. Other times listeners get an error that says something to the effect of "Access Denied." I can't corroborate that because I have never personally seen it happen, and the listeners that have seen it haven't been able to give me exact wording.
The server itself is behaving well - I'm not convinced yet that it's an OS issue. I can hammer the network card and OS with other tasks and it shrugs it off quite well. Resource Monitor's networking tab shows the following when IceCast is choking: http://i.imgur.com/jQcpX.png
Here's my sanitized config file:
| Quote: |
<icecast>
<limits>
<sources>999</sources>
</limits>
<authentication>
<source-password>password</source-password>
<admin-user>admin</admin-user>
<admin-password>password</admin-password>
</authentication>
<hostname>my.server.com</hostname>
<location>USA</location>
<admin>me@email.com</admin>
<listen-socket>
<port>80</port>
<shoutcast-mount>/live</shoutcast-mount>
</listen-socket>
<fileserve>1</fileserve>
<paths>
<logdir>./logs</logdir>
<webroot>./web</webroot>
<adminroot>./admin</adminroot>
<alias source="/" dest="/index.html"/>
</paths>
<logging>
<accesslog>access.log</accesslog>
<errorlog>error.log</errorlog>
<loglevel>3</loglevel> <!-- 4 Debug, 3 Info, 2 Warn, 1 Error -->
</logging>
<mount>
40 mount points
</mount>
</icecast> |
I'm currently looking through potential config file changes that would have an effect on this situation. I've got 220+ concurrent listeners at this very moment and am loathe to bump them in order to reload the configuration file. "But you don't have to restart the service to reload the config file!" You do when the administrative URLs are not loading. 
I thought the thread pool option might be of help, but that option seems to not exist after 2.0 in the KH branch. I'll keep looking, but was desirous of some sage advice from those who have administered IceCast on higher-load implementations before.
Offered wisdom is appreciate whilst I attempt to earn it on my own.  |
|
| Back to top |
|
 |
Nonapeptide
Joined: 08 Sep 2012
Posts: 40
Location: Scottsdale, Arizona
|
| Posted: Sat Sep 15, 2012 6:31 am Post subject: |
|
|
When in doubt, consult the log files! I noticed this jewel in error.log:
| Quote: |
| [2012-09-14 18:52:46]WARN connection/_handle_get_request server client limit reached (256/257) |
Apparently when a listener limit is not explicitly defined, there is 256 listener connection limit. I bumped it up to 20,000 and then pounded it to death with curl requests. I've had up over 1,000 concurrent listeners now and the server barely stops snoring long enough to roll over.
So... yeah.  |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2002 phpBB Group
subRebel style by ktauber
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
