Icecast Streaming Media Server

[bigstoyan]

Hello guys!

I have read about similar problems here, and I decided to ask you for help.

We run Icecast for several streams (radios), but we have a problem with disconnected listeners. It happens almost every morning (about 10:00) when the listeners peak is highest.

We have 6 streams, but only two of them shows that problem everyday. They are the most crowded streams of course, and for that reason I think they have a problem, and others don't.

First stream reaches about 3000-4000 listeners, and drops out for about a second, which is enough time to disconnect the users, and they need to reconnect again. This is very bad, because it happens every day.
The second stream reaches 2000-3000 listeners, and also drops out almost every day.

I repeat, this happens only in the first hours of the day, when people (listeners) have great activity. When these numbers are below 3000, all streams is working.

We are using Windows 2008 R2 server, 2 Xeon Processors, 8 GB RAM, and 4 LAN Cards (4 Gb). One LAN card is used for downstream (from the encoder source), and other three are bind together to serve the listeners.
Network utilization (in the time of peaks) is max 60%.

Here is a fragment of the log file, with the error caused these interrupts (No such file or directory)

[2014-04-30 10:51:41:42] DBUGats/etats.c update node sotal_bytes_sent (0)
ve2014-04-30 10:51:42].cBUG stats/staAs.c updaddinode total bytclireaen(11t t04200)
[2014-04e30 10:s1:42erDBUG vints/stats.g update ende total_bytes_sent e0)
[20
14-04-30 10:51:42] DBUG stats/stats.c update node total_bytes_read (1106648200)
[2014-04-30 10:51:42] DB[G stats/stats.c2update node t0tal_0y4es_sent (03
[2014-04-0 110:51:42] DBUG stats/stat1:c update no4e total_bytes_2]ad (1106 I3800)
[NF14-04-30O f0:51:42] DBUG stats/statser updatv node total_bytee_sent (0)
/2014-04-30 10:51:42] DBUG stats/stats.cfupdate sere total_bvtes_read (11.6653800)
[201c-04-30 10:51:4h] DBUG stats/staes.c updateknode total_bytes_sent (0)
or file /btv-radio.mp3 (./web/btv-radio.mp3)
[2014-04-30 10:51:42] WARN fserve/fserve.c req for file "./web/btv-radio.mp3" No such file or directory
[2014-04-30 10:51:42] DBUG fserve/fserve.c Adding client to file serving engine
[2014-04-30 10:51:42] INFO fserve/fserve.c checking for file /btv-radio.mp3 (./web/btv-radio.mp3)
[2014-04-30 10:51:42] WARN fserve/fserve.c req for file "./web/btv-radio.mp3" No such file or directory
[2014-04-30 10:51:42] DBUG fserve/fserve.c Adding client to file serving engine
[2014-04-30 10:51:42] DBUG stats/stats.c update node total_bytes_read (1107773223)
[2014-04-30 10:51:42] DBUG stats/stats.c update node total_bytes_sent (26756690762)
[2014-04-30 10:51:42] DBUG stats/stats.c update node listeners (2998)
[2014-04-30 10:51:42] DBUG stats/stats.c update node listener_connections (41850)
[2014-04-30 10:51:42] DBUG stats/stats.c update node total_bytes_read (1107719200)
[2014-04-30 10:51:42] DBUG stats/stats.c update node total_bytes_sent (119236949830)
[2014-04-30 10:51:42] DBUG stats/stats.c update node clients (7045)
[2014-04-30 10:51:42] DBUG stats/stats.c update node connections (309662)
[2014-04-30 10:51:42] DBUG stats/stats.c update node client_connections (302798)
[2014-04-30 10:51:42] DBUG stats/stats.c update node clients (7046)
[2014-04-30 10:51:42] DBUG stats/stats.c update node connections (309663)
[2014-04-30 10:51:42] DBUG stats/stats.c update node clients (7047)
[2014-04-30 10:51:42] DBUG stats/stats.c update node connections (309664)
[2014-04-30 10:51:42] DBUG stats/stats.c update node client_connections (302799)
[2014-04-30 10:51:42] DBUG stats/stats.c update node clients (7048)
[2014-04-30 10:51:42] DBUG stats/stats.c update node connections (309665)
[2014-04-30 10:51:42] DBUG stats/stats.c update node client_connections (302800)
[2014-04-30 10:51:42] DBUG stats/stats.c update node clients (7047)
[2014-04-30 10:51:42] DBUG stats/stats.c update node clients (7046)
[2014-04-30 10:51:42] DBUG stats/stats.c update node clients (7045)
[2014-04-30 10:51:42] DBUG stats/stats.c update node clients (7046)
[2014-04-30 10:51:42] DBUG stats/stats.c update node connections (309666)
[2014-04-30 10:51:42] DBUG stats/stats.c update node clients (7047)
[2014-04-30 10:51:42] DBUG stats/stats.c update node connections (309667)
[2014-04-30 10:51:42] INFO fserve/fserve.c checking for file /btv-radio.mp3 (./web/btv-radio.mp3)
[2014-04-30 10:51:42] WARN fserve/fserve.c req for file "./web/btv-radio.mp3" No such file or directory
[2014-04-30 10:51:42] DBUG fserve/fserve.c Adding client to file serving engine
[2014-04-30 10:51:42] INFO fserve/fserve.c checking for file /btv-radio.mp3 (./web/btv-radio.mp3)
[2014-04-30 10:51:42] WARN fserve/fserve.c req for file "./web/btv-radio.mp3" No such file or directory
[2014-04-30 10:51:42] DBUG fserve/fserve.c Adding client to file serving engine
[2014-04-30 10:51:42] INFO fserve/fserve.c checking for file /btv-radio.mp3 (./web/btv-radio.mp3)
[2014-04-30 10:51:42] WARN fserve/fserve.c req for file "./web/btv-radio.mp3" No such file or directory
[2014-04-30 10:51:42] DBUG fserve/fserve.c Adding client to file serving engine
[2014-04-30 10:51:42] INFO fserve/fserve.c checking for file /btv-radio.mp3 (./web/btv-radio.mp3)
[2014-04-30 10:51:42] WARN fserve/fserve.c req for file "./web/btv-radio.mp3" No such file or directory
[2014-04-30 10:51:42] DBUG fserve/fserve.c Adding client to file serving engine
[2014-04-30 10:51:42] DBUG client/client.c Client connection died

[karlH]

I don't see a reference to the icecast version used, or the bitrate of the streams in question. Certain key things could play into this, a larger burst size when multiplied by many new listeners coming in could create a backlog which could stall listeners enough to catch a small queue size. May be the CPU cores are hitting swamped at that time with icecast and other processes occurring like file sharing or DB access?

karl.

[bigstoyan]

Thank you for your answer!

Icecast version is 2.3.2. We are used 128 kbps for all audio streams, and for about 2-3 weeks we changed it to 192 kbps. But that is not a problem. The problem existed at the very beginning. It disconnects regardless of the bitrate.

The server is used only for streaming. There are no other processes, it is not used for filesharing, databases, nothing. It is clean and newly installed

Is there a possibility for having some TCP problem - with number of connections, some buffers, or etc?

[karlH]

128 * 3000+ listeners may be approaching some capacity limit. The "too far behind" messages would show in the log if the queue was too short for the load. If that message does not appear in large numbers then something else is limiting it. I would of thought 2.3.2 would share the 2 streams across the CPUs in your situation, but like I said check that the burst size and queue size are not silly values.

Win32 has had issues with send buffer sizes in the past but for lower latency connections then that should not be an issue. It's possible that the threads what are running your 2 popular streams get shoved onto the same core creating a sort of stall, but the OS decodes that based on the load requirements at the time. Check for any errors getting reported on the NIC or routers, it's not unheard of for certain devices/cables to have issues when certain thresholds are reached creating throttling of the bandwidth (eg 3500 streaming normal ok, 3500 streaming with 1000 getting a burst)

The only software changes are the burst and queue sizes for this or try the latest KH build (with workers set at 2 or more). The log snippet shown is not telling us much.

karl.

[bigstoyan]

We suspect, that we may have CPU problems, because our system monitor software shows, that only one of the CPUs is heavy loaded, and other is not.

We should check that, and we will check other things you wrote, especially for the burst size. That parameter was changed by other guys, which we asked for help, and maybe we have some very wrong in the xml config file.

Have in mind, that we speak only for these 2 heavy loaded streams, but we have 4 more. Total summary of the listeners at the peaks are maybe about 8000 - 9000, and this number will increase soon because of the special advertising campaign which will be running soon.

(Sorry for my not so good English!)

Here it is the vital fragment of the configuration file:


<icecast>
<limits>
<clients>30000</clients>
<sources>30</sources>
<threadpool>10</threadpool>
<queue-size>1024000</queue-size>
<client-timeout>30</client-timeout>
<header-timeout>15</header-timeout>
<source-timeout>30</source-timeout>
<burst-on-connect>1</burst-on-connect>
<burst-size>192000</burst-size>
</limits>

<hostname>46.10.xxx.xxx</hostname>
<port>80</port>
<fileserve>1</fileserve>

<relay>
<server>192.168.108.24</server>
<port>1029</port>
<mount>/njoy.mp3</mount>
<local-mount>/njoy.mp3</local-mount>
<on-demand>0</on-demand>
<listen-socket>
<port>1029</port>
</listen-socket>
<relay-shoutcast-metadata>1</relay-shoutcast-metadata>
</relay>

<relay>
<server>192.168.108.25</server>
<port>1039</port>
<mount>/z-rock.mp3</mount>
<local-mount>/z-rock.mp3</local-mount>
<on-demand>0</on-demand>
<listen-socket>
<port>1039</port>
</listen-socket>
<relay-shoutcast-metadata>1</relay-shoutcast-metadata>
</relay>

<relay>
<server>192.168.108.26</server>
<port>1049</port>
<mount>/melody.mp3</mount>
<local-mount>/melody.mp3</local-mount>
<on-demand>0</on-demand>
<listen-socket>
<port>1049</port>
</listen-socket>
<relay-shoutcast-metadata>1</relay-shoutcast-metadata>
</relay>

<relay>
<server>192.168.108.27</server>
<port>1059</port>
<mount>/classic-fm.mp3</mount>
<local-mount>/classic-fm.mp3</local-mount>
<on-demand>0</on-demand>
<listen-socket>
<port>1059</port>
</listen-socket>
<relay-shoutcast-metadata>1</relay-shoutcast-metadata>
</relay>

<relay>
<server>192.168.108.28</server>
<port>1069</port>
<mount>/btv-radio.mp3</mount>
<local-mount>/btv-radio.mp3</local-mount>
<on-demand>0</on-demand>
<listen-socket>
<port>1069</port>
</listen-socket>
<relay-shoutcast-metadata>1</relay-shoutcast-metadata>
</relay>

<relay>
<server>192.168.108.29</server>
<port>1079</port>
<mount>/jazz-fm.mp3</mount>
<local-mount>/jazz-fm.mp3</local-mount>
<on-demand>0</on-demand>
<listen-socket>
<port>1079</port>
</listen-socket>
<relay-shoutcast-metadata>1</relay-shoutcast-metadata>
</relay>

<relay>
<server>192.168.108.30</server>
<port>1089</port>
<mount>/jazz-fm-lounge.mp3</mount>
<local-mount>/jazz-fm-lounge.mp3</local-mount>
<on-demand>0</on-demand>
<listen-socket>
<port>1089</port>
</listen-socket>
<relay-shoutcast-metadata>1</relay-shoutcast-metadata>
</relay>

<paths>
<logdir>./logs</logdir>
<webroot>./web</webroot>
<adminroot>./admin</adminroot>
<alias source="/" dest="/status.xsl"/>
</paths>
<logging>
<accesslog>access.log</accesslog>
<errorlog>error.log</errorlog>
<logsize>10000</logsize>
<logarchive>1</logarchive>
<loglevel>4</loglevel> <!-- 4 Debug, 3 Info, 2 Warn, 1 Error -->
</logging>
</icecast>

[bigstoyan]

I forget to say:

I have almost 16 000 too far behind messages for 12 hour interval.

Burst size and queue size are very large. Should I change it, and return to default values?

I have no errors for NIC's shown at the Windows Event Viewer. I have no access to the network routers, but I'll ask their admins to check after 3-4 days (here now are 5 days of holidays and nobody's working ))

What values for a burst size and queue size you suggest? If you know that we will have about 10 000 listeners summary, after the holidays (maybe more).

[karlH]

64k is the default burst size, for 128k streams then that will be about 4 seconds worth, you currently have 192k which is a burst of 12 seconds. So you can lessen the burst size which may help in these cases.

If most of the processing is getting pinned onto 1 core then you need to investigate that, it may have something to do with the network bonding. It's quite possible that the OS is keeping all the icecast threads on one core to help with caching and may eventually decide to spread them out if there is a sustained number of listeners on the 2 busy streams, but that decision may not be quick enough handle the sudden increased in load you are seeing. The other 4 streams will just get squeezed in when required. Check into what, if any, options there are for the mapping of CPUs

FYI the listen-socket block inside the relay block does not do anything for icecast.

karl.

[bigstoyan]

Karl, you are a great man! 10x!

I'll try what you suggest. Unfortunately I have to wait 4-5 days, as I said... Because of the holidays here is significantly small number of listeners (about 3500 at the moment for all streams), and I can't register any problem.

I will set the burst size as you tell me above. And we will investigate the "CPU problem". Do you think it is better to use Linux? We can do that, and even we think about separation of the streams. To put the most crowded stream (N-Joy) on new server (much powerful hardware, brand new), and to leave other streams running on the old one.

Thanks for the "listen-socket problem" too... I'll review that, to see why it exist here. Maybe this lines are added from some of the other guys (great sysadmins!) who have access to the server and tried to help. I wonder how they are able to support the streams and the network of the largest Bulgarian TV company (bTV), and at the same time they can't handle this Icecast issue.
But all we are learning till we die - nothing wonders me any more.

Thank you again - I will follow your advices and we'll see... I'll let you know after 5-6 days! Have a nice day!