Icecast Streaming Media Server

[Anonymous]

Hello everyone, this is my first post here. I recently have been asked by someone I'm working with to fix a problem he has been having with icecast. Apparantly in the past he has run streams with several hundred connected users, but recently he says that whenever it gets to around 50-60 people the stream will die and everyone will lose their connection.

I noticed this error in the log (posted below) and was wondering if anyone here could tell me what might be causing this. The critical section seems to be this:

[2006-02-20 00:06:02] DBUG source/get_next_buffer last 1140415551, timeout 10, now 1140415562
[2006-02-20 00:06:02] WARN source/get_next_buffer Disconnecting source due to socket timeout

after that everyone is disconnected as you can see.

Do you think this is something wrong with the configuration, or is it something like a memory/bandwidth resource limit, a bug, or something else? Thanks for any insight.

edit: also btw this was icecast 2.2.0. I have since upgraded to 2.3.1 but have not tested it with that version yet.

---BEGIN LOG----

[2006-02-20 00:01:39] DBUG stats/modify_node_event update node clients (61)
[2006-02-20 00:01:39] DBUG stats/modify_node_event update node listeners (61)
[2006-02-20 00:01:46] DBUG connection/_handle_get_request Source found for client
[2006-02-20 00:01:46] DBUG source/source_main Client added
[2006-02-20 00:01:46] INFO source/source_main listener count on /live now 62
[2006-02-20 00:01:46] DBUG stats/modify_node_event update node connections (1622)
[2006-02-20 00:01:46] DBUG stats/modify_node_event update node client_connections (1604)
[2006-02-20 00:01:46] DBUG stats/modify_node_event update node clients (62)
[2006-02-20 00:01:46] DBUG stats/modify_node_event update node listeners (62)
[2006-02-20 00:01:52] DBUG admin/admin_handle_request Admin request (/admin/stats.xml)
[2006-02-20 00:01:52] DBUG admin/admin_handle_request Got command (stats.xml)
[2006-02-20 00:01:52] DBUG admin/command_stats Stats request, sending xml stats
[2006-02-20 00:01:52] DBUG stats/modify_node_event update node connections (1623)
[2006-02-20 00:01:52] DBUG stats/modify_node_event update node client_connections (1605)
[2006-02-20 00:01:52] DBUG client/client_send_bytes Client connection died
[2006-02-20 00:01:52] DBUG source/source_main Client removed
[2006-02-20 00:01:52] INFO source/source_main listener count on /live now 61
[2006-02-20 00:01:52] DBUG stats/modify_node_event update node clients (61)
[2006-02-20 00:01:52] DBUG stats/modify_node_event update node listeners (61)
[2006-02-20 00:02:07] DBUG source/send_to_listener Client has fallen too far behind, removing
[2006-02-20 00:02:07] DBUG source/source_main Client removed
[2006-02-20 00:02:07] INFO source/source_main listener count on /live now 60
[2006-02-20 00:02:07] DBUG stats/modify_node_event update node clients (60)
[2006-02-20 00:02:07] DBUG stats/modify_node_event update node listeners (60)
[2006-02-20 00:02:27] DBUG slave/_slave_thread checking master stream list
[2006-02-20 00:02:50] DBUG stats/modify_node_event update node connections (1624)
[2006-02-20 00:02:53] DBUG connection/_handle_get_request Source found for client
[2006-02-20 00:02:53] DBUG client/client_send_bytes Client connection died
[2006-02-20 00:02:53] DBUG source/source_main Client removed
[2006-02-20 00:02:53] DBUG source/source_main Client added
[2006-02-20 00:02:53] DBUG stats/modify_node_event update node client_connections (1606)
[2006-02-20 00:02:53] DBUG stats/modify_node_event update node clients (59)
[2006-02-20 00:02:53] DBUG stats/modify_node_event update node clients (60)
[2006-02-20 00:02:56] DBUG admin/admin_handle_request Admin request (/admin/stats.xml)
[2006-02-20 00:02:56] DBUG admin/admin_handle_request Got command (stats.xml)
[2006-02-20 00:02:56] DBUG admin/command_stats Stats request, sending xml stats
[2006-02-20 00:02:56] DBUG stats/modify_node_event update node connections (1625)
[2006-02-20 00:02:56] DBUG stats/modify_node_event update node client_connections (1607)
[2006-02-20 00:03:10] DBUG source/send_to_listener Client has fallen too far behind, removing
[2006-02-20 00:03:10] DBUG source/source_main Client removed
[2006-02-20 00:03:10] INFO source/source_main listener count on /live now 59
[2006-02-20 00:03:11] DBUG stats/modify_node_event update node clients (59)
[2006-02-20 00:03:11] DBUG stats/modify_node_event update node listeners (59)
[2006-02-20 00:03:32] DBUG source/send_to_listener Client has fallen too far behind, removing
[2006-02-20 00:03:32] DBUG source/source_main Client removed
[2006-02-20 00:03:32] INFO source/source_main listener count on /live now 58
[2006-02-20 00:03:32] DBUG stats/modify_node_event update node clients (58)
[2006-02-20 00:03:32] DBUG stats/modify_node_event update node listeners (58)
[2006-02-20 00:03:53] DBUG admin/admin_handle_request Admin request (/admin/stats.xml)
[2006-02-20 00:03:53] DBUG admin/admin_handle_request Got command (stats.xml)
[2006-02-20 00:03:53] DBUG admin/command_stats Stats request, sending xml stats
[2006-02-20 00:03:53] DBUG stats/modify_node_event update node connections (1626)
[2006-02-20 00:03:53] DBUG stats/modify_node_event update node client_connections (1608)
[2006-02-20 00:04:16] DBUG connection/_handle_get_request Source found for client
[2006-02-20 00:04:16] DBUG stats/modify_node_event update node connections (1627)
[2006-02-20 00:04:16] DBUG stats/modify_node_event update node client_connections (1609)
[2006-02-20 00:04:16] DBUG source/source_main Client added
[2006-02-20 00:04:16] INFO source/source_main listener count on /live now 59
[2006-02-20 00:04:16] DBUG stats/modify_node_event update node clients (59)
[2006-02-20 00:04:16] DBUG stats/modify_node_event update node listeners (59)
[2006-02-20 00:04:28] DBUG slave/_slave_thread checking master stream list
[2006-02-20 00:04:30] DBUG source/send_to_listener Client has fallen too far behind, removing
[2006-02-20 00:04:30] DBUG source/source_main Client removed
[2006-02-20 00:04:30] INFO source/source_main listener count on /live now 58
[2006-02-20 00:04:30] DBUG stats/modify_node_event update node clients (58)
[2006-02-20 00:04:30] DBUG stats/modify_node_event update node listeners (58)
[2006-02-20 00:04:47] DBUG connection/_handle_get_request Source found for client
[2006-02-20 00:04:47] DBUG source/source_main Client added
[2006-02-20 00:04:47] INFO source/source_main listener count on /live now 59
[2006-02-20 00:04:47] DBUG stats/modify_node_event update node connections (1628)
[2006-02-20 00:04:47] DBUG stats/modify_node_event update node client_connections (1610)
[2006-02-20 00:04:47] DBUG stats/modify_node_event update node clients (59)
[2006-02-20 00:04:47] DBUG stats/modify_node_event update node listeners (59)
[2006-02-20 00:04:53] DBUG admin/admin_handle_request Admin request (/admin/stats.xml)
[2006-02-20 00:04:53] DBUG admin/admin_handle_request Got command (stats.xml)
[2006-02-20 00:04:53] DBUG admin/command_stats Stats request, sending xml stats
[2006-02-20 00:04:53] DBUG stats/modify_node_event update node connections (1629)
[2006-02-20 00:04:53] DBUG stats/modify_node_event update node client_connections (1611)
[2006-02-20 00:05:08] DBUG source/send_to_listener Client has fallen too far behind, removing
[2006-02-20 00:05:08] DBUG source/source_main Client removed
[2006-02-20 00:05:08] INFO source/source_main listener count on /live now 58
[2006-02-20 00:05:08] DBUG stats/modify_node_event update node clients (58)
[2006-02-20 00:05:08] DBUG stats/modify_node_event update node listeners (58)
[2006-02-20 00:05:37] DBUG source/send_to_listener Client has fallen too far behind, removing
[2006-02-20 00:05:37] DBUG source/source_main Client removed
[2006-02-20 00:05:37] INFO source/source_main listener count on /live now 57
[2006-02-20 00:05:37] DBUG stats/modify_node_event update node clients (57)
[2006-02-20 00:05:37] DBUG stats/modify_node_event update node listeners (57)
[2006-02-20 00:05:54] DBUG admin/admin_handle_request Admin request (/admin/stats.xml)
[2006-02-20 00:05:54] DBUG admin/admin_handle_request Got command (stats.xml)
[2006-02-20 00:05:54] DBUG admin/command_stats Stats request, sending xml stats
[2006-02-20 00:05:54] DBUG stats/modify_node_event update node connections (1630)
[2006-02-20 00:05:54] DBUG stats/modify_node_event update node client_connections (1612)
[2006-02-20 00:06:02] DBUG source/get_next_buffer last 1140415551, timeout 10, now 1140415562
[2006-02-20 00:06:02] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2006-02-20 00:06:02] INFO source/source_shutdown Source "/live" exiting
[2006-02-20 00:06:02] DBUG source/source_clear_source clearing source "/live"
[2006-02-20 00:06:02] DBUG source/source_free_source freeing source "/live"
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node sources (0)
[2006-02-20 00:06:02] DBUG stats/process_source_event delete source node /live
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (56)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (55)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (54)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (53)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (52)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (51)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (50)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (49)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (48)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (47)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (46)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (45)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (44)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (43)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (42)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (41)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (40)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (39)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (38)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (37)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (36)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (35)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (34)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (33)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (32)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (31)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (30)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (29)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (28)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (27)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (26)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (25)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (24)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (23)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (22)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (21)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (20)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (19)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (18)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (18)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (17)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (16)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (15)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (14)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (13)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (12)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (11)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (10)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (9)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (8)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (7)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (6)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (5)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (4)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (3)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (2)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (1)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node clients (0)
[2006-02-20 00:06:02] DBUG connection/_handle_get_request Source not found for client
[2006-02-20 00:06:02] DBUG connection/_handle_get_request Source not found for client
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node connections (1631)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node client_connections (1613)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node connections (1632)
[2006-02-20 00:06:02] DBUG stats/modify_node_event update node client_connections (1614)
[2006-02-20 00:06:03] DBUG connection/_handle_get_request Source not found for client
[2006-02-20 00:06:04] DBUG stats/modify_node_event update node connections (1633)
[2006-02-20 00:06:04] DBUG stats/modify_node_event update node client_connections (1615)
[2006-02-20 00:06:05] DBUG connection/_handle_get_request Source not found for client
[2006-02-20 00:06:05] DBUG stats/modify_node_event update node connections (1634)
[2006-02-20 00:06:05] DBUG stats/modify_node_event update node client_connections (1616)
[2006-02-20 00:06:05] DBUG connection/_handle_get_request Source not found for client
[2006-02-20 00:06:05] DBUG stats/modify_node_event update node connections (1635)
[2006-02-20 00:06:05] DBUG stats/modify_node_event update node client_connections (1617)
[2006-02-20 00:06:06] DBUG connection/_handle_get_request Source not found for client
[2006-02-20 00:06:06] DBUG stats/modify_node_event update node connections (1636)
[2006-02-20 00:06:06] DBUG stats/modify_node_event update node client_connections (1618)
[2006-02-20 00:06:06] DBUG connection/_handle_get_request Source not found for client
[2006-02-20 00:06:06] DBUG stats/modify_node_event update node connections (1637)
[2006-02-20 00:06:06] DBUG stats/modify_node_event update node client_connections (1619)
[2006-02-20 00:06:07] DBUG connection/_handle_get_request Source not found for client
[2006-02-20 00:06:07] DBUG connection/_handle_get_request Source not found for client
[2006-02-20 00:06:07] DBUG stats/modify_node_event update node connections (1638)
[2006-02-20 00:06:07] DBUG stats/modify_node_event update node connections (1639)
[2006-02-20 00:06:07] DBUG stats/modify_node_event update node client_connections (1620)
[2006-02-20 00:06:07] DBUG stats/modify_node_event update node client_connections (1621)
[2006-02-20 00:06:07] DBUG connection/_handle_get_request Source not found for client

[karlH]

with a message like

2006-02-20 00:06:02] DBUG source/get_next_buffer last 1140415551, timeout 10, now 1140415562

this indicates that nothing has come in from the source client/relay for 10 seconds, so icecast is shuuting down the stream, it's assuming that the source client/relay is having some problem. source-timeout controls the trigger.

As to the cause of the problem, that's hard to say as icecast is just reacting to a situation. A maxmum bandwidth case would probably cause player skipping for your listener first but it's not impossible for a busy period to starve the link for that amount of time. Another possibility is something is occurring on the network which effectively halts the connection eg IP changes.

karl.

[Anonymous]

Thanks for the help, I will try increasing source-timeout.

Also I'm fairly sure it is not a max bandwidth problem because I noticed in the log that it happens occasionally with only 1-2 connected listeners.

[karlH]

increasing the source timeout is unlikely to help, if nothing has been sent in the last 10 seconds then the listeners are going to have problems.

karl.