Icecast Streaming Media Server Forum Index Icecast Streaming Media Server
Icecast is a Xiph Foundation Project
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Listeners dumped at remote DJ takeover and finish
Goto page 1, 2  Next
 
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Icecast Server
View previous topic :: View next topic  
Author Message
square_eyes



Joined: 18 Oct 2011
Posts: 83

PostPosted: Mon Feb 04, 2013 1:23 am    Post subject: Listeners dumped at remote DJ takeover and finish Reply with quote

I have a simple cascading setup of fallbacks for my radio station.

Live, remote DJs were always accidentally getting locked out of their /live mount as the auto DJ (and backup auto DJ) were also configured to use this mount. So I recently started using an /autodj mount and now the /live stream takes priority. This means live DJs should be able to get on air, even if the autodj source clients are in a mess. A new problem has come out of this. Listeners seem to get dumped, or left in limbo at the start and finish of live remote shows, and they don't come back without restarting their listening clients.

Here's my setup (simplified)

:: Listeners connect to /stream (which never has audio). This falls back to /live
:: Live, remote DJs connect to /live, which is setup to fall back to /autodj
:: My auto DJ source client streams to /autodj, which falls back to /fallback.mp3

The process for a changeover involves an audio fade out, then disconnect of the auto DJ, which I'm thinking drops the listeners to /fallback.mp3 for a brief moment. Then the DJ starts streaming to /live, EXISTING listeners are not moved up to /live. NEW or restarted listeners do land in the right place however.

Questions
1. Is it because the listeners briefly dropped from /autodj during the changeover, and that the override of /live only applies to /autodj, and not other levels of fallbacks below it (in this case /fallback.mp3)?
2. If that's the case, at the end of a live show, listeners are presumably dropped from /live, through a (briefly) non existent /autodj to /fallback.mp3. Why then, are those listeners not picked up when the /autodj is connected a moment later?

Note, I have all audio set at 128kbs stereo 44.1 mp3

Help will be much appreciated. I can provide config and logs if needed. But I was hoping that I'm missing something obvious.


Last edited by square_eyes on Thu Feb 14, 2013 12:25 pm; edited 3 times in total
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Mon Feb 04, 2013 5:00 am    Post subject: Reply with quote

assuming you have fallback override set on both /live and /autodj then I suppose it is possible that you have introduced a race case between live and autodj if they are connecting at the same time. It would not explain a client drop or being left in limbo though, you may need expand on that.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
square_eyes



Joined: 18 Oct 2011
Posts: 83

PostPosted: Mon Feb 04, 2013 11:40 am    Post subject: Reply with quote

Ah OK so you only have one fallback override in a chain of fallbacks? That might make sense. I see what you mean by race case, as I swear sometimes the listeners aren't dropped. But I have seen it and experienced it, so I know it's a thing.

To expand, when I was listening to a live DJ takeover, I tuned to /stream, and was listening at /autodj. I heard the music fade out to silence. At that point I must have been on /fallback.mp3 (looping silence file). When the DJ commenced at /live, I never returned to music. Looking at the station stats, neither did any of the other listeners.

I had to re-open the /stream url to hear the live DJ.

Note: I just tested killing the /autodj encoder, and I dropped back to silence, then returned when I reconnected. So I know the /fallback.mp3 works and I'm not dropping off the bottom, so to speak.

I will post the config file in the next post. Sorry it's a bit messy, but it's how my hosting provider has it (I think it's configured by Centova Cast panel anyway).


Last edited by square_eyes on Thu Feb 14, 2013 12:27 pm; edited 1 time in total
Back to top
View user's profile Send private message
square_eyes



Joined: 18 Oct 2011
Posts: 83

PostPosted: Mon Feb 04, 2013 12:21 pm    Post subject: Reply with quote

http://www.icecast.org/docs/icecast-2.2.0/icecast2_config_file.html

fallback-override
When enabled, this allows a connecting source client or relay on this mountpoint to move listening clients back from the fallback mount.

...I thought override was a mount specific setting. Now I'm confused as to how they might be competing for 'dominance'
Back to top
View user's profile Send private message
square_eyes



Joined: 18 Oct 2011
Posts: 83

PostPosted: Tue Feb 05, 2013 12:58 am    Post subject: Reply with quote

Code:
<icecast>
<limits>
<clients>50</clients>
<sources>3</sources>
<threadpool>5</threadpool>
<queue-size>102400</queue-size>
<client-timeout>30</client-timeout>
<header-timeout>15</header-timeout>
<source-timeout>10</source-timeout>
<!-- If enabled, this will provide a burst of data when a client
first connects, thereby significantly reducing the startup
time for listeners that do substantial buffering. However,
it also significantly increases latency between the source
client and listening client. For low-latency setups, you
might want to disable this. -->
<burst-on-connect>1</burst-on-connect>
<!-- same as burst-on-connect, but this allows for being more
specific on how much to burst. Most people won't need to
change from the default 64k. Applies to all mountpoints -->
<burst-size>65535</burst-size>
</limits>

<authentication>
<!-- Sources log in with username 'source' -->
<source-password>XXXX</source-password>
<!-- Relays log in username 'relay' -->
<relay-password>XXXX</relay-password>

<!-- Admin logs in with the username given below -->
<admin-user>admin</admin-user>
<admin-password>XXXX</admin-password>
</authentication>

<!-- This is the hostname other people will use to connect to your server.
It affects mainly the urls generated by Icecast for playlists and yp
listings. -->
<hostname>50.23.115.122</hostname>

<!-- You can use these two if you only want a single listener -->
<port>8496</port>
<bind-address>50.23.115.122</bind-address>

<!-- Only define a <mount> section if you want to use advanced options,
like alternative usernames or passwords -->
<mount cc="main">
<mount-name>/stream</mount-name>
<intro></intro>
<fallback-mount>/live</fallback-mount>
<fallback-override>1</fallback-override>
<fallback-when-full>0</fallback-when-full>
<!-- {%comment-open-if:icecast.suppressmeta=="0"%} --> <!--
<mp3-metadata-interval>0</mp3-metadata-interval>
--> <!-- {%comment-close-if:icecast.suppressmeta=="0"%} -->

<no-yp>1</no-yp>
</mount>

<mount cc="live">
<mount-name>/live</mount-name>

<intro></intro>
<fallback-mount>/autodj</fallback-mount>
<fallback-override>1</fallback-override>
<fallback-when-full>0</fallback-when-full>
<!-- {%comment-open-if:icecast.suppressmeta=="0"%} --> <!--
<mp3-metadata-interval>0</mp3-metadata-interval>
--> <!-- {%comment-close-if:icecast.suppressmeta=="0"%} -->

<no-yp>1</no-yp>
</mount>

<mount cc="autodj">
<mount-name>/autodj</mount-name>
<bitrate>128</bitrate>

<intro></intro>
<fallback-mount>/sounds/fallbackfile.mp3</fallback-mount>
<fallback-override>1</fallback-override>
<fallback-when-full>0</fallback-when-full>
<!-- {%comment-open-if:icecast.suppressmeta=="0"%} --> <!--
<mp3-metadata-interval>0</mp3-metadata-interval>
--> <!-- {%comment-close-if:icecast.suppressmeta=="0"%} -->

<no-yp>1</no-yp>
<charset>UTF-8</charset>
</mount>


Last edited by square_eyes on Thu Feb 07, 2013 6:55 pm; edited 1 time in total
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Tue Feb 05, 2013 1:18 am    Post subject: Reply with quote

again as with all centova setups, 100k is not a large queue size, change that to at least 500k.

hard to know what is going on without the error log at the time of the problem being reported.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
square_eyes



Joined: 18 Oct 2011
Posts: 83

PostPosted: Tue Feb 05, 2013 2:29 am    Post subject: Reply with quote

OK adjusted the queue size to 512000 (although that doesn't appear to have had a noticeable impact)

I might not be able to get the log off the hosting company. I'll try.

I can replicate the issue below consistently.

When tuned to /stream (remote live dj playing to /live, nothing going to /autodj) ...stopping the source client drops the listener completely. So it seems that going from /live to /fallback.mp3 a level below /autodj does not work, even though it's set up to cascade through to an available mount.

Note: When I stream (silence) to /autodj I can disconnect the /live source client and comfortably fall back, and get picked up (as a listener) when /live reconnects. It seems to be the skipping of a level that drops the listener.


Last edited by square_eyes on Tue Feb 05, 2013 2:55 am; edited 2 times in total
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Tue Feb 05, 2013 2:36 am    Post subject: Reply with quote

and this is withe version 2.3.3 ?

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
square_eyes



Joined: 18 Oct 2011
Posts: 83

PostPosted: Tue Feb 05, 2013 2:51 am    Post subject: Reply with quote

Icecast 2.3.2-kh31
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Tue Feb 05, 2013 3:13 am    Post subject: Reply with quote

kh31 is from a while ago, github contains the most recent fixes, 2.3.3-kh6 should be out shortly

karl
Back to top
View user's profile Send private message Send e-mail Visit poster's website
square_eyes



Joined: 18 Oct 2011
Posts: 83

PostPosted: Thu Feb 07, 2013 4:38 pm    Post subject: Reply with quote

I have been talking with my host for a week or so. Haven't really moved past 'Try rebooting the server'.

I have acess to logs via Centovacast. But when I ran a test nothing came through. Config says '4' debugging, but I'm not sure I'm getting all the info.

All this is too rough to put listeners through, I'm loosing a bunch of people at every change-over so I'm going to have set up a test environment with my prod config file.

It's so odd. Each mount and fallback works independently. It's just going back to /autodj from /live, and happens whether I leave silence streaming to /autodj or not during the live show.

FYI I have asked for a version upgrade, but not heard back on that. Are there specific bugs fixed in 2.3.3 that relate to this? I see 2.3.4 beta has a race condition bug fix. Is that related?
Back to top
View user's profile Send private message
square_eyes



Joined: 18 Oct 2011
Posts: 83

PostPosted: Thu Feb 07, 2013 6:53 pm    Post subject: Reply with quote

Here are the results from the test environment. All were successful, without dumping the listener.

Version: Icecast 2.3.2 Windows. All source and listener clients were identical.

I copied the mount settings from my Centova based config file across to my test server, repointed to a white noise fallbackfile.mp3 and removed some comments from the config for ease of reading. These can be seen above if needed.

:: Test 1. Success - Stream auto DJ to /autodj, listen at /stream. Break encoder, hear fallbackfile.mp3 white noise, and back again when reconnected.
:: Test 2. Success - Stream auto DJ to /autodj, listen at /stream. Stream 2nd source to /live (while /auto DJ continues). Listener cuts to /live and back again without dropping.
:: Test 3. Success - Stream auto DJ to /autodj, listen at /stream. Disconnect auto DJ (first this time). Hear fallbackfile.mp3 white noise. Stream 2nd source to /live. Listener cuts to /live and back again, via white noise, then to /autodj after that is reconnected.

I can't think of why it's not working at the hosting centre.

Here's my working test config file if that helps. Now I'll look to get some meaningful logs off my prod server.

Code:
<icecast>
    <limits>
        <sources>5</sources>
   <clients>3</clients>
   <threadpool>5</threadpool>
   <queue-size>512000</queue-size>
   <source-timeout>10</source-timeout>
   <burst-on-connect>1</burst-on-connect>
   <burst-size>64000</burst-size>
    </limits>
    <authentication>
        <source-password>XXXX</source-password>
        <relay-password>XXXX</relay-password>
        <admin-user>admin</admin-user>
        <admin-password>XXXX</admin-password>
    </authentication>
    <hostname>XXXX</hostname>
    <listen-socket>
        <port>8456</port>
    </listen-socket>
    <fileserve>1</fileserve>

<mount cc="main">
<mount-name>/stream</mount-name>
<intro></intro>
<fallback-mount>/live</fallback-mount>
<fallback-override>1</fallback-override>
<fallback-when-full>0</fallback-when-full>
<no-yp>1</no-yp>
</mount>

<mount cc="live">
<mount-name>/live</mount-name>
<intro></intro>
<fallback-mount>/autodj</fallback-mount>
<fallback-override>1</fallback-override>
<fallback-when-full>0</fallback-when-full>
<no-yp>1</no-yp>
</mount>

<mount cc="autodj">
<mount-name>/autodj</mount-name>
<bitrate>128</bitrate>
<intro></intro>
<fallback-mount>/fallback128Stereo_test_noise.mp3</fallback-mount>
<fallback-override>1</fallback-override>
<fallback-when-full>0</fallback-when-full>
<no-yp>1</no-yp>
<charset>UTF-8</charset>
</mount>

    <paths>
        <logdir>./logs</logdir>
        <webroot>XXXXWeb</webroot>
        <adminroot>./admin</adminroot>
        <alias source="/" dest="/status.xsl"/>
    </paths>
    <logging>
        <accesslog>access.log</accesslog>
        <errorlog>error.log</errorlog>
         <loglevel>3</loglevel> <!-- 4 Debug, 3 Info, 2 Warn, 1 Error -->
    </logging>

</icecast>


Last edited by square_eyes on Thu Feb 14, 2013 12:30 pm; edited 1 time in total
Back to top
View user's profile Send private message
square_eyes



Joined: 18 Oct 2011
Posts: 83

PostPosted: Fri Feb 08, 2013 1:10 pm    Post subject: Reply with quote

Here are some logs, around the time I tried a /live test on the production server. Or have I just found myself restarting the server?

Code:
[2013-02-08  06:29:20] DBUG stats/modify_node_event update "global" banned_IPs (0)
[2013-02-08  06:29:20] DBUG stats/modify_node_event update "global" outgoing_kbitrate (0)
[2013-02-08  06:29:21] DBUG stats/modify_node_event update "global" banned_IPs (0)
[2013-02-08  06:29:21] DBUG stats/modify_node_event update "global" outgoing_kbitrate (0)
[2013-02-08  06:29:22] DBUG stats/modify_node_event update "global" banned_IPs (0)
[2013-02-08  06:29:22] DBUG stats/modify_node_event update "global" outgoing_kbitrate (0)
[2013-02-08  06:29:23] INFO connection/wait_for_serversock HUP received, reread scheduled
[2013-02-08  06:29:23] INFO connection/connection_thread connection thread finished
[2013-02-08  06:29:23] INFO event/event_config_read Re-reading XML
[2013-02-08  06:29:23] WARN cfgfile/parse_xml_tags unknown element "threadpool" parsing "limits" at line 5
[2013-02-08  06:29:23] WARN cfgfile/parse_xml_tags unknown element "burst-on-connect" parsing "limits" at line 16
[2013-02-08  06:29:23] WARN /home/centovacast/vhosts/budgiecollective/etc/server.conf warning:
[2013-02-08  06:29:23] WARN /home/centovacast/vhosts/budgiecollective/etc/server.conf skipping element "intro" parsing "mount" at line 99
[2013-02-08  06:29:23] WARN cfgfile/parse_xml_tags unknown element "no-yp" parsing "mount" at line 107
[2013-02-08  06:29:23] WARN /home/centovacast/vhosts/budgiecollective/etc/server.conf warning:
[2013-02-08  06:29:23] WARN /home/centovacast/vhosts/budgiecollective/etc/server.conf skipping element "intro" parsing "mount" at line 113
[2013-02-08  06:29:23] WARN cfgfile/parse_xml_tags unknown element "no-yp" parsing "mount" at line 121
[2013-02-08  06:29:23] WARN /home/centovacast/vhosts/budgiecollective/etc/server.conf warning:
[2013-02-08  06:29:23] WARN /home/centovacast/vhosts/budgiecollective/etc/server.conf skipping element "intro" parsing "mount" at line 128
[2013-02-08  06:29:23] WARN cfgfile/parse_xml_tags unknown element "no-yp" parsing "mount" at line 136
[2013-02-08  06:29:23] INFO thread/ lock abort set to 0
[2013-02-08  06:29:23] INFO connection/connection_thread_shutdown shutting down connection thread
[2013-02-08  06:29:23] DBUG yp/yp_recheck_config Updating YP configuration
[2013-02-08  06:29:23] DBUG stats/modify_node_event update "global" server_id (Icecast 2.3.2-kh31)
[2013-02-08  06:29:23] DBUG stats/modify_node_event update "global" host (50.23.115.122)
[2013-02-08  06:29:23] DBUG stats/modify_node_event update "global" location (Earth)
[2013-02-08  06:29:23] DBUG stats/modify_node_event update "global" admin (icemaster@localhost)
[2013-02-08  06:29:23] INFO client/workers_adjust requested worker count 1
[2013-02-08  06:29:23] DBUG stats/modify_node_event update "global" banned_IPs (0)
[2013-02-08  06:29:23] INFO connection/connection_thread connection thread started
[2013-02-08  06:29:23] DBUG stats/modify_node_event update "global" outgoing_kbitrate (0)
[2013-02-08  06:29:23] INFO connection/get_ssl_certificate No SSL capability on any configured ports
[2013-02-08  06:29:23] EROR connection/connection_setup_sockets Could not create listener socket on port 8496 bind 50.23.115.122
[2013-02-08  06:29:23] EROR connection/connection_setup_sockets No listening sockets established
[2013-02-08  06:29:24] DBUG stats/modify_node_event update "global" banned_IPs (0)
[2013-02-08  06:29:24] DBUG stats/modify_node_event update "global" outgoing_kbitrate (0)
[2013-02-08  06:29:25] DBUG stats/modify_node_event update "global" banned_IPs (0)
[2013-02-08  06:29:25] DBUG stats/modify_node_event update "global" outgoing_kbitrate (0)
[2013-02-08  06:29:26] DBUG stats/modify_node_event update "global" banned_IPs (0)
[2013-02-08  06:29:26] DBUG stats/modify_node_event update "global" outgoing_kbitrate (0)
[2013-02-08  06:29:27] DBUG stats/modify_node_event update "global" banned_IPs (0)
[2013-02-08  06:29:27] DBUG stats/modify_node_event update "global" outgoing_kbitrate (0)
[2013-02-08  06:29:28] DBUG connection/wait_for_serversock signalfd received a termination
[2013-02-08  06:29:28] INFO connection/connection_thread connection thread finished
[2013-02-08  06:29:28] INFO connection/connection_thread_shutdown shutting down connection thread
[2013-02-08  06:29:28] INFO slave/_slave_thread shutting down current relays
[2013-02-08  06:29:28] INFO slave/_slave_thread Slave thread shutdown complete
[2013-02-08  06:29:28] DBUG yp/yp_stop YP client is now stopped
[2013-02-08  06:29:28] DBUG client/worker 0x7a2160 now has 0 clients
[2013-02-08  06:29:28] INFO client/workers_adjust requested worker count 0
[2013-02-08  06:29:28] INFO client/worker shutting down
[2013-02-08  06:29:28] INFO main/server_process Shutting down
[2013-02-08  06:29:28] INFO auth/auth_shutdown Auth shutdown complete
[2013-02-08  06:29:28] DBUG yp/yp_shutdown releasing directory details
[2013-02-08  06:29:28] DBUG yp/destroy_yp_server Removing YP server entry for http://dir.xiph.org/cgi-bin/yp-cgi
[2013-02-08  06:29:28] INFO yp/yp_shutdown YP cleanup complete
[2013-02-08  06:29:28] INFO fserve/fserve_shutdown file serving stopped
[2013-02-08  06:29:31] INFO thread/ lock abort set to 0
[2013-02-08  06:29:31] INFO main/server_process Icecast 2.3.2-kh31 server started
[2013-02-08  06:29:31] WARN slave/slave_startup process has 16384 max file descriptor limit
[2013-02-08  06:29:31] INFO client/workers_adjust requested worker count 1
[2013-02-08  06:29:31] DBUG yp/yp_recheck_config Updating YP configuration
[2013-02-08  06:29:31] DBUG client/worker 0x10f5160 now has 0 clients
[2013-02-08  06:29:31] INFO yp/yp_recheck_config Adding new YP server "http://dir.xiph.org/cgi-bin/yp-cgi" (timeout 15s, default interval 30s)
[2013-02-08  06:29:31] INFO yp/yp_client_add Starting Directory client for YP processing
[2013-02-08  06:29:31] DBUG client/worker_add_pending_clients Added 1 pending clients to 0x10f5160
[2013-02-08  06:29:31] DBUG stats/modify_node_event update "global" banned_IPs (0)
[2013-02-08  06:29:31] DBUG stats/modify_node_event update "global" outgoing_kbitrate (0)
[2013-02-08  06:29:31] DBUG client/worker 0x10f5160 now has 1 clients
[2013-02-08  06:29:31] INFO connection/connection_thread connection thread started
[2013-02-08  06:29:31] DBUG yp/check_servers Add pending yps http://dir.xiph.org/cgi-bin/yp-cgi
[2013-02-08  06:29:31] INFO connection/get_ssl_certificate No SSL capability on any configured ports
[2013-02-08  06:29:31] EROR connection/connection_setup_sockets Could not create listener socket on port 8496 bind 50.23.115.122
[2013-02-08  06:29:31] EROR connection/connection_setup_sockets No listening sockets established
[2013-02-08  06:29:32] DBUG stats/modify_node_event update "global" banned_IPs (0)
[2013-02-08  06:29:32] DBUG stats/modify_node_event update "global" outgoing_kbitrate (0)
[2013-02-08  06:29:33] DBUG stats/modify_node_event update "global" banned_IPs (0)
[2013-02-08  06:29:33] DBUG stats/modify_node_event update "global" outgoing_kbitrate (0)
[2013-02-08  06:29:34] DBUG stats/modify_node_event update "global" banned_IPs (0)
[2013-02-08  06:29:34] DBUG stats/modify_node_event update "global" outgoing_kbitrate (0)
[2013-02-08  06:29:35] DBUG stats/modify_node_event update "global" banned_IPs (0)
[2013-02-08  06:29:35] DBUG stats/modify_node_event update "global" outgoing_kbitrate (0)
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Fri Feb 08, 2013 8:34 pm    Post subject: Reply with quote

just looks to be reloaded a few times and restarted once.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
square_eyes



Joined: 18 Oct 2011
Posts: 83

PostPosted: Mon Feb 11, 2013 11:27 am    Post subject: Reply with quote

OK, it looks like the hosting company has agreed to look at the icecast version. Will update when I hear more.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Icecast Server All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2002 phpBB Group
subRebel style by ktauber