Icecast Streaming Media Server Forum Index Icecast Streaming Media Server
Icecast is a Xiph Foundation Project
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Icecast is disconnecting listeners after one hour
Goto page 1, 2  Next
 
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Icecast Server
View previous topic :: View next topic  
Author Message
PlayRadio



Joined: 12 Mar 2009
Posts: 9
Location: Romania

PostPosted: Tue Nov 17, 2009 6:37 pm    Post subject: Icecast is disconnecting listeners after one hour Reply with quote

Hello.
We've been using Icecast for some time now and it's the first time we have a serious problem.
We moved our hosting to another provider. They gave us a VPS with enough resources to support over 500 listeners.

The OS we installed is Slackware 13.0 64bit edition. We run Icecast 2.3.2 with an 128 kbps stream. Everything runs well but after one hour of listening, every individual user gets disconnected just like it pressed stop button. It is not a buffer problem. That happents even with only 2 listeners on the server. It is not a bandwith problem because we have 100 mbit connection.
We thought to increase the queue-size from 102400 to 512000. The connection time increases to almost 2 hours. After that every user gets disconnected each at the exact same time.

So we ask you, why is this happening because it is very embarrasing to increase the queue-size just to increase the listening time. We want every person to listen to our station as much as they like... even if that time is one week without intrerruption.

Could this problem come from the 64bit OS? Should we switch to 32bit slackware? We also have a server with Slackware 12.2 32bit edition and we have no problems on it with a similar 128 kbps stream.

Thank you.
_________________
The best dance hits of the 90's only @ Play Radio
Back to top
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Tue Nov 17, 2009 7:19 pm    Post subject: Reply with quote

If the problem was down to queue length then you would see the "too far behind" messages in the log. Seeing that you have not reported that happening then it will be down to something else. It sounds like you have <max-listener-duration> set but that would sound too obvious.

Just to be clear, this only applies to the listeners, not source clients? and this forced disconnection occurs to the second but nothing is logged to indicate why?

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
PlayRadio



Joined: 12 Mar 2009
Posts: 9
Location: Romania

PostPosted: Tue Nov 17, 2009 9:16 pm    Post subject: Reply with quote

Well, I do get a "source/send_to_listener Client *** has fallen too far behind, removing" error in the log directory. This disconnection applies only to the listeners and occurs to each listener at the absolute 60th minute of listening. It's very odd.
Like I said before, we use the same provider on Slackware 12.2 32bit OS @ 128 kbps stream and we have no problems with that.

This problem only occures on a VPS with Slackware 13.0 64bit OS @ 128 kbps. Same conf parameters as the one above.
_________________
The best dance hits of the 90's only @ Play Radio
Back to top
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Tue Nov 17, 2009 9:34 pm    Post subject: Reply with quote

If listener 1 connects and then 30 mins later, listener 2 connects, will both listeners be dropped at the same time or will listener 1 go first then 30 mins later listener 2.

If there is some starvation (eg VPS scheduling or network stall) then that could be enough to prevent the writing and push the listeners to the end of the queue for dropping (too far behind etc).

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
PlayRadio



Joined: 12 Mar 2009
Posts: 9
Location: Romania

PostPosted: Wed Nov 18, 2009 5:26 am    Post subject: Reply with quote

karlH wrote:
If listener 1 connects and then 30 mins later, listener 2 connects, will both listeners be dropped at the same time or will listener 1 go first then 30 mins later listener 2.


The listeners are not dropped at the same time. Each listener is disconnected when the listening time reaches exactly one hour. If I increase the queue-size from 102400 to 512000 the listening time increases to about 2 hours. Right now the queue-size is set to 102400000000000000000000000000000000000 and there are no problems after 3 hours of listening. That's the thing I can't understand and it's kind of embarrasing. I mean... if a listener wants to stay connected to our station for one week without disconnection, how much would the queue-size should be? Is this normal?
Is there a solution to this problem?
_________________
The best dance hits of the 90's only @ Play Radio
Back to top
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Wed Nov 18, 2009 4:39 pm    Post subject: Reply with quote

the queue size has nothing to do with a timed duration for listener timeout. The only way the 'too far behind' check would tie into playback duration is if the stream settings did not match the expected feed, eg expect to encode from DSP at 48000hz but only get 44100hz from DSP, in such cases the player would only read from the connection to playback 44100 and over time icecast would delay the writing until eventually getting to the end of the queue.

A simple test for that would be to see if curl to /dev/null gets terminated.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
PlayRadio



Joined: 12 Mar 2009
Posts: 9
Location: Romania

PostPosted: Wed Nov 18, 2009 8:40 pm    Post subject: Reply with quote

karlH wrote:
the queue size has nothing to do with a timed duration for listener timeout.


It seems that in this case queue size has a lot to do with the timed duration. I added lots of zero's in the end of queue-size of the icecast.xml file.
We had lots of users that listened over 5 hours and had no disconnection. After that I decreased the queue-size to 102400 and after exactly one hour users got disconnected again.

The strange thing is that in the error log I cannot find my IP with the the end of the queue error. There are maximum 5 IP's showing the end of queue error. We had more than that on the stream. So it seems like this disconnection event comes from something alse. Come to think of it, why increaseing the queue time increases the listening time, and when I get disconnected I can't find my IP in the error log at all?

Isn't that odd?
_________________
The best dance hits of the 90's only @ Play Radio
Back to top
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Wed Nov 18, 2009 8:50 pm    Post subject: Reply with quote

The code for dropping a listener based on the time is very different from the code dropping a listener based on the end of queue. Look at source.c: send_to_listener, the first logs as "time limit reached for client..." then second logs at the end of the function.

try again with curl reading off a 100k queue and see if the drop occurs at 60 mins. You can match the time in the access log to the messages in the error log.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
PlayRadio



Joined: 12 Mar 2009
Posts: 9
Location: Romania

PostPosted: Fri Nov 20, 2009 7:47 am    Post subject: Reply with quote

We can't solve this problem, so we migrated this server to Shoutcast. We don't have any of these problems on their platform.

Thanks anyway.
_________________
The best dance hits of the 90's only @ Play Radio
Back to top
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Fri Nov 20, 2009 8:46 pm    Post subject: Reply with quote

I did offer to track down the issue, and even made suggestions on proceeding but if you don't want to proceed on this then so be it. I've not had any other reports like this so if anyone else is then get in touch.

karl
Back to top
View user's profile Send private message Send e-mail Visit poster's website
SideSwipe



Joined: 24 Feb 2010
Posts: 3
Location: Northern Ireland

PostPosted: Wed Feb 24, 2010 3:03 pm    Post subject: Reply with quote

Sorry for raising this old thread from the graveyard, but it is remarkably similar to one that I'm suffering from.

Currently, I've got the following set up. Audio passes from a mixing desk to a volume normaliser. After the normaliser it passes into an encoding machine.

The encoder runs Darkice 0.19, sitting ontop of Slackware 10.1 (I realise this is an old version, the encoder is due for an upgrade shortly), and encodes the audio into three streams, 128Kbps, 96Kbps and 24Kbps, all in MP3. The encoder then hands off to our Icecast server on a different machine. All of our data passes over a 100Mb connection to the Icecast server.

The Icecast server itself is running on 64bit CentOS 5.3, kernel 2.6.18. It has bandwidth in excess of 100Mbit. I would like to give you a more exact figure on that one, but the person who provides the Icecast server for ourselves cannot currently remember what the capactity of the switch it's currently connected to. It is very possible it is 1Gbit however.

The issue is that after a set amount of time, which varies on each of the different streams (1.5 hours @ 128kbps, ~3 hours @ 96kpbs, unknown but high @ 24kbps), the listener will be disconnected. It doesn't appear to matter if there is one listener, or thirty. After they have been connected for one of the times that matches the stream they are on, they will be disconnected.

When the disconnect occurs, the following is written into the error log:
"INFO source/send_to_listener Client xx (xxx.xxx.xxx.xxx) has fallen too far behind, removing"

Adjusting the queue-size will effect how long a listener can stay connected to. The higher the queue-size, the longer they can stay connected.

I'm not sure where to proceed from here, and would greatly appreciate any assistance that can be given. If you need any further information, just ask Smile
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Wed Feb 24, 2010 3:59 pm    Post subject: Reply with quote

Whenever you get the "fallen too far behind" message, what you are seeing is that the player end is not reading as much stream data as the source end is sending. Sometimes could be because of a network interruption but it could also be down to network restrictions (congestion or throttling) or even the player not reading quickly enough.

As players will regulate the stream being read (for playback) you can instead test the link side of things by just downloading the stream to a file. A normal downloading app (eg curl or wget or any other) will just read as quick as possible, and if that connection is dropped for being too far behind then you know it will be down to the network slowing you down. This is most likely the cause and if so check your network settings for errors, run other tests, like a http/ftp transfer and time it from the same machines.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
SideSwipe



Joined: 24 Feb 2010
Posts: 3
Location: Northern Ireland

PostPosted: Thu Feb 25, 2010 2:25 pm    Post subject: Reply with quote

Thanks for that.

I've used wget to download the mountpoint that drops the users the earliest for several hours now (well past when it normally drops) and it's still downloading strong on several different computers. This includes computers not connected to the same network as the Icecast and Darkice servers. The same computers I've been using wget on will drop at the times I stated in my previous post when using Winamp, Windows Media Player, iTunes, and a custom Flash player.

I'm somewhat at a loss again as to what is causing this issue.
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Thu Feb 25, 2010 3:03 pm    Post subject: Reply with quote

ok, so that means the throughput of stream data on the network is not a factor. This issue will be that the player reading that data will be proceeding at a different rate that the encoder placing content on the network.

If this affects all players no matter where they are located then I suspect the timing that darkice is using is out of whack. If it is running fast then more data will be pushed on. One possibility (assuming dsp input is used) is that the soundcard clock is fixed at 48000 and not an expected 44100. This would mean more data is sent per second which will translate to a few kbytes/hr extra on the queue. I don't know what your tolerances are defaults are 512k queue and 64k burst but some use larger bursts and/or smaller queues

So I would check the darkice messages and see what bandwidth the source client is sending on each stream (iptraf might be useful to use on that).

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
SideSwipe



Joined: 24 Feb 2010
Posts: 3
Location: Northern Ireland

PostPosted: Thu Feb 25, 2010 3:26 pm    Post subject: Reply with quote

Interesting!

Currently DarkIce is set to use jackd to get input from the soundcard on the encoder. According to the jackd log file and the DarkIce configuration file, both are using 44100Hz. The DarkIce log file is currently very small, and the contents are below. Jackd is using alsa to access the sound card.

I'll investigate further on the encoder, and attempt to get in contact with the person who set it up initially. Thanks for all your help so far! Smile

Code:
DarkIce 0.19 live audio streamer, http://darkice.tyrell.hu/
Copyright (c) 2000-2007, Tyrell Hungary, http://tyrell.hu/

Using config file: /root/darkice.cfg
Using JACK audio server as input device.
Using POSIX real-time scheduling, priority 98
Registering as JACK client darkice-13196
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Icecast Server All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2002 phpBB Group
subRebel style by ktauber