View previous topic :: View next topic |
Author |
Message |
shards
Joined: 07 May 2014 Posts: 3
|
Posted: Wed May 07, 2014 1:13 pm Post subject: Running multiple streams of ~320kbit with few listeners each |
|
|
We're a few thinking about creating some personalized streams, customizable via a webpage which then causes a liquidsoap script to run that outputs to icecast.
This means that there will be alot of streams with maybe only 1 listener each.
I have created a small proof of concept on a linux box my local network but i seem to run into alot of buffering after creating around 20-25 streams running side by side each defined by their own liquidsoap script and then outputted with a unique stream name to port 8000 on icecast, the streams are fairly simple with a 5 song m3u of 320 kbit mp3s with a small spot occasionally playing atop the music with a fadein/out.
My config on the test environment is set to allow 200 sources, 200 clients and 200 threads in threadpool.
Is it feasible to run that many icecast streams side by side (and just a matter of hardware?)
Is it a bad idea overall and should i rather have few streams with many listeners and avoid the customization options that would spawn many concurrent streams?
Any info or similar projects to go further from here perhaps?
(can provide sample script if needed) |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Wed May 07, 2014 11:25 pm Post subject: |
|
|
The main trunk releases can be poor on this as each stream takes one thread and therefore many streams leads to heavy switching, low numbers of threads won't be a problem but 200+ can get into problems. The threadpool setting will not make any difference as it's deprecated.
The KH tree allows for better scaling with this sort of load, as you can specify a <workers> setting in the <limits> to fix the number of threads to process clients, this way you can have thousands of sources (yes that does exist). Note that clients includes listeners, sources and file requests.
karl. |
|
Back to top |
|
|
shards
Joined: 07 May 2014 Posts: 3
|
Posted: Fri May 09, 2014 9:41 am Post subject: |
|
|
Thanks for your reply karlH.
I will try the KH edition.
On the regular version I seem to run into buffering already at around 30 streams on my virtual box with "4" cpus, however i guess it's not nescessarily icecast causing this as im using liquidsoap to output to icecast?
I'm not entirely sure where liquidsoap's responsibility ends and icecast2's starts since i'm kinda new to this area, however im using liquidsoap to output to it, so my performance could also be caused by whatever processing liquidsoap does. Any other software i should be looking at for attaching sources to icecast (or can i do it without and still be able to perform somewhat of the same in terms of spots/volume control etc.)?
Ideally i'd be able to run atleast 100 streams on such a setup, since it otherwise starts getting expensive hosting wise for a box with more cpus. |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Fri May 09, 2014 10:26 am Post subject: |
|
|
a couple of points to note. virtual box, while useful, is a virtual machine so does have an impact on the scheduling of tasks, liquidsoap deals with the encoding of the stream (which can include any processing of the audio before encoding) so can be quite cpu intensive if you have many of them (the cpu load should help determine in that regard). liquidsoap, like any source client, sends the encoded stream to icecast via the TCP link so it is not required to be on the same machine as icecast so can help in spreading the encoding load if required.
karl. |
|
Back to top |
|
|
shards
Joined: 07 May 2014 Posts: 3
|
Posted: Fri May 09, 2014 10:45 am Post subject: |
|
|
Ok yeah, any source clients similar to liquidsoap that may be a bit lighter on the cpu (or is it negligible since they have to perform more or less the same task)? |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Fri May 09, 2014 12:10 pm Post subject: |
|
|
encoding will be the same, no matter what the source client (not sure if there is much difference with different encoder libraries these days), but if there is other work being done like resampling then that will add to it.
karl. |
|
Back to top |
|
|
|