Icecast Streaming Media Server

[Anonymous]

Hi,

I'm using Ices 0.4 with Icecast and using my own playlist generator to play MP3s. Ices reencode function is enabled and set to 96kbps 44kHz Stereo.

The server I use is 2x64bit @ 2.5GHz , 1Gb RAM, RAID1 SATA 230Mb

Everything looks working fine until some tasks like copying files or tar or gzip occurs. It looks like that disk I/O operations makes ices to start lagging or even restart, as it does not get enough I/O resources (The CPU usage is fine and is under 25% by that time taking in mind all processes).

This makes streaming lag when backups are performed, so I'm looking for any solution for this problem.

I tried setting ices to highest possible I/O priority with ionice command, however the problem is still there. The only solution I see here would be increasing cache for ices however could not find any documentation on how it is possible to set it.

Would appreciate for any useful information on that.

PS: Please do not recommend playing with hdparm as with SATA (SCSI driver) it does not work , and sdparm is already optimized properly.

Sincerely,
Radioshark.net

[karlH]

have you used the same ionice utility to lower the priority of the backup process. Should be easier as lowering the priority requires no special privileges like increasing does.

karl.

[Anonymous]

Yep, I have root access , so I'm
renicing ices and icecast to -15
ionicing ices and icecast to realtime 0

and lowering all backup related processes to 15 and ionice to idle (c3)

Still no result... Lag occurs. But what is really strange , that the disk usage/RAM/CPU is not such high when backup is running in idle mode, and ices are still lagging ...

Can I somehow recompile ices changing the way it reads MP3 form disk ?

[karlH]

There are no changes that you can make in the way it reads mp3 from disk. If you are sure that ices is lagging then you have to check the messages from it, how long a file is and how long it takes to go through that file. Other than a slow clock or some broken virtual machine I'm not sure what else could cause it.

karl.

[Anonymous]

Hi,

There are no messages in ices log/error file neither in icecast log file that could somehow help to clarify the situation. The hardware/drivers/kernel are up to date and checked several times. The disk read speed is quite normal and assuming ionice are set the disk IO usage is less than 20% of it maximum capacities.

If you, karlH, are the developper maybe you can give me a clue on how the read of MP3 is happening? Does it read to buffer and then works with it (then what is the buffer size) , or is it a direct read from the file? As lag occurs ~1-2 times per second I can make an assumption that if ices reads the MP3 in portion of lets say ~30Kb (0.5-2seconds preread) then it could be the reading that lags, otherwise (if lets say ices reads the block of lets say 500-1000Kb , then there is no way the lag could be caused by I/O operation and the problem should be somewhere elsewhere)

Will be waiting for your answer.


PS: My Disk read speed measured while backup is running.
Timing cached reads: 3064 MB in 2.00 seconds = 1533.22 MB/sec
Timing buffered disk reads: 132 MB in 3.02 seconds = 43.64 MB/sec

[karlH]

I'm not involved in ices 0.x so I can only go by what I've seen reported. The reading of the mp3 data is based on short reads with frequent syncs to regulate the timing of the stream. use strace to see that if you wish. You could tell what the difference is by comparing how long a file is supposed to be versus how long ices takes to play it (logging with help).

karl.

[Anonymous]

Sorry, I'm not sure how can I measure that ? Of course I can get the length of mp3 , but where can I take the time it took ices to play the file ? Can you advice me on how I can perform this ?

[karlH]

A watch, clock or just running the date command (can't remember if timestamps are in the log file) when you see the filenames appear. If you really know how long the file is then you have a good idea how long it should take. At least then you can confirm it's the source client timing and not the network.

karl.

[Anonymous]

Hi again,

Yes I managed measuring everything manually, as the delay was/is ~30seconds per 3minutes song. So should be ices that produces a lag , am I correct ?

[karlH]

if 30 seconds difference occurs then either the clock (which ices uses for timing) is drifting or it's having problems parsing the files accurately which is not something I've heard of before.

If it was just down to lagging because of contention with CPU or disk then you would find that ices will catch up after the contention is over. I've only every seen this sort of difference on systems that are overloaded or on virtual machines not scheduling tasks correctly.

karl.

[Anonymous]

HI,

Sent you PM. AAnd would like to add that lagging stops when no tar/backup operation is running. However please check my stats in PM.

[karlH]

Try a different IO scheduler to the one you are currently running with

karl.

[Anonymous]

Hi Karl,
Thanx for helping,

However I'm not a pro in linux kernel tuning
Maybe you can advice me on how I can play around with I/O schedulers or recommend some tutorials/info on the web (As it seems that google gives so many crap for older kernels and different OSes, that I can't find good tutorial)

I'm running Fedora Core 6 64bit with 2.6.23.16 kernel and grub loader.

Would really appreciate your help.

[karlH]

I replied to your pm exactly how to do that. To see the possibilities, the current is marked with [], do

cat /sys/block/sda/queue/scheduler

to change, do the following

echo cfq > /sys/block/sda/queue/scheduler

obviously change sda and/or cfq to whatever applies to your situation.

karl.

[Anonymous]

Hi, karlH,

You are genius ! Thanx a lot!

I have tried all of I/O schedulers and found out that [noop] is the one that fixes all my problems. Maybe the overall disk I/O speed have lost some 10-15% , however on the whole the system performance increased i would say some 500% (I mean while doing backups or copying/taring files ). No more lagging ices, no more freezing applications!

I own you a box of beer once you're in London