Icecast Streaming Media Server

[DJ-Zath]

Hi everyone! Karl!

Okay, this ones probably going to be strange to explain, so i ask to read carefully!

in the past, I have NEVER ran into this problem, untill RECENTLY with the last two hosts I have used for the remote server.


when Icecast starts up.. and as peeps log in... the box will begin reporting "write errors" and "connection closed - permission denied" errors and OTHER apps on the server will be KILLED and/or disconnect..

this became really aggravated once I ran a load test.. and I really noticed the issue getting progressively WORSE until around 250 users when all, everything (including Icecast itself) will crash and disconnect itself from the net.

at which point the box itself MUST be rebooted to regain access to the drive. and/or net

this is a dedicated box, too.. and with an unmetered 100m connection to the net, it should be able to handle 780 connections @ 128 Kbps; initial tests won't even get CLOSE to HALF that!

ALL other apps seem effected by this, as well.. for example, an IRCD will start dropping connections- one-by-one, and even localhost connections as well.. other apps will also lose their connections to the net and/or be unable to write to the drive/log and/or eventually be shut-down somehow (or crash entirely!) and will/can not regain their sockets or drive access until the box is completely rebooted.

when I see this happen, I see "write error","disk access failure", connection refused", etc etc etc in logs (in that which can log before the crash(es))

the CPU load never drops below 98.6 IDLE

I also see a LOT of <defunct> processes in the pid (ps ax) list when its all about to die.

the server:

FreeBSD
Intel dual core E8400
4 Gigs RAM
1 TB hard drive
dedicated (not shared or virtual)
100 Base T ethernet to pipe (unmetered)

Resource limits (current):
cputime infinity secs
filesize infinity kB
datasize-cur 131072 kB
stacksize-cur 8192 kB
coredumpsize infinity kB
memoryuse-cur 3994580 kB
memorylocked-cur 1331527 kB
maxprocesses 5547
openfiles 11095
sbsize infinity bytes
vmemoryuse infinity kB

I have tried tweaking and tuning it up, but not having ANY luck..

the more the connections (even test ones from localhost to /dev/null) the worse the problem and then, quicker it CRASHES!

even at LOW loads (around 10 connections) the problem still esists, it just take longer to reach "critical mass" so to speak.

I have scoured the net for answers to this, but can't find ANY

ANY advice would be appreciated!

thanks in advance!

-DjZ-

* partially Solved - see last post for explanation...
_________________
-DjZ-

[karlH]

check for the kernel allocation settings, like mbufs. You may be hitting a limit. netstat -m for example

karl.

[DJ-Zath]

hi Karl and thanks for the reply..

actually I posted limits to show various settings.. I did check mbufs and stuff, as well

here's what netstat -m shows:

warp-radio# netstat -m

258/777/1035 mbufs in use (current/cache/total)
256/396/652/25600 mbuf clusters in use (current/cache/total/max)
256/384 mbuf+clusters out of packet secondary zone in use (current/cache)
0/35/35/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
576K/1126K/1702K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
11 requests for I/O initiated by sendfile
0 calls to protocol drain routines

warp-radio#

again, thanks for the reply! this one has me stumped!

-DjZ-
_________________
-DjZ-

[karlH]

looks ok, have you done a decent memory test? sounds more like a unlying hardware failure.

karl.

[DJ-Zath]

funny that you ask..

I have had 2 PSUs, 2 motherboards (2 NICs) and 2 different hard drives replaced in the thing already; everything but the case and possibly RAM)

I donno if they changed the memory.. but perhaps tonight I should have them run an overhaul/system-wide test

but, before I did that, I wanted to know if it was something I did (or didn't do) and/or something with IceCast itself.

a system overhaul check will take me offline for many many hours!

-DjZ-
_________________
-DjZ-

[karlH]

Any incorrect setting with icecast would not have the effect of causing disk write errors, maybe if the disk were full but that is an obvious thing to test for. A max'd out CPU would usually indicate a single process but you didn't say which, it could be unrelated to a single app. While there's always a chance of issues with drivers, a memory check is usually one to check for. No need for anything lasting hours, just a good pounding by the memtest utility should be sufficient, could be cpu related (I assume there's no overclocking)

karl.

[DJ-Zath]

Hi Karl!

Happy holidays to you and your family


Yes, I had the host provider run a massive memtest session and SMART checks last night/this morning (I don't have KVM services for the box, so no local console access) and, although they didn't see anything wrong with the drive or RAM itself, they suggested that perhaps the SATA controller or whatnot may be defective and recommended swapping out another motherboard... after giving the OKay to do so, they replaced basically, EVERYthing... I saw an immediate improvement in login/shell performance, but by then, it was getting daylight, and I then decided to wait on the load tests for later in the day... well, now is that time! I will be starting up some load tests in just a few minutes..

more to come...

UPDATE:

the problem returns!

at around 270 logins, I lost irc services and a few peeps "fell out" of chat due to their sockets disappearing.. services can not regain disk access, and the overall number of file discriptors seems to have cut in half...


its like Icecast is causing some sort of a huge MEMORY LEAK or something...

is there anything else I should check for? or confirm?

-sigh-

-DjZ-
_________________
-DjZ-

[DJ-Zath]

okay.. back from the holiday crunch.. and, now, back to work!

What should I do from here?

the system was, pretty much, rebuilt from scratch..

Karl:

the CPU isn't being overloaded from what I can tell from a remote console (using top, for example); I never see it fall below 95% idle.

but I have determined that even with 2 or 3 listeners, the box WILL eventually run out of descriptors/sockets for other processes, and, in theory, will eventually result in the same effect, even for itself.. it just will take a while!

I have also asked on the FreeBSD forums about this as well, however, its currently gone "ignored or unnoticed".

-sigh-
_________________
-DjZ-

[karlH]

you can always access whether a particular process is consuming too many resources, the ps or top can show the CPU and memory usage (they should never be that high for this type of task), lsof can show the open file descriptors for a running process.

While an OS has per-process FD limits (eg 1024 by default) they also have system wide limits. On BSD I think it's shown as kern.maxfiles, but reaching that limit should spew out messages like file table full in dmesg

karl.

[DJ-Zath]

UPDATE:

I think I have found the issue... somewhat..

it seems that Icecast is having problems with amd64 and FBSD 7.4

I have now moved it to FBSD 9.0 and SOME of the problems went away..

well, not entirely.. its still causing "permission denied"/disk write errors, but now its happening around 500 connections instead of 20..

-DjZ-
_________________
-DjZ-

[DJ-Zath]

UPDATE:

it turns out that I did NOT solve the issue, it is still there..

I have been running some tests tonight.. using the "load test" script

I can better word it, now..

it seems that, under amd64 architecture, icecast will spend-out file/connection descriptors.. and eventually, enough descriptors will be removed and all other apps and services will be effected.. causing bottlenecking, dropped connections, instability, disk corruption, apps/services crashing and a few other strange things..

not sure if its Icecast, itself, thats causing it, however, icecast will make the problem "happen a lot more quickly"

the only remedy, so far, is to reboot the box..

more details on this as I learn them.
_________________
-DjZ-

[karlH]

check the sysctl list matching the 'kern.*files', see what numbers are thrown up.

karl

[DJ-Zath]

hi Karl!

As requested:

# sysctl kern. | grep files
kern.maxfiles: 36958
kern.maxfilesperproc: 18450
kern.openfiles: 3740

heres some more numbers (if they help)

# limits
Resource limits (current):
cputime infinity secs
filesize infinity kB
datasize 33554432 kB
stacksize 524288 kB
coredumpsize infinity kB
memoryuse infinity kB
memorylocked infinity kB
maxprocesses 5547
openfiles 18450
sbsize infinity bytes
vmemoryuse infinity kB
pseudo-terminals infinity
swapuse infinity kB


and.. perhaps this can help, too?

security.bsd.see_other_uids=0

net.inet.ip.fw.enable=1
net.inet.ip.fw.verbose=0
net.inet.ip.fw.verbose_limit=1
net.inet.ip.fw.dyn_short_lifetime=10
net.inet.ip.fw.dyn_udp_lifetime=10
net.inet.ip.fw.dyn_rst_lifetime=5
net.inet.ip.fw.dyn_fin_lifetime=5
net.inet.ip.fw.dyn_syn_lifetime=10
net.inet.ip.fw.dyn_ack_lifetime=30

net.inet.tcp.nolocaltimewait=1
net.inet.tcp.msl=10000
net.inet.tcp.delayed_ack=0
net.inet.tcp.finwait2_timeout=10
net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1

net.inet.ip.fastforwarding=1
net.inet.ip.redirect=0
net.inet.ip.random_id=1
net.inet.ip.portrange.first=2048
net.inet.ip.portrange.last=63500
net.inet.ip.portrange.randomized=0

net.inet.icmp.icmplim=2000
net.inet.icmp.icmplim_output=0

kern.ipc.somaxconn=32500
kern.maxfiles=36958
kern.maxfilesperproc=18450
kern.maxproc=1016384
kern.coredump=0


I hope this can shed some light on thew issue, any help/advice/suggestions are appreciated!
_________________
-DjZ-

[DJ-Zath]

UPDATE:

I still haven't come up with a solution to the issue... I'm "drawing straws" at this point..

I have tried recompiling/installing, from v2.3 to v2.4 to see if that makes any difference; running tests now...
_________________
-DjZ-

[karlH]

I very much doubt that anything any icecast could do is going to change the effect on this. If the app is staying within sane limits for cpu, memory and file descriptors and it triggers disk write errors for other apps then there has to be some bad reaction triggerred by the OS or the drivers. Could it be an external attack like synflood? is there any stange activity being logged beforehand like lots of dummy requests to icecast or any other serivces? If you can fake a listener load that always causes a problem then what are you requesting?

karl.