Icecast Streaming Media Server

[Murrawhip]

Ubuntu
2.3.3-kh1

All of a sudden today 7 hours ago Icecast crashed.
There was nothing in Icecast's log about it.
It had never happened in the last 6 months or so of using icecast. I figured probably just bad luck, restarted it, and it's happened twice in the last 30 minutes.
I searched through /var/log and in /var/log/messages I found:

[karlH]

Most likely a bug from the recent rwlock merge. That particular function looks like it's still using the reference counts. I'll drop those operations now, as it could lead to bad pointers. This is a master feeding a slave relay of a similar version icecast.

This and another fix for restarting relays will mean a new release required.


karl.

[Murrawhip]

Okay sweet, thanks for looking into it. I was worried it was my system being screwy.

[Murrawhip]

I was running KH3 up until today, when it started crashing via segfault every 5-40 minutes.
It had been running for months just fine until today. No software/hardware changes that I know of.

I upgraded to the latest github release and was receiving the same errors (I didn't have time to gdb, as it's a live system).

I downgraded back to KH3 and ran through gdb.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7fd5700 (LWP 18313)]
send_iceblock_to_client (client=0x7fffe40027c0) at format_mp3.c:562
562 if (refbuf->associated != client_mpg->associated)

(gdb) backtrace
#0 send_iceblock_to_client (client=0x7fffe40027c0) at format_mp3.c:562
#1 write_mpeg_buf_to_client (client=0x7fffe40027c0) at format_mp3.c:602
#2 0x0000000000415b7e in send_listener (client=0x7fffe40027c0, source=0x7fffec00eef0) at source.c:1078
#3 send_to_listener (client=0x7fffe40027c0) at source.c:953
#4 0x0000000000419e35 in worker (arg=0x692070) at client.c:561
#5 0x0000000000431c94 in _start_routine (arg=0x692130) at thread.c:673
#6 0x00007ffff6b3ee9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007ffff686bcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()


Anything I can do to get this resolved?

[Edit]: I've ruled out bad RAM.
[Edit2]: It crashed while on DEBUG. It's a server with ~25 streams and a few hundred listeners, so it's pretty chatty. This is the log entries from the last second:

[Murrawhip]

I upgraded to the latest git release again, as I noticed you made a change in format_mp3.c

[karlH]

the abort is being triggered because of the buffer referenced is incorrect. What is not clear yet is how that came about. I suspect it's a lagging listener tripping this based on the lag values I've seen so far, but the more information I get the better. Any details about the stream in question?

karl.

[Murrawhip]

I'm not sure how to determine which stream is causing it.

Is there any other information out of gdb I could get that might help? I'm not particularly experienced in it.


I would say that this version from github is having the issue more often than kh3.

Github version crashes:
30/01/2013 01:59:12 AM
30/01/2013 02:06:38 AM
30/01/2013 02:10:38 AM
30/01/2013 02:20:49 AM
30/01/2013 02:33:20 AM
30/01/2013 02:41:50 AM
30/01/2013 02:43:01 AM
30/01/2013 02:50:21 AM
30/01/2013 02:54:46 AM
30/01/2013 03:05:07 AM
30/01/2013 03:17:53 AM

kh3 crashes:
29/01/2013 07:50:21 AM
29/01/2013 08:13:43 AM
29/01/2013 10:38:52 AM
29/01/2013 10:44:57 AM
29/01/2013 10:50:22 AM
29/01/2013 11:26:40 AM
29/01/2013 12:53:01 PM
29/01/2013 01:48:50 PM
29/01/2013 02:07:12 PM
29/01/2013 02:55:37 PM


Edit:
This is the entry in access.log immediately before the last crash:
removedIP - - [30/Jan/2013:04:03:05 -0800] "GET /CKQR HTTP/1.1" 200 3359507 "-" "Lavf52.111.0" 262

And the crash before that:
removedIP - - [30/Jan/2013:04:08:32 -0800] "GET /CJCD HTTP/1.1" 200 45260 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0

Doesn't look too relevant.

[karlH]

in gdb, in this case, you can do 'up' twice and then do p source->mount to show the mountpoint.

I've reverted the previous patch, but I put in a trap for a possible 0 length buffer case.

karl.

[Murrawhip]

Okay I'm running your new version under gdb. Will update when it crashes.

[Murrawhip]

[Murrawhip]

[karlH]

it seems that you can reproduce it easily, do you have a procedure that I could duplicate?

karl.

[Murrawhip]

I'm not really doing anything special, that I know of.
I just wait for 10-30 minutes and it happens.
I've just emailed you a link to my icecast.xml, and my stats.xsl. There's barely any listeners right now and it's still happening.

[karlH]

When you can try it, can you test out the latest updates.

karl.