Icecast Streaming Media Server

humanclay · Joined: 02 Jun 2009 Posts: 16

I am getting a segfault on 2.3.2kh-24 on FreeBSD 8 AMD64

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Core was generated by `icecast'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/local/lib/libcurl.so.5...done.
Loaded symbols for /usr/local/lib/libcurl.so.5
Reading symbols from /usr/lib/libssl.so.6...done.
Loaded symbols for /usr/lib/libssl.so.6
Reading symbols from /lib/libcrypto.so.6...done.
Loaded symbols for /lib/libcrypto.so.6
Reading symbols from /usr/local/lib/libtheora.so.0...done.
Loaded symbols for /usr/local/lib/libtheora.so.0
Reading symbols from /usr/local/lib/libvorbis.so.4...done.
Loaded symbols for /usr/local/lib/libvorbis.so.4
Reading symbols from /usr/local/lib/libogg.so.6...done.
Loaded symbols for /usr/local/lib/libogg.so.6
Reading symbols from /usr/local/lib/libxslt.so.2...done.
Loaded symbols for /usr/local/lib/libxslt.so.2
Reading symbols from /usr/local/lib/libxml2.so.5...done.
Loaded symbols for /usr/local/lib/libxml2.so.5
Reading symbols from /lib/libz.so.5...done.
Loaded symbols for /lib/libz.so.5
Reading symbols from /usr/local/lib/libiconv.so.3...done.
Loaded symbols for /usr/local/lib/libiconv.so.3
Reading symbols from /lib/libm.so.5...done.
Loaded symbols for /lib/libm.so.5
Reading symbols from /lib/libthr.so.3...done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0 0x00000008018d8504 in strcmp () from /lib/libc.so.7
[New Thread 801f76900 (LWP 101564)]
[New Thread 801f76ac0 (LWP 101563)]
[New Thread 801f76c80 (LWP 101452)]
[New Thread 801dc11c0 (LWP 101253)]
[New Thread 801dc1380 (LWP 100894)]
[New Thread 801b021c0 (LWP 101362)]
(gdb) bt
#0 0x00000008018d8504 in strcmp () from /lib/libc.so.7
#1 0x000000000042142c in command_metadata (client=0x801f6bb00, source=0x801bcf2c0, response=2)
at admin.c:1117
#2 0x000000000041f66f in admin_mount_request (client=0x801f6bb00, uri=0x801f12ea7 "metadata.xsl")
at admin.c:384
#3 0x000000000041f92e in admin_handle_request (client=0x801f6bb00, uri=0x801f12ea7 "metadata.xsl")
at admin.c:467
#4 0x000000000040d6fa in _handle_get_request (client=0x801f6bb00) at connection.c:1343
#5 0x000000000041b380 in worker (arg=0x801b40880) at client.c:420
#6 0x0000000000434ade in _start_routine (arg=0x801b74c40) at thread.c:660
#7 0x00000008016ee4b1 in pthread_getprio () from /lib/libthr.so.3
#8 0x0000000000000000 in ?? ()
Cannot access memory at address 0x7fffffbff000
(gdb)

This was all that showed up in the error log prior to the crash:
[2010-06-16 11:42:20] EROR thread/ lock abort set to 0

karlH · Code Warrior Joined: 13 Jun 2005 Posts: 5476 Location: UK

The line numbers seem a bit out there for kh24, could be because of compiler optimisation I suppose. Assuming it's what I think it is, then I have a fix for that. It was a NULL deref on a metadata update of an inactive stream (on-demand relay?).

I've uploaded an update that fixes that and the relay retry delay you mentioned and a couple of others things.

www.xiphicecast.webspace.virginmedia.com/icecast-2.3.2-kh24.3.tar.gz

karl.

humanclay · Joined: 02 Jun 2009 Posts: 16

This version seems to break all metadata updates. This is the message I receive when updating either Ogg or MP3 streams:

Message : Mountpoint will not accept this URL update
Return Code: 1

karlH · Code Warrior Joined: 13 Jun 2005 Posts: 5476 Location: UK

The change I did since kh24 would not affect this being reported. Without any context it's hard to say, maybe sending the logs will be helpful?

karl.

humanclay · Joined: 02 Jun 2009 Posts: 16

I figured the metadata issue out. It was not related. My apologies.

KH24.3 seemed to work fine in my testing, but when I went to deploy it live today I ran into some problems.

For the first few hours everything went fine, but between 9AM and 9:30AM approximately 2000 listeners began connecting to the stream. After this, the server became unresponsive to anything but a ping and I was forced to reboot. After rebooting the same thing occurred one more time, after which I reverted to KH21a, which handled the load fine.

I have over 200 mountpoints, all of which are on-demand relays. Some of those mountpoints will be up and down at any given time. I am suspicious of the new 5 second wait time, and will try again on Monday with that code commented out.

karlH · Code Warrior Joined: 13 Jun 2005 Posts: 5476 Location: UK

You could send the part of the log that refers to the "hang" period. The timeout could have triggered a reconnect storm from listeners but wouldn't explain why the stream died. Are you getting a lot of those /mountpoint/index.html botnet requests?

karl.

humanclay · Joined: 02 Jun 2009 Posts: 16

Adding <workers>4</workers> seems to have resolved this issue. Thanks so much for the help!

One thing you might want to consider adding to Icecast is an Expires header. I have found that some proxies ignore the Cache-Control and Pragma headers and attempt to cache the stream. Adding the following header to all live streams, including FLV streams, seems to prevent this:

"Expires: Thu, 01 Jan 1970 00:00:01 GMT"

karlH · Code Warrior Joined: 13 Jun 2005 Posts: 5476 Location: UK

You just added the workers, nothing else? I did see a botnet attack on your logs so the ban-client hook would be useful.

karl.

humanclay · Joined: 02 Jun 2009 Posts: 16

Yes, that's all that I added. I will implement the banning of bots later. Thanks for the tip on that.

Brutish · Joined: 18 Mar 2010 Posts: 62

Hey Karl, is there a link to the Win32/KH24.exe

I heard its out there, I didnt get the link location though Smile

_________________
www.Hobbycaster.com

karlH · Code Warrior Joined: 13 Jun 2005 Posts: 5476 Location: UK

there is a kh24 exe on my web site

karl.

m3gab0y · Joined: 15 Aug 2009 Posts: 47 Location: London, UK

Is there a reason for Icecast KH24 not allowing to stream out more than 192 kbit/s, but can take in a source client even at 320 kbit/s with no problems ?
The same simple config (no limit rate or stuff like that, only minimal config) was launched on icecast 2.3.2 and there were no problems.
Over the logs the listeners get dropped out, because of "falling behind" when high bitrate (192+ kbit/s) source is used.

karlH · Code Warrior Joined: 13 Jun 2005 Posts: 5476 Location: UK

probably because the reschedule time for clients (on short writes) is too long. This shows up on win32 more than other platforms. I have an updated tree on my site if you would be kind enough to verify.

web address plus either of the following.

icecast-2.3.2-kh24.16.tar.gz
icecast2_win32_v2.3.2-kh24.16_setup.exe

I'm getting to the point of kh25, so let me know if things are working out there or not.

karl.

m3gab0y · Joined: 15 Aug 2009 Posts: 47 Location: London, UK

karlH · Code Warrior Joined: 13 Jun 2005 Posts: 5476 Location: UK

the burst on connect setting was next to useless over burst-size, which is why it was deprecated in the 2.3 release. If that is of sufficient size (default 64k) then bursting that should be quicker than others post kh10. These delays being tuned should only apply when there is a stall required but if you do see an issue then let me know the specifics.

karl.