Icecast Streaming Media Server Forum Index Icecast Streaming Media Server
Icecast is a Xiph Foundation Project
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Icecast KH
Goto page Previous  1, 2, 3 ... 9, 10, 11 ... 17, 18, 19  Next
 
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Dev Branches
View previous topic :: View next topic  
Author Message
stevemj



Joined: 19 Feb 2010
Posts: 9

PostPosted: Tue Mar 02, 2010 3:09 pm    Post subject: kh21a Reply with quote

We're seeing a memory issue on slaves when connected to a "master" with over 100 mount points.

Were running 2.3.2 on Win32 (master and slave) and I'm currently testing kh20 for slaves, so that I can use the xml config reload option and run in "demand" mode.

The server memory utilisation grows on the slave until there's about 40% memory left available, and its stable at this level for about 5 days. We get between 300 and 500 concurrent listeners every day (at peak).

After about 5 / 6 days, the Icecast service starts to grab more and more memory until the server runs out of resources, and we have to restart the Icecast service.

Is this memory issue addresses in the current KH release (or earlier - sorry if I've missed it)
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Tue Mar 02, 2010 3:47 pm    Post subject: Reply with quote

The kh21a update fixes all identified memory usage cases and I'm not aware of any outstanding memory issue to be fixed in trunk. However I am wondering if recent botnet-like activity is triggering some higher memory usage for some people and possibly exposing a memory leak. Check your access log for many 404 requests for /X/index.html typically where X is a stream.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Mon Mar 29, 2010 2:14 am    Post subject: Reply with quote

kh22 is up now. Various error checking updates, global bandwidth limit tunings and mpeg buffer alignment work for incoming streams. Let me know if any issue crops up with that last one as any mp3/aac feed will go through that and cuts up the buffers slightly differently.

The only xml change is the new per-mount option to skip logging client details to the access log with the <skip-accesslog> boolean setting.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
stevemj



Joined: 19 Feb 2010
Posts: 9

PostPosted: Wed Mar 31, 2010 10:30 am    Post subject: EROR thread/ lock abort set to 0 Reply with quote

I have tested in KH21a the lab with no issue

Going live, I have had the service fail several times on a slave/relay (Win32). The following entries can be found in the error log at the time the service fails:

Error log:
[2010-03-30 19:42:00] WA[2010-03-30 20:02:35] EROR thread/ lock abort set to 0
[2010-03-30 20:02:35] INFO main/ Icecast 2.3.2-kh21a server started

In the access log at this time, I have a large amount of 404 entries from 2 clients trying to access one stream each, both of which happen to be offline.

2 questions:
1 - any idea why the service is failing?
2 - can I limit the amount of retries a client makes for an un-available service?

Thanks

Steve
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Wed Mar 31, 2010 2:37 pm    Post subject: Reply with quote

failing is such a vague term, if you mean the service has to be restarted then you'll be seeing many of those INFO main/ Icecast 2.3.2-kh21a server started messages

I'm not sure what you are asking for the second query.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
stevemj



Joined: 19 Feb 2010
Posts: 9

PostPosted: Wed Mar 31, 2010 2:47 pm    Post subject: EROR thread/ lock abort set to 0 Reply with quote

The service stops and has to be manually restarted.

I am asking the second query because I am being hammered by iPhone apps and webpages constantly requesting files that are not available, as if they have a constantr re-try coded into them.

ANy idea why the service is failing? How can I start to tell?
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Wed Mar 31, 2010 3:27 pm    Post subject: Reply with quote

Regarding the second query, I'm not sure what icecast can do for that, the clients are not under icecast control so they are allowed to retry as much as possible. The latest update does have a <skip-accesslog> mount option to prevent heavy access logging, you could try to send a 302 redirect back to the client via url auth but that is more complicated (not sure you want that).

The more important issue is the 'failing service', obviously very little has bee reported from the log and win32 is a really bad debug environment. An application crash usually gets reported as such in the event logs but a stalling app could be another possibility which can be trapped for by setting the ICE_LOCK_ABORT environment variable to 2 before starting the service.

I would make sure the service is crashing or stalling first, if the former, check for entries i the event log, if the latter check to see if any new connections can be made (eg status page). You could email me the error log

karl
Back to top
View user's profile Send private message Send e-mail Visit poster's website
stevemj



Joined: 19 Feb 2010
Posts: 9

PostPosted: Wed Mar 31, 2010 4:12 pm    Post subject: Reply with quote

Event Type: Error
Event Source: Application Error
Event Category: (100)
Event ID: 1000
Date: 31/03/2010
Time: 15:10:53
User: N/A
Computer:nnnn
Description:
Faulting application icecastService.exe, version 0.0.0.0, faulting module kernel32.dll, version 5.2.3790.4480, fault address 0x00021675.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74 Applicat
0008: 69 6f 6e 20 46 61 69 6c ion Fail
0010: 75 72 65 20 20 69 63 65 ure ice
0018: 63 61 73 74 53 65 72 76 castServ
0020: 69 63 65 2e 65 78 65 20 ice.exe
0028: 30 2e 30 2e 30 2e 30 20 0.0.0.0
0030: 69 6e 20 6b 65 72 6e 65 in kerne
0038: 6c 33 32 2e 64 6c 6c 20 l32.dll
0040: 35 2e 32 2e 33 37 39 30 5.2.3790
0048: 2e 34 34 38 30 20 61 74 .4480 at
0050: 20 6f 66 66 73 65 74 20 offset
0058: 30 30 30 32 31 36 37 35 00021675
Back to top
View user's profile Send private message
Oele



Joined: 21 Apr 2010
Posts: 6

PostPosted: Sat May 08, 2010 2:51 am    Post subject: Icecast 2.3.2-kh22 crashes Reply with quote

I'm running Icecast 2.3.2-kh22 on Ubuntu 9.10 (production) and Ubuntu 8.04.4 (test). It's configured as a "master-slave" relay to an Apache webserver that "emulates" the behaviour of an Icecast server.

There's a CGI script that generates an /admin/streams list and behind the stream url's there are a few other CGI scripts that generate AAC and FLAC streams (in fact they're transcoding existing internet audio streams - first to FLAC and from there to 3 bitrates AAC). If a source stream goes down, these scripts just play an 'error tone' and stop and icecast tries to reconnect. If a source stream stays down for some time, it can happen that Icecast connects to the source, only receives the error tone and a disconnect, reconnects, receives the error tone and disconnect, etc.

This works OK most of the time but every now and then (varying from a few hours to almost a week) the Icecast server either:

- keeps running and accepting TCP connections but stops serving any content (existing and new connections just don't output any bytes anymore and the log shows a ""global" outgoing_kbitrate" of 0)

OR

- it crashes with a "Segmentation fault".

I realize that i'm trying to use icecast in a way that it wasn't designed for but i think this problem shows that there's something wrong somewhere in the icecast code. Whatever happens, an external (possibly instable) server should not cause Icecast to crash IMHO.

If i put an Icecast-2.3.2 (non-kh) between Apache and icecast-kh (icecast relays from apache, icecast-kh relays from icecast) , the crashes happen +/- once a week instead of once every few hours. The non-kh icecast never crashes.

karlH, could you please tell me what you would need to diagnose this problem? I can off course provide the config xml, log files, or even shell access to the test server.

Thanks in advance!
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Sat May 08, 2010 4:05 am    Post subject: Reply with quote

kh23 is up and ready.

I've had a number of fixes pending for relays (possible lockup or crashes with listeners present) and parsing of non-ogg streams.

For those suffering those /mountpoint/index.html botnet requests, you can now add
<mount>
<mount-name>/*/index.html</mount-name>
<ban-client>3600</ban-client>
<mount>

and those IPs will be added to the internal banned IP list for 1 hour. You will need a deny file even if it is empty.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Oele



Joined: 21 Apr 2010
Posts: 6

PostPosted: Mon May 10, 2010 11:50 am    Post subject: Reply with quote

Thanks!

I have installed kh23 on my test server and it is a *lot* more stable than kh22.

I did however see another "Segmentation fault" crash.

That's one crash in 3 days instead of one crash every 2-3 hours.

Is there anything in particular i can do to provide you with sufficient info about the crash when it happens again?
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Mon May 10, 2010 3:39 pm    Post subject: Reply with quote

Getting a backtrace of the crash is the most useful thing. Getting that will depend upon on your setup. The traditional unix approach is to get a core file and use gdb to extract the backtrace, use ulimit -c unlimited before starting icecast, but if you use changeowner then that will prevent core dumps unless you tell the system otherwise (eg on linux echo 2 >/proc/sys/fs/suid_dumpable). Another approach is to use catchsegv ..../icecast .... and the backtrace will appear on the output on crash. Email me if you need the specifics on your precise setup.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Oele



Joined: 21 Apr 2010
Posts: 6

PostPosted: Mon May 10, 2010 7:37 pm    Post subject: Reply with quote

Thanks, i will do that.

What about the situations where Icecast keeps running but stops sending responses to HTTP requests? This just happened again on my production server, about 6 ours after i installed kh23.

What happens is:
--------------------------------------
# telnet localhost 8000
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
Host: localhost

--------------------------------------

.. and then nothing happens, until i kill -9 icecast, after which the telnet connection is closed.
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Mon May 10, 2010 8:17 pm    Post subject: Reply with quote

you can set an environment variable (export ICE_LOCK_ABORT=2) before starting icecast and that will force a timer on any locks being used and if it occurs and times out (a few seconds) then an abort is done and core is dumped if enabled. In such cases a full backtrace is wanted for all threads, in gdb

thread apply all bt

and look at the bottom of the error log as well, there should be a message to indicate a trigger case. I don't usually enable it as it adds extra work but it's not excessive.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Oele



Joined: 21 Apr 2010
Posts: 6

PostPosted: Wed May 12, 2010 2:07 pm    Post subject: Reply with quote

I just had another "Segmentation fault" crash.

Is this the output you need?

(gdb) bt
#0 wait_for_restart (client=0xb6a108e0) at source.c:821
#1 0x0805d1a9 in worker (arg=0x8090e90) at client.c:420
#2 0x08072cd9 in _start_routine (arg=0x8090f40) at thread.c:660
#3 0xb709a4fb in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#4 0xb7183e5e in clone () from /lib/tls/i686/cmov/libc.so.6
(gdb)


The last few lines from the error log:

[2010-05-12 15:51:19] DBUG slave/relay_free freeing relay /flac/pp-culpabalky (0x80d7258)
[2010-05-12 15:51:19] INFO source/source_free_source source /flac/pp-culpabalky to be freed
[2010-05-12 15:51:19] DBUG source/source_free_source removing source /flac/pp-culpabalky from tree
[2010-05-12 15:51:19] DBUG source/source_clear_source clearing source "/flac/pp-culpabalky"
[2010-05-12 15:51:19] DBUG format-ogg/format_ogg_free_headers releasing header pages
[2010-05-12 15:51:19] DBUG format-ogg/free_ogg_codecs freeing codecs
[2010-05-12 15:51:19] DBUG format-flac/flac_codec_free freeing FLAC codec
[2010-05-12 15:51:19] INFO source/_free_source freeing source "/flac/pp-culpabalky"
[2010-05-12 15:51:19] DBUG stats/modify_node_event update "global" clients (9)
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Dev Branches All times are GMT
Goto page Previous  1, 2, 3 ... 9, 10, 11 ... 17, 18, 19  Next
Page 10 of 19

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2002 phpBB Group
subRebel style by ktauber