| View previous topic :: View next topic |
| Author |
Message |
vdborght
Joined: 09 Sep 2005
Posts: 14
Location: Belgium
|
 Posted: Wed Sep 23, 2009 3:56 pm Post subject: Multiple Master Relays? |
 |
|
Hi,
I'm in a bit of a jam here.
I'm currently hosting my icecast servers on AWS to minimize bandwidth costs and have now started using their auto-scaling feature.
The setup is as follows:
- Master server & sources in the datacenter
- First slave on amazon connects to the master server and acts as master for the other slaves on amazon while serving clients on its own too
- Other slaves connect to the first slave and serve clients
My problem is that if the "first slave" dies for some reason (232KH15B seems not 100% stable and VM on amazon can die too) then all my other slaves are pointing to a dead master.
Is there any way to specify a second master-server that will be used as a backup? Or even to specify a whole chain of master-servers?
Regards,
Peter
_________________
Regards,
Peter
TAATU - Create the World
www.taatu.com |
|
| Back to top |
|
 |
karlH
Code Warrior
Joined: 13 Jun 2005
Posts: 5476
Location: UK
|
| Posted: Wed Sep 23, 2009 7:55 pm Post subject: |
|
|
regarding 15b, I have found a possible crash bug if you are using URL based auth, but if there other issues then I would like to hear about them.
the kh tree does allow multiple masters in the relay, just look in the conf subdirectory in the tree.
karl. |
|
| Back to top |
|
|
vdborght
Joined: 09 Sep 2005
Posts: 14
Location: Belgium
|
| Posted: Thu Sep 24, 2009 7:59 am Post subject: |
|
|
| karlH wrote: |
the kh tree does allow multiple masters in the relay, just look in the conf subdirectory in the tree.
|
What I see is that it allows multiple <master> tags under a <relay>.
But what I'd like to get is multiple <master-server> tags.
Since I have over 100 streams running, I want to avoid having to define all of them on each slave, especially since we add new ones regularly.
| karlH wrote: |
regarding 15b, I have found a possible crash bug if you are using URL based auth, but if there other issues then I would like to hear about them.
|
Is that bug caused by having the authentication params in the URL or by configuring URL authentication in the config?
Reason I'm asking is that we've stopped using URL auth some time ago (so it's no longer configured), but haven't bothered updating our client code (so the request still contains it).
Apart from that, I haven't managed to put my finger on the crash issue yet. It seems to happen only under reasonable load (300+ clients ~ 30Mbit total bandwidth) and without anything out of the ordinary in the log (even at loglevel 4).
One thing I did notice is that your client counts are way off.
At 150 concurrent connections (netstat -an | grep ...), it's reporting 500-700 connections in the log.
_________________
Regards,
Peter
TAATU - Create the World
www.taatu.com |
|
| Back to top |
|
|
karlH
Code Warrior
Joined: 13 Jun 2005
Posts: 5476
Location: UK
|
| Posted: Thu Sep 24, 2009 9:23 am Post subject: |
|
|
I'll look into the multiple server definition for a slave. some of the internals are in place for that but it's not complete.
The bug I mentioned was relating to a client getting service after auth has completed, there is a race there that could lead to memory corruption but a very small window. So if no url transactions then not possible to trigger.
To help pinpoint any crash case, do a "make debug" build and setup for a core dump or even run catchsegv .../icecast ....
Any stats mis-accounting could be identified with the error log at level 4. The 'clients' stat is not typically one that goes wrong.
karl. |
|
| Back to top |
|
|
vdborght
Joined: 09 Sep 2005
Posts: 14
Location: Belgium
|
| Posted: Thu Sep 24, 2009 2:57 pm Post subject: |
|
|
| karlH wrote: |
I'll look into the multiple server definition for a slave. some of the internals are in place for that but it's not complete.
|
Thanks, that would help a lot.
| karlH wrote: |
To help pinpoint any crash case, do a "make debug" build and setup for a core dump or even run catchsegv .../icecast ....
|
I'm on it... will let you know the results
| karlH wrote: |
Any stats mis-accounting could be identified with the error log at level 4. The 'clients' stat is not typically one that goes wrong.
|
That's the one I've noticed though... All "connected client" stats in the status pages look ok, but the client count in the logfile is way too high.
I'll see if I can get you some snippets tomorrow.
_________________
Regards,
Peter
TAATU - Create the World
www.taatu.com |
|
| Back to top |
|
|
vdborght
Joined: 09 Sep 2005
Posts: 14
Location: Belgium
|
| Posted: Thu Oct 01, 2009 8:09 am Post subject: |
|
|
| karlH wrote: |
To help pinpoint any crash case, do a "make debug" build and setup for a core dump or even run catchsegv .../icecast ....
|
I've zipped up my logs for you, I'll send you a PM with the URL.
| karlH wrote: |
Any stats mis-accounting could be identified with the error log at level 4. The 'clients' stat is not typically one that goes wrong.
karl. |
If you look at the snippet below, you'll agree there's something not right...
This was grabbed from error.log on a slave server when my master server was down (and had been down for hours) and "netstat -an | grep ESTAB" showed 0 connections on port 80.
| Code: |
[2009-10-01 03:59:08] DBUG stats/modify_node_event update "global" outgoing_kbitrate (89)
[2009-10-01 03:59:08] DBUG stats/modify_node_event update "global" clients (391)
[2009-10-01 03:59:08] DBUG stats/modify_node_event update "global" connections (187891)
[2009-10-01 03:59:08] DBUG stats/modify_node_event update "global" connections (187892)
[2009-10-01 03:59:08] DBUG client/worker 0x808e840 now has 391 clients
|
_________________
Regards,
Peter
TAATU - Create the World
www.taatu.com |
|
| Back to top |
|
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
