View previous topic :: View next topic |
Author |
Message |
jaba
Joined: 21 Sep 2009 Posts: 8 Location: Helsinki, Finland
|
Posted: Mon Sep 21, 2009 12:40 pm Post subject: 2.3.2 SIGHUP not working correctly? |
|
|
Hello,
I'm relatively new with Icecast 2, so this might well be a problem between my keyboard and chair, but nevertheless I've confirmed this bug with both FreeBSD 6.2 and Debian Lenny 5.0.3. I even tried to fetch the latest Icecast build from Debian Sid (which claims to have some logrotate related problems fixed), but no go.
With Icecast 2.3.2 logrotate makes things go wrong. icecast2 will still be hanging around in the process list and currently connected clients will be able to continue listening to radio, but admin interface will show a blank screen and new clients are unable to connect.
This is somehow related to configuration reloading and parsing. My best guess is that even though Icecast thinks it was able to correctly re-read and parse the config file, in reality it was unable to do so. Perhaps some kind of race condition with all the locks Icecast seems to create under the hood?
I have no real knowledge about Icecast source code - today I glimpsed through its code for the first time a bit. But anyway, as soon as I slightly changed the src/slave.c, logrotate started to work as I want.
Code: |
diff --git a/src/slave.c b/src/slave.c
index cb792ac..9de9157 100644
--- a/src/slave.c
+++ b/src/slave.c
@@ -695,9 +695,10 @@ static void *_slave_thread(void *arg)
/* re-read xml file if requested */
if (global . schedule_config_reread)
{
- event_config_read (NULL);
+ /*event_config_read (NULL);*/
+ restart_logging (config);
global . schedule_config_reread = 0;
- }
+ }
|
In my case on-the-fly config modifications are very rare, so I can live with the fact I have to do a full stop and start for every config change. On the other hand, I do need to make Icecast survive during logrotate, so this arguably stupid little patch made my day for now.
Still a proper fix should be made, or if you have any suggestions what could trigger this logrotate crash behaviour, all the hints are gladly accepted. |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Mon Sep 21, 2009 1:43 pm Post subject: |
|
|
more detail on the failure case would be helpful. One possible issue with reload is if changeowner or chroot is enabled because obviously you are changing your access after starting up, although symlinks/permissions can get around that if you are careful.
karl. |
|
Back to top |
|
|
jaba
Joined: 21 Sep 2009 Posts: 8 Location: Helsinki, Finland
|
Posted: Mon Sep 21, 2009 1:48 pm Post subject: |
|
|
changeowner and chroot are not in use, Icecast2 will start as icecast2 user thanks to Debian /etc/init.d/icecast2 init script.
I'll provide you a more detailed report tomorrow, I'll need to investigate this a bit further. |
|
Back to top |
|
|
jaba
Joined: 21 Sep 2009 Posts: 8 Location: Helsinki, Finland
|
Posted: Tue Sep 22, 2009 3:32 am Post subject: |
|
|
I will soon start my further investigation, but here's couple of other random notes about the things I made before I started hacking the source code:
- To make sure /etc/init.d/icecast2 init script is not to blame, I started icecast2 manually with icecast2 -b -c /etc/icecast2/icecast.xml, and also tried the foregrounded version without the -b option.
- To make sure SMP is not to blame, I started icecast manually with taskset 0x1 icecast2 -b -c /etc/icecast2/icecast.xml, and also without the -b option. On the other hand, my FreeBSD 6.2 server is a very old Pentium III, so surely that's a uni-processor system, anyway.
- Logs did not reveal any odd error messages.
- I tried to strace the icecast2 processes whenever they were stuck, but did not find anything. I'll try again today and paste the results for you to see. |
|
Back to top |
|
|
jaba
Joined: 21 Sep 2009 Posts: 8 Location: Helsinki, Finland
|
Posted: Tue Sep 22, 2009 7:02 am Post subject: |
|
|
I still have no idea what is going on, but here's one more finding: after HUP the web admin interface stops working because suddenly the password doesn't match. Icecast thinks that password the client supplied was GET /status.xsl HTTP/1.0 or whatever http request I make.
Code: |
2009-09-22 09:49:24 INFO connection/_handle_shoutcast_compatible password does not match "GET /status.xsl HTTP/1.0" |
|
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Tue Sep 22, 2009 3:08 pm Post subject: |
|
|
so icecast believes that is coming in on a shoutcast port. The thing is that 2.3.2 does not reopen the listening sockets over HUP so those should stay the same. I'll check here but I don't have FBSD for comparison.
karl. |
|
Back to top |
|
|
jaba
Joined: 21 Sep 2009 Posts: 8 Location: Helsinki, Finland
|
Posted: Wed Sep 23, 2009 5:53 am Post subject: |
|
|
But this happens with Debian Lenny 5.0.3 for me, too. Actually that's the main server where I'm trying to track down this issue. FreeBSD was just checking out if that happens elsewhere or if that's a bug in Debian. |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Wed Sep 23, 2009 10:11 am Post subject: |
|
|
Did you try the current trunk code at people.xiph.org/~brendan/snapshots? I can't try 2.3.2 here as a glibc symbol redefinition is stopping that.
karl. |
|
Back to top |
|
|
jaba
Joined: 21 Sep 2009 Posts: 8 Location: Helsinki, Finland
|
Posted: Wed Sep 23, 2009 10:25 am Post subject: |
|
|
No, I didn't try the trunk code yet. Downloading it as we speak. |
|
Back to top |
|
|
jaba
Joined: 21 Sep 2009 Posts: 8 Location: Helsinki, Finland
|
Posted: Wed Sep 23, 2009 10:54 am Post subject: |
|
|
Or, to be more accurate, I tried to download the latest trunk tgz. For some reason my wget is not amused at the moment, it just tries to connect to people.xiph.org, but does not begin the download. Will try again later. |
|
Back to top |
|
|
jaba
Joined: 21 Sep 2009 Posts: 8 Location: Helsinki, Finland
|
Posted: Thu Sep 24, 2009 6:33 am Post subject: |
|
|
I'm sorry to tell you, but the trunk version does not work any better. The behaviour is exactly the same. New clients are unable to connect and admin page shows just a blank page.
Here's what the log tells immediately after HUP signal:
---
[2009-09-24 09:28:40] DBUG stats/modify_node_event update "/ourradiourl" total_bytes_read (306366)
[2009-09-24 09:28:40] DBUG stats/modify_node_event update "/ourradiourl" total_bytes_sent (242473)
[2009-09-24 09:28:42] DBUG yp/yp_recheck_config Updating YP configuration
[2009-09-24 09:28:42] DBUG stats/modify_node_event update global server_id (Icecast trunk)
[2009-09-24 09:28:42] DBUG stats/modify_node_event update global host (ouricecastserver.ourdomain.com)
[2009-09-24 09:28:42] DBUG stats/modify_node_event update global location (Earth)
[2009-09-24 09:28:42] DBUG stats/modify_node_event update global admin (icemaster@localhost)
[2009-09-24 09:28:43] DBUG slave/_slave_thread checking master stream list
[2009-09-24 09:28:44] INFO connection/_handle_shoutcast_compatible password does not match "GET /ourradiourl HTTP/1.1 Accept: */*"
[2009-09-24 09:28:44] DBUG stats/modify_node_event update global clients (3)
[2009-09-24 09:28:44] DBUG stats/modify_node_event update global connections (62)
[2009-09-24 09:28:44] DBUG stats/modify_node_event update global clients (2)
[2009-09-24 09:28:45] DBUG stats/modify_node_event update "/ourradiourl" total_bytes_read (386204)
[2009-09-24 09:28:45] DBUG stats/modify_node_event update "/ourradiourl" total_bytes_sent (322278)
[2009-09-24 09:28:45] INFO connection/_handle_shoutcast_compatible password does not match "GET /admin/ HTTP/1.1"
[2009-09-24 09:28:45] DBUG stats/modify_node_event update global clients (3)
[2009-09-24 09:28:45] DBUG stats/modify_node_event update global connections (63)
[2009-09-24 09:28:45] DBUG stats/modify_node_event update global clients (2)
[2009-09-24 09:28:45] INFO connection/_handle_shoutcast_compatible password does not match "GET /admin/ HTTP/1.1"
--- |
|
Back to top |
|
|
|