View previous topic :: View next topic |
Author |
Message |
sven
Joined: 09 Apr 2014 Posts: 34
|
Posted: Mon Oct 20, 2014 8:38 pm Post subject: Segfault |
|
|
Icecast crashed unexpectedly, leaving the following error in /var/log/messages:
Code: |
Oct 20 22:22:03 ibcs01 kernel: [6736769.035579] icecast2[12499]: segfault at a ip 000000000000000a sp 00007f87f4a5e438 error 14 in icecast[400000+4b000] |
Using the following version:
Code: |
icecast2 -v
Icecast 2.3.3-kh11-20140922134918 |
Could not find any other info in the Icecast logs....
What could have caused this? |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Tue Oct 21, 2014 2:24 am Post subject: |
|
|
no idea, any information in the icecast error log? a segfault is definitely something to chase up but with no logging or xml or core file then it's complete guesswork. If you think you can reproduce it then so a "make debug" then run icecast to get a core file and then I can look at the specific fault.
karl. |
|
Back to top |
|
|
sven
Joined: 09 Apr 2014 Posts: 34
|
Posted: Sun Mar 29, 2015 8:03 pm Post subject: Segfault in libc-2.13.so |
|
|
Hi Carl,
Getting more segfaults lately.
Hope you can point me in a direction?
Code: |
[3752961.744434] icecast2[25544]: segfault at 8 ip 00007f4f2b5de4ed sp 00007fff3f6c3130 error 4 in libc-2.13.so (deleted)[7f4f2b566000+182000] |
|
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Tue Mar 31, 2015 4:39 pm Post subject: |
|
|
can you try the latest from my github repo? It may be already fixed
karl. |
|
Back to top |
|
|
sven
Joined: 09 Apr 2014 Posts: 34
|
Posted: Tue Apr 07, 2015 2:59 pm Post subject: |
|
|
Hi Karl,
I have installed latest vesion now.
That includes your latest commit up to Apr 3, 2015.
If the segfault returns, I will let you know.
Thanks for your fast response!
Best regards,
Sven |
|
Back to top |
|
|
sven
Joined: 09 Apr 2014 Posts: 34
|
Posted: Tue Jun 09, 2015 8:31 pm Post subject: |
|
|
So far the segfault has not returned.
Looks like it is fixed. |
|
Back to top |
|
|
sven
Joined: 09 Apr 2014 Posts: 34
|
Posted: Tue Jul 21, 2015 3:20 pm Post subject: |
|
|
Problem is back, now on new server, new distro; latest Debian 8.0.
Serverinfo: Icecast: 2.4.0-kh1-20150716231913
Kernel: Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux
Icecast is running on SSL (HTTPS).
These are the only usable log items:
Code: |
[1185340.663368] icecast2[22783]: segfault at 7f4100001160 ip 00007f4100001160 sp 00007f410c5e9ab8 error 15
[1566953.998768] traps: icecast2[44204] general protection ip:7f4966ac163e sp:7f494b7fdac0 error:0 in libcrypto.so.1.0.0[7f49669e3000+1cb000]
|
Seems like it happens some times when a mountpoint is requested once every 2 or 3 seconds while it is offline.
In the acces log, these entries can be found at the time it occurs:
Access log:
Code: |
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:00 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:05 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:05 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:10 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:10 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:15 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:15 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:20 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:21 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:26 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:26 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:31 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:31 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:36 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:36 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:41 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:42 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:47 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:47 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:52 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:52 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:57 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:57 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
|
Error log:
Code: |
[2015-07-20 20:22:00] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:05] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:05] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:10] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:10] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:15] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:15] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:20] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:21] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:26] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:26] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:31] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:31] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:36] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:36] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:41] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:42] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:47] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:47] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:52] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:52] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:57] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20 20:22:57] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
|
These lines repeat lots of times; 2- 3- or maybe 4000 after each other.
Might have something to do with it, more logs are unavailable. |
|
Back to top |
|
|
sven
Joined: 09 Apr 2014 Posts: 34
|
Posted: Mon Jul 27, 2015 11:07 am Post subject: |
|
|
Once agian Icecast crashed with the message:
Code: |
[467016.453692] traps: icecast2[28531] general protection ip:7f9ad20e063e sp:7f9acc6a3ac0 error:0 in libcrypto.so.1.0.0[7f9ad2002000+1cb000] |
Could it be that I use an unsupported ssl version?
Code: |
root@server:/# dpkg -l | grep -i libssl
ii libssl-dev:amd64 1.0.1k-3+deb8u1 amd64 Secure Sockets Layer toolkit - development files
ii libssl-doc 1.0.1k-3+deb8u1 all Secure Sockets Layer toolkit - development documentation
ii libssl1.0.0:amd64 1.0.1k-3+deb8u1 amd64 Secure Sockets Layer toolkit - shared libraries
|
|
|
Back to top |
|
|
sven
Joined: 09 Apr 2014 Posts: 34
|
Posted: Mon Jul 27, 2015 12:32 pm Post subject: |
|
|
The sever has 1500 active mountpoints listed in icecast.xml.
Could that cause any stability issues? |
|
Back to top |
|
|
sven
Joined: 09 Apr 2014 Posts: 34
|
Posted: Sat Aug 01, 2015 9:54 am Post subject: |
|
|
I reduced the amount of mountpoints to approximately 150.
Last night, Icecast crashed again, with the following error:
Code: |
[408622.979613] traps: icecast[17250] general protection ip:7f8ddd90663e sp:7f8dd3ffeac0 error:0 in libcrypto.so.1.0.0[7f8ddd828000+1cb000] |
|
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Sat Aug 01, 2015 5:33 pm Post subject: |
|
|
The number of clients should not have any bearing on this sort of stability. If you are hitting any sort of bottleneck then you would just get a slower response and/or timeout. Make sure you have a 'make debug' build (may need a 'make clean' beforehand) and get a core file from the crash. You can simulate a crash with kill -SEGV `pidof of icecast`.
I've fixed all known issues so far, so if you are running on the latest master and it faults with a core file then let me know so I can investigate it.
karl. |
|
Back to top |
|
|
sven
Joined: 09 Apr 2014 Posts: 34
|
Posted: Thu Sep 10, 2015 8:03 pm Post subject: |
|
|
Hi Karl,
You where right, it had nothing to do with the amount of mountpoints.
Problem was still there after we reduced it.
But, at the time i write this message, Icecast is up for 1 full month without crashing and handled thousands of connections.
Works great!
So what did happen?
Well, we had a script that restarted Icecast occasionally using the SIGHUP command.
We reduced the amount of restarts and the problem did not return.
To be clear, we issued the command 'killalll -HUP icecast2' via PHP.
Is there a better way to restart Icecast using PHP, what do you suggest?
The reason we want to reload is:
1. To add a new mountpoint.
2. When a mountpoint is removed.
We add / remove additional (mp3) mountpoints for a transcoder that starts when a user starts a broadcasting in non-mp3 format. |
|
Back to top |
|
|
sven
Joined: 09 Apr 2014 Posts: 34
|
Posted: Fri Sep 11, 2015 6:24 pm Post subject: |
|
|
Hmm, one day later and Icecats crashed again:
Code: |
[2681137.688200] icecast[45481]: segfault at 29 ip 00007f6332150079 sp 00007f632c6f5a40 error 4 in libcrypto.so.1.0.0[7f6332054000+1cb000]
|
I think it must have something to do with the way we restart using SIGHUP.
Isn't there an API call to add or remove mountpoints on the fly while Icecast is running, without the use of reloading or restarting?
That would be great... |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Mon Sep 21, 2015 1:19 pm Post subject: |
|
|
updated the release, let me know if the issue still persists.
karl. |
|
Back to top |
|
|
sven
Joined: 09 Apr 2014 Posts: 34
|
Posted: Mon Sep 21, 2015 2:59 pm Post subject: |
|
|
Hi Karl,
Yesterday another segfault.
This time a new message:
Code: |
# *** Error in `/usr/local/bin/icecast': double free or corruption (!prev): 0x00000000020772c0 *** |
Just installed the latest Github release.
I will let you know if the problem returns.
Thanks for you help! |
|
Back to top |
|
|
|