Icecast Streaming Media Server Forum Index Icecast Streaming Media Server
Icecast is a Xiph Foundation Project
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Segfault
Goto page 1, 2, 3  Next
 
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Icecast Server
View previous topic :: View next topic  
Author Message
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Mon Oct 20, 2014 8:38 pm    Post subject: Segfault Reply with quote

Icecast crashed unexpectedly, leaving the following error in /var/log/messages:
Code:
Oct 20 22:22:03 ibcs01 kernel: [6736769.035579] icecast2[12499]: segfault at a ip 000000000000000a sp 00007f87f4a5e438 error 14 in icecast[400000+4b000]


Using the following version:
Code:
icecast2 -v
Icecast 2.3.3-kh11-20140922134918


Could not find any other info in the Icecast logs....
What could have caused this?
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Tue Oct 21, 2014 2:24 am    Post subject: Reply with quote

no idea, any information in the icecast error log? a segfault is definitely something to chase up but with no logging or xml or core file then it's complete guesswork. If you think you can reproduce it then so a "make debug" then run icecast to get a core file and then I can look at the specific fault.


karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Sun Mar 29, 2015 8:03 pm    Post subject: Segfault in libc-2.13.so Reply with quote

Hi Carl,

Getting more segfaults lately.
Hope you can point me in a direction?
Code:
[3752961.744434] icecast2[25544]: segfault at 8 ip 00007f4f2b5de4ed sp 00007fff3f6c3130 error 4 in libc-2.13.so (deleted)[7f4f2b566000+182000]
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Tue Mar 31, 2015 4:39 pm    Post subject: Reply with quote

can you try the latest from my github repo? It may be already fixed

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Tue Apr 07, 2015 2:59 pm    Post subject: Reply with quote

Hi Karl,

I have installed latest vesion now.
That includes your latest commit up to Apr 3, 2015.

If the segfault returns, I will let you know.

Thanks for your fast response!

Best regards,
Sven
Back to top
View user's profile Send private message
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Tue Jun 09, 2015 8:31 pm    Post subject: Reply with quote

So far the segfault has not returned.
Looks like it is fixed.
Back to top
View user's profile Send private message
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Tue Jul 21, 2015 3:20 pm    Post subject: Reply with quote

Problem is back, now on new server, new distro; latest Debian 8.0.

Serverinfo: Icecast: 2.4.0-kh1-20150716231913
Kernel: Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux

Icecast is running on SSL (HTTPS).

These are the only usable log items:

Code:
[1185340.663368] icecast2[22783]: segfault at 7f4100001160 ip 00007f4100001160 sp 00007f410c5e9ab8 error 15
[1566953.998768] traps: icecast2[44204] general protection ip:7f4966ac163e sp:7f494b7fdac0 error:0 in libcrypto.so.1.0.0[7f49669e3000+1cb000]


Seems like it happens some times when a mountpoint is requested once every 2 or 3 seconds while it is offline.

In the acces log, these entries can be found at the time it occurs:

Access log:
Code:
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:00 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:05 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:05 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:10 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:10 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:15 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:15 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:20 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:21 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:26 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:26 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:31 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:31 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:36 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:36 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:41 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:42 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:47 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:47 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:52 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:52 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:57 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "BASS/2.4" 0
/var/log/icecast2/access.log:92.87.123.110 - - [20/Jul/2015:20:22:57 +0200] "GET /djcosmin.mp3 HTTP/1.1" 404 96 "-" "NSPlayer/11.0.5721.5275 WMFSDK/11.0" 0


Error log:
Code:
[2015-07-20  20:22:00] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:05] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:05] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:10] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:10] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:15] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:15] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:20] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:21] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:26] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:26] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:31] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:31] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:36] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:36] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:41] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:42] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:47] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:47] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:52] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:52] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:57] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory
[2015-07-20  20:22:57] WARN fserve/fserve_client_create req for file "/usr/share/icecast2/web/djcosmin.mp3" No such file or directory


These lines repeat lots of times; 2- 3- or maybe 4000 after each other.
Might have something to do with it, more logs are unavailable.
Back to top
View user's profile Send private message
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Mon Jul 27, 2015 11:07 am    Post subject: Reply with quote

Once agian Icecast crashed with the message:
Code:
[467016.453692] traps: icecast2[28531] general protection ip:7f9ad20e063e sp:7f9acc6a3ac0 error:0 in libcrypto.so.1.0.0[7f9ad2002000+1cb000]


Could it be that I use an unsupported ssl version?

Code:
root@server:/# dpkg -l | grep -i libssl
ii  libssl-dev:amd64                       1.0.1k-3+deb8u1              amd64        Secure Sockets Layer toolkit - development files
ii  libssl-doc                             1.0.1k-3+deb8u1              all          Secure Sockets Layer toolkit - development documentation
ii  libssl1.0.0:amd64                      1.0.1k-3+deb8u1              amd64        Secure Sockets Layer toolkit - shared libraries
Back to top
View user's profile Send private message
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Mon Jul 27, 2015 12:32 pm    Post subject: Reply with quote

The sever has 1500 active mountpoints listed in icecast.xml.
Could that cause any stability issues?
Back to top
View user's profile Send private message
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Sat Aug 01, 2015 9:54 am    Post subject: Reply with quote

I reduced the amount of mountpoints to approximately 150.
Last night, Icecast crashed again, with the following error:
Code:
[408622.979613] traps: icecast[17250] general protection ip:7f8ddd90663e sp:7f8dd3ffeac0 error:0 in libcrypto.so.1.0.0[7f8ddd828000+1cb000]
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Sat Aug 01, 2015 5:33 pm    Post subject: Reply with quote

The number of clients should not have any bearing on this sort of stability. If you are hitting any sort of bottleneck then you would just get a slower response and/or timeout. Make sure you have a 'make debug' build (may need a 'make clean' beforehand) and get a core file from the crash. You can simulate a crash with kill -SEGV `pidof of icecast`.

I've fixed all known issues so far, so if you are running on the latest master and it faults with a core file then let me know so I can investigate it.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Thu Sep 10, 2015 8:03 pm    Post subject: Reply with quote

Hi Karl,

You where right, it had nothing to do with the amount of mountpoints.
Problem was still there after we reduced it.

But, at the time i write this message, Icecast is up for 1 full month without crashing and handled thousands of connections.
Works great!

So what did happen?
Well, we had a script that restarted Icecast occasionally using the SIGHUP command.
We reduced the amount of restarts and the problem did not return.

To be clear, we issued the command 'killalll -HUP icecast2' via PHP.
Is there a better way to restart Icecast using PHP, what do you suggest?

The reason we want to reload is:
1. To add a new mountpoint.
2. When a mountpoint is removed.

We add / remove additional (mp3) mountpoints for a transcoder that starts when a user starts a broadcasting in non-mp3 format.
Back to top
View user's profile Send private message
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Fri Sep 11, 2015 6:24 pm    Post subject: Reply with quote

Hmm, one day later and Icecats crashed again:
Code:
[2681137.688200] icecast[45481]: segfault at 29 ip 00007f6332150079 sp 00007f632c6f5a40 error 4 in libcrypto.so.1.0.0[7f6332054000+1cb000]


I think it must have something to do with the way we restart using SIGHUP.

Isn't there an API call to add or remove mountpoints on the fly while Icecast is running, without the use of reloading or restarting?

That would be great...
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Mon Sep 21, 2015 1:19 pm    Post subject: Reply with quote

updated the release, let me know if the issue still persists.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Mon Sep 21, 2015 2:59 pm    Post subject: Reply with quote

Hi Karl,

Yesterday another segfault.
This time a new message:
Code:
# *** Error in `/usr/local/bin/icecast': double free or corruption (!prev): 0x00000000020772c0 ***


Just installed the latest Github release.
I will let you know if the problem returns.

Thanks for you help!
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Icecast Server All times are GMT
Goto page 1, 2, 3  Next
Page 1 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2002 phpBB Group
subRebel style by ktauber