Icecast Streaming Media Server Forum Index Icecast Streaming Media Server
Icecast is a Xiph Foundation Project
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Segfault
Goto page Previous  1, 2, 3  Next
 
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Icecast Server
View previous topic :: View next topic  
Author Message
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Fri Oct 09, 2015 12:39 pm    Post subject: Reply with quote

Hi Karl,

The segfault keeps coming back.
Code:
[817906.051069] icecast[1634]: segfault at 49 ip 00007f2aa713a576 sp 00007f2aa06f1ac0 error 4 in libcrypto.so.1.0.0[7f2aa705c000+1cb000]
[1253969.780681] icecast[14928]: segfault at 21 ip 0000000000000021 sp 00007f4e17bfca88 error 14 in icecast[400000+4d000]


I have downloaded the latest master and build Icecast using 'make debug' and then 'make install'.
Now waiting for the problem to return so i can send you the core dump.

Question; what is the name and the location where the core dump will be stored?

Grt Sven
Back to top
View user's profile Send private message
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Sun Oct 18, 2015 3:43 pm    Post subject: Reply with quote

Hi Karl,

Icecast just crashed again, with the following segfault:
Code:
*** Error in `/usr/local/bin/icecast': double free or corruption (!prev): 0x00000000022022c0 ***


I have compiled Icecast using "make debug".
Could you tell me where i can find the core dump?
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Fri Oct 23, 2015 11:25 pm    Post subject: Reply with quote

There was a fix for iphone on on-demand relays that I just fixed, was not a double free bug though.

core file creation is OS dependent really, and you may need some settings if you run with changeowner or chroot set.

On linux, it's placed at the location specified in /proc/sys/kernel/core_pattern which can be just "core" for the current directory or even piped to another program.

echo /tmp/core > /proc/sys/kernel/core_pattern
echo 1 > /proc/sys/kernel/core_uses_pid
echo 2 > /proc/sys/fs/suid_dumpable

Those should give you enough to create a core file as /tmp/core.XXX

karl
Back to top
View user's profile Send private message Send e-mail Visit poster's website
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Sat Oct 24, 2015 9:49 am    Post subject: Reply with quote

I have changed the settings like you mentioned.
If i have a report, i will let you know.
Thanks!
Back to top
View user's profile Send private message
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Mon Dec 14, 2015 12:34 pm    Post subject: Reply with quote

Hi Karl,

Icecast crashed again with message:
Code:
[1202527.904573] icecast[2000]: segfault at 29 ip 00007f0c024fb079 sp 00007fffd64af120 error 4 in libcrypto.so.1.0.0 (deleted)[7f0c023ff000+1cb000]

Unfortunately there was no core dump created, at least i couldn't find it in /tmp.

It looks like the segfault occasionally happens when in reload Icecast after changing the contents of icecast.xml via PHP using:
Code:
killall -HUP icecast2

We do this when a new mount point is added, or an old mountpoint is removed.

Is there a better way to reload Icecast's config from a script?
How do you recommend to do this?
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Mon Dec 14, 2015 6:47 pm    Post subject: Reply with quote

You would need to inspect those files I mentioned again to see if they have been reset, they only apply at the time of the crash, so something else could change it back. The icecast itself does set the ulimit so that is already done for you.

The part of the message that says libcrypto.so.1.0.0 (deleted) would indicate at least the ssl libs have been updated since you last started. Do you use SSL for anything in particular?

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Mon Dec 14, 2015 8:28 pm    Post subject: Reply with quote

The files are not found in /tmp and i cannot find them anywhere else.
Seems that the settings i applied where erased after a reboot:
Code:
root@server:~# cat /proc/sys/kernel/core_pattern
core

Don't know where this dump is stored with only core defined.
I just restarted Icecast because nobody could connect.

I re-ran the core settings you suggested:
Code:
root@mx:~# cat /proc/sys/kernel/core_pattern
/tmp/core

Will now wait until the next segfault, hoping we then have more data available.

But i must say, after the last update the stability improved allot.
It did not crashed once after my last post and that was on Okt 24th!

And i checked it and the was un upgrade of libcrypto.so.1.0.0 on Dec 3 18:46.
And Icecast wasn't restarted since.

The reason we use SSL is because the entire website runs on https.
You can visit the site here, so you know what we use it for: http://tiny.cc/hb8a7x

Do you have any suggestions regarding the reload of Icecast with PHP in a fashioned manner?
Or even better, add or remove mountpoints while running without restarting?
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Mon Dec 14, 2015 9:04 pm    Post subject: Reply with quote

it's possible that you have some daemon or startup service updating those details, abrt is one example although that but "core" is the default in the kernel.

Regarding the SSL query, I meant in relation to icecast, obviously SSL can be used by other apps. I have seen mention of odd crashes in libcrypto because of the openssl version installed, while I cannot confirm that the issue you have is down to that, obviously there seems to be some use of SSL on your icecast so was just wondering what part.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Tue Dec 15, 2015 11:48 am    Post subject: Reply with quote

Hi Karl,

Thanks for your fast response.

We use SSL only for listeners.
Te reason we do that, is because we offer the streams in HTML5.
If we do not use SSL visitors get a warning about insecure content.

Here are some details about our setup:
- We use an valid SSL cert defined with the ssl-certificate directive in icecast.xml.
- Icecast listens on a dedicated IP on ports: 80, 443 and 8000.
- Sources broadcast in OGG format to port 80.
- Listeners play streams in mp3 format (transcoded) on port 443 via HTML5.

Hope this clears things our for you, if not please let me know.
Back to top
View user's profile Send private message
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Mon Dec 21, 2015 10:07 pm    Post subject: Reply with quote

Hi Karl,

Icecast crashed again today with the following error:
Code:
[250564.011911] icecast[5349]: segfault at 21 ip 0000000000000021 sp 00007fffd9cc68e8 error 14 in icecast[400000+5e000]


Finally got a core dump, you can download it here:
http://tiny.cc/wgvh7x

Could you have a look at it?
If you need any more information, please let me know.

Sven
Back to top
View user's profile Send private message
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Tue Dec 29, 2015 2:38 pm    Post subject: Reply with quote

Hi Karl,

Icecast Just crashed again with the error:
Code:
[952534.669738] icecast[48146]: segfault at 7fa200000048 ip 00007fa2c7e50576 sp 00007fa2c0602be0 error 4 in libcrypto.so.1.0.0[7fa2c7d72000+1cb000]

A cordump was again generated and can be found here:
http://tiny.cc/nyxo7x

If you find the time, could you have a look at it?
Hoping the coredump gives you enough info to solv this issue for once and for all.

Thanks for your help.

Best regards,
Sven
Back to top
View user's profile Send private message
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Tue Dec 29, 2015 3:05 pm    Post subject: Reply with quote

Another uodate regadring this issue,

I just upgraded to that latest release ( the commit of Dec 20, 2015).
AFter compiling with ./configure and make degug and make install, icecast now crashes every time i start it.

These are the errors:
Code:
[954494.931906] icecast[6499]: segfault at 38 ip 0000000000433b09 sp 00007f8d7f091d40 error 6 in icecast[400000+5e000]
[954562.552451] icecast[6538]: segfault at 38 ip 0000000000433b09 sp 00007f0ae9932d40 error 6 in icecast[400000+5e000]
[954663.618195] icecast[6691]: segfault at 38 ip 0000000000433b09 sp 00007f28c134ad40 error 6 in icecast[400000+5e000]
[954926.942596] icecast[6936]: segfault at 38 ip 0000000000433b09 sp 00007f5120617d40 error 6 in icecast[400000+5e000]
[954987.268685] icecast[7008]: segfault at 38 ip 0000000000433b09 sp 00007f7b72c3ad40 error 6 in icecast[400000+5e000]
[955181.306160] icecast[7216]: segfault at 38 ip 0000000000433b09 sp 00007fa9d0ac3d40 error 6 in icecast[400000+5e000]
[955629.372769] icecast[17896]: segfault at 38 ip 0000000000433b09 sp 00007efecc2c0d40 error 6 in icecast[400000+5e000]

When i go back and install the commit of Nov 14, 2015 Icecast start normal again.

This is the core it generates now every time i start Icecast:
http://tiny.cc/ksyo7x

Might hold some info about the cause.
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Tue Dec 29, 2015 4:59 pm    Post subject: Reply with quote

I might be able to if you also send the binary you built

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
sven



Joined: 09 Apr 2014
Posts: 34

PostPosted: Wed Dec 30, 2015 2:34 pm    Post subject: Reply with quote

Hi Karl,

I moved all files to another place so you can access them more easily.
The previous URL's do not work anymore and can be ignored.

The new file location is here: http://tiny.cc/rg6p7x
I have added the commit date to each core dump (staring with core) and added the corresponding binary (starting with icecast).

Here a short overview of issues per date (commit) version:
(View list here: https://github.com/karlheyes/icecast-kh/commits/master )

2015-11-14
Runs well and crashed only after upgrading libcrypto.so.1.0.0 with the following message:
[1202527.904573] icecast[2000]: segfault at 29 ip 00007f0c024fb079 sp 00007fffd64af120 error 4 in libcrypto.so.1.0.0 (deleted)[7f0c023ff000+1cb000]

2015-12-10
Crashed after running for a few days with the following error:
[250564.011911] icecast[5349]: segfault at 21 ip 0000000000000021 sp 00007fffd9cc68e8 error 14 in icecast[400000+5e000]

2015-12-20
Does nor run at all. Always crashes directly after starting Icecast with the following message:
[954987.268685] icecast[7008]: segfault at 38 ip 0000000000433b09 sp 00007f7b72c3ad40 error 6 in icecast[400000+5e000]

We are now running the 2015-11-14 commit in production.

Hope you can point out what causes these crashes.
If you need any more info, please let me know.

Thanks!

Grt Sven
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Wed Dec 30, 2015 6:21 pm    Post subject: Reply with quote

I've fixed the last one reported, down to the auth stream start/end trigger not being handled correctly (recent change tripped over that). The other two cores are harder to assess as the core files are not yielding much info. The build dates seem to be way different from the dates indicated


karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Icecast Server All times are GMT
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2002 phpBB Group
subRebel style by ktauber