Icecast Streaming Media Server Forum Index Icecast Streaming Media Server
Icecast is a Xiph Foundation Project
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

directory dump dir.xiph.org/yp.xml is erroneous transmitted

 
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Bug Reports
View previous topic :: View next topic  
Author Message
Joe-L



Joined: 10 Jan 2009
Posts: 9

PostPosted: Sat Jan 10, 2009 11:09 pm    Post subject: directory dump dir.xiph.org/yp.xml is erroneous transmitted Reply with quote

Hi,

currently i am integrating the icecast directory in a Stream Browser Application. Therefore i download the xml file which is available from http://dir.xiph.org/yp.xml

This was working fine and reliable at the beginning. But then the transmission failed very often. The download took a long time and a truncated xml file was received.

Temporally i got around this problem by activating the HTTP request header field "Accept-Encoding: gzip,deflate". Instead of 600kb xml data now around 60kb gzipped compressed xml was received.

Unfortunately now the problem occures again, but a bit different than before. Now i am receiving often non welformed xml data. The file seems to be ok until a certain line is reached, then the data becomes random and looks like snipets from old versions of the yp.xml.

The problem is only occuring sometimes but quiet often. Perhaps when there is much load on the webserver.

I have opened a ticket at trac, see https://trac.xiph.org/ticket/1470#preview

Btw. the branch location of the stream browser which was mentioned in the ticket changed to Streamtastic-icecast.

Is anyone else affected by this problem or can someone help me with this issue?

Greetings Joachim
Back to top
View user's profile Send private message
resus



Joined: 18 Jan 2009
Posts: 2

PostPosted: Sun Jan 18, 2009 11:24 am    Post subject: Reply with quote

I can only emphasize this bug report. if you don't plan to resolve this issue imediatly you can stop the delivery of the yp.xml.

every streambrowser application i know fails to get your yp.xml nearly everytime i try. even my browser fails to deal with your erroneous configured server. this means even with compression there is no way to get the yellow-pages...

programs that can't deal with your lame performance servers

-internet explorer
-firefox 3.05
-various winamp-plugins
-various w-lan-webradio hardware

your service is just useless for at least 2 months... (nearly everytime i intend to use it...)


Last edited by resus on Sun Jan 18, 2009 12:10 pm; edited 1 time in total
Back to top
View user's profile Send private message
resus



Joined: 18 Jan 2009
Posts: 2

PostPosted: Sun Jan 18, 2009 11:50 am    Post subject: Reply with quote

ehm... sorry for my bad english and the strong language.
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Sun Jan 18, 2009 2:26 pm    Post subject: Reply with quote

I raised this with balbinus, so I don't know the current bug tracking on this is but the initial thinking was some sort of race condition with multiple updates hitting the same file at certain times. This has been avoided so that aspect should not be an issue now and the last few times I've checked the file has been valid but are you are saying that this is still a problem?.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Joe-L



Joined: 10 Jan 2009
Posts: 9

PostPosted: Sun Jan 18, 2009 4:41 pm    Post subject: Reply with quote

Hi karl,

thanks for your response.

Yes, actually it is still a problem. I tried to open the yp.xml 5 minutes ago. This is what i get from my browser: Screenshot (It's german, and says: xml is not welformed)

I also attached a gzipped version of an erroneous yp.xml to the ticket i opened at trac (see ticket). In case you want to have a look on a complete erroneous file.

It really would be nice, if this issue could be resolved soon. If you need anything else, feel free to ask.

Greetings Joachim
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Sun Jan 18, 2009 11:31 pm    Post subject: Reply with quote

I see the output corruption in the screenshot, but so for I don't see it in the actual file on the server. I'm running a regular xmllint on the generated file to flag up any errors.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Joe-L



Joined: 10 Jan 2009
Posts: 9

PostPosted: Tue Jan 20, 2009 9:36 pm    Post subject: Reply with quote

Hi karl,

i don't think that the file on the server is corrupt. I think it is a problem with the transmission. I suppose that the non-welformed xml is caused by decompression of the truncated gzip stream.

I ve got your private message. It seems that for you the download of the file is working fine. But for me it still fails very often. Quiet every time i try. It only works seldom.

The strange thing is, that it worked fine at the beginning even without compression. Then it failed often. I added the header "Accept-Encoding: gzip,deflate" and again it worked fine for a while. But soon stopped working.

I tried to download the file using wget like you did. These are my results:
a) without compression using wget -t 3 http://dir.xiph.org/yp.xml
Code:
--2009-01-20 22:08:53--  http://dir.xiph.org/yp.xml
Resolving dir.xiph.org... 193.218.105.163
Connecting to dir.xiph.org|193.218.105.163|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 742508 (725K) [application/xml]
Saving to: `yp.xml'

     0K .......... .......... .......... .......... ..........  6% 8.31K 81s
    50K .......... .......... ...                              10% 6.17K=9.8s

2009-01-20 22:09:05 (7.48 KB/s) - Connection closed at byte 75259. Retrying.

--2009-01-20 22:09:06--  (try: 2)  http://dir.xiph.org/yp.xml
Connecting to dir.xiph.org|193.218.105.163|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 742508 (725K), 667249 (652K) remaining [application/xml]
Saving to: `yp.xml'

        [ skipping 50K ]
    50K ,,,,,,,,,, ,,,,,,,,,, ,,,....... .......... .......... 13% 9.41K 2m5s
   100K ...                                                    14% 69.9K=5.4s

2009-01-20 22:09:13 (5.65 KB/s) - Connection closed at byte 106363. Retrying.

--2009-01-20 22:09:15--  (try: 3)  http://dir.xiph.org/yp.xml
Connecting to dir.xiph.org|193.218.105.163|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 742508 (725K), 636145 (621K) remaining [application/xml]
Saving to: `yp.xml'

        [ skipping 100K ]
   100K ,,,....... .......... ...                              16%  478 =41s

2009-01-20 22:10:06 (478 B/s) - Connection closed at byte 126126. Giving up.

For the sample log i limited wget to 3 retries to keep the log short. Actually the download still wouldn't be finished after 10 retries. But wget would be able to receive the file if the number of retries is set to infinite. But it would take a very long time (more than 5 minutes)

b) with compression using wget http://dir.xiph.org/yp.xml --header "Accept-Encoding: gzip,deflate"
Code:
--2009-01-20 22:18:25--  http://dir.xiph.org/yp.xml
Resolving dir.xiph.org... 193.218.105.163
Connecting to dir.xiph.org|193.218.105.163|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/xml]
Saving to: `yp.xml'

     0K .......... .......... .......... .......... .......... 4.50K
    50K ..                                                     1.34K=13s

2009-01-20 22:18:39 (4.03 KB/s) - `yp.xml' saved [53839]

The transmission is aborted without retries. If i zcat the incomplete downloaded file it looks ok until a certain line and then becomes random.

I hope this helps to find the cause of the problem.

Joachim
Back to top
View user's profile Send private message
karlH
Code Warrior
Code Warrior


Joined: 13 Jun 2005
Posts: 5476
Location: UK

PostPosted: Tue Jan 20, 2009 11:31 pm    Post subject: Reply with quote

I tried with your wget command but with an added rate limit to test a slow connection and it does a similar thing, but if I use curl to transfer using the same approach then it works fine, can you verify whether curl is any better for you?

curl -o yp.xml.gz --limit-rate 1k -H "Accept-Encoding: gzip,deflate" http://dir.xiph.org/yp.xml

The speed through wget seems to be a factor as I've tried it with other files not just compressed ones.

karl.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Joe-L



Joined: 10 Jan 2009
Posts: 9

PostPosted: Wed Jan 21, 2009 1:08 am    Post subject: Reply with quote

I tried curl. It is more reliable than wget. Now the download works more often but still fails regularly. If I use the --limit-rate 1k parameter it seems to work not at all. The error message is also different (see below).

If the download works it looks like this:
Code:
$ curl -o yp.xml.gz -H "Accept-Encoding: gzip,deflate" http://dir.xiph.org/yp.xml
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 82530    0 82530    0     0  10013      0 --:--:--  0:00:08 --:--:-- 14423


When it fails i get this:
Code:
$ curl -o yp.xml.gz -H "Accept-Encoding: gzip,deflate" http://dir.xiph.org/yp.xml
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 23549    0 23549    0     0   1816      0 --:--:--  0:00:12 --:--:--  2448
curl: (56) Received problem 3 in the chunky parser


There is another error message if --limit-rate 1k is used:
Code:
$ curl -o yp.xml.gz --limit-rate 1k -H "Accept-Encoding: gzip,deflate" http://dir.xiph.org/yp.xml
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 57545    0 57545    0     0   1012      0 --:--:--  0:00:56 --:--:--   914
curl: (18) transfer closed with outstanding read data remaining


Joachim
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    Icecast Streaming Media Server Forum Index -> Bug Reports All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2002 phpBB Group
subRebel style by ktauber