View previous topic :: View next topic |
Author |
Message |
Joe-L
Joined: 10 Jan 2009 Posts: 9
|
Posted: Sat Jan 10, 2009 11:09 pm Post subject: directory dump dir.xiph.org/yp.xml is erroneous transmitted |
|
|
Hi,
currently i am integrating the icecast directory in a Stream Browser Application. Therefore i download the xml file which is available from http://dir.xiph.org/yp.xml
This was working fine and reliable at the beginning. But then the transmission failed very often. The download took a long time and a truncated xml file was received.
Temporally i got around this problem by activating the HTTP request header field "Accept-Encoding: gzip,deflate". Instead of 600kb xml data now around 60kb gzipped compressed xml was received.
Unfortunately now the problem occures again, but a bit different than before. Now i am receiving often non welformed xml data. The file seems to be ok until a certain line is reached, then the data becomes random and looks like snipets from old versions of the yp.xml.
The problem is only occuring sometimes but quiet often. Perhaps when there is much load on the webserver.
I have opened a ticket at trac, see https://trac.xiph.org/ticket/1470#preview
Btw. the branch location of the stream browser which was mentioned in the ticket changed to Streamtastic-icecast.
Is anyone else affected by this problem or can someone help me with this issue?
Greetings Joachim |
|
Back to top |
|
|
resus
Joined: 18 Jan 2009 Posts: 2
|
Posted: Sun Jan 18, 2009 11:24 am Post subject: |
|
|
I can only emphasize this bug report. if you don't plan to resolve this issue imediatly you can stop the delivery of the yp.xml.
every streambrowser application i know fails to get your yp.xml nearly everytime i try. even my browser fails to deal with your erroneous configured server. this means even with compression there is no way to get the yellow-pages...
programs that can't deal with your lame performance servers
-internet explorer
-firefox 3.05
-various winamp-plugins
-various w-lan-webradio hardware
your service is just useless for at least 2 months... (nearly everytime i intend to use it...)
Last edited by resus on Sun Jan 18, 2009 12:10 pm; edited 1 time in total |
|
Back to top |
|
|
resus
Joined: 18 Jan 2009 Posts: 2
|
Posted: Sun Jan 18, 2009 11:50 am Post subject: |
|
|
ehm... sorry for my bad english and the strong language. |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Sun Jan 18, 2009 2:26 pm Post subject: |
|
|
I raised this with balbinus, so I don't know the current bug tracking on this is but the initial thinking was some sort of race condition with multiple updates hitting the same file at certain times. This has been avoided so that aspect should not be an issue now and the last few times I've checked the file has been valid but are you are saying that this is still a problem?.
karl. |
|
Back to top |
|
|
Joe-L
Joined: 10 Jan 2009 Posts: 9
|
Posted: Sun Jan 18, 2009 4:41 pm Post subject: |
|
|
Hi karl,
thanks for your response.
Yes, actually it is still a problem. I tried to open the yp.xml 5 minutes ago. This is what i get from my browser: Screenshot (It's german, and says: xml is not welformed)
I also attached a gzipped version of an erroneous yp.xml to the ticket i opened at trac (see ticket). In case you want to have a look on a complete erroneous file.
It really would be nice, if this issue could be resolved soon. If you need anything else, feel free to ask.
Greetings Joachim |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Sun Jan 18, 2009 11:31 pm Post subject: |
|
|
I see the output corruption in the screenshot, but so for I don't see it in the actual file on the server. I'm running a regular xmllint on the generated file to flag up any errors.
karl. |
|
Back to top |
|
|
Joe-L
Joined: 10 Jan 2009 Posts: 9
|
Posted: Tue Jan 20, 2009 9:36 pm Post subject: |
|
|
Hi karl,
i don't think that the file on the server is corrupt. I think it is a problem with the transmission. I suppose that the non-welformed xml is caused by decompression of the truncated gzip stream.
I ve got your private message. It seems that for you the download of the file is working fine. But for me it still fails very often. Quiet every time i try. It only works seldom.
The strange thing is, that it worked fine at the beginning even without compression. Then it failed often. I added the header "Accept-Encoding: gzip,deflate" and again it worked fine for a while. But soon stopped working.
I tried to download the file using wget like you did. These are my results:
a) without compression using wget -t 3 http://dir.xiph.org/yp.xml
Code: |
--2009-01-20 22:08:53-- http://dir.xiph.org/yp.xml
Resolving dir.xiph.org... 193.218.105.163
Connecting to dir.xiph.org|193.218.105.163|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 742508 (725K) [application/xml]
Saving to: `yp.xml'
0K .......... .......... .......... .......... .......... 6% 8.31K 81s
50K .......... .......... ... 10% 6.17K=9.8s
2009-01-20 22:09:05 (7.48 KB/s) - Connection closed at byte 75259. Retrying.
--2009-01-20 22:09:06-- (try: 2) http://dir.xiph.org/yp.xml
Connecting to dir.xiph.org|193.218.105.163|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 742508 (725K), 667249 (652K) remaining [application/xml]
Saving to: `yp.xml'
[ skipping 50K ]
50K ,,,,,,,,,, ,,,,,,,,,, ,,,....... .......... .......... 13% 9.41K 2m5s
100K ... 14% 69.9K=5.4s
2009-01-20 22:09:13 (5.65 KB/s) - Connection closed at byte 106363. Retrying.
--2009-01-20 22:09:15-- (try: 3) http://dir.xiph.org/yp.xml
Connecting to dir.xiph.org|193.218.105.163|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 742508 (725K), 636145 (621K) remaining [application/xml]
Saving to: `yp.xml'
[ skipping 100K ]
100K ,,,....... .......... ... 16% 478 =41s
2009-01-20 22:10:06 (478 B/s) - Connection closed at byte 126126. Giving up.
|
For the sample log i limited wget to 3 retries to keep the log short. Actually the download still wouldn't be finished after 10 retries. But wget would be able to receive the file if the number of retries is set to infinite. But it would take a very long time (more than 5 minutes)
b) with compression using wget http://dir.xiph.org/yp.xml --header "Accept-Encoding: gzip,deflate"
Code: |
--2009-01-20 22:18:25-- http://dir.xiph.org/yp.xml
Resolving dir.xiph.org... 193.218.105.163
Connecting to dir.xiph.org|193.218.105.163|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/xml]
Saving to: `yp.xml'
0K .......... .......... .......... .......... .......... 4.50K
50K .. 1.34K=13s
2009-01-20 22:18:39 (4.03 KB/s) - `yp.xml' saved [53839]
|
The transmission is aborted without retries. If i zcat the incomplete downloaded file it looks ok until a certain line and then becomes random.
I hope this helps to find the cause of the problem.
Joachim |
|
Back to top |
|
|
karlH Code Warrior
Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Tue Jan 20, 2009 11:31 pm Post subject: |
|
|
I tried with your wget command but with an added rate limit to test a slow connection and it does a similar thing, but if I use curl to transfer using the same approach then it works fine, can you verify whether curl is any better for you?
curl -o yp.xml.gz --limit-rate 1k -H "Accept-Encoding: gzip,deflate" http://dir.xiph.org/yp.xml
The speed through wget seems to be a factor as I've tried it with other files not just compressed ones.
karl. |
|
Back to top |
|
|
Joe-L
Joined: 10 Jan 2009 Posts: 9
|
Posted: Wed Jan 21, 2009 1:08 am Post subject: |
|
|
I tried curl. It is more reliable than wget. Now the download works more often but still fails regularly. If I use the --limit-rate 1k parameter it seems to work not at all. The error message is also different (see below).
If the download works it looks like this:
Code: |
$ curl -o yp.xml.gz -H "Accept-Encoding: gzip,deflate" http://dir.xiph.org/yp.xml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 82530 0 82530 0 0 10013 0 --:--:-- 0:00:08 --:--:-- 14423
|
When it fails i get this:
Code: |
$ curl -o yp.xml.gz -H "Accept-Encoding: gzip,deflate" http://dir.xiph.org/yp.xml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 23549 0 23549 0 0 1816 0 --:--:-- 0:00:12 --:--:-- 2448
curl: (56) Received problem 3 in the chunky parser |
There is another error message if --limit-rate 1k is used:
Code: |
$ curl -o yp.xml.gz --limit-rate 1k -H "Accept-Encoding: gzip,deflate" http://dir.xiph.org/yp.xml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 57545 0 57545 0 0 1012 0 --:--:-- 0:00:56 --:--:-- 914
curl: (18) transfer closed with outstanding read data remaining |
Joachim |
|
Back to top |
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2002 phpBB Group subRebel style by ktauber
|