| View previous topic :: View next topic |
| Author |
Message |
Anonymous
Guest
|
 Posted: Mon Dec 31, 2007 5:48 pm Post subject: ices + idv3 tag encoding |
 |
|
We have a problem with how ices handles idv3 encoding. The problem lies in that ices (correct me if I'm wrong) is trying to autodetect idv3 tag encoding.
What we see is that some cp1251 tags are displayed in UTF8, while some other idv3 tags are displayed normally in cp1251.
To make it short:
We have track1.mp3 that has idv3 tags in CP1251 and when you call status.xsl or if you're listening stream via player you see unreadable characters.
Yet you have different track2 mp3 with idv3 tags in CP1251 and status.xsl and players are showing everything correctly.
We've double checked mp3 files and we're sure they all have CP1251 idv3 tags, yet the displayed information is different. In one time the characters are "unreadable (utf8 translation of cp1251)" in other time they are OK - cp1251.
The shorter idv3 tags are the higher is the probability that the displayed information is going to be corrupted.
By displayed information we understand icecasts's status.xsl output and media player track's info.
The question is: How to make icecast/ices to understand CP1251 idv3 tags? I've searched the manuals, tried <charset>cp1251</charset> in ices' confing and icecast's explicit <mount></mount> with no avail.
It WOULD be very cool to specify playlist's idv3 tags encoding. Is is possible? Thanks in advance. |
|
| Back to top |
|
 |
karlH
Code Warrior
Joined: 13 Jun 2005
Posts: 5476
Location: UK
|
| Posted: Mon Dec 31, 2007 7:14 pm Post subject: |
|
|
icecast does not attempt to handle ID3, it gets any metadata information from a source client via the /admin requests. I don't know if ices does any conversion but if it does and gets it wrong then that is probably why. Th charset option you mentioned is only in the trunk and -kh releases of icecast, 2.3.1 will not do anything with it. You may find that those 2 versions are better at handling the metadata if it is sent as CP1251.
karl. |
|
| Back to top |
|
|
Anonymous
Guest
|
| Posted: Mon Dec 31, 2007 10:58 pm Post subject: |
|
|
| karlH wrote: |
icecast does not attempt to handle ID3, it gets any metadata information from a source client via the /admin requests. I don't know if ices does any conversion but if it does and gets it wrong then that is probably why. Th charset option you mentioned is only in the trunk and -kh releases of icecast, 2.3.1 will not do anything with it. You may find that those 2 versions are better at handling the metadata if it is sent as CP1251.
karl. |
Thanks, Karl, I thought the problem is most probably in ices. Don't you know where I could get ices developers? I admit I was under impression that icecast and ices are developed by one team. |
|
| Back to top |
|
|
karlH
Code Warrior
Joined: 13 Jun 2005
Posts: 5476
Location: UK
|
| Posted: Mon Dec 31, 2007 11:13 pm Post subject: |
|
|
ices 0.4 is not really maintained by anyone these days, MP3 doesn't really fit within Xiph as it's not a free codec. It could be that you just need to pdated your icecast so that the charset setting applies.
karl. |
|
| Back to top |
|
|
Anonymous
Guest
|
| Posted: Tue Jan 01, 2008 1:14 am Post subject: |
|
|
| karlH wrote: |
ices 0.4 is not really maintained by anyone these days, MP3 doesn't really fit within Xiph as it's not a free codec. It could be that you just need to pdated your icecast so that the charset setting applies.
karl. |
Thanks. How do I get icecast version with <charset> tag support? I've been looking through ices 0.4 sources and I found no conversions at all. It looks like IceS 0.4 is sending to icecast what it gets from mp3 files and the problem with encoding (or lack of support) is actually in icecast server. It probably thinks that metadata it receives is in UTF8 and there is no way to make it think otherwise. I don't understand then why some of idv3 tags are correctly displayed and some are not. Is there any autodetection in icecast server? |
|
| Back to top |
|
|
karlH
Code Warrior
Joined: 13 Jun 2005
Posts: 5476
Location: UK
|
| Posted: Tue Jan 01, 2008 3:07 am Post subject: |
|
|
Icecast has traditionally assumed UTF8 for metadata, as unicode is really the only character set that can handle multiple languages, just imagine a status page with many streams each with a different charset.
With post 2.3.1 work I've made it so that the non-ogg streams are latin1 by default (Ogg is still UTF8). There's no definitive rules for non-ascii data so, for the time being, latin1 (iso-8859-1) is assumed. You can specify an alternative charset in <mount> and you can supply a charset parameter in the /admin/metadata request to do conversions.
I haven't seen any autodetection mechanism that works to enable converting from an unknown charset to UTF8, namely because the data that you do get may translate to valid but incorrect characters.
look at my website address for specific builds, the -kh releases are on there and I have a trunk build for win32, the directory containing nightly snapshots for the trunk source code is also on there.
karl. |
|
| Back to top |
|
|
Anonymous
Guest
|
| Posted: Wed Jan 02, 2008 2:03 pm Post subject: |
|
|
| karlH wrote: |
look at my website address for specific builds, the -kh releases are on there and I have a trunk build for win32, the directory containing nightly snapshots for the trunk source code is also on there.
karl. |
Thanks! I made some research on my problem and need some advise from an expert. I have Icecast 2.3.1 running under Gentoo.
I have specified charset in mout section of icecasts's config. (You told me earlier that only -kh release does support charset)
All playing files have id3v1, id3v2 tags filled in CP1251.
Some files have only id3v1 tags filled the others have id3v1 & id3v2 filled (with the same information)
If mp3 file has only id3v1 tags in CP1251 the status.xsl shows correct information, the tags are extracted correctly and the displayed information is just fine (I don't know how it happens since icecast 2.3.1 does not support charset option???, (it looks like gentoo release does suport it??))
If mp3 file has id3v1 & id3v2 tags in CP1251 the status.xsl shows corrupted information. And it is not simply all corrupted, it is partially corrupted, one tag could be normal, the other one could be courrputed in unreadable way.
To discard possible browser influence I've checked raw results with curl and there is positively no influence from my browsers.
Could you tell me what this problem could be related to - iceS or icecast?
Why, if 2.3.1 doesn't support charset the tid3v1 tags in CP1251 are shown correctly in status.xsl.
PS I understand that many people write here only when they have problems and I would like to tell you that icecast is a cool software, thanks for keeping it up! |
|
| Back to top |
|
|
karlH
Code Warrior
Joined: 13 Jun 2005
Posts: 5476
Location: UK
|
| Posted: Wed Jan 02, 2008 3:07 pm Post subject: |
|
|
Just to clear up the status of <charset>, the -kh releases have had the setting available for some time but was merged into the trunk code a few months back.
The question you need to determine is, what charset is metadata from the source client sent as? If the metadata is in UTF8 then 2.3.1 will be fine and kh/trunk with charset of CP1251 will be wrong (a charset of utf8 will give you want you had in 2.3.1).
Could it be possible that you have one id3 tag in CP1251 and the other id3 tag in UTF8?
karl. |
|
| Back to top |
|
|
Anonymous
Guest
|
| Posted: Wed Jan 02, 2008 3:39 pm Post subject: |
|
|
| karlH wrote: |
Could it be possible that you have one id3 tag in CP1251 and the other id3 tag in UTF8?
karl. |
No, I thought that might be the case and I've checked id3 tags in the questionalbe files and they are all in CP1251, that's granted. Right now I'm going to turn on verbose mode is IceS, hope this will clarify the case. |
|
| Back to top |
|
|
karlH
Code Warrior
Joined: 13 Jun 2005
Posts: 5476
Location: UK
|
| Posted: Wed Jan 02, 2008 4:03 pm Post subject: |
|
|
check the error log in icecast as well, that should show the metadata
karl. |
|
| Back to top |
|
|
Anonymous
Guest
|
| Posted: Wed Jan 02, 2008 4:15 pm Post subject: |
|
|
| karlH wrote: |
Could it be possible that you have one id3 tag in CP1251 and the other id3 tag in UTF8?
karl. |
After turning on verbose mode the status of the case is confirmed.
Here comes the log file from IceS when it is playing file that will have corrupted information in status.xsl
| Code: |
DEBUG: ID3v1: Title: Нам обоим (Конец)
DEBUG: ID3v1: Artist: VerMat
DEBUG: ID3v2: version 3.0. Tag size is 100363 bytes.
DEBUG: ID3v2: Title found: Нам обоим (Конце)
DEBUG: ID3v2: Artist found: VerMat
|
see, it has ID3v1 & ID3v2 tags all filled in (they are all in CP1251 and IceS recognized them OK)
Another file with "incorrect" idv3 display
| Code: |
DEBUG: ID3v1: Title: М
DEBUG: ID3v1: Artist: VerMat
DEBUG: ID3v2: version 3.0. Tag size is 122414 bytes.
DEBUG: ID3v2: Title found: М
DEBUG: ID3v2: Artist found: VerMat
|
It is interesting to see that though IDV3 tags do not contain any CP1251 data the status.xsl still shows corrupted data.
another "bugged" output
| Code: |
DEBUG: ID3v1: Title: Переживание
DEBUG: ID3v1: Artist: VerMat
DEBUG: ID3v2: version 3.0. Tag size is 122424 bytes.
DEBUG: ID3v2: Title found: Переживание
DEBUG: ID3v2: Artist found: VerMat
|
I've noticed that on this particular file when I press Ctrl+R, for this mount point the information status.xls shows is always ends differently. The number of unreadable characters differs with each Ctrl+R. i.e.
| Code: |
vermart 98sdfasfd
vermart 0x98sdfasfdssadf
vermart 0x98sdfasfdssadf
|
it always stars with the correct data and ends differently as though there is an unchecked buffer or something... the file icecast playing is the same.
Now comes the correct file. It doesn't have any idv3 tags filled and information shown by icecast consists of the file name, the file name is correct.
Now another correct file
| Code: |
DEBUG: ID3v1: Title: WEST.mp3
DEBUG: ID3v1: Artist: Pustylnik
|
it doesn't have ID3v2 tags....
and again comes uncorrupted file,
| Code: |
DEBUG: ID3v1: Title: Summer is Gone
DEBUG: ID3v1: Artist: Zentaura&MaxComa
|
it has no ID3v2 tags filled in.
And the most interesting file:
| Code: |
DEBUG: ID3v1: Title: Девочка
DEBUG: ID3v1: Artist: MaxComa
|
it has cp1251 tags but only ID3v1 tags are filled in. The displayed information is correct.
So it looks like the problem occurs when we have both id3v1 and id3v2 data sent to icecast. When only ID3v1 tags are present - everything is fine. When ID3v1 & ID3v2 tags are present - the data is corrupted, no matter what codepage is (the last is just a guess).
May be the bug was introduced with iconv library support (charset). Could you comment on this issue? May by I just did wrong conclusions.
[/quote] |
|
| Back to top |
|
|
Anonymous
Guest
|
| Posted: Wed Jan 02, 2008 4:36 pm Post subject: |
|
|
| karlH wrote: |
check the error log in icecast as well, that should show the metadata
karl. |
[pre]
Here is the snippets from icecast error log. It is mostly filled with
[2008-01-02 19:30:02] DBUG stats/modify_node_event update node total_bytes_sent (0)
[2008-01-02 19:30:02] DBUG stats/modify_node_event update node total_bytes_read (36758597)
[2008-01-02 19:30:02] DBUG stats/modify_node_event update node total_bytes_sent (0)
[2008-01-02 19:30:02] DBUG stats/modify_node_event update node total_bytes_read (26557983)
[2008-01-02 19:30:02] DBUG stats/modify_node_event update node total_bytes_sent (0)
[2008-01-02 19:30:02] DBUG stats/modify_node_event update node total_bytes_read (26357876)
[2008-01-02 19:30:02] DBUG stats/modify_node_event update node total_bytes_sent (0)
[/pre]
sometimes there are records like:
[pre]
[2008-01-02 19:29:41] DBUG stats/modify_node_event update node title (Дмитрий Мрачный - Нарисованные чувства)
[/pre]
Дмитрий Мрачный - Нарисованные чувства - is CP1251, Author Name - Track name
the information is in CP1251 and is a correct one. There are no events of other type in icecast's errorlog.
(I have no idea what they mean) |
|
| Back to top |
|
|
karlH
Code Warrior
Joined: 13 Jun 2005
Posts: 5476
Location: UK
|
| Posted: Wed Jan 02, 2008 7:10 pm Post subject: |
|
|
I've tried emailing you but it bounces
You have to be careful when checking these characters. They may appear correct on your terminal display but I don't know what your terminal display is set to, it could be UTF8 and the charset on the webpage is Latin1. The cp1251 tables are available to look at so showing how a particular character is converted through from the mp3 file to icecast will help. I'd need an example of a failing case.
karl. |
|
| Back to top |
|
|
Anonymous
Guest
|
| Posted: Wed Jan 02, 2008 8:53 pm Post subject: |
|
|
| karlH wrote: |
I've tried emailing you but it bounces
You have to be careful when checking these characters. They may appear correct on your terminal display but I don't know what your terminal display is set to, it could be UTF8 and the charset on the webpage is Latin1. The cp1251 tables are available to look at so showing how a particular character is converted through from the mp3 file to icecast will help. I'd need an example of a failing case.
karl. |
I'm pretty sure that my terminal is working in CP1251, since I've been using FAR's SSH client from XP. I can provide you with failing mp3 files and with a good ones. I've just sent you e-mail at karl @ xiph with the subject "C3PO". (Sorry we've a long holiday here due to New Year and my box was overquoted). I'll all so provide you with screen shots, of "unreadable" output. Thanks for your help. If you received my e-mail please answer back. If it will bounce by other reason than overquoting see the second e-mail with my other boxes. |
|
| Back to top |
|
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
