Icecast Streaming Media Server

[unnamed_devel]

Hi guys, i really need some help from U...

I have servers that stream live with national-characters in metadata.
My contry codepage is win-cp1251. In players (example: jetaudio, winamp, windows media player) i get correct streamname and metadata infos.... but in icecast stats and admin page i will get something like "$&%^".

if i request untranslated (XML) stats it will have an errorneus characters in metadata sections AND xml-file isn't COMPLETE: data stops after errorneus characters. example: </server_description><server_name>Êˀè


I think that logic:
source client stream some data with metadata in national characters in it (cp1251 for example).
icecast2 will receive it and send over all clients.
and of course icecast will simply copy-paste it in XML-document!
This is NOT correct way because xml-specifications says we need to CONVERT characters to proper style. Something like ˀłÈ
and specify encoding="UTF8" or simular but not you currently default "ISO-8859-1"


basically: icecast2 deal with national characters, but XML-part of it - ISN'T


How soon there will be a correct version of icecast?
or i will need to handy re-write XML-part of icecast2 from a scratch?

[MikeS]

It sounds like the problem is that icecast is (incorrectly) accepting invalid data from source clients. You didn't say what format you were streaming in, or how the stream metadata was being passed from the source client to the icecast server, so I can't give much more detail on that part.

However, if we fixed that bug, you'd just get no metadata at all, because icecast would be refusing it.

Icecast will not accept incoming metadata in anything other than unicode, so it's up to the source client to do this, if neccesary.

Mike

[karlH]

For ogg and xml, the standard used is UTF-8 as that deals with the different character sets nicely, for mp3 metadata though, well nothing was defined, and little information is passed (if any) about the character set to convert from. Feel free to discuss a possible solution to this.

karl.

[unnamed_devel]

2MikeS:

im specially write about *CORRECT* displayning of all metadata characters in *ALL* listeners clients (wmp, winamp, z-play, jetaudio, etc).
I think there is BIG problem with icecast2 XML-part logic. :)
Listen man, if i pass *incorrect* data (in metadata!) to icecast2 server HOW *almost all* listener clients will *CORRECTLY* receive and display that data? :)
Understood, icecast2 will CORRECTLY broadcast received from a source metadata characters "AS-IS" to all clients! The problem is in the /admin/ interface (xml).
My source protocol is mp3 data over HTTP protocol.



2KarlH:

Hi again man! I loved your "icecast technology" about ....hmmm 5 years!
Thanks for it! Now i want to carry in some useful things because it MUST be number one in the world! (hi-customizable)

My suggestion logic is this:

after running icecast2 we all have one GLOBAL structure holding ALL our important stats. By the way that stat will include stream names and other codepage-sensitivity data.
We MUST:
1. after *ESTABILISHING* source client connection, convert data received from source to XML-standart (UTF8). after that - pass it in GLOBAL XML-structure. in fuction apply_mountpoint_data thats may be good place KarlH?

2. after EACH receive of matadata fragment from source we must convert received data to XML-standart (UTF8). and after that - pass it to our GLOBAL XML-stats structure.

3. functions that broadcast metadata (and audio data) to listeners clients *MUST* pass data "as-is" received from a source! (same way as do it now!)


if we do this - we got correct deal with listeners cliens AND admin-clients like browsers (/admin/* pageS)

[karlH]

There are several points need clarifying before any non-utf8 mp3 processing is done

a xml/xsl page often contain details from multiple streams, so conversion to a common spec (ie utf8) is needed.

"convert data received from source to XML-standart (UTF8)"
A determination of what the source character set is needs defining, each time I've managed to check the information passed, no indication of the character set is given. Maybe a per-mount tag would be sufficient if source clients are inflexable enough.

Are you sure passing metadata data through as-is to the player works as well as you inidicate? Again I think the listener doesn't get enough information to always work.

karl.

[unnamed_devel]

>>There are several points need clarifying before any non-utf8 mp3 >>processing is done

Yes! Of course, KarlH! We need some clarifying...


>>a xml/xsl page often contain details from multiple streams, so >>conversion to a common spec (ie utf8) is needed.

Yes, I also spoke about it! Because if server internally works with XML-documents we must correctly follows XML-specifications.



>>"convert data received from source to XML-standart (UTF8)"
>>A determination of what the source character set is needs defining, each >>time I've managed to check the information passed, no indication of the >>character set is given. Maybe a per-mount tag would be sufficient if >>source clients are inflexable enough.

why? Currently NO ONE client players support codepage setting by information defined in 'stream protocol' (i.e. x-audiocast, shoutcast icy, icecast2 ice (http))
i consider for now we must pass metadata over icecast2 servers "as-is" for LISTENERS clients(!) and convert metadata chars to univesal UTF-8 (for example) in server heart to correctly deal withing data in /admin/* interface.



i've looked once again at icecast2 sources and investigating libxml2 API...
probably we need to use module 'encoding' from libxml2
funtions like: isolat1ToUTF8(), UTF8Toisolat1()

injection place in icecast2 sources starts at
format_mp3.c function filter_shoutcast_metadata()
{
....
logging_playlist (source->mount, p, source->listeners);
stats_event (source->mount, "title", p);
yp_touch (source->mount);
....
}

BEFORE doing stats_event(source->mount, "title", p) we MUST call isolat1ToUTF8() and only after that pass properly encoded data to stats_event() function!


SAME pathing need be done (or OGG streaming internally works with UTF8???)
format_ogg.c function update_comments()
{
...
stats_event (source->mount, "artist", artist);
stats_event (source->mount, "title", title);
...
}

encode 'artist' and 'title' strings to UTF8 by calling isolat1ToUTF8() and when call to stats_event()


probably in icecast2 server we have another place(s) where we need carefuly deal with encoding. investigating admin.c (the /admin/* interface)
gives me function command_metadata().
consider if we are changing metadata on mount (source) throught BROWSER we are using UTF8....but maybe we need additionaly parse HTTP protocol headers (or any other) to identify encoding. logic may be like this:
function command_metadata()
...
if (encoding_of_HTTP_GET_METHOD == UTF8) {
UTF8Toisolat1()
}
....

[unnamed_devel]

source.c
we must patch function source_apply_mount() too:

BEFORE any of those 3 functions do isolat1ToUTF8():

stats_event (source->mount, "server_name", str);

stats_event (source->mount, "server_description", str);

stats_event (source->mount, "genre", str);