Icecast Streaming Media Server

[Anonymous]

Hi all.
Any hint about logging single ip access to icecast with date and connection time?
That is need for copyright fees.


Thanks

[karlH]

Can you be clear on what you are asking, is there some information that the access log does not provide that you think it should?

karl.

[Anonymous]

Sure. The access log file looks unreadable to the excel guy; he just need to know 3 points: ip (user), date and time of connection to calculate copyright fees.

After reading http://jwz.livejournal.com/637628.html I'm trying a program (sawmill) to make the reports more readable ...
The problem is that the log reports too many field. Here is an example ->

62.110.136.201 - - [13/Jun/2006:06:06:56 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 3
217.133.103.35 - - [13/Jun/2006:06:06:56 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 3
81.208.36.85 - - [13/Jun/2006:06:06:56 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 0
62.110.136.201 - - [13/Jun/2006:06:06:56 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 0
217.133.103.35 - - [13/Jun/2006:06:06:56 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 0
81.208.36.85 - - [13/Jun/2006:06:06:56 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 0
62.110.136.201 - - [13/Jun/2006:06:06:57 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 1
217.133.103.35 - - [13/Jun/2006:06:06:57 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 1
81.208.36.85 - - [13/Jun/2006:06:06:57 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 1
217.133.103.35 - - [13/Jun/2006:06:06:57 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 0
81.208.36.85 - - [13/Jun/2006:06:06:57 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 0
81.208.36.85 - - [13/Jun/2006:06:06:57 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 0
217.133.103.35 - - [13/Jun/2006:06:06:57 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 0
81.208.36.85 - - [13/Jun/2006:06:06:57 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 0

Reports many lines every second. This way we are having giga of logs in a few month. 26k lines just this morning, a 2mega log from 6 oclock.
I need to configure icecast to report only the 3 data needed and avoid reporting every single breath.
For some reason also the duration of the connection cant be calculated by sawmill from the log.


Thx for answer.

[unnamed_devel]

u can try write some script to automate processing of huge logs. we have this problem too.
After investigating trouble we write solution for our needs:
1. move one-day(original) log to some temporary place
2. delete original log
3. tell icecast to restart logging (done with simple wget request, NOT with usual SIGHUP)
4. parse logs info SQL-database
5. delete parsed log.

this script run on crond every one day and able to process huge amounts of data :)

maybe developers of icecast will simplify this solution. for fist do standart unix-thing: SIGHUP must do config reload AND reopen all log files.

[karlH]

logs can be cycled based on file size are archived with date

based on the example provided

62.110.136.201 - - [13/Jun/2006:06:06:56 +0200] "GET /made HTTP/1.1" 404 106 "-" "Barix Exstreamer" 3

you have IP, not used, no username, time of exit, request, http code, bytes sent, referrer, user agent, duration of connection.

The 404 is saying that /made is unknown at that time of the request and duration is to within a second. It's very close to the CLF spec.

karl.

[Anonymous]

Hi unnamed_devel. Your idea of parsing the logs to a db is a good one, but not actualy the priority. As a first we need to know the duratoin o connection.

karl, sorry about that, but what is CLF spec.?
Also about /made there is a 404 unknown error but the service is working properly, this is may be couse "user agent" is "Baris Exstreamer" a streaming device.
Finaly the duration connection which is 3 (meant in seconds or milliseconds?). How is this calculated? I mean we have a "time of exit" but no "time of enter": here there is an error: streaming to this ip lasts for many hours a day.
Here another line of log:
217.133.103.35 - - [13/Jun/2006:09:28:18 +0200] "GET /made HTTP/1.1" 200 7367036 "-" "Barix Exstreamer" 651
This was reports more time.
And a couple more lines from an http client...
213.175.2.50 - - [13/Jun/2006:09:13:00 +0200] "GET /e-fm HTTP/1.0" 200 148650 "-" "NSPlayer/8.0.0.4487" 7
62.101.126.208 - - [13/Jun/2006:09:13:02 +0200] "GET /e-fm HTTP/1.1" 200 8650 "-" "NSPlayer/11.0.5358.4827 WMFSDK/11.0" 1
Here again time isnt actualy true. This ones are from www.e-fm.it.

I missed the docs with the specifics about the log fields you just declared in your post (ip-not used-no username-timeofexit..and so on) and more info on logging; any link?

Thx for your answer.
Cheers
Jo

[karlH]

CLF - common log format, check the apache web site

I can't comment on the reason for why the 404 occurs, but listening app is connecting, requesting /made and is being sent 404 indicateing that it's not there. There may be other mounpoint in use at the time but /made is not.

the 3 is in seconds, I don't think there is much point is referring to milliseconds. When a client connects, the time in seconds is taken and at exit the difference is reported. The time of entering has to be calculated.

karl.

[Anonymous]

[unnamed_devel]

2snowbird: that question u must ask icecast developers :) I hope only he can *FULLY* document all features of moster server :)

After digging thru sources i found *NON STANDART* (say: not a true linux way) solution to reset (reopen) logs.

u must point your browser (or wget, or other tool) to http://your-server:port/admin/function.xsl?perform=reopenlog

after that - your logs will be recreated :)

I mean standart linux-way is doing this job on SIGHUP signal :)

[karlH]

This should be clarified, the reopen of logs via a url is only currently available in the -kh branch. kh6c is the latest version (on http://karl.mediacast1.com), I've also managed to build a version for win32. It's not documented because basically I haven't got around to it. You can also reread the xml and apply settings to mountpoints just like the unix platforms by using ?perform=updatecfg.

karl.