Icecast Streaming Media Server

[DJ-Zath]

hi Karl and gang!

I have an issue with Icecast "trunk" and Fetch that started when I rebuilt my server (it seems).

Without getting into things too deeply, I have an eggdrop that calls for stats from Icecast once every 2 seconds, and it does this through using "fetch" to call through the admin console via its auth connection on localhost; the eggdrop, fetch, and Icecast are all on the same box.

All works well, except, I have noticed lately that the connections between Icecast and Fetch seem to be getting severed before "last ack" can be sent/received... the result is, theres a bunch of "dead connections" pileing up on the tree! The connections DO eventually "time out", but at 2 mins or so per connection, the "backlog" of them piles up into the HUNDREDS!

Changing the time cycle on the eggdrop helps to reduce this number- but that can only go so far.. I mean, setting the eggdrop to call for stats once every 5 MINUTES is just "too infrequent".. and then entire songs (and other "twitch" data) are completely MISSED in the stats.

but, even setting it to 30 seconds to a minute will still result in over 70+ "dead connections" still hanging about..

So, I'm kinda puzzled as to why this is happening, and why NOW since the server rebuild.. the libs are the same, and the OS is the same.. and even Icecast trunk is the same.. (as far as I know) yet now, this issue has come up..

There is a firewall, and its settings are the same as before, too..

Connections work, the streams work.. the bot works.. all seems to work- except the sockets/connections aren't CLOSING; they seem to be stuck at "last_ack" and they really pile up QUICK!

it may also be that external connectios to Icecast are also "getting cut off" from last_ack as well..

any suggestions would help!

thanks!

-DjZ-

_________________
-DjZ-

[karlH]

This does not sound like a an application thing, looks more like a firewall or OS bug. The last_ack state is the last state really, the server has sent a FIN and is waiting for the ack to come back, timing out eventually.

The strange aspect of this is that it should be over localhost and therefore avoid the firewall. You should be able to stress the case with multiple requests for xml data with curl/wget etc and monitor the loopback interface for the packets. I've seen a couple of old comments about this on FBSD but whether you are running that I don't know.

karl.

[DJ-Zath]

Hi Karl!

Yeah, its FBSD...

but, I DID find the culprit!

PROBLEM:

in FBSD, if you set your "kernel security level" to 3 (anything higher than 1..), the kernel will set the firewall state to "block all, then allow" and..

THIS INCLUDES LOCALHOST CONNECTIONS!

I have set "kern_Secure_level=3" and "set Blackhole=2" and, although, I knew it affected localhost, and I DID set a rule to allow passage for icecast's port... the problem came from the "other numbered" port on the RETURN!

let me see if I can explain this in an example:

a rule exists to:

"allow tcp/udp in/out on port 8000, localhost"

but after this connection, another "return" port on some OTHER number opens as well... and it is COMPLETELY REJECTED/BLOCKED!

netstat:

tcp4 0 0 localhost.8000 localhost.2578 ESTABLISHED
tcp4 0 183 localhost.2578 localhost.8000 ESTABLISHED

The one port opens, no problem.. because the rule allows it.. but it was the OTHER port which was NOT ruled.. that was being "dropped"

and that was the PROBLEM!

SOLUTION:

For this, it was easy:

"allow TCP/UDP to/from ALL ports via localhost"

Now BOTH ports are opening and "established" (as you can see in the example) and, even at 1 sec to 1 sec, everything closes properly and PERFECTLY there are NO pileups!

BUT...

although this works well for localhost, I will have a challenge to figure out for another set of ports from an external machine... though I think an ip-based ruleset will take care of that.. the point, here, is to NOT open a "random return" FROM THE INTERNET on ANY port; that would not be wise.. and, in fact DEFEATS the firewall in the first place! but this is beyond the scope of this post...

The important thing is, for anyone running Icecast on FBSD.. be VERY CAREFUL on how you set up your security; you can, in fact, be TOO secure for Icecast to operate properly!

Thanks karl for the reply!

UPDATE:
Actually, I didn't explain myself correctly, let me try this again:

when you open a connection on localhost, it uses TWO ports- one incoming; the other outgoing (even though its still localhost) one of these ports will be random, of course.. its THAT port that doesn't get passage, though.. so its basically a one-way trip! DING...

an external connectio won't have this problem because the "other port" is on the remote machine.. I get it!
_________________
-DjZ-

[karlH]

ah so on the fetch side, receiving that last FIN triggers the firewall to re-block that terminating "fetch" port before it has had a chance to send the last_ack back to icecast.

karl.

[DJ-Zath]

you got it!

and.. another update...

I have confirmed I have the problem on EVERY port- incoming and outgoing as well.. at this point I have to conclude either I have a mis-configured firewall, or the firewall is broken...

more on this later...
_________________
-DjZ-

[DJ-Zath]

UPDATE:

althogh Icecast is working "as it should" and the ports are "wide open" on the localhost side, the problem still exists on incoming connections.. ANY and ALL incoming connections.. Icecast is hanging the sockets to said connections.. and they really do "pile up" after a while!

I don't think theres a firewall misconfiiguration, nor, is the firewall broken; its EXACTLY what the firewall is SUPPOSED to do!... Its terminating the connections AT ONCE.. and I think Icecast doesn't know (or understand) that aspect of it or perhaps it simply doesn't know the connections were severed!

As you said in the post above, Karl... thats EXACTLY whats happening.

perhaps a setting of <port_timeout> needs to be added?
_________________
-DjZ-

[karlH]

TCP applications do not see the individual state changes of sockets. There's a 3 stage handshake at the beginning and something similar during the shutdown of the socket. The state of the sockets you described before does indicate that the sockets were closed from an icecast point of view, but the OS maintains those sockets to make sure the last remaining data (if any) and closure is confirmed as received by the peer and that has a timeout. While that timeout could be lessened it does not fix the cause of the problem, which is that the normal path for closing sockets is not completed.

karl.