| View previous topic :: View next topic |
| Author |
Message |
Anonymous Guest
|
Posted: Fri Mar 10, 2006 8:28 am Post subject: Icecast 2.3.1 optimization help for large server needed |
|
|
| I'm running Icecast 2.3.1 for a group of radio stations, and am having trouble with Icecast's status.xsl page seeming to die periodically (it works one minute, and then you get a page not found -- only way to get it back is to restart the icecast process). I'm also experiencing a bandwidth bottleneck somewhere that is limiting the number of simultaneous listeners to the streams, even though the total number of listeners on the server is less than half of the maximums specified in the .xml config file. The system is a dual xeon 2.4GHz system running SuSE 9.2 Professional, connected to a 100Mb pipe to the Internet (and it is a full 100Mb pipe costing me $2000 a month). This server is hosting 12 MP3/MP3pro streams ranging from 16k to 96k in bandwidth, and nearly 2200 simultaneous users (and growing). Average throughput is around 45Mbs, before traffic seems to bottleneck. Hardware has been ruled out, as well as things on my server co-location end. Any suggestions on where to start looking at problems? |
|
| Back to top |
|
 |
karlH Code Warrior

Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Fri Mar 10, 2006 11:27 am Post subject: |
|
|
The error log is the place to look for failures or limits being reached. In these error cases what does icecast log.
karl. |
|
| Back to top |
|
 |
Anonymous Guest
|
Posted: Fri Mar 10, 2006 3:48 pm Post subject: |
|
|
Here's a snippit from the error log...
| Quote: |
[2006-03-02 13:27:16] WARN fserve/fserve_client_create req for file "/web/wayh_high.smi" No such file or directory
[2006-03-02 13:29:59] WARN fserve/fserve_client_create req for file "/web/wayw_high.smi" No such file or directory
[2006-03-02 13:40:39] WARN fserve/fserve_client_create req for file "/web/wayw_high.smi" No such file or directory
[2006-03-02 13:45:30] WARN fserve/fserve_client_create req for file "/web/wayf_high.smi" No such file or directory
[2006-03-02 13:47:18] WARN fserve/fserve_client_create req for file "/web/wayf_high.smi" No such file or directory
[2006-03-02 13:48:34] WARN fserve/fserve_client_create req for file "/web/wayw_high.smi" No such file or directory
[2006-03-02 14:01:00] WARN client/client_read_bytes source connection has died
[2006-03-02 14:01:02] WARN client/client_read_bytes source connection has died
[2006-03-02 14:03:22] WARN fserve/fserve_client_create req for file "/web/wayf_high.smi" No such file or directory
[2006-03-02 14:05:33] WARN client/client_read_bytes source connection has died
[2006-03-02 14:07:04] WARN fserve/fserve_client_create req for file "/web/wayw_high.smi" No such file or directory
[2006-03-02 14:12:07] WARN fserve/fserve_client_create req for file "/web/wayf_high.smi" No such file or directory
[2006-03-02 14:13:55] WARN client/client_read_bytes source connection has died
[2006-03-02 14:14:04] WARN client/client_read_bytes source connection has died
[2006-03-02 14:25:55] WARN fserve/fserve_client_create req for file "/web/wayw_high.smi" No such file or directory
[2006-03-02 14:28:09] WARN fserve/fserve_client_create req for file "/web/wayw_high.smi" No such file or directory
[2006-03-02 14:28:40] WARN fserve/fserve_client_create req for file "/web/wayh_high.smi" No such file or directory
[2006-03-02 14:28:59] WARN fserve/fserve_client_create req for file "/web/wayf_high.smi" No such file or directory
[2006-03-02 14:32:33] WARN fserve/fserve_client_create req for file "/web/wayf_high.smi" No such file or directory
[2006-03-02 14:41:27] WARN fserve/fserve_client_create req for file "/web/wayw_high.smi" No such file or directory
[2006-03-02 14:48:05] WARN fserve/fserve_client_create req for file "/web/wayw_high.smi" No such file or directory
[2006-03-02 14:49:31] WARN fserve/fserve_client_create req for file "/web/wayf_high.smi" No such file or directory
[2006-03-02 14:57:28] WARN fserve/fserve_client_create req for file "/web/wayf_high.smi" No such file or directory
[2006-03-02 15:15:26] WARN fserve/fserve_client_create req for file "/web/kxwa_high.smi" No such file or directory
[2006-03-02 15:40:57] WARN fserve/fserve_client_create req for file "/web/wayh_high.smi" No such file or directory
[2006-03-02 15:51:18] WARN fserve/fserve_client_create req for file "/web/kxwa_high.smi" No such file or directory
[2006-03-02 16:09:28] WARN fserve/fserve_client_create req for file "/web/kxwa_high.smi" No such file or directory
[2006-03-02 16:15:05] WARN fserve/fserve_client_create req for file "/web/kxwa_high.smi" No such file or directory
[2006-03-02 16:28:14] WARN fserve/fserve_client_create req for file "/web/wayw_high.smi" No such file or directory
[2006-03-02 16:31:06] WARN fserve/fserve_client_create req for file "/web/wayf_high.smi" No such file or directory
[2006-03-02 16:32:53] WARN fserve/fserve_client_create req for file "/web/wayf_high.smi" No such file or directory
[2006-03-02 16:37:31] WARN fserve/fserve_client_create req for file "/web/wayw_high.smi" No such file or directory
[2006-03-02 16:52:26] WARN fserve/fserve_client_create req for file "/web/wayw_high.smi" No such file or directory
[2006-03-02 17:03:05] WARN fserve/fserve_client_create req for file "/web/wayf_high.smi" No such file or directory
[2006-03-02 17:03:47] WARN fserve/fserve_client_create req for file "/web/kxwa_high.smi" No such file or directory
[2006-03-02 17:19:31] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:19:31] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:18] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:18] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:19] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:19] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:20] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:20] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:20] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:20] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:21] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:21] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:21] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:21] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:21] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:22] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:22] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:23] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:23] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:23] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:24] WARN connection/_accept_connection accept() failed with error 24: Too many open files
[2006-03-02 17:20:24] WARN connection/_accept_connection accept() failed with error 24: Too many open files
|
Once the "Too many open files" line appears in the log, Icecast will no longer display any status page, the admin interface dies, and it will start dropping listeners over time. In Icecast.xml, the max listeners parameter is set to 3000, while just before the "crash", there were only 1500-1600 listeners to the various streams. There are no relays and no files being served by icecast's webserver, other than the status page. |
|
| Back to top |
|
 |
karlH Code Warrior

Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Fri Mar 10, 2006 5:13 pm Post subject: |
|
|
Do you know what your process file descriptor limit is ? The accept fails because that is being reached and it's not an icecast setting. The setting, if set, will be in /etc/security/limits.conf (nofile)
Can you also try some fixes that have been collected since the 2.3.1 release, I have a tarball at http://mediacast1.com/~karl/testing/icecast-2.3.1a.tar.gz
karl. |
|
| Back to top |
|
 |
Anonymous Guest
|
Posted: Fri Mar 10, 2006 7:36 pm Post subject: |
|
|
I don't know what the file descriptor limit is, unfortunately. The install is an out-of-the-box Suse 9.2 Professional load, with the usual security updates and some lockdowns to keep the script kiddies out. /etc/security/limits.conf is set at its defaults, with all lines commented out. Is there another place I should look?
I'll check out the fixes and see if any help. Thanks for your help on all of this! |
|
| Back to top |
|
 |
karlH Code Warrior

Joined: 13 Jun 2005 Posts: 5476 Location: UK
|
Posted: Fri Mar 10, 2006 8:22 pm Post subject: |
|
|
The limits file states various limits that you want to impose, ulimit -a and ulimit -aH will show what the soft/hard limits currently are but typically the limit has been 1024 descriptors. I guess you will need to increase this limit if you want a lot more listeners to connect.
karl. |
|
| Back to top |
|
 |
Anonymous Guest
|
Posted: Fri Mar 10, 2006 11:12 pm Post subject: |
|
|
Check the file-max entry in proc:
| Code: |
cat /proc/sys/fs/file-max
|
And you can raise the system default value from /etc/sysctl.conf
add the following line to it:
| Code: |
sys.fs.file-max=VALUE
|
And run the sysctl program to apply:
If you run icecast as a non-root user, you have the raise the ulimit. This can't be higher than the system value displayed in proc. |
|
| Back to top |
|
 |
Anonymous Guest
|
Posted: Sat Mar 11, 2006 5:52 pm Post subject: |
|
|
/proc/sys/fs/file-max was set to over 200000. Trying to use /etc/security/limits.conf didn't work either at first. I ended up doing a little more googling and found an article about Oracle and the mods needed to make the Oracle dB work on Suse 9.2. Turns out you have to install the ulimits package using Yast2 in order to control anything with the ulimit command, otherwise you will get a "operation not permitted" error. Going to test this over the next week and see if there is any improvement.
Thanks again. |
|
| Back to top |
|
 |
|