Icecast Streaming Media Server

[rserling]

We're serving 6 live mp3 streams at varying rates from 48 to 92kbps, all sourced from an AudioTX WebStream by MDOUK (v1.0a-AAC). We have a long-standing problem with icecast apparently using up all available sockets on its local server and hanging the network interface. Anywhere from 2 to 12 hours of streaming (yes, it varies arbitrarily) brings it to some sort of limit and all clients have to be dropped before the interface will respond. It more commonly happens at peak usage times (around 1000 concurrent listeners), but not always. The local desktop gets a beachball during these events and is generally extremely unresponsive, so you can't really monitor it in realtime. Then after about a minute (after all streaming clients have been dropped), everything is great again and all services resume without intervention.

Tweaking of sysctl values (e.g. net.inet.tcp.*, kern.ipc.maxsockbuf) has been required for any sort of icecast reliability, but I haven't found the magic combination to eliminate this problem and it doesn't really seem like there is one. Here's what i've done with that:
net.inet.tcp.sendspace=2097152
net.inet.tcp.recvspace=2097152
kern.ipc.maxsockbuf=8388608
kern.maxproc=2500
kern.maxprocperuid=1500
net.inet.tcp.sockthreshold=64
kern.ipc.somaxconn=2500

It has the general feel of a memory leak, except with sockets. It just seems like icecast accumulates far more sockets than it should need and doesn't give them back. Nothing in the system logs corresponds with these outages. In the icecast error log there are these, but i believe they are just symptoms:
[2010-10-22 10:33:33] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:33:33] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:33:33] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:33:33] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:33:33] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:33:33] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:33:33] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:33:33] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:33:53] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:34:30] WARN connection/_handle_source_request Mountpoint /stream5.mp3 in use
[2010-10-22 10:34:42] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:34:43] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:34:43] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:34:43] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:34:43] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:34:43] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:34:43] WARN source/get_next_buffer Disconnecting source due to socket timeout
[2010-10-22 10:36:24] WARN admin/admin_handle_request Admin command metadata on non-existent source /stream2.mp3

Here's the server:

Model Name: Xserve
Model Identifier: Xserve3,1
Processor Name: Quad-Core Intel Xeon
Processor Speed: 2.26 GHz
Number Of Processors: 2
Total Number Of Cores: 8
L2 Cache (per core): 256 KB
L3 Cache (per processor): 8 MB
Memory: 16 GB
Processor Interconnect Speed: 5.86 GT/s
Boot ROM Version: XS31.0081.B04
SMC Version (system): 1.43f4
LOM Revision: 1.1.8

Here is icecast.xml (genericized):
<icecast>
<limits>
<clients>3500</clients>
<sources>12</sources>
<threadpool>20</threadpool>
<queue-size>102400</queue-size>
<client-timeout>60</client-timeout>
<header-timeout>15</header-timeout>
<source-timeout>10</source-timeout>
<burst-on-connect>1</burst-on-connect>
<burst-size>65536</burst-size>
</limits>

<authentication>
<source-password>********</source-password>
<relay-password>********</relay-password>
<admin-user>admin</admin-user>
<admin-password>********</admin-password>
</authentication>

<hostname>servername.talk.org</hostname>
<listen-socket>
<port>80</port>
</listen-socket>
<listen-socket>
<port>8000</port>
</listen-socket>

<mount>
<mount-name>/radio_player.mp3</mount-name>
<stream-name>TALK Radio</stream-name>
<bitrate>64</bitrate>
<stream-description>Live FM Stream TALK WebPlayer</stream-description>
<max-listeners>700</max-listeners>
</mount>

<mount>
<mount-name>/radio.mp3</mount-name>
<stream-name>TALK Radio</stream-name>
<stream-description>Live FM Stream from TALK</stream-description>
<bitrate>48</bitrate>
<max-listeners>1000</max-listeners>
<intro>/sponsors/talkintro4824.mp3</intro>
<hidden>0</hidden>
</mount>

<mount>
<mount-name>/jazz_player.mp3</mount-name>
<stream-name>JAZZ Radio</stream-name>
<stream-description>JAZZ FM Stream</stream-description>
<bitrate>56</bitrate>
<max-listeners>1000</max-listeners>
<hidden>0</hidden>
</mount>

<mount>
<mount-name>/jazzhbr.mp3</mount-name>
<stream-name>JAZZ Radio</stream-name>
<stream-description>JAZZ FM Stream</stream-description>
<bitrate>96</bitrate>
<max-listeners>1000</max-listeners>
<hidden>0</hidden>
</mount>

<mount>
<mount-name>/jazz.mp3</mount-name>
<stream-name>JAZZ Radio</stream-name>
<stream-description>JAZZ FM Stream</stream-description>
<bitrate>96</bitrate>
<max-listeners>1000</max-listeners>
<hidden>0</hidden>
</mount>

<fileserve>1</fileserve>

<paths>
<alias source="/" dest="/status.xsl"/>
</paths>

<logging>
<accesslog>access.log</accesslog>
<errorlog>error.log</errorlog>
<playlistlog>playlist.log</playlistlog>
<loglevel>2</loglevel><!-- 4 Debug, 3 Info, 2 Warn, 1 Error -->
<logsize>20000</logsize> <!-- Max size of a logfile -->
<logarchive>1</logarchive>
</logging>

<security>
<chroot>0</chroot>
<changeowner>
<user>daemon</user>
<group>daemon</group>
</changeowner>
</security>
</icecast>

I've looked at serveral other posts here with similar problems, but they either were different enough to not be applicable or they recommended fixes that didn't work for me. But i may have missed something....

Any hints/tips on troubleshooting this would be greatly appreciated. Specifically, i'd like to know what limit is being reached and (of course) how to increase it.


Thanks!

[karlH]

I suspect you are on the right lines but have not changed the right setting. I've seen this sort of thing before on BSD where a limit like this was reached and that caused major stalls within applications like icecast which are network heavy. I don't know the precise name for the settings, I'd have to see which ones are available to see. It won't be the per-socket settings, more likely the max memory that can be allocated to socket buffers.

karl.

[rserling]

Ok that's somewhat encouraging.

Also of note, in the miserable, helpless minutes leading up to these failures i can see this from netstat -m:
43817 KB allocated to network (76.7% in use)
43817 KB allocated to network (84.4% in use)
43817 KB allocated to network (87.5% in use)
43817 KB allocated to network (92.7% in use)

...and that's about where the shell session hangs. The remote shell session returns if you just wait a minute or two, it is usually not terminated.

Here is a complete list of my sysctl settings/values:
kern.exec: unknown type returned
kern.ostype = Darwin
kern.osrelease = 9.8.0
kern.osrevision = 199506
kern.version = Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386
kern.maxvnodes = 280000
kern.maxproc = 2500
kern.maxfiles = 49152
kern.argmax = 262144
kern.securelevel = 0
kern.hostid = 0
kern.clockrate: hz = 100, tick = 10000, profhz = 100, stathz = 100
kern.posix1version = 200112
kern.ngroups = 16
kern.job_control = 1
kern.saved_ids = 1
kern.boottime = Wed Oct 6 15:30:59 2010

kern.nisdomainname =
kern.maxfilesperproc = 40960
kern.maxprocperuid = 1500
kern.dummy = 0
kern.dummy = 0
kern.usrstack = -1073741824
kern.dummy = 0
kern.netboot = 0
kern.dummy = 0
kern.dummy = 0
kern.aiomax = 90
kern.aioprocmax = 16
kern.aiothreads = 4
kern.corefile = /cores/core.%P
kern.coredump = 1
kern.sugid_coredump = 0
kern.delayterm = 0
kern.shreg_private = 0
kern.proc_low_pri_io = 0
kern.usrstack64 = 3221225472
kern.nx = 1
kern.procname =
kern.speculative_reads_disabled = 0
kern.osversion = 9L34
kern.safeboot = 0
kern.rage_vnode = 0
vfs.nfs has 1 mounted instance
vfs.fdesc has 1 mounted instance
vfs.hfs has 3 mounted instances
vfs.devfs has 1 mounted instance
vfs.autofs has 3 mounted instances
hw.machine = i386
hw.model = Xserve3,1
hw.ncpu = 16
hw.byteorder = 1234
hw.physmem = 2147483648
hw.usermem = 967467008
hw.pagesize = 4096
hw.epoch = 0
hw.vectorunit = 1
hw.busfrequency = 4294967295
hw.cpufrequency = 2260000000
hw.cachelinesize = 64
hw.l1icachesize = 32768
hw.l1dcachesize = 32768
hw.l2settings = 1
hw.l2cachesize = 262144
hw.l3settings = 1
hw.l3cachesize = 8388608
hw.tbfrequency = 1000000000
hw.memsize = 17179869184
hw.availcpu = 16
user.cs_path = /usr/bin:/bin:/usr/sbin:/sbin
user.bc_base_max = 99
user.bc_dim_max = 2048
user.bc_scale_max = 99
user.bc_string_max = 1000
user.coll_weights_max = 2
user.expr_nest_max = 32
user.line_max = 2048
user.re_dup_max = 255
user.posix2_version = 200112
user.posix2_c_bind = 0
user.posix2_c_dev = 0
user.posix2_char_term = 0
user.posix2_fort_dev = 0
user.posix2_fort_run = 0
user.posix2_localedef = 0
user.posix2_sw_dev = 0
user.posix2_upe = 0
user.stream_max = 20
user.tzname_max = 255
kern.ostype: Darwin
kern.osrelease: 9.8.0
kern.osrevision: 199506
kern.version: Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386
kern.maxvnodes: 280000
kern.maxproc: 2500
kern.maxfiles: 49152
kern.argmax: 262144
kern.securelevel: 0
kern.hostid: 0
kern.clockrate: { hz = 100, tick = 10000, tickadj = 1, profhz = 100, stathz = 100 }
kern.posix1version: 200112
kern.ngroups: 16
kern.job_control: 1
kern.saved_ids: 1
kern.boottime: { sec = 1286404259, usec = 0 } Wed Oct 6 15:30:59 2010
kern.nisdomainname:
kern.maxfilesperproc: 40960
kern.maxprocperuid: 1500
kern.ipc.maxsockbuf: 8388608
kern.ipc.sockbuf_waste_factor: 8
kern.ipc.somaxconn: 2500
kern.ipc.nmbclusters: 32768
kern.ipc.soqlimitcompat: 1
kern.ipc.mb_normalized: 0
kern.ipc.sosendjcl_ignore_capab: 0
kern.ipc.sosendjcl: 1
kern.ipc.sorecvmincopy: 16384
kern.ipc.sosendminchain: 16384
kern.ipc.soqlencomp: 0
kern.ipc.njclbytes: 16384
kern.ipc.njcl: 10920
kern.ipc.sbspace_factor: 8
kern.ipc.maxsockets: 512
kern.ipc.sendfileuiobufs: 64
kern.dummy: 0
kern.usrstack: -1073741824
kern.netboot: 0
kern.sysv.shmall: 4096
kern.sysv.shmseg: 8
kern.sysv.shmmni: 32
kern.sysv.shmmin: 1
kern.sysv.shmmax: 8388608
kern.sysv.semume: 10
kern.sysv.semmsl: 87381
kern.sysv.semmnu: 87381
kern.sysv.semmns: 87381
kern.sysv.semmni: 87381
kern.exec.archhandler.powerpc: /usr/libexec/oah/translate
kern.aiomax: 90
kern.aioprocmax: 16
kern.aiothreads: 4
kern.corefile: /cores/core.%P
kern.coredump: 1
kern.sugid_coredump: 0
kern.delayterm: 0
kern.shreg_private: 0
kern.proc_low_pri_io: 0
kern.posix.sem.max: 10000
kern.usrstack64: 3221225472
kern.nx: 1
kern.tfp.policy: 2
kern.procname:
kern.speculative_reads_disabled: 0
kern.osversion: 9L34
kern.safeboot: 0
kern.lctx.max: 8192
kern.lctx.count: 0
kern.lctx.last: 1
kern.rage_vnode: 0
kern.tty.ptmx_max: 127
kern.copyregionmax: 8
kern.lowpagemax: 10953
kern.himemorymode: 0
kern.sleeptime: { sec = 0, usec = 0 } Wed Dec 31 16:00:00 1969
kern.waketime: { sec = 0, usec = 0 } Wed Dec 31 16:00:00 1969
kern.hibernatefile:
kern.bootsignature:
kern.hibernatemode: 0
kern.maxnbuf: 60000
kern.nbuf: 32511
kern.flush_cache_on_write: 0
kern.hfs_last_io_wait_time: 125000
kern.always_do_fullfsync: 0
kern.sugid_scripts: 0
kern.affinity_sets_mapping: 1
kern.affinity_sets_enabled: 1
kern.singleuser: 0
kern.bootargs:
kern.msgbuf: 4096
kern.wq_timer_interval_msecs: 40
kern.wq_max_run_latency_usecs: 500
kern.wq_reduce_pool_window_usecs: 3000000
kern.wq_stalled_window_usecs: 20000
kern.secure_kernel: 0
vm.loadavg: { 0.17 0.18 0.16 }
vm.swapusage: total = 64.00M used = 0.00M free = 64.00M
vm.cs_debug: 0
vm.cs_force_hard: 0
vm.cs_force_kill: 0
vm.user_wire_limit: 17179868184
vm.global_user_wire_limit: 17179868184
vm.cs_blob_size_max: 29952
vm.cs_blob_size_peak: 1033664
vm.cs_blob_count_peak: 171
vm.cs_blob_size: 874896
vm.cs_blob_count: 141
vm.cs_validation: 1
vm.vm_page_free_target: 2000
vm.shared_region_persistence: 1
vm.shared_region_version: 3
vm.shared_region_trace_level: 1
vm.allow_data_exec: 1
vm.allow_stack_exec: 0
vfs.generic.nfs.server.nfsd_thread_count: 0
vfs.generic.nfs.server.nfsd_thread_max: 0
vfs.generic.nfs.server.fsevents: 1
vfs.generic.nfs.server.user_stats: 1
vfs.generic.nfs.server.request_queue_length: 128
vfs.generic.nfs.server.reqcache_size: 64
vfs.generic.nfs.server.async: 0
vfs.generic.nfs.server.require_resv_port: 0
vfs.generic.nfs.server.wg_delay_v3: 0
vfs.generic.nfs.server.wg_delay: 1000
vfs.generic.nfs.client.max_async_writes: 128
vfs.generic.nfs.client.lockd_mounts: 0
vfs.generic.nfs.client.nfsiod_thread_count: 0
vfs.generic.nfs.client.nfsiod_thread_max: 16
vfs.generic.nfs.client.statfs_rate_limit: 10
vfs.generic.nfs.client.allow_async: 0
vfs.generic.nfs.client.access_cache_timeout: 60
vfs.generic.nfs.client.iosize: 1048576
vfs.generic.nfs.client.nextdowndelay: 30
vfs.generic.nfs.client.initialdowndelay: 12
net.local.stream.recvspace: 8192
net.local.stream.sendspace: 8192
net.local.dgram.recvspace: 4096
net.local.dgram.maxdgram: 2048
net.local.inflight: 0
net.inet.ip.portrange.hilast: 65535
net.inet.ip.portrange.hifirst: 49152
net.inet.ip.portrange.last: 65535
net.inet.ip.portrange.first: 49152
net.inet.ip.portrange.lowlast: 600
net.inet.ip.portrange.lowfirst: 1023
net.inet.ip.forwarding: 0
net.inet.ip.redirect: 1
net.inet.ip.ttl: 64
net.inet.ip.rtexpire: 10
net.inet.ip.rtminexpire: 10
net.inet.ip.rtmaxcache: 128
net.inet.ip.sourceroute: 0
net.inet.ip.intr_queue_maxlen: 50
net.inet.ip.intr_queue_drops: 0
net.inet.ip.accept_sourceroute: 0
net.inet.ip.fastforwarding: 0
net.inet.ip.keepfaith: 0
net.inet.ip.gifttl: 30
net.inet.ip.subnets_are_local: 0
net.inet.ip.use_route_genid: 1
net.inet.ip.check_route_selfref: 1
net.inet.ip.dummynet.debug: 0
net.inet.ip.dummynet.red_max_pkt_size: 1500
net.inet.ip.dummynet.red_avg_pkt_size: 512
net.inet.ip.dummynet.red_lookup_depth: 256
net.inet.ip.dummynet.max_chain_len: 16
net.inet.ip.dummynet.expire: 1
net.inet.ip.dummynet.search_steps: 0
net.inet.ip.dummynet.searches: 0
net.inet.ip.dummynet.extract_heap: 0
net.inet.ip.dummynet.ready_heap: 0
net.inet.ip.dummynet.curr_time: 0
net.inet.ip.dummynet.hash_size: 64
net.inet.ip.fw.dyn_keepalive: 1
net.inet.ip.fw.dyn_short_lifetime: 5
net.inet.ip.fw.dyn_udp_lifetime: 10
net.inet.ip.fw.dyn_rst_lifetime: 1
net.inet.ip.fw.dyn_fin_lifetime: 1
net.inet.ip.fw.dyn_syn_lifetime: 20
net.inet.ip.fw.dyn_ack_lifetime: 300
net.inet.ip.fw.static_count: 20
net.inet.ip.fw.dyn_max: 4096
net.inet.ip.fw.dyn_count: 0
net.inet.ip.fw.curr_dyn_buckets: 256
net.inet.ip.fw.dyn_buckets: 256
net.inet.ip.fw.verbose_limit: 1000
net.inet.ip.fw.verbose: 2
net.inet.ip.fw.debug: 0
net.inet.ip.fw.one_pass: 0
net.inet.ip.fw.autoinc_step: 100
net.inet.ip.fw.enable: 0
net.inet.ip.random_id: 1
net.inet.ip.linklocal.in.allowbadttl: 1
net.inet.ip.check_interface: 0
net.inet.ip.scopedroute: 0
net.inet.ip.maxfrags: 2048
net.inet.ip.maxfragsperpacket: 128
net.inet.ip.maxfragpackets: 1024
net.inet.ip.select_srcif_debug: 0
net.inet.ip.maxchainsent: 49
net.inet.icmp.maskrepl: 0
net.inet.icmp.icmplim: 250
net.inet.icmp.timestamp: 0
net.inet.icmp.bmcastecho: 1
net.inet.icmp.log_redirect: 0
net.inet.icmp.drop_redirect: 0
net.inet.tcp.rfc1323: 1
net.inet.tcp.rfc1644: 0
net.inet.tcp.mssdflt: 512
net.inet.tcp.keepidle: 7200000
net.inet.tcp.keepintvl: 75000
net.inet.tcp.sendspace: 2097152
net.inet.tcp.recvspace: 2097152
net.inet.tcp.keepinit: 75000
net.inet.tcp.v6mssdflt: 1024
net.inet.tcp.rexmt_thresh: 2
net.inet.tcp.rfc3465: 1
net.inet.tcp.maxseg_unacked: 8
net.inet.tcp.slowlink_wsize: 8192
net.inet.tcp.reass.overflows: 0
net.inet.tcp.reass.cursegments: 0
net.inet.tcp.reass.maxsegments: 2048
net.inet.tcp.drop_synfin: 1
net.inet.tcp.tcp_lq_overflow: 1
net.inet.tcp.delayed_ack: 2
net.inet.tcp.blackhole: 0
net.inet.tcp.log_in_vain: 0
net.inet.tcp.socket_unlocked_on_output: 1
net.inet.tcp.packetchain: 50
net.inet.tcp.ecn_negotiate_in: 0
net.inet.tcp.ecn_initiate_out: 0
net.inet.tcp.newreno: 0
net.inet.tcp.local_slowstart_flightsize: 8
net.inet.tcp.slowstart_flightsize: 1
net.inet.tcp.path_mtu_discovery: 1
net.inet.tcp.sack_globalholes: 0
net.inet.tcp.sack_globalmaxholes: 65536
net.inet.tcp.sack_maxholes: 128
net.inet.tcp.sack: 1
net.inet.tcp.rtt_min: 1
net.inet.tcp.background_io_enabled: 1
net.inet.tcp.isn_reseed_interval: 0
net.inet.tcp.strict_rfc1948: 0
net.inet.tcp.icmp_may_rst: 1
net.inet.tcp.pcbcount: 483
net.inet.tcp.do_tcpdrain: 0
net.inet.tcp.tcbhashsize: 4096
net.inet.tcp.minmssoverload: 0
net.inet.tcp.minmss: 216
net.inet.tcp.always_keepalive: 0
net.inet.tcp.msl: 15000
net.inet.tcp.background_io_trigger: 5
net.inet.tcp.sockthreshold: 64
net.inet.tcp.out_sw_cksum_bytes: 0
net.inet.tcp.out_sw_cksum: 0
net.inet.tcp.in_sw_cksum_bytes: 32529804399
net.inet.tcp.in_sw_cksum: 813056016
net.inet.tcp.win_scale_factor: 3
net.inet.udp.checksum: 1
net.inet.udp.maxdgram: 9216
net.inet.udp.recvspace: 42080
net.inet.udp.pcbcount: 19
net.inet.udp.blackhole: 0
net.inet.udp.log_in_vain: 0
net.inet.udp.out_sw_cksum_bytes: 0
net.inet.udp.out_sw_cksum: 0
net.inet.udp.in_sw_cksum_bytes: 10452
net.inet.udp.in_sw_cksum: 824
net.inet.ipsec.def_policy: 1
net.inet.ipsec.esp_trans_deflev: 1
net.inet.ipsec.esp_net_deflev: 1
net.inet.ipsec.ah_trans_deflev: 1
net.inet.ipsec.ah_net_deflev: 1
net.inet.ipsec.ah_cleartos: 1
net.inet.ipsec.ah_offsetmask: 0
net.inet.ipsec.dfbit: 0
net.inet.ipsec.ecn: 0
net.inet.ipsec.debug: 0
net.inet.ipsec.esp_randpad: -1
net.inet.ipsec.esp_port: 0
net.inet.ipsec.bypass: 1
net.inet.raw.recvspace: 8192
net.inet.raw.maxdgram: 8192
net.appletalk.atp_resp_seqno2big: 0
net.appletalk.routermix: 2000
net.link.generic.system.ifcount: 6
net.link.generic.system.dlil_input_sanity_check: 0
net.link.generic.system.multi_threaded_input: 1
net.link.ether.inet.send_conflicting_probes: 1
net.link.ether.inet.keep_announcements: 1
net.link.ether.inet.log_arp_warnings: 0
net.link.ether.inet.sendllconflict: 0
net.link.ether.inet.proxyall: 0
net.link.ether.inet.useloopback: 1
net.link.ether.inet.maxtries: 5
net.link.ether.inet.apple_hwcksum_rx: 1
net.link.ether.inet.apple_hwcksum_tx: 1
net.link.ether.inet.host_down_time: 20
net.link.ether.inet.max_age: 1200
net.link.ether.inet.prune_intvl: 300
net.key.debug: 0
net.key.spi_trycnt: 1000
net.key.spi_minval: 256
net.key.spi_maxval: 268435455
net.key.int_random: 60
net.key.larval_lifetime: 30
net.key.blockacq_count: 10
net.key.blockacq_lifetime: 20
net.key.esp_keymin: 256
net.key.esp_auth: 0
net.key.ah_keymin: 128
net.key.prefered_oldsa: 0
net.key.natt_keepalive_interval: 20
net.inet6.ip6.forwarding: 0
net.inet6.ip6.redirect: 1
net.inet6.ip6.hlim: 64
net.inet6.ip6.maxfragpackets: 1024
net.inet6.ip6.accept_rtadv: 0
net.inet6.ip6.keepfaith: 0
net.inet6.ip6.log_interval: 5
net.inet6.ip6.hdrnestlimit: 50
net.inet6.ip6.dad_count: 1
net.inet6.ip6.auto_flowlabel: 1
net.inet6.ip6.defmcasthlim: 1
net.inet6.ip6.gifhlim: 0
net.inet6.ip6.kame_version: 20010528/apple-darwin
net.inet6.ip6.use_deprecated: 1
net.inet6.ip6.rr_prune: 5
net.inet6.ip6.v6only: 0
net.inet6.ip6.rtexpire: 3600
net.inet6.ip6.rtminexpire: 10
net.inet6.ip6.rtmaxcache: 128
net.inet6.ip6.use_tempaddr: 0
net.inet6.ip6.temppltime: 86400
net.inet6.ip6.tempvltime: 604800
net.inet6.ip6.auto_linklocal: 1
net.inet6.ip6.maxfrags: 8192
net.inet6.ip6.neighborgcthresh: 2048
net.inet6.ip6.maxifprefixes: 16
net.inet6.ip6.maxifdefrouters: 16
net.inet6.ip6.maxdynroutes: 4096
net.inet6.ip6.fw.verbose_limit: 0
net.inet6.ip6.fw.verbose: 0
net.inet6.ip6.fw.debug: 0
net.inet6.ip6.fw.enable: 0
net.inet6.ipsec6.def_policy: 1
net.inet6.ipsec6.esp_trans_deflev: 1
net.inet6.ipsec6.esp_net_deflev: 1
net.inet6.ipsec6.ah_trans_deflev: 1
net.inet6.ipsec6.ah_net_deflev: 1
net.inet6.ipsec6.ecn: 0
net.inet6.ipsec6.debug: 0
net.inet6.ipsec6.esp_randpad: -1
net.inet6.icmp6.rediraccept: 1
net.inet6.icmp6.redirtimeout: 600
net.inet6.icmp6.nd6_prune: 1
net.inet6.icmp6.nd6_delay: 5
net.inet6.icmp6.nd6_umaxtries: 3
net.inet6.icmp6.nd6_mmaxtries: 3
net.inet6.icmp6.nd6_useloopback: 1
net.inet6.icmp6.nodeinfo: 3
net.inet6.icmp6.errppslimit: 100
net.inet6.icmp6.nd6_maxnudhint: 0
net.inet6.icmp6.nd6_debug: 0
debug.lowpri_max_waiting_msecs: 200
debug.lowpri_max_window_msecs: 200
debug.lowpri_IO_window_inc: 50
debug.lowpri_IO_initial_window_msecs: 100
debug.bpf_maxdevices: 256
debug.bpf_maxbufsize: 524288
debug.bpf_bufsize: 4096
debug.iokit: 0
hw.ncpu: 16
hw.byteorder: 1234
hw.memsize: 17179869184
hw.activecpu: 16
hw.optional.x86_64: 1
hw.optional.sse4_2: 1
hw.optional.sse4_1: 1
hw.optional.supplementalsse3: 1
hw.optional.sse3: 1
hw.optional.sse2: 1
hw.optional.sse: 1
hw.optional.mmx: 1
hw.optional.floatingpoint: 1
hw.packages: 2
hw.tbfrequency: 1000000000
hw.l3cachesize: 8388608
hw.l2cachesize: 262144
hw.l1dcachesize: 32768
hw.l1icachesize: 32768
hw.cachelinesize: 64
hw.cpufrequency_max: 2260000000
hw.cpufrequency_min: 2260000000
hw.cpufrequency: 2260000000
hw.busfrequency_max: 5866000000
hw.busfrequency_min: 5866000000
hw.busfrequency: 5866000000
hw.pagesize: 4096
hw.cachesize: 17179869184 32768 262144 8388608 0 0 0 0 0 0
hw.cacheconfig: 16 2 2 8 0 0 0 0 0 0
hw.cpufamily: 1801080018
hw.cpu64bit_capable: 1
hw.cpusubtype: 4
hw.cputype: 7
hw.logicalcpu_max: 16
hw.logicalcpu: 16
hw.physicalcpu_max: 8
hw.physicalcpu: 8
hw.cputhreadtype: 1
machdep.pmap.hashmax: 8
machdep.pmap.hashcnts: 11419204
machdep.pmap.hashwalks: 11307931
machdep.cpu.flex_ratio.max: 17
machdep.cpu.flex_ratio.min: 12
machdep.cpu.flex_ratio.desired: 63
machdep.cpu.thread_count: 8
machdep.cpu.core_count: 4
machdep.cpu.address_bits.virtual: 48
machdep.cpu.address_bits.physical: 40
machdep.cpu.tlb.data_large: 32
machdep.cpu.tlb.inst_large: 7
machdep.cpu.tlb.data_small: 64
machdep.cpu.tlb.inst_small: 0
machdep.cpu.cache.size: 256
machdep.cpu.cache.L2_associativity: 6
machdep.cpu.cache.linesize: 64
machdep.cpu.arch_perf.fixed_width: 48
machdep.cpu.arch_perf.fixed_number: 3
machdep.cpu.arch_perf.events: 68
machdep.cpu.arch_perf.events_number: 7
machdep.cpu.arch_perf.width: 48
machdep.cpu.arch_perf.number: 4
machdep.cpu.arch_perf.version: 3
machdep.cpu.thermal.ACNT_MCNT: 1
machdep.cpu.thermal.thresholds: 2
machdep.cpu.thermal.dynamic_acceleration: 1
machdep.cpu.thermal.sensor: 1
machdep.cpu.mwait.sub_Cstates: 4384
machdep.cpu.mwait.extensions: 3
machdep.cpu.mwait.linesize_max: 64
machdep.cpu.mwait.linesize_min: 64
machdep.cpu.microcode_version: 15
machdep.cpu.cores_per_package: 8
machdep.cpu.logical_per_package: 16
machdep.cpu.extfeatures: XD EM64T RDTSCP TSCI
machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM SSE3 MON DSCPL VMX EST TM2 SSSE3 CX16 TPR PDCM SSE4.1 SSE4.2 POPCNT
machdep.cpu.brand: 0
machdep.cpu.signature: 67237
machdep.cpu.extfeature_bits: 672137472 1
machdep.cpu.feature_bits: -1075053569 10281917
machdep.cpu.stepping: 5
machdep.cpu.extfamily: 0
machdep.cpu.extmodel: 1
machdep.cpu.model: 26
machdep.cpu.family: 6
machdep.cpu.brand_string: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
machdep.cpu.vendor: GenuineIntel
security.mac.seatbelt.debug: 0
security.mac.seatbelt.profile_refcount: 21
security.mac.seatbelt.qtnstate_refcount: 0
security.mac.seatbelt.cred_label_refcount: 12
security.mac.vnode_enforce: 1
security.mac.vm_enforce: 1
security.mac.sysvshm_enforce: 1
security.mac.sysvsem_enforce: 1
security.mac.sysvmsg_enforce: 1
security.mac.system_enforce: 1
security.mac.socket_enforce: 1
security.mac.proc_enforce: 1
security.mac.posixshm_enforce: 1
security.mac.posixsem_enforce: 1
security.mac.pipe_enforce: 1
security.mac.iokit_enforce: 0
security.mac.file_enforce: 0
security.mac.device_enforce: 1
security.mac.mmap_revocation_via_cow: 0
security.mac.mmap_revocation: 0
security.mac.max_slots: 8

[karlH]

When I last saw this sort of problem on BSD, the effect was that the process went into an unusual state (ps) stalling I presume waiting for memory. What was apparent was that if you did a netstat -m, the buffers (mbufs) had reached the maximum allowed so needed increasing to allow for more socket buffers. The entry in that list that we saw on BSD that helped was kern.ipc.nmbclusters.

Obviously your platform is not exactly the same but it will be BSD derived so increasing that (say double) should help. Whether other setting need altering is something to check up on.

karl.