help needed - state of california needs a benchmark

[don at bowenvale.co.nz (Don Gould)]

Morning Mike,

The *New Zealand Government* don't use speedtest.net as a benchmark.  
Our Government uses a consulting company to provide a range of tests 
that address the issues you're talking about and benchmarks are 
published each year.  http://www.comcom.govt.nz/broadband-reports

The user and network communities are not 100% happy with the way this 
testing is done either.  
http://www.geekzone.co.nz/forums.asp?forumid=49&topicid=73698  Some 
providers are know to fudge the results by putting QoS on the test paths.

http://weathermap.karen.net.nz/ is a New Zealand academic project that 
shows their network performance in real time.  This is a very useful 
site for demonstrating the sort of tools that Governments should be 
looking for when doing performance measuring.

Recent work done by Jared Kells, in Australia, on consumer level network 
performance shows a very interesting picture (pictures are best for 
political people).  
http://forums.whirlpool.net.au/forum-replies.cfm?t=1579142  Kells 
demonstrates that providers deliver very different results for national 
and international sites.  Kells provides a set of Open Source tools to 
do your own testing.

http://www.truenet.co.nz - John Butt - is a commercial start up 
providing another range of testing metrics which the user community at 
www.geekzone.co.nz seem to be much happier with as a proper indication 
of network performance.  I have talked with John personally and can 
attest that the testing is fairly robust and addresses issues that 
you've raised.  http://www.truenet.co.nz/how-does-it-work

The recent upgrades of www.telstraclear.co.nz HFC network from DOCIS2.0 
(25/2 max) to DOCIS3.0 (100/10 testing introduction speed) presented a 
range of challenges for John's testing.  http ramp up speeds to 100mbit 
cause impact on test results, so John had to change the way they were 
testing to get a better performance presentation.

Internode in Australia have learnt the hard way recently that consumer 
expectation of their new NBN FTTH network needs to be managed 
carefully.  As a result of some very poor media press over the 
performance of an education site recently installed in Tasmania, they 
have engaged in quite a bit of consumer education around network 
performance.  
http://www.theaustralian.com.au/news/nation/governments-broadband-not-up-to-speed-at-tasmanian-school/story-e6frg6nf-1225961150410  
-  http://whrl.pl/Rcyrhz - 
<http://forums.whirlpool.net.au/user/6258>Simon Hackett - Internode CEO 
responds.

*Speedtest.net* will only provide a BIR/PIR measure, and not CIR, which 
is not an indicator of service quality.

In New Zealand SpeedTest.net is used extensively with a number of 
hosting servers.  The information is fundamentally flawed as you have no 
control over what testing the end user performs.  In my case I can 
product three different tests from a 15/2 HFC service and get 3 
different results.

http://www.speedtest.net/result/1133639492.png - Test 1 - The 
application has identified that I am located in Christchurch New Zealand 
so has selected a Christchurch based server for testing 
(www.snap.co.nz).  As you can see the results show ~7.5/2.1mbits/s.

http://www.speedtest.net/result/1133642520.png - Test 2 - This time I've 
chosen the CityLink (www.citylink.co.nz) server in Wellington New 
Zealand.  ~6.2/1.97bits/s.

http://www.speedtest.net/result/1052386636.png - Test 3 - from 12/12/10 
shows ~15.1/2.15.  This was tested to an Auckland, New Zealand server.

I did run a set of tests this morning to the Auckland servers as well, 
however they are all being limited to the same numbers as the 
Christchurch test (1) now.  None of the servers are on my providers 
network and performance is governed by the peering/hand overs between 
the networks.

Christchurch - Wellington - 320km - Christchurch - Auckland -  750km 
straight line distances according to Google Earth.

The HFC service I'm using will deliver a through put of 15/2 for some 
time even at peek usage times when pulling content off the providers own 
network.

Ok, that's enough for now.  I hope this helps and let me know if you 
need any more assistance.

Cheers Don