Measuring Broadband America - Open Methodology | Federal Communications Commission

Notice: As of August 1, 2023, the FCC no longer partners with SamKnows on fixed broadband performance data collection, and SamKnows is no longer authorized to collect MBA data or provide fixed broadband performance analysis on the FCC’s behalf. The FCC will continue its Measuring Broadband America program work internally as it assesses Agency requirements for future potential procurement action related to the measurement of fixed broadband performance.

To stop unauthorized MBA data collection by SamKnows, a user should remove the Whitebox from their network after sending a Termination Notice to community@samknows.com stating that they would like to terminate SamKnows’ Measurement Services. If the Termination Notice is emailed before 5:00 pm Eastern Time on a business day, the user should remove the Whitebox from their network on the same day after they email SamKnows the Termination Notice. Otherwise, the user should remove the Whitebox from their network on the next following business day. Questions can be directed to Mr. Rajender Razdan at rajender.razdan@fcc.gov.

The FCC's Measuring Broadband America program is built on principles of openness and transparency. The FCC has made available to stakeholders and the general public the open source software used on both its fixed and mobile applications, the data collected, and detailed information regarding the FCC's technical methodology for analyzing the collected data.

The measurement methodology for the Measuring Broadband America program has been developed in collaboration with SamKnows, the FCC's contractor supporting the Measuring Broadband America program, who perform similar projects for other countries around the world.

Over the course of the multi-year program the FCC has released the comprehensive measurement methodology used to collect the data and produce reports, and in addition to the various data sets, the actual software source code that was used for the testing has been made available for academic and other researchers for non-commercial purposes by SamKnows.

The goal of SamKnows is to help create a standard methodology for measuring Internet performance globally, and in pursuit of this goal, SamKnows is now making the source code of the actual tests available as open source under a GNU General Public License.

Open Source Software

Source Code

Fixed Source Code https://github.com/SamKnows/skandroid-fcc
Mobile Github Repository https://github.com/FCC/mobile-mba-androidapp

Over the course of the two-year program the FCC has released the comprehensive measurement methodology used to collect the data and produce reports, and in addition to the various data sets, the actual software source code that was used for the testing has been made available for academic and other researchers for non-commercial purposes by SamKnows.

The FCC would like to thank SamKnows for their commitment to the openness and transparency principles of the program, and welcome the opportunity to share the open source contributions of any other technology providers interested in supporting the program.

Data Methodology

Measuring Fixed Broadband

After a call for volunteers for testing, SamKnows selected 12,000 participants. Each participant received a Whitebox, an off-the-shelf router with tests embedded in the software. The Whiteboxes performed the following tests: [same table as seen on Methodology page]

Download speed	Throughput in Megabits per second (Mbps) utilizing three concurrent TCP connections
Upload speed	Throughput in Mbps utilizing three concurrent TCP connections
Web browsing	Total time to fetch a page and all of its resources from a popular website
UDP latency	Average round trip time of a series of randomly transmitted UDP packets distributed over a long timeframe
UDP packet loss	Fraction of UDP packets lost from UDP latency test
Video streaming	Initial time to buffer, number of buffer under-runs and total time for buffer delays
Voice over IP	Upstream packet loss, downstream packet loss, upstream jitter, downstream jitter, round trip latency
DNS resolution	Time taken for the ISP‚Äôs recursive DNS resolver to return an A record25 for a popular website domain name
DNS failures	Percentage of DNS requests performed in the DNS resolution test that failed
ICMP latency	Round trip time of five regularly spaced ICMP packets
ICMP packet loss	Percentage of packets lost in the ICMP latency test
Latency under load	Average round trip time for a series of regularly spaced UDP packets sent during downstream/upstream sustained tests
Availability	Total time the connection was deemed unavailable for any purpose, which could include a network fault or unavailability of a measurement point
Consumption	A simple record of the total bytes downloaded and uploaded by the router

For more information, please refer to the Technical Appendix.

Test Schedule

The scheduled timing for each of these tests is described in the test schedule whitepaper.

Raw Data Processing

In order to maintain the integrity of the data set, samples that had invalid characteristics, such as samples outside the testing period and samples in which the ISP changed, were eliminated. For more details, please refer to the following report: Validated Data Cleansing

Statistics and Analytics

Statistical averages for the most recent validated data can be found at: Statistical Averages Sept. 2015. The results within these bands are further broken out by ISP and service tier. Where an ISP does not offer a service tier within a specific band or a representative sample could not be formed for tier(s) in that band, the ISP will not appear in that speed band.

Validation and Sampling Checks

In the most recent study, the methodology employed in this study included verifying each panelist's service tier and ISP against the record base of participating ISPs. 14 Initial throughput tests were used to confirm reported speeds.

The broadband service tier reported by each panelist was authenticated in the following way:

At the time of recruitment, each panelist was required to complete a speed test using an M-Lab server. This test provided a rough approximation of the panelist's service tier that served to identify panelists with targeted demographics, and highlighted anomalies in the panelist's survey response to measured speed.
At the time the panelist installed the Whitebox, the device automatically ran an IP test to check that the ISP identified by the volunteer was correct.
The Whitebox also ran an initial test that flooded each panelist's connection in order to accurately detect the throughput speed when their deployed Whitebox connected to a test node.
Each ISP was asked to confirm the broadband service tier reported by each selected panelist.
SamKnows then took the validated speed tier information that was provided by the ISPS and compared this to both the panelist-provided information, and the actual test results obtained, in order to ensure accurate tier validation.

SamKnows manually completed the following four steps for each panelist:

Verified that the IP address was in a valid range for those served by the ISP in question.
Reviewed data for each panelist and removed data where speed changes such as tier upgrade or downgrade appeared to have occurred, either due to a service change on the part of the consumer or a network change on the part of the ISP.
Identified panelists whose throughput appeared inconsistent with the provisioned service tier.
Such anomalies were re-certified with the panelist's ISP.
Verified that the resulting downstream-upstream test results corresponded to the ISP-provided speed tiers, and updated accordingly if required.