Documentation / Browsers

Browsers

You can fetch timings, run your own JavaScript and record a video of the screen. The following browsers are supported: Firefox, Chrome and Chrome on Android. If you run our Docker containers, we always update them when we tested the latest stable release of the browsers (at the moment we run a beta of Firefox :)).

Firefox

You will need Firefox 61 or later (current beta). In Firefox 55 the HAR export trigger was broken, and there’s a new version that works in Firefox 61. You can use older Firefoxes but you will then miss out on the HAR file.

Firefox profile setup

At the moment we setup a new profile for each run the browser do. We set up the profiles preferences like this. We use Mozillas own configuration as default with some changes + some extra configuration for performance and privacy.

We try to disable all Firefox ping home:

For performance and deterministic reasons we disable the Tracking protection. The problem with the current implementation of the Tracking protection is that it calls home (during a page load) to download the latest blacklist for scripts that should be disabled.

You can also configure your own preferences for the profile.

Starting with a total blank profile isn’t supported at the moment but if you need it, please create an issue and let us know!

Collecting the HAR

To collect the HAR from Firefox we use HAR Export trigger. It needs Firefox 61 to work (if you run a earlier version you will automatically not get the HAR). The trigger is in OMHO a superior version of getting the HAR than parsing the MOZ HTTP log since it adds less overhead to metrics.

If you for some reason don’t need the HAR you can disable it by --browsertime.skipHar.

What to include in the HAR

If you use Firefox you can choose to include response bodies in the HAR file. The HAR file will be larger but it can make things easier to debug on your site.

You can choose what do include by --firefox.includeResponseBodies and choose between none (default) , all (all response bodies for the type text/JS/CSS or html (only save the body of the HTML response).

Choosing Firefox version

Running Firefox on Mac OS X you can choose what version to run with sitespeed.io:

--firefox will use stable, --firefox.nightly, --firefox.beta or --firefox.developer will choose between the others. Remember that you need to install them first before you use them :)

If you run on Linux you need to set the full path to the binary: --firefox.binaryPath

The current default Docker container only contains one version of Firefox. If you want to test on more versions, let us know so we can fix that.

Set your own Firefox preferences

Firefox preferences are all the preferences that you can set on your Firefox instance using about:config. Since we start with a fresh profile (except some defaults) of Firefox for each page load, we are not reusing the setup you have in your Firefox instance.

You set a preference by adding --firefox.preference with the format key:value. If you want to add multiple preferences, repeat --firefox.preference once per argument.

Collect the MOZ HTTP log

You can turn on Firefox HTTP log by adding --firefox.collectMozLog to your run. That can be useful if you want to file upstream issues to Mozilla.

It is setup with timestamp,nsHttp:5,cache2:5,nsSocketTransport:5,nsHostResolver:5 and will create one HTTP log file per run.

Accept insecure certificates

If you want to accept insecure certificates add --firefox.acceptInsecureCerts to your run.

Collect trace logs

We have no way to get trace data from Firefox today (by trace data we mean time spent in JavaScript/paint etc). You can follow the upstream request to make that happen.

Chrome

The latest version of Chrome should work out of the box.

Chrome setup

When we start Chrome it is setup with these command line switches.

Add your own Chrome args

Chrome has a long list of command line switches that you can use to make Chrome act differently than the default setup. You can add those switched to Chrome with --chrome.args (repeat the argument if you have multiple arguments).

When you add your command line switched you should skip the minus. For example: You want to use --deterministic-fetch then add it like --chrome.args deterministic-fetch.

Collect trace logs

You can get the trace log from Chrome by adding --chrome.timeline. Doing that you will see how much time the CPU spend in different categories and a trace log file that you can drag and drop into your devtools timeline.

docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:7.7.2 --chrome.timeline https://www.sitespeed.io/

You can also choose which Chrome trace categories you want to collect by adding --chrome.traceCategories to your parameters.

Collect the console log

If you use Chrome you can collect everything that is logged to the console. You will see the result in the PageXray tab for each run and if you have errors, the numbers are errors are sent to Graphite/InfluxDB. Collect the console log by adding --chrome.collectConsoleLog.

Collect the net log

Collect Chromes net log with --chrome.collectNetLog. This is useful if you want to debug exact what happens with Chrome and your web page. You will get one log file per run.

Choosing Chrome version

You can choose which version of Chrome you want to run by using the --chrome.binaryPath and the full path to the Chrome binary.

Our Docker container only contains one version of Chrome and let us know if you need help to add more versions.

Choose when to end your test

By default the browser will collect data until window.performance.timing.loadEventEnd happens + aprox 2 seconds more. That is perfectly fine for most sites, but if you do Ajax loading and you mark them with user timings, you probably want to include them in your test. Do that by changing the script that will end the test (–browsertime.pageCompleteCheck). When the scripts returns true the browser will close or if the timeout time is reached.

In this example we wait 10 seconds until the loadEventEnd happens, but you can also choose to trigger it at a specific event.

docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:7.7.2 https://www.sitespeed.io --browsertime.pageCompleteCheck 'return (function() {try { return (Date.now() - window.performance.timing.loadEventEnd) > 10000;} catch(e) {} return true;})()'

Yoy can also choose to end the test after 5 seconds of inactivity that happens after loadEventEnd. Do that by adding --browsertime.pageCompleteCheckInactivity to your run. The test will then wait for loadEventEnd to happen and no requests in the Resource Timing API the last 5 seconds. Be-aware though that the script will empty the resource timing API data for every check so if you have your own script collecting data using the Resource Timing API it will fail.

If you add your own complete check you can also choose when your check is run. By default we wait until onLoad happens (by using pageLoadStrategy normal). If you want control direct after the navigation, you can get that by adding --pageLoadStrategy none to your run.

Custom metrics

You can collect your own metrics in the browser by supplying Javascript file(s). By default we collect all metrics inside these folders, but you might have something else you want to collect.

Each javascript file need to return a metric/value which will be picked up and returned in the JSON. If you return a number, statistics will automatically be generated for the value (like median/percentiles etc).

For example say we have one file called scripts.js that checks how many scripts tags exist on a page. The script would look like this:

(function() {
  return document.getElementsByTagName("script").length;
})();

Then to pick up the script, you would run it like this:

docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:7.7.2 https://www.sitespeed.io --browsertime.script scripts.js -b firefox

You will get a custom script section in the Browsertime tab. Custom scripts individual page

And in the summary and detailed summary section. Summary page

Bonus: All custom scripts values will be sent to Graphite, no extra configuration needed!

Visual Metrics

Visual metrics (Speed Index, Perceptual Speed Index, First and Last Visual Complete, and 85-95-99% Visual Complete) can be collected if you also record a video of the screen. If you use our Docker container you automagically get all what you need. Video and Visual Metrics is turned on by default.

docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:7.7.2 https://www.sitespeed.io/

On Android you need to follow these instructions.

Using Browsertime

Everything you can do in Browsertime, you can also do in sitespeed.io. Prefixing browsertime to a CLI parameter will pass that parameter on to Browsertime.

You can check what Browsertime can do.

For example if you want to pass on an extra native arguments to Chrome. In standalone Browsertime you do that with --chrome.args. If you want to do that through sitespeed.io you just prefix browsertime to the param: --browsertime.chrome.args. Yes we know, pretty sweet! :)

How can I disable HTTP/2 (I only want to test HTTP/1.x)?

In Chrome, you just add the switches --browsertime.chrome.args disable-http2.

For Firefox, you need to turn off HTTP/2 and SPDY, and you do that by setting the Firefox preferences: --browsertime.firefox.preference network.http.spdy.enabled:false --browsertime.firefox.preference network.http.spdy.enabled.http2:false --browsertime.firefox.preference network.http.spdy.enabled.v3-1:false

How does it work behind the scene?

We use Browsertime to drive the browser. This is the flow per URL you test:

  1. We setup connectivity for the browser using different engines depending on your configuration.
  2. Open the browser with a new user session (cleared cache etc).
  3. If you add a request header, a cookie, use Basic Auth, block domains or clear the cache browser side the browser will open the Browsertime extension and do what you asked.
  4. If you configured a --preScript it runs next.
  5. If you configured a --preURL the browser navigates to that URL (you should only do that if you don’t use a preScript).
  6. If you configured the video, the video starts to record the screen.
  7. We ask the browser to navigate to the URL (using JavaScript).
  8. Check if the URL in the browser has changed to configured URL (check every 500 ms, time out after 50 s).
  9. Loop to 2. until the URL in the browser has changed.
  10. Check if the page has finished loading using the pre configured pageCompleteCheck or --pageCompleteCheck or --pageCompleteCheckInactivity.
  11. Loop to 4 until the check is done (return true).
  12. Stop the video.
  13. Collect all the default metrics using JavaScript and your own configured scripts --script.
  14. If you configured a --postScript it runs next.
  15. The browser is closed.
  16. Start over in step 2 for the next run for that URL.