Documentation / Browsers

Browsers

You can fetch timings, run your own JavaScript and record a video of the screen. The following browsers are supported: Firefox, Safari, Chrome, Chrome on Android and Safari on iOS. If you run our Docker containers, we always update them when we tested the latest stable release of Chrome and Firefox. Safari and Safari on iOS needs Mac OS X Catalina.

Firefox #

The latest version of Firefox should work out of the box.

Firefox profile setup #

At the moment we setup a new profile for each run the browser do. We set up the profiles preferences like this. We use Mozillas own configuration as default with some changes + some extra configuration for performance and privacy.

We try to disable all Firefox ping home:

For performance and deterministic reasons we disable the Tracking protection. The problem with the current implementation of the Tracking protection is that it calls home (during a page load) to download the latest blacklist for scripts that should be disabled.

You can also configure your own preferences for the profile.

Starting with a total blank profile isn’t supported at the moment but if you need it, please create an issue and let us know!

Collecting the HAR #

To collect the HAR from Firefox we use HAR Export trigger. It needs Firefox 61 to work (if you run a earlier version you will automatically not get the HAR). The trigger is in OMHO a superior version of getting the HAR than parsing the MOZ HTTP log since it adds less overhead to metrics.

If you for some reason don’t need the HAR you can disable it by --browsertime.skipHar.

What to include in the HAR #

If you use Firefox you can choose to include response bodies in the HAR file. The HAR file will be larger but it can make things easier to debug on your site.

You can choose what do include by --firefox.includeResponseBodies and choose between none (default) , all (all response bodies for the type text/JS/CSS or html (only save the body of the HTML response).

Choosing Firefox version #

Running Firefox on Mac OS X you can choose what version to run with sitespeed.io:

--firefox will use stable, --firefox.nightly, --firefox.beta or --firefox.developer will choose between the others. Remember that you need to install them first before you use them :)

If you run on Linux you need to set the full path to the binary: --firefox.binaryPath

The current default Docker container only contains one version of Firefox. If you want to test on more versions, let us know so we can fix that.

Set your own Firefox preferences #

Firefox preferences are all the preferences that you can set on your Firefox instance using about:config. Since we start with a fresh profile (except some defaults) of Firefox for each page load, we are not reusing the setup you have in your Firefox instance.

You set a preference by adding --firefox.preference with the format key:value. If you want to add multiple preferences, repeat --firefox.preference once per argument.

Collect the MOZ HTTP log #

You can turn on Firefox HTTP log by adding --firefox.collectMozLog to your run. That can be useful if you want to file upstream issues to Mozilla.

It is setup with timestamp,nsHttp:5,cache2:5,nsSocketTransport:5,nsHostResolver:5 and will create one HTTP log file per run.

Accept insecure certificates #

If you want to accept insecure certificates add --firefox.acceptInsecureCerts to your run.

Collect trace logs #

We have no way to get trace data from Firefox today (by trace data we mean time spent in JavaScript/paint etc). You can follow the upstream request to make that happen.

More memory #

When you run Firefox in Docker you should use --shm-size 2g to make sure Firefox get enough shared memory (for Chrome we disabled the use of shm with –disable-dev-shm-usage).

docker run --shm-size 2g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:11.2.0 https://www.sitespeed.io -b firefox

Chrome #

The latest version of Chrome should work out of the box. Latest version of stable ChromeDriver is bundled in sitespeed.io and needs to match your Chrome version.

Chrome setup #

When we start Chrome it is setup with these command line switches.

Add your own Chrome args #

Chrome has a long list of command line switches that you can use to make Chrome act differently than the default setup. You can add those switched to Chrome with --chrome.args (repeat the argument if you have multiple arguments).

When you add your command line switched you should skip the minus. For example: You want to use --deterministic-fetch then add it like --chrome.args deterministic-fetch.

Collect trace logs #

You can get the trace log from Chrome by adding --chrome.timeline. Doing that you will see how much time the CPU spend in different categories and a trace log file that you can drag and drop into your devtools timeline.

docker run --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:11.2.0 --chrome.timeline https://www.sitespeed.io/

You can also choose which Chrome trace categories you want to collect by adding --chrome.traceCategories to your parameters.

Collect the console log #

If you use Chrome you can collect everything that is logged to the console. You will see the result in the PageXray tab for each run and if you have errors, the numbers are errors are sent to Graphite/InfluxDB. Collect the console log by adding --chrome.collectConsoleLog.

Collect the net log #

Collect Chromes net log with --chrome.collectNetLog. This is useful if you want to debug exact what happens with Chrome and your web page. You will get one log file per run.

Choosing Chrome version #

You can choose which version of Chrome you want to run by using the --chrome.binaryPath and the full path to the Chrome binary.

Our Docker container only contains one version of Chrome and let us know if you need help to add more versions.

Use a newer version of ChromeDriver #

ChromeDriver is the driver that handles the communication with Chrome. At the moment the ChromeDriver version needs to match the Chrome version. By default sitespeed.io and Browsertime comes with the ChromeDriver version that matches the Chrome version in the Docker container. If you wanna run tests on Chrome Beta/Canary you probably need to download a later version of ChromeDriver.

You download ChromeDriver from http://chromedriver.chromium.org and then use --chrome.chromedriverPath to set the path to the new version of the ChromeDriver.

Safari #

You can run Safari on Mac OS X. To run on iOS you need Catalina and iOS 13. To see more what you can do with the SafariDriver you can run man safaridriver in your terminal.

Limitations #

We do not support HAR, video, cookies/request headers in Safari at the moment.

Configuration #

There are a couple of different specific Safari configurations.

Run on iOS #

To run on iOS you need to add:

sitespeed.io --safari.ios -b safari

Choose which device #

There are a couple of different ways to choose which device to use:

  • --safari.deviceName set the device name. Device names for connected devices are shown in iTunes.
  • --safari.deviceUDID set the device UDID. If Xcode is installed, UDIDs for connected devices are available via the output of instruments(1) and in the Device and Simulators window accessed in Xcode via “Window > Devices and Simulators”)
  • --safari.deviceType set the device type. If the value of safari:deviceType is iPhone, safaridriver will only create a sessio using an iPhone device or iPhone simulator. If the value of safari:deviceType is iPad, safaridriver will only create a session using an iPad device or iPad simulator.
  • --safari.useSimulator if the value of useSimulator is true, safaridriver will only use iOS Simulator hosts. If the value of safari:useSimulator is false, safaridriver will not use iOS Simulator hosts. NOTE: An Xcode installation is required in order to run WebDriver tests on iOS

Diagnose problems #

If you need to file a bug with SafariDriver, you also want to include diagnostics generated by SafariDriver. You do that by adding --safari.diagnose to your run.

sitespeed.io --safari.ios -b safari --safari.diagnose

The log file will be stored in ~/Library/Logs/com.apple.WebDriver/.

Choose when to end your test #

By default the browser will collect data until window.performance.timing.loadEventEnd happens + aprox 5 seconds more. That is perfectly fine for most sites, but if you do Ajax loading and you mark them with user timings, you probably want to include them in your test. Do that by changing the script that will end the test (--browsertime.pageCompleteCheck). When the scripts returns true the browser will close or if the timeout time is reached.

In this example we wait 10 seconds until the loadEventEnd happens, but you can also choose to trigger it at a specific event.

docker run --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:11.2.0 https://www.sitespeed.io --browsertime.pageCompleteCheck 'return (function() {try { return (Date.now() - window.performance.timing.loadEventEnd) > 10000;} catch(e) {} return true;})()'

You can also configure how long time your current check will wait until completing with --pageCompleteWaitTime. By default the pageCompleteCheck waits for 5000 ms after the onLoad event to happen. If you want to increase that to 10 seconds use --pageCompleteWaitTime 10000. This is also useful if you test with pageCompleteCheckInactivity and it takes long time for the server to respond, you can use the pageCompleteWaitTime to wait longer than the default value.

You can also choose to end the test after 5 seconds of inactivity that happens after loadEventEnd. Do that by adding --browsertime.pageCompleteCheckInactivity to your run. The test will then wait for loadEventEnd to happen and no requests in the Resource Timing API the last 5 seconds. Be-aware though that the script will empty the resource timing API data for every check so if you have your own script collecting data using the Resource Timing API it will fail.

There’s is also another alternative: use --spa to automatically wait for 5 seconds of inactivity in the Resource Timing API (independently if the load event end has fired or not). If you need to wait longer, use --pageCompleteWaitTime.

If you add your own complete check you can also choose when your check is run. By default we wait until onLoad happens (by using pageLoadStrategy normal). If you want control direct after the navigation, you can get that by adding --pageLoadStrategy none to your run.

Custom metrics #

You can collect your own metrics in the browser by supplying JavaScript file(s). By default we collect all metrics inside these folders, but you might have something else you want to collect.

Each javascript file need to return a metric/value which will be picked up and returned in the JSON. If you return a number, statistics will automatically be generated for the value (like median/percentiles etc).

For example say we have one file called scripts.js that checks how many scripts tags exist on a page. The script would look like this:

(function() {
  return document.getElementsByTagName("script").length;
})();

Then to pick up the script, you would run it like this:

docker run --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:11.2.0 https://www.sitespeed.io --browsertime.script scripts.js -b firefox

You will get a custom script section in the Browsertime tab. Custom scripts individual page

And in the summary and detailed summary section. Summary page

Bonus: All custom scripts values will be sent to Graphite, no extra configuration needed!

Visual Metrics #

Visual metrics (Speed Index, Perceptual Speed Index, First and Last Visual Complete, and 85-95-99% Visual Complete) can be collected if you also record a video of the screen. If you use our Docker container you automagically get all what you need. Video and Visual Metrics is turned on by default.

docker run --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:11.2.0 https://www.sitespeed.io/

On Android you need to follow these instructions.

Using Browsertime #

Everything you can do in Browsertime, you can also do in sitespeed.io. Prefixing browsertime to a CLI parameter will pass that parameter on to Browsertime.

You can check what Browsertime can do.

For example if you want to pass on an extra native arguments to Chrome. In standalone Browsertime you do that with --chrome.args. If you want to do that through sitespeed.io you just prefix browsertime to the param: --browsertime.chrome.args. Yes we know, pretty sweet! :)

How can I disable HTTP/2 (I only want to test HTTP/1.x)? #

In Chrome, you just add the switches --browsertime.chrome.args disable-http2.

For Firefox, you need to turn off HTTP/2 and SPDY, and you do that by setting the Firefox preferences: --browsertime.firefox.preference network.http.spdy.enabled:false --browsertime.firefox.preference network.http.spdy.enabled.http2:false --browsertime.firefox.preference network.http.spdy.enabled.v3-1:false

How does it work behind the scene? #

We use Browsertime to drive the browser. This is the flow per URL you test:

  1. We setup connectivity for the browser using different engines depending on your configuration.
  2. Open the browser with a new user session (cleared cache etc).
  3. If you add a request header, a cookie, use Basic Auth, block domains or clear the cache browser side the browser will open the Browsertime extension and do what you asked.
  4. If you configured a --preScript it runs next.
  5. If you configured a --preURL the browser navigates to that URL (you should only do that if you don’t use a preScript).
  6. If you configured the video, the video starts to record the screen.
  7. We ask the browser to navigate to the URL (using JavaScript).
  8. Check if the URL in the browser has changed to configured URL (check every 500 ms, time out after 50 s).
  9. Loop to 2. until the URL in the browser has changed.
  10. Check if the page has finished loading using the pre configured pageCompleteCheck or --pageCompleteCheck or --pageCompleteCheckInactivity.
  11. Loop to 4 until the check is done (return true).
  12. Stop the video.
  13. Collect all the default metrics using JavaScript and your own configured scripts --script.
  14. If you configured a --postScript it runs next.
  15. The browser is closed.
  16. Start over in step 2 for the next run for that URL.