Documentation / Continuously run your tests

Continuously run your tests

We have an example setup that we use to collect metrics for dashboard.sitespeed.io that you can use as inspiration. Or you can just run your tests in the crontab or as an infinite loop on your server. In our example we use the Dockerized version of sitespeed.io. You can use Docker or you can run sitespeed.io standalone for more control.

Configuration in Git (our example setup)

We have an example setup that we use to collect the metrics for dashboard.sitespeed.io that you can use as a starting point. You need a server running Linux with Docker and Git installed. We have all our configuration in Git so it is version controlled.

You can check out our setup at https://github.com/sitespeedio/dashboard.sitespeed.io.

It works like this: you fork our repo (or copy what you need into your own repo) and edit the URLs/script files, choosing which URLs and tests you want to run.

You go to your server, clone the repo, and start the script by pointing out which group of tests you want to run (each server has its own base folder). The start script will do a git pull for each iteration to get the latest versions of the URLs/scripts you want to run. Then it runs all your tests, using the configuration files you have in /config/.

The script creates a file called sitespeed.run in your current folder. If you want to stop your tests gracefully, remove that file (rm sitespeed.run) and wait for the tests to finish (tail -f /tmp/sitespeed.io.log).

Do you want to add a new URL to test on desktop? Navigate to desktop and create your new file there. Want to add a user journey? Add the script in the same place, ending with .js.

Our example runs tests for desktop and emulated mobile (both URLs and scripts).

The structure looks like this:

.
├── config
│   ├── alexaDesktop.json
│   ├── alexaMobile.json
│   ├── crux.json
│   ├── desktop.json
│   ├── desktopMulti.json
│   ├── emulatedMobile.json
│   ├── emulatedMobileMulti.json
│   ├── loginWikipedia.json
│   ├── news.json
│   ├── replay.json
│   └── spa.json
├── loop.sh
├── run.sh
└── tests
    ├── desktop
    │   ├── alexaDesktop.txt
    │   ├── crux.txt
    │   ├── desktop.txt
    │   ├── desktopMulti.js
    │   ├── loginWikipedia.js
    │   ├── news.txt
    │   ├── replay.replay
    │   └── spa.js
    └── emulatedMobile
        ├── alexaMobile.txt
        ├── emulatedMobile.txt
        └── emulatedMobileMulti.js

The loop.sh is the starting point. Run it. That script will git pull the repo for every iteration and run the script run.sh.

Then run.sh will use the right configuration in /config/ and run the URLs/scripts that are configured. Our configuration files extend configuration files that only exist on the server, where we hold secret information like usernames and passwords. You don't need to set it up that way if you use a private Git repo.

What you need to do

You need to modify our tests and scripts so that you don't test the exact same URLs as us :)

Configuration

In our example we have two configuration files on the server that we extend. These configuration files hold the secrets that we don't want to expose on our public GitHub repo. In our example they look like this:

/conf/secrets.json

{
  "graphite": {
    "host": "OUR_HOST",
    "auth": "THE_AUTH",
    "annotationScreenshot": true
  },
  "slack": {
    "hookUrl": "https://hooks.slack.com/services/THE/SECRET/S"
  },
  "resultBaseURL": "https://s3.amazonaws.com/results.sitespeed.io",
  "s3": {
    "key": "S3_KEY",
    "secret": "S3_SECRET",
    "bucketname": "BUCKET_NAME",
    "removeLocalResult": true
  }
}

/conf/webpagetest-secrets.json

{
  "extends": "/config/secrets.json",
  "influxdb": {
    "host": "OUR_HOST",
    "database": "DATABASE",
    "username": "USER",
    "password": "PASSWORD"
  },
  "webpagetest": {
    "timeline": true,
    "key": "WPT_KEY"
  }
}

Then our configuration files in /config/ extends these config files. They look something like this:

{
  "extends": "/config/secrets.json",
  ...
}

And when we run our tests, we map the volume on the server /config to our Docker container. You can see that in the run.sh file. Look for -v /config:/config. That is the magic line.

We then also map the current working dir to -v "$(pwd):/sitespeed.io" and feed the config file to sitespeed: --config /sitespeed.io/config. That way, inside the Docker container, we have /config/ that has the secret configuration files and /sitespeed.io/config with the configuration we want to use for our tests.

Change the tests

Stop! Before you move on you need to change the tests you wanna run. Our current test structure looks like this:

└── tests
    ├── desktop
    │   ├── alexaDesktop.txt
    │   ├── crux.txt
    │   ├── desktop.txt
    │   ├── desktopMulti.js
    │   ├── loginWikipedia.js
    │   ├── news.txt
    │   ├── replay.replay
    │   └── spa.js
    └── emulatedMobile
        ├── alexaMobile.txt
        ├── emulatedMobile.txt
        └── emulatedMobileMulti.js

Adding a text file in the desktop folder with URLs will test those URLs. If you add a scripting file (ending with .js) a user journey will be tested.

All tests need a corresponding configuration file. Say that you have a test file named alexaMobile.txt. You then need an alexaMobile.json configuration file in the config/ directory for the test to work.

Change how you test

The most important thing is to change this so you run the latest stable version of sitespeed.io. In our test environment we run the latest build (to make sure we catch bugs/regressions before we release them). It looks like this:

DOCKER_CONTAINER=sitespeedio/sitespeed.io-autobuild:latest

YOU NEED TO CHANGE THAT! Your version should look like:

DOCKER_CONTAINER=sitespeedio/sitespeed.io:40.4.0

That way you have a safe and easy way to upgrade and roll back versions of sitespeed.io. When a new sitespeed.io version is released, you edit the file and change the version number. The change will be picked up on the next iteration of the loop.

In our tests we run all URL tests on Chrome and Firefox, test the scripts in Chrome, and run URLs and scripts on emulated mobile using Chrome.

We run all these tests because we use them to verify that all the functionality is working on our side. You probably don't need to run all of them, in which case you can just remove those lines from run.sh.

Run

Go into the directory where you cloned the repo: cd dashboard.sitespeed.io. Then start: nohup ./loop.sh &

To verify that everything works you should tail the log: tail -f /tmp/sitespeed.io

Run on Mac

If you run on Mac you should use screen instead of nohup. First open a new screen instance: screen. Then start your tests: ./loop.sh. Then detach your screen: ctrl+A and then press D. To resume back to the screen use screen -x.

Stop your tests

Starting your test creates a file named sitespeed.run in your current folder. The script on the server will continue to run forever until you remove the control file: rm sitespeed.run

The script will then stop when it has finished the current run(s). Wait for it to stop by looking at the log: tail -f /tmp/sitespeed.io.

Start on reboot

Sometimes your cloud server reboots. To make sure it auto-starts your tests, you can add it to the crontab. Edit the crontab with crontab -e and add (make sure to change the path to your installation):

@reboot rm /root/dashboard.sitespeed.io/sitespeed.run;cd /root/dashboard.sitespeed.io/ && ./loop.sh

Keeping your instance updated

We constantly do new Docker releases: bug fixes, new functionality and new versions of the browser. To keep your instance updated, follow this workflow.

Update your run.sh file so it uses the new version (40.4.0 in this case). On the next iteration of the loop, it will use the new version.

If you test many URLs you may want to shorten the wait time. Log into the server and stop your tests by removing the sitespeed.run file. Wait until the tests stop, then restart.

Go into the Grafana dashboard and create a new annotation, telling your team mates that you updated to the new version. This is really important so you can keep track of browser updates and other changes that can affect your metrics.

Crontab

If for some reason you don't want to follow our example setup, you can choose to run the tests in the crontab of your server.

Using the crontab (on a standalone server) you do it like this: crontab -e to edit the crontab. Make sure your cron user can run Docker.

You can have a script crontab.sh file:

Shell script

#!/bin/bash
# Specify the exact version of sitespeed.io. When you upgrade to the next version, pull it down and change the tag
DOCKER_CONTAINER=sitespeedio/sitespeed.io:40.4.0

# Simplify some configurations
CONFIG="--config /sitespeed.io/default.json"
DOCKER_SETUP="--shm-size=1g --rm -v /home/ubuntu/config:/sitespeed.io -v /result:/result -v /etc/localtime:/etc/localtime:ro --name sitespeed"

# Make sure we can throttle the connection
sudo modprobe ifb numifbs=1

# Start running the tests
# We run more tests on our test server but this gives you an idea of how you can configure it
docker run $DOCKER_SETUP $DOCKER_CONTAINER -n 11 --browsertime.viewPort 1920x1080 /sitespeed.io/wikipedia.org.txt $CONFIG
docker run $DOCKER_SETUP $DOCKER_CONTAINER -n 11 --browsertime.viewPort 1920x1080 /sitespeed.io/wikipedia.org.txt -b firefox $CONFIG
docker run $DOCKER_SETUP $DOCKER_CONTAINER --graphite.namespace sitespeed_io.emulatedMobile --cpu /sitespeed.io/m.wikipedia.org.txt -c 3g --connectivity.engine throttle --mobile true $CONFIG

# We remove all docker stuff to get a clean next run
docker system prune --all --volumes -f

# Get the container so we have it the next time we wanna use it
docker pull $DOCKER_CONTAINER

Edit the crontab

Then you can trigger the script from the crontab. In this example we run every hour but it depends on how many tests you run. crontab -e.

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
0 * * * * /root/crontab.sh >> /tmp/sitespeed.io.log 2>&1

Infinite loop

Another way is to just run the script in an infinite loop and have a file that you remove (so the run stops) when you want to update your instance. This example script is for Ubuntu.

#!/bin/bash
LOGFILE=/tmp/s.log
exec > $LOGFILE 2>&1
CONTROL_FILE=/home/ubuntu/sitespeed.run

if [ -f "$CONTROL_FILE" ]
then
  echo "$CONTROL_FILE exist, do you have running tests?"
  exit 1;
else
  touch $CONTROL_FILE
fi

DOCKER_CONTAINER=sitespeedio/sitespeed.io:40.4.0

function cleanup() {
  docker system prune --all --volumes -f
}

function control() {
  if [ -f "$CONTROL_FILE" ]
  then
    echo "$CONTROL_FILE found. Make another run ..."
  else
    echo "$CONTROL_FILE not found - stopping after cleaning up ..."
    cleanup
    echo "Exit"
    exit 0;
  fi
}

while true
do

  DOCKER_SETUP="--shm-size=1g --rm -v /home/ubuntu/config:/sitespeed.io -v /result:/result -v /etc/localtime:/etc/localtime:ro "
  CONFIG="--config /sitespeed.io/default.json"
  echo 'Start a new loop '
 
  docker run $DOCKER_SETUP $DOCKER_CONTAINER -n 7 --browsertime.viewPort 1920x1080 --browsertime.cacheClearRaw true /sitespeed.io/wikipedia.org.txt $CONFIG
  control
  docker run $DOCKER_SETUP $DOCKER_CONTAINER -n 7 --browsertime.viewPort 1920x1080 /sitespeed.io/wikipedia.org.txt -b firefox $CONFIG
  cleanup
done

And make sure the script starts on server restart. Edit the crontab crontab -e and add (loop.sh is the name of your loop script file):

@reboot rm /home/ubuntu/sitespeed.run;/home/ubuntu/loop.sh

And start it like this:

nohup /home/ubuntu/loop.sh &

Continuously run your tests

Configuration in Git (our example setup) #

What you need to do #

Configuration #

Change the tests #

Change how you test #

Run #

Run on Mac #

Stop your tests #

Start on reboot #

Keeping your instance updated #

Crontab #

Shell script #

Edit the crontab #

Infinite loop #