Documentation / S3

S3

You can store the result HTML/images and videos at Amazon S3 (or S3-compliant storage). That's what we are using for https://dashboard.sitespeed.io/. What's good about it is that it is cheap and you can easily control how long you want to keep your files. And it is not too much work to set up.

Set up an S3 bucket

To set up a bucket on S3 you need to have an Amazon account.

Create an IAM User and follow the best practice guide - you need an IAM user to get the key and secret to be able to upload the result from sitespeed.io.
Create a bucket - set up one bucket per server that runs tests (then you are sure there will be no collisions, and it will be easier when you want to remove tests).
Configure your bucket for static website hosting - so that you can access the result pages, and make sure you add a bucket policy (for access).
Add a lifecycle rule - you want to make sure you remove old result pages/videos/screenshots to save money.
If you want to show screenshots/videos/use the metadata from S3 in Grafana, you need to make sure that your Grafana instance can access S3 through your browser by setting up the correct CORS.

Here's an example of setting up CORS to give compare.sitespeed.io access to get HARs and other files.

[
    {
        "AllowedHeaders": [],
        "AllowedMethods": [
            "GET"
        ],
        "AllowedOrigins": [
            "https://compare.sitespeed.io"
        ],
        "ExposeHeaders": []
    }
]

Do you need more help? First dive into the AWS S3 docs then if it doesn't help, create an issue and we can try to help you.

sitespeed.io configuration

To push the metrics to S3 you need either Explicit Credentials (Key, Secret) or an IAM instance profile if running on EC2. Here are both approaches:

Using Explicit Credentials (Key, Secret)

You need the key, the secret and your bucketname. You get the --s3.key and --s3.secret from your IAM User. The --s3.bucketname is the name you picked for your bucket.

Depending on the setup, you sometimes want to set the S3 region (--s3.region if you don't use the default one) and the canned access control --s3.acl of the uploaded files (you can set up the access control when you set up the bucket too).

Using IAM Instance Profile

When running sitespeed.io on an EC2 instance, you can use IAM instance profiles for S3 access instead of providing explicit AWS credentials. This is recommended as it's more secure than handling access keys directly.

Your EC2 instance needs an IAM role with S3 permissions (s3:PutObject and s3:ListBucket). You only need to specify the --s3.bucketname and optionally the --s3.region if you don't use the default one. The instance profile credentials will be automatically used without needing to specify --s3.key or --s3.secret.

Extra configuration

You can also pass all parameters that the official AWS JavaScript SDK uses.

Pass parameters to S3 upload

Extra params passed when you do the S3.upload: --s3.params. Check out the AWS upload property docs for all properties.

Example - set expire to one year: --s3.params.Expires=31536000

Pass extra options when you create the S3 object

Extra options passed when you create the S3 object. Check out the S3 Object documentation.

Example - lock to a specific API version: --s3.options.apiVersion=2006-03-01

Using S3 as server

Running on S3, you should also set up a URL to your S3 instance(s). That way the annotation links in Graphite/InfluxDB will appear. You can then go from a result in Grafana to the actual HTML result. That is super useful when you want to understand a regression.

For the dashboard we have set up a domain https://results.sitespeed.io, so when we run we add the following: --resultBaseURL https://results.sitespeed.io.

Extra options you should use

When you push to S3, there are a couple of extra configurations you should use.

By default the HAR file is bundled in the HTML, because if you run the HTML files locally on your machine, that's the only way they can be loaded. But on S3 you want to separate the HAR and the HTML (it will save space). Do that by adding --html.fetchHARFiles.

Then you want to make sure the HAR files are gzipped (to save even more space): --gzipHAR.

Screenshots are PNG by default, but you probably want them to be JPG: --screenshot.type jpg.

You should also make sure that all the result files (HTML/videos/screenshots) are removed from your local server and only exist on S3. Add --s3.removeLocalResult.

As a last thing you should also add --copyLatestFilesToBase, which will make it possible to view the latest screenshot and video in Grafana from S3.

MinIO

If you want to deploy the storage yourself, you can use the Open Source https://min.io. You can deploy that using Docker. There's an example of how you can set that up in the sitespeed.io online test repo (see the docker-compose.*.yml files at the root).

Digital Ocean Spaces

Digital Ocean is compatible with the S3 API, so all that is required after setting up your space and acquiring a key and secret is to modify the endpoint that the s3 results are passed to as shown below.

Make sure that your endpoint starts with http/https.

JSON configuration file

If the endpoint is not passed this will default to AWS's endpoint. You may safely exclude it for AWS integration. If you use a JSON configuration file you should make sure you add this to get S3 to work:

{
  ...
  "resultBaseURL": "https://your.bucket.url",
  "gzipHAR": true,
  "html": {
    "fetchHARFiles": true
  },
  "s3": {
     "endpoint": "OPTIONALLY_YOUR_ENDPOINT_FOR_DO_OR_AWS",
     "key": "YOUR_KEY",
     "secret": "YOUR_SECRET",
     "bucketname": "YOUR_BUCKETNAME",
     "removeLocalResult": true
  },
  "screenshot": {
    "type": "jpg"
    }
  }
}

S3

Set up an S3 bucket #

sitespeed.io configuration #

Using Explicit Credentials (Key, Secret) #

Using IAM Instance Profile #

Extra configuration #

Pass parameters to S3 upload #

Pass extra options when you create the S3 object #

Using S3 as server #

Extra options you should use #