Google says “webp”

.webp is an example of Google controlling the internet by producing a new standard and then penalising sites that don’t use it.

Introduced in 2010, the webp  image format, though open, was not supported by Apple and Microsoft and to this date (May 2020) does not appear on the W3C graphics page https://www.w3.org/Graphics/. Webp was only supported by Microsoft as a side effect of the Edge browser switching to use Google Chromium in 2020, and is not supported at all in Internet Explorer.  Firefox introduced support in 2019 (v65) and Apple Safari not at all.  Looking at browser market share, it’s still likely that >20% of browser sessions do not support webp.

Nevertheless sites that support webp will be given preference by Google who in 2018 officially moved to a “mobile-first” ranking system preferring the smaller image formats of webp for mobile performance.  Google Pagespeed test reports may give the impression that serving an ordinary .jpg amounts to an error.

Webserver implementation

WordPress

EWWW Image Optimizer handles image compression and optimisation for over 800,000 active installations.  EWWW options are scattered throughout the admin interface, particularly:

  • Settings, EWWW Image Optimizer, WebP – for server-side solution select Force WebP to try to ensure a webp image exists for every jpg so that blanket server-side redirection will work.
  • Media, Bulk Optimise with option WebP only will create the .webp versions of existing images created before installing the EWWW tool.
  • EWWW Image Optimizer manages history of optimised images

See also their advice on .webp implementation in different scenarios.

Offload Media Lite is designed to work with EWWW image optimiser to synchronise optimised images including the .webp files to S3.

Apache

use mod_rewrite and mod_headers to rewrite .jpg requests to .webp if the browser header indicates support for .webp

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_ACCEPT} image/webp
RewriteCond %{REQUEST_FILENAME} (.*)\.(jpe?g|png)$
RewriteCond %{REQUEST_FILENAME}\.webp -f
RewriteCond %{QUERY_STRING} !type=original
RewriteRule (.+)\.(jpe?g|png)$ %{REQUEST_FILENAME}.webp [T=image/webp,E=accept:1,L]
</IfModule>
<IfModule mod_headers.c>
Header append Vary Accept env=REDIRECT_accept
</IfModule>
AddType image/webp .webp

Also, configure Apache to defend against invalid image requests:

<IfModule mod_rewrite.c>
RewriteEngine On
#fast fail missing standard images
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule \.(jpg|jpeg|png|gif|ico|swf|bmp)$ - [nocase,redirect=404,last]
#if .webp requested but was not created return image without .webp suffix
RewriteCond %{DOCUMENT_ROOT}/$1.$2.webp !-f
RewriteRule (.*).(png|jpe?g|gif).webp$ $1.$2 [T=image/$2,L]
# Rewrite images to non-WebP if webp are not supported and they exist
RewriteCond %{HTTP_ACCEPT} !image/webp
RewriteCond %{DOCUMENT_ROOT}/$1.$2.webp -f
RewriteRule (.*).(png|jpe?g|gif).webp$ $1.$2 [T=image/$2,L]
</IfModule>

These rules are assuming the common scenario that .webp files are generated for each image and saved along with the original image, so for each file mypic.jpg there should be a corresponding file “mypic.jpg.webp”.

  1. The first rule detects if a standard non-webp image does not exist and does a fast fail to 404 not found error code – this is advisable when using frameworks like WordPress otherwise the standard wordpress rules would redirect the not found file to index.php, search virtual ‘permalink’ paths for a match and invoke a php 404 page, which is unnecessarily expensive processing.
  2. The second rule detects if a .webp file does not exist (eg it has not been created yet) and rewrites the request to serve the original image without the .webp extension.
  3. The third rule detects if .webp is requested (and does exist) but the browser does not support .webp and similarly downgrades the request.  Of course this shouldn’t happen but if a .webp reference was hardcoded into a page, this rule would attempt to downgrade the request gracefully.

If there is a proxy in front of Apache (whether nginx or CloudFront, CloudFlare or other CDN, then the Apache Vary Accept header should not be used, otherwise the proxy may cache the “wrong” version of the image.  Similarly the defensive rule 3 may not take effect since if the .webp is requested, the proxy server may serve it from cache without rechecking the request against Apache.  The file not found rules are worth keeping though!

nginx / Engintron

If using nginx as a proxy in front of Apache, nginx should do the replacement work

Replace this in /etc/nginx/common_http.conf:

    # Enable browser cache for images (TTL is 60 days)
    location ~* \.(?:ico|jpg|jpeg|gif|png|webp)$ {
        include proxy_params_common;
        include proxy_params_static;
        expires 60d;
    }

with this:

    # Enable browser cache for images (TTL is 365 days)
    location ~* \.(?:jpg|jpeg|png)$ {
        include proxy_params_common;
        include proxy_params_static;
        expires 365d;

	set $rw "";
	if ($host ~* 'mysite.com') {
		set $rw "A";
	}
	if ($http_accept ~* "webp") {
		set $rw "${rw}B";
	}
	if (-f $request_filename.webp) {
		set $rw "${rw}C";
	}
	if ($rw = "ABC") {
		add_header Vary Accept;
		rewrite (.*) $1.webp;
                #OR actually redirect which will send 302 and additional browser round-trip
		#rewrite (.*) $1.webp redirect;
	}
    }

    location ~* \.(?:ico|gif|webp)$ {
        include proxy_params_common;
        include proxy_params_static;
        expires 365d;
    }

replacing mysite.com as appropriate.
Note that we can choose to rewrite the request, which is transparent to the browser (but also to any intermediate caching layers) or redirect the request (expensive in terms of browser redirects but at least any cached jpg is a real jpg and won’t be inadvertently cached as a webp by any intermediary layer.

The third test (C) only works if nginx knows where the website files are located, if nginx is a caching proxy over apache remove the third statement and the test for C and let Apache handle the file test as per the previous section.

Equally, if a CDN is involved on top of nginx then the implementation needs to move to the CDN..

CDN Implementation

CloudFront over Apache / nginx

With CloudFront in front of Apache or nginx, one possibility is to vary response by Accept header: CloudFront, Edit Behavior, Cache Based on Selected Request Headers, Whitelist, Whitelist Headers >> Accept.

The header Whitelist – Accept works by caching a different version for each header and  depends on the backend webserver understanding how to server up a .webp image where supported as per above examples for Apache and nginx (if NOT using the accept header, better to turn off the support in Apache/nginx to avoid caching webp where not supported by the browser).

A disadvantage of the Accept header method in CloudFront is that it is not a binary rule (webp or jpg), instead the CDN would need to cache as many versions of the image as there are different browsers and versions as these [mostly] send different accept headers, for example:

  • Chrome: accept: image/webp,image/apng,image/*,*/*;q=0.8
  • Firefox (since v65) accept: image/webp,*/*
  • Safari: accept: image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5
  • Opera: accept: image/webp,image/apng,image/*,*/*;q=0.8
  • Microsoft Edge ( 44.18362.449.0): image/png, image/svg+xml, image/*; q=0.8, */*; q=0.5
  • Microsoft Edge (new Chromium version) 11.836.18362.0 accept: image/webp,image/apng,image/*,*/*;q=0.8  [same as Chrome]
  • Internet Explorer 11: Accept: image/png, image/svg+xml, image/jxr, image/*; q=0.8, */*; q=0.5  (tested 11.719.18362.0 and 11.836.18362.0)

CloudFront over S3

S3 does not implement logic based on browser Accept header or other webserver-like rules, so if the static content is coming from S3, any variation for .webp support must be implemented at the CDN layer.

In the case of CloudFront this can be done via Lambda@Edge:

  • Go to AWS console for Lambda
  • Switch to us-east-1 region: Lambda@Edge functions must be created in the US East (N. Virginia) Region.
  • Functions, Create Function
  • create a function called eg cfwebp, example below will use node.js, create a new role from Policy Templates select Basic Lambda@Edge permissions (for CloudFront trigger), code as below.
  • Create Test events using one of the CloudFront templates and add appropriate test cases for different url and accept headers
  • when happy with function, publish a Version (CloudFront requires a published version and does not accept LATEST)
  • edit the relevant CloudFront Behaviour (for example the behaviour may be scoped to Path Pattern .jpg), go to Lambda Function Associations and on Viewer Request add the ARN of the function.

Here’s a simple code example in node.js that adds .webp to a jpg request if the browser accept includes webp:

exports.handler = async (event, context, callback) => {
	const request = event.Records[0].cf.request
	const headers = request.headers
	const acceptHeader = headers['accept'] && headers['accept'][0] ? headers['accept'][0].value : ''

        //if browser accepts webp
	if (acceptHeader.indexOf('webp')>0 ) {
	    //add .webp to requests for which webp should be available
	    const suffix = request.uri.split(".").pop()
	    const ext = suffix.split("?").shift()
            //console.info('testing suffix: ' + suffix + ' ext: ' + ext);
	    switch (ext){
	        case 'png':
	        case 'jpg':
	        case 'jpeg':
        		// Replace supported image formats with WebP
        		request.uri = request.uri + '.webp'
        		break
	        default:
	            //console.info('skipped: ' + request.uri)
	    }
	} else {
	    //console.info('skipped unsupported browser accept: ' + String(acceptHeader))
	}
	return callback(null, request)
};

Important notes:

  • this example is not ‘safe’ – there is no test that the .webp file actually exists and S3 will offer no fallback if it does not.  Depending on the use case another lambda function could dynamically create missing .webp files or could fall back to strip the .webp extension.
  • if the Lambda@Edge solution is attached to the Viewer Request it is NOT necessary to to “Cache Based on Selected Request Headers” , “Accept”: the function is modifying the request and CloudFront is then serving and caching the .webp and original image as requested.
  • simply updating the lambda function has no effect, the lambda function has to be created in N.Virginia region and each update has to be specifically published to the CloudFront Edge locations to take effect.
  • allow up to 15 minutes for CloudFront changes to distribute across locations
  • always debug with new files, don’t get caught out by files already cached from the previous test
  • the Lambda@Edge function executions cannot be monitored through the Lambda and Lambda consoles since the execution is happening at the edge location not the Lambda function location.  Instead using the monitoring within CloudFront which will also link through to the CloudWatch log files from each location.
  • Lambda@Edge pricing is pricing is $0.60 per 1 million requests with no free tier – which is 3x more expensive than standard Lambda executions at $0.20 per 1M requests (with free tier of 1M free requests per month).
  • for all the above reasons, unit test the Edge function thoroughly before deploying.  Use some monitoring code – any console.info type statements will be echoed to the log.

CloudFlare

CloudFlare generally has a lot more pre-built functionality which is a lot simpler to implement than CloudFront, and their Polish functionality since dec 2016 has been offering image optimisation including .webp.  This is not available on the CloudFlare free plans (which are also great and easy to use but may be best avoided unless your server and clients are in the US).

Conclusions

Server-side varying of response based on client capability (in this case serving .webp when .jpg was asked for can be problematic:

  • .webp may become cached by a proxy server and served to a browser which doesn’t support it
  • expected .webp versions may be missing
  • solution may be brittle / easily broken by changes to server or CDN configuration

A simpler option is javascript: a javascript function which rewrites the src/srcset of images, combined with a lazy loader will give a fast page load and solve main performance concerns without any compatibility issues, degrading gracefully back to .jpg.  For more WordPress specific image handling see also Images for all sizes.

part of a series on cPanel migration to AWS

3 thoughts on “Google says “webp”

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s