Time and again working on big web applications we customize files based on user’s platform, and their preferences. We can send different files to legacy browsers, different CSS and JS to mobile browsers depending on their form factor, different images to accomodate bandwidth requirements, and so on.
This post was prompted by my desire to serve sprites produced by
as WebP images to WebP-capable browsers falling back to “classic” image formats for the rest using nginx.
While it is hardly a new topic, I was not satisfied with existing solutions, which all used
instead of simpler methods.
Obviously the core solution can be used to serve almost any file conditionally, not only images, but all examples will be about WebP.
The problem at hand
One common technique to conserve connections and bandwidth, especially on mobile platforms, is to use sprites instead of individual images. See Sprites by CSS on Wikipedia for more details. While the upcoming HTTP 2.0 will reduce the need for sprites by using one connection to request images (and other files) asynchronously, the technique will still be valid due to a possible bandwidth gain by compressing several images together.
Generally sprites (or sprite sheets) are relatively big images, so every bit of extra compression helps. That’s why we crush resulting sprites with zopflipng or with jpegtran. We take our images to the limit.
But can we do better? Enter WebP.
WebP is “a new image format that provides lossless and lossy compression for images on the web.” It improves on lossless PNG and lossy JPEG without compromizing on visual quality, and implements new features, like lossy alpha channel.
WebP-capable browsers advertise themselves by including
"image/webp" string in an HTTP
Given the popularity of Chrome, and gazillions of Android devices on the market, it makes perfect sense to support WebP in addition to legacy formats.
Now we can sketch the final goal:
- Let’s clone our images and produce WebP version of them.
- If user’s browser supports WebP, we will send them a WebP version.
- If we don’t have a WebP version for some reason, we will send a legacy version.
- All other users will receive a PNG or JPEG image.
Now we have to serve files making decisions. Let’s remove from the table web services, and other high-level solutions — they are too heavy for our needs. Let’s leverage our web server: both Apache and nginx can do it using “native” tools.
Hitting the web we will find a lot of good battle-tested recipes for nginx. The problem is that all
of them “smell” like Apache + C: they use
rewrite to achive their goal.
One of the best articles on this topic is Deploying WebP via Accept Content Negotiation
(code repository) by Ilya Grigorik
of Google — it is small, concise, practical. Yet again, it relies on
What’s wrong with
- According to nginx Wiki “if is evil”
(yes, this is how it is phrased in Wiki).
ifhas problems when used in location context, in some cases it doesn’t do what you expect but something completely different instead. In some cases it even segfaults. It’s generally a good idea to avoid it if possible.”
- It is frequently misunderstood, and misused. In short: it is not
- Performance problem: server-level
ifis evaluated for every request, even when non-images are requested. The same is true for unrestricted location-level
- We make assumptions, while rewriting. For example, it will redirect, even if new URI points
to a non-existent WebP file. We should serve a legacy file instead, yet there is no simple recourse for that.
- Performance problem: the best existing solutions check for a file first, and account for its absense.
Yet they do the check even when we know that user cannot accept WebP due to limitations of
- Performance problem: the best existing solutions check for a file first, and account for its absense. Yet they do the check even when we know that user cannot accept WebP due to limitations of
- Performance problem: it is a subject of several pitfalls including so-called “Taxing Rewrites”, when no regular expression is required, yet evaluated.
- We make assumptions, while rewriting. For example, it will redirect, even if new URI points to a non-existent WebP file. We should serve a legacy file instead, yet there is no simple recourse for that.
All those problems are usually present in existing solutions I found on the web.
Let’s try something completely different.
The idea is still the same: we check HTTP
Accept header for
"webp", and if it is there, we serve
an alternative file. But instead of
rewrite we will use lightweight
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
This is it. The real configuration file is likely to use more cache-control headers, but what we have here is realistic enough to demonstrate the technique.
For your convenience, the snippet above is available as GitHub Gist.
How it works
Only three parts are important:
mime.types should list
webp, we should define a variable depending on
Accept header, and we should use it in
nginx uses a file to list mappings from file extensions to MIME types. Usually it is called
mime.types, and included externally. Make sure that it lists
Otherwise, a file will be sent as
application/octet-stream, which will prompt a user to download it
rather than show it inline.
map defines a variable that depends on values of other variables (see
ngx_http_map_module for details).
This module is included in nginx by default, you don’t need to recompile anything.
1 2 3 4
http-level snippet defines a variable called
$webp_suffix, which depends on
$http_accept (our HTTP
Accept header). If the header contains
"webp" substring (using a case-insensitive regular expression),
than our variable will be set to
".webp", otherwise it will be an empty string.
Strictly speaking defining a default as an empty string is superfluous: this is the default behavior anyway. I added this excessive line here for clarity.
Interesting thing about nginx’s variables is that they are all lazily calculated, so we can define a lot of them without slowing down our server — only variables, which we actually use, will be evaluated. In our case, it means that adding our variable does not affect serving other non-image files.
This is a workhorse of the solution:
location-level directive checks files conditionally breaking on success:
- Checks a file + a possible
".webp"suffix, and serves it, if it is found.
- Checks a file as it was requested, and serves it, if it is found.
HTTP404(AKA “not found”), if not found.
Let’s go over it in details. We assume that we have file called
image.png and its possible counterpart
- User comes with a WebP-capable browser:
$webp_suffixis set to
image.png.webp. It is served, if found.
image.png(the original file). It is served, if found.
- Otherwise “not found” is returned.
- User comes with a browser that knows nothing about WebP:
$webp_suffixis set to
image.png. It is served, if found.
image.png(the original file).
- While it is clearly redundant, it is unlikely that an image is not found. If this is a common situation for an application you develop, remember that nginx caches files, so it will likely be amortized.
- Otherwise “not found” is returned.
try_files is a core module directive. It may check files on disk, redirect to proxies, or internal locations, and return error codes, all in one directive. See try_files for more details.
Our solution uses a simplified approach based on
try_files. It avoids common performance pitfalls
without complicating our logic.
Appendix: JPEG XR
Google is not alone in defining a better image specification. Microsoft came out to play with its JPEG XR, which is a serious contender too. It is supported in their line of Internet Explorer browsers starting with version 9, and grunt-tight-sprite has a recipe for producing JPEG XR files.
Obviously it would be nice to send JPEG XR-encoded sprites to IE9+. Unfortunately it appears that those browsers do not indicate that they accept this advanced image format. The only way is to parse and interpret User Agent string, which is unreliable, and generally discouraged even by Microsoft.
Why did they do that? Apparently there is a school of thought that if we add all accepted types to HTTP
Accept header, which is sent with every HTTP request, all communications will be extremely bloated.
The latter is exacerbated by the fact that headers are sent uncompressed (will be fixed in HTTP 2.0).
What is their solution? I didn’t see any yet. Which leaves us with User Agent, sigh.
So I will leave the implementation of sending JPEG XR conditionally as an exercise for my readers.
To get you started, here is a hint: don’t forget to add necessary MIME types to