Measure Performance

Summary: 
Establish a base metric before the switch to ImageEngine.

Use a third-party tool like WebPageTest to test from multiple locations around the globe, using real browsers and real consumer connection speeds.

User-centric metrics are ideal. These include:

• "Largest Contentful Paint”
• “Time to First Meaningful Paint"
• Lighthouse or other site's “Performance” scores

Don’t forget to monitor the actual payload reduction related to images.

Metrics like “Time To First Byte” are not as important because they don’t reflect user experience.

Still not getting the results you expect? It may be time to investigate your CSS and JS.

There are many tools and services out there that can help to monitor and measure web performance. Performance is a moving target and different metrics serve different purposes. In the following, we will look at a few legacy metrics in addition to user-centric metrics. Traditional performance metrics like load time or DOMContentLoaded time are extremely unreliable since when they occur may or may not correspond to when the user thinks the webpage is loaded. Unfortunately, these legacy metrics are still reported by many analytics tools and considered a KPI for many sites.

ImageEngine is all about improving the user experience. Improved user experience has a direct impact on the bottom line. But how should you measure the impact of ImageEngine?

Let’s start by looking at some traditional metrics.

Traditional Performance Metrics

Traditional metrics can be described as metrics that are easily collected programmatically based on events fired in the browser or HTTP traffic timings. These metrics are dubbed “traditional” because the way a webpage is implemented has developed significantly in the last couple of years. Additionally, the definition of performance has changed. For example, a low TTFB, doesn’t necessarily mean that the key content is visible to the end-user any sooner.

Time To First Byte

Time to first byte (TTFB) may be defined as:

“Time spent waiting for the initial response, also known as the Time To First Byte. This time captures the latency of a round trip to the server in addition to the time spent waiting for the server to deliver the response.”

In the traditional CDN space, a low TTFB has always been an important metric. TTFB is definitely still very important, as a delay here will add directly to any other metric. Looking at this only will however give a skewed picture of how fast a webpage is from the users’ perspective.

Page Load Time

Page load time is usually reported by the load event or DOMContentLoaded event.

The problem with page load time on today’s modern web is that these events can fire any time regardless of when content is shown or the page is usable for the user. Reasons for this may be asynchronous loading of resources, lazy loading, third-party dependencies, or pages rendered client-side using JavaScript. Page Load Time related metrics are available by default in most tools, including Google Analytics, but should be used with care.

Averages

Another problem with Google Analytics and other tools is that they report averages.

Because of the complexity of the modern web, and the vast number of variables in play when a page has loaded, an average is heavily skewed by the long tail of outliers. Whenever possible, stick to percentiles.

User-Centric Performance Metrics

One way to define user-centric performance metrics is thinking of the characteristics of a page which from the users’ perspective is fast.

The above image illustrates when some user-centric metrics occur during the page load process.
Note that TTFB and page load time do not correlate well to any of these.

Google also talks about user-centric metrics as “Web vitals”.

Web Vitals is an initiative by Google to provide unified guidance for quality signals that are essential to delivering a great user experience on the web.

A complete definition of user-centric metrics can be found elsewhere. Google illustrates the most important, Core Web Vitals like this:

Largest Contentful Paint

The most recent metric implemented by Google Lighthouse is Largest Contentful Paint (LCP), deprecating First Meaningful Paint, in Lighthouse 6.0.

“LCP marks the point in the page load timeline when the page’s main content has likely loaded—a fast LCP helps reassure the user that the page is useful.”
(from web.dev)

It is similar to the other paint-based metrics and it measures how soon content is visible to the user.

Lighthouse and Page Speed Insights

Google, with its focus on web performance, has implemented user-centric metrics in Lighthouse and PageSpeed Insights. The webpage will get a “performance grade” composed of many different metrics, such as First Contentful Paint, Speed Index, Time to Interactive, and more. How these metrics are weighted vary slightly across versions. Here is a neat tool that illustrates the weighting and how the score is computed.

The main takeaway is that improvements in user-centric metrics are more likely to impact your business than the traditional metrics.

Metrics to Monitor ImageEngine Performance

When evaluating ImageEngine performance it is important to establish a current baseline or in other ways create a data set (like A/B tests) to which ImageEngine performance metrics can be compared. As explained in this document, user-centric metrics are recommended. If you don’t have these metrics implemented it is highly recommended to do so.

At the same time, you should stick to whatever datasets you have at hand and datasets that your business use on a daily basis. These sets can for example be whatever is available by default in Google Analytics or other analytics tools you may use. It is very likely that ImageEngine will have a positive impact on all performance-related metrics.

Make sure the ImageEngine cache is warm (few cache missses) before running performance tests! If not there will be a high degree of cache misses which will slow down delivery.

Payload Reduction

Monitoring the reduction in payload generated by images is probably the most isolated metric to look at when evaluating ImageEngine. Comparing the byte size of images with- and without ImageEngine enabled clearly illustrate the core purpose of ImageEngine.

The easiest way to monitor this is either through the control panel or using tools like webpagetest.org to compare before and after data.

Time To First (Last?) Byte

TTFB may vary more with ImageEngine than with other CDNs because ImageEngine does real-time optimization. In the case when ImageEngine has to produce a new version of an image, a few extra milliseconds are spent server-side before returning the payload. You’ll notice this as outliers in your datasets.

ImageEngine performs quite well compared to other CDNs with regards to TTFB when the cache is properly warmed:

It is also important to remember that even if ImageEngine might sometimes be slower in TTFB, ImageEngine will send less data over the wire. Especially in slow mobile networks, less data transfer will weigh up for a delayed TTFB because most of the time on slow networks is spent on downloading data to the browser. In these settings what really counts is “Time to Last Byte”.

Largest Contentful Paint and Images

How fast an image is loaded and painted on the screen has a direct impact on LCP. From ImageEngine’s perspective, this is then one of the more important metrics to keep an eye on.

In addition to blocking CSS or slow JavaScript rendered content, Server response time and asset/sub-resource (typically images) load time are the primary factors affecting LPC.

As currently specified in the Largest Contentful Paint API, the types of elements considered for Largest Contentful Paint are:

• <img> elements
• <image> elements inside an <svg> element
• <video> elements (the poster image is used)
• An element with a background image loaded via the url() function (as opposed to a CSS gradient)
• Block-level elements containing text nodes or other inline-level text elements children.

How to measure LCP

LCP can be monitored in Chrome User Experience Report or by using the Largest Contentful Paint API. The following example shows how to create a PerformanceObserver that listens for largest-contentful-paint entries and logs the LCP value to the console:

// Create a variable to hold the latest LCP value (since it can change).
let lcp;

// Create the PerformanceObserver instance.
const po = new PerformanceObserver((entryList) => {
  const entries = entryList.getEntries();
  const lastEntry = entries[entries.length - 1];

// Update `lcp` to the latest value, using `renderTime` if it's available,
// otherwise using `loadTime`. (Note: `renderTime` may not be available if
// the element is an image and it's loaded cross-origin without the
// `Timing-Allow-Origin` header.)
  lcp = lastEntry.renderTime || lastEntry.loadTime;
});

// Observe entries of type `largest-contentful-paint`, including buffered
// entries, i.e. entries that occurred before calling `observe()`.
po.observe({type: 'largest-contentful-paint', buffered: true});

// Send the latest LCP value to your analytics server once the user
// leaves the tab.
addEventListener('visibilitychange', function fn() {
  if (lcp && document.visibilityState === 'hidden') {
   console.log('LCP:', lcp);
   removeEventListener('visibilitychange', fn, true);
  }
}, true);