Author: kenharker

Categories
DevOps

Ideas for Incorporating RUM into Your Release Cycle

Programmatically Interact with RUM data

Real User Measurement (RUM) can significantly help you manage the quality of your releases, by providing near-immediate feedback on site health. One of the most powerful ways to leverage this feedback is through API calls. Most RUM platforms offer API access to their system for a variety of tasks and queries. When considering RUM, look for a platform that support using API calls to create and update data collection configurations, to add annotations to the RUM data, and to query the RUM data on demand.

You can leverage APIs to automate tasks during builds and releases. The most valuable use of APIs is to annotate RUM data when you push out changes so you can later track changes back to the specific change/release that might be responsible. Another valuable use, but one that is more complex to deploy is to use RUM data to validate that your site performance is still meeting target levels after the release.

Breadcrumbs for Diagnostics

Annotations are like breadcrumbs for diagnostics. Not every issue introduced during a release will be immediately obvious. Some performance issues may take a while to identify, either because they affect some smaller subset of the visitor population, because the degradation is only in one key performance timers, or because the degradation is subtle and only something that can be identified with a large sample size. Whatever the problem, if the regression can be traced back to a point in time, and that point in time matches an annotation in your RUM data that says “we pushed out a release”, you will likely save a lot of time and energy tracking the source of the issue.

Annotations should not be limited to externally-visible site changes, of course. You should use API calls to annotate RUM data for all meaningful changes in the delivery architecture. This should include updates or changes to servers, load balancers, web application firewalls, and other components even if the end user would not see a change in page design or content. You might consider annotations for major business cycle events, such as the beginning and end of major sales or promotional periods as well.

Integrating mPulse into Release Pipelines

The next step beyond simply annotating the RUM data with information about changes is to leverage the RUM data itself to make informed decisions about the health of the build process. With API calls, most RUM platforms will let you pull summary data about the health of your web site. This gives you the opportunity to do a spot check shortly after a deployment to see if the performance of the site has regressed. Or, at the very least, archive a set of performance benchmarks with each release for later manual analysis and consideration.

The most advanced users of RUM in a release pipeline might even fail and rollback a build if the performance numbers are not up to specification. Even if you are not ready for that level of integration, archiving a performance benchmark, and perhaps automatically reporting the performance numbers to critical team members as a checkpoint in the release can help build a culture of performance within your organization. Knowing that RUM will keep everyone honest when it comes to performance will ensure that performance is valued within your teams.

Categories
Web Performance Basics

A Brief Introduction to Real User Measurement

How RUM Works

Real User Measurement (also known by its acronym, RUM) collects performance (and other) information about the real user experience of a web page load or other navigation on your site. It does this by asking your visitors’ own web browsers to report on their experiences by sending a beacon of data to a collection repository where it can be aggregated and analyzed. All RUM systems accomplish this through JavaScript, and there are four primary steps involved.

First, a JavaScript loader snippet or tag is placed on every page that you want to have measured. This small chunk of code executes during the page load to kick things off. Next, this loader snippet will request a core JavaScript library that contains the logic to interface with the browser through W3C standard APIs for data collection. For most RUM platforms, this core JavaScript library is a variant of the open source Boomerang software. Next, a configuration file (typically in JSON format) loads in the browser that has site and/or page-specific data collection config details. Finally, when the page load or navigation is complete, the JavaScript that has loaded interacts with the browser through W3C APIs and other standards to assemble a “beacon” of data that is delivered to a data repository and visualization system.

Where the Data Comes From

RUM systems collect data from the browsers using a variety of industry standard JavaScript APIs. The core data collection relies upon four standard APIs defined by the World Wide Web Consortium (W3C) These standards have been adopted by all major browsers:

  • Navigation Timing API
  • Resource Timing API
  • Paint Timing API
  • User Timing API

You should encounter very, very few real users who are still on browsers that predate these standards. Google browsers have been the earliest adopters of these standards. Microsoft browsers have supported these standards wince 2012. The final major browser to adopt Navigation Timing was Apple Mobile Safari, which included support beginning with version 8 in 2014. For details on browser adoption: https://www.caniuse.com/.

What Kind of Timers to Measure?

“Page Load Time” is just the starting point when it comes to Real User Measurements. Depending on the RUM platform, data collection may include some or all of the following timers and metrics. Which ones you focus on may change depending on what problems you are trying to solve with the data. Highly popular at the moment are the “core web vitals” measurements of Largest Contentful Paint, Cumulative Layout Shirt, and First Input Delay. These three measurements speak more directly to the user experience than traditional timers that speak to the network or infrastructure performance.

  • Time to First Byte
  • First Contentful Paint
  • Largest Contentful Paint
  • DOM Loading
  • Time to Visually Ready
  • Page Load Time
  • Time to Interactive
  • Time to First Interaction
  • First Input Delay
  • Cumulative Layout Shift
  • DNS Lookup Time
  • TCP Connection Time
  • SSL Connection Time
  • Resource Load Time
  • CDN Edge Delivery Time
  • Origin Time
  • User Timers
  • Rage Clicks

What Kind of Dimensions to Capture?

In addition to capture timers and other performance-related data, any effective RUM solution will capture a wide variety of dimensional data as well. Performance is almost never uniform across all kinds of site content or the wide variety of site visitors. Dimensional breakdowns are critical to finding those “soft spots” in the user experience on your site. Depending on the RUM platform, data collection may include some or all of the following dimensional data:

  • Geographic location
  • Browser
  • Operating System
  • Device Details
  • Page Group, Page Name, or Page Type
  • IP Version
  • TLS Version
  • ISP
  • Network Connection Type
  • Multivariate Test and Test Segment
  • Custom Dimensions
  • Visibility State

RUM and Business Intelligence

While all RUM platforms offer real-time dashboards, basic reports, and alerts, one of the most powerful uses of a RUM system is the ability to connect your visitors’ page load or navigational experiences with the business outcome. You can use RUM data to answer questions like “How fast do I need to be to maximize my primary business metric?” or “What pages on the site are most impactful to my primary business objective?”.

We have the ability to answer these questions because RUM system collect performance data from visitors at the exact same time that it can collect business metric information. For example, if your site is an online retail service, then your business objective is to convert a visitor into a purchaser, and you can measure the success of that objective for each visit by whether the visit includes an order confirmation page view or not. By looking at the web performance experienced by visitors who convert, and comparing it to those who do not, you can gain significant insights to help your business.

The most advanced RUM services and consultants can even prepare data models to let you extrapolate from current site experiences to possible future results. We call this “What If” modeling, and it can be a powerful tool for planning, or even simply justifying, your investment in web performance

Categories
Web Performance Basics

A Brief Overview of Web Performance Management

Being Slow Can Be Worse Than Being Down

While availability is obviously critical to a successful enterprise on the web, the impact of slow page load performance is often underestimated. Performance affects the user perception of site quality, and not just at the moment in time when the user is trying to use your site. The effect of slow experiences persists into the future. Visitors who fail to load your site might be more forgiving and willing to try again in the future, whereas those whose experiences were bad may shy away from a return visit.

This is hard to appreciate fully.

We have, however, known of the importance of fast page loads for some time. As early as 2006, Google published a landmark study showing that 500 ms of delay reduced web searches by 25%. In 2009, Bing and Google published joint experiments that showed that as little as 200 ms of additional delay in page load times was enough to measurably change user engagement. 500 ms of additional delay was all it took to impact revenue on a commercial web site.

And nothing has really changed in the past decade. In 2013, Walmart revealed that every 100 ms improvement in page load time resulted in a 1% increase in revenue. In 2017, Akamai publishes biannual State of Online Retail Performance report that showed that a 100 ms delay in web site load time can hurt conversion rates by 7%. In 2018, Fashion retailer Missguided removed BazaarVoice for Android visitors; median page load time improved by 4 seconds, and revenue increased by 26%. There are dozens of other published studies and assessments showing the strong connection between web performance and business results, and a great resource to find them is https://wpostats.com/.

So You Need to Measure Performance

To fully understand how site performance is affecting your business, you need to take control with measurements to guide you. Web performance measurements give you a baseline to know where you are starting from. As you make changes to improve performance (or simply to add or change features on the site) you need to have a baseline to know whether the site is getting faster or slower. Web performance measurements also let you track changes over time. Even if you are not making major changes to the site design, it is important to track performance to ensure that users continue to get the same experience.

How we measure web performance also matters. Your measurements need to be as accurate and complete as possible. The measurements and the tools you use to visualize them should help you readily identify success and failure. It is not enough to know if the site has a 3 second median page load time, for example. You need to know that page load time is the correct metric for assessing the health of the online site, and whether 3 seconds is a successful or unsuccessful result.

The Ecosystem of Web Performance Measurement

Without getting into too much detail, web performance is commonly measured in four approaches today:

APM Short for Application Performance Monitoring, APM solutions help you assess the health and performance of the “nuts and bolts” of your infrastructure and application components. APM solutions primarily track events inside your own data center or in your own private cloud.

Synthetic This solution uses “agents” or “probes” to visit your web site robotically on a fixed schedule, measuring the performance from the (simulated) visitor’s point of view. Synthetic measurements may be scripted to complete multi-page transactions. Because each measurement is made using controlled variables (same locations, same browsers, etc.) they are great for probing for weaknesses and establishing benchmarks.

Web Analytics While many web analytics offerings include some performance measurement capability, these solutions are primarily aimed at categorizing and understanding your visitors and their behaviors on your site. Web analytics are popular because some of the basic offerings are free.

RUM Short for Real User Measurements, RUM solutions leverage JavaScript embedded on your web pages to query your visitors’ browsers for detailed performance information about the page load experience and send that information to a central repository through “beacons”. RUM offers the ability to get very comprehensive coverage up to 100% of all visitor experiences.

These four main approaches to web performance assessment are more complementary to each other than in direct competition. They each have their strengths and weaknesses. APM services offer less direct insight into overall visitor experience, and are generally unable to track performance of third-party or platform services on your pages. Synthetic measurements offer no insight into visitor engagement, and no insight into conversion and revenue. Web analytics offers only very light, to no, insights into performance. Real User Measurement services collect data sets that can be extremely large, the data can be noisy with many outliers, and distilling this data into business intelligence can be a challenge.

All of that said, Real User Measurements offer some significant benefits. You should think of them as the “ground truth” for visitor experience. They are not simulating real-world conditions or looking at a limited portion of the user experience. These measurement come directly from your visitors’ own browsers. RUM measurements will covers all variety of site visitors (across geography, device type, connection type). The stronger recommendation for RUM, however, is that it can captures both performance and business metric data at the same time. Knowing what the user experience was like when they complete a purchase or choose to view an another article, and what it was like if they did not, is key to making good business decisions about site performance.