In this Watershed client spotlight, see how Caterpillar is using Scatter reports to highlight their best videos and use that information to improve future L&D initiatives. We've also included a guide to help you start exploring the outstanding content and resources in your organization.
An Overview
Caterpillar, Inc. (CAT) is a Fortune 100 corporation that designs, develops, engineers, manufactures, and sells machinery, engines, financial products, and insurance via 172 dealers worldwide.
CAT is implementing a visionary learning ecosystem with multiple learning platforms and tools integrated with xAPI. (Read more about this vision in their xAPI Case Study.)
CAT has created hundreds of Watershed reports across multiple dashboards that explore their learning ecosystem data from different angles. And while we can tell many stories from this data, this client story focuses on a handful of related reports showing data about one learner activity in one application: watching videos on Kaltura MediaSpace.
The Challenge: What videos stand out?
With thousands of learners watching thousands of different videos each month, Alfonso Riley, learning and performance strategist at CAT, wanted to highlight the videos that stood out as especially popular and were being watched most often.
In particular, he wanted to pinpoint the videos with the most watches per person per month. In other words, which videos are learners coming back to again and again?
The Solution: The Kaltura Anomaly Detection Dashboard
To answer this question, Alfonso created a dashboard called Kaltura Anomaly Detection. This dashboard contains a series of scatter reports in Watershed, one per month, each designed to highlight anomalous videos in a given month.
The y-axis (left side) of the following scatter report measures unique users watching a video, while the x-axis (bottom) shows the number of times a quarter of a video is watched. So, watching a whole video from start to finish will generate four interactions.
This means videos that are watched many times per person are represented by dots that are significantly more to the right compared to videos with a similar number of viewers, which appear on the left side of the report.
You can immediately see how easy it is to identify outliers. Alfonso also can hover his cursor over each dot to see a video’s name with its respective person and interaction counts.
Recommended Reading
What does it mean?
The first thing to note about the report is how the background shows quadrants to highlight the top and bottom 50% of videos both in terms of the number of people who watched each video and the number of times each video was watched.
- Quadrant A: This is the white (and green) space that takes up the majority of the report and represents videos in the top 50% in terms of both viewers and watches.
- Quadrant B: This is the thin gray bar on the left side and shows the top 50% of viewers, but the bottom 50% of watches.
- Quadrant C: This is the thin gray bar along the bottom and shows videos in the top 50% of watches, but the bottom 50% of viewers.
- Quadrant D: This is the area in the lower left corner and shows the bottom 50% on both metrics. It covers such a small range that it is entirely buried in overlapping dots.
This report shows the majority of videos are watched only a few times per month. (This is why the B, C, and D quadrants are so small—the dots are concentrated into a smaller space.)
Any dot on the report that is distinguishable from the others is one of the top videos. And there are a handful of dots representing “exceptional” videos that stand out above the top videos.
The second thing to note is there’s a clear line showing most videos have approximately 4 “watch” interactions per person. This is unsurprising since we are counting quarters of a video watched: on average, people watch each whole video once.
That’s not to say most people complete the whole video, but these people are balanced out by other people who might watch part or all of that same video multiple times. As a result, people watch an average of four video quarter segments each.
Videos that buck this trend are interesting anomalies to explore. There is one video in the top left that was watched by 99 people, which is higher than any other video.
However, quarter segments were only watched 275 times—an average of only 2.8 quarter segments watched per person. This video continued to be a popular video, but with comparatively low watch counts in January and February 2019. This is a good video to explore in more detail to see which video segments are not being watched and why.
For instance, if the first two video segments comprise the majority of views, perhaps the video is too long and people are dropping out before they reach the end.
Other videos appear to the right of this line in varying degrees. People are watching these videos multiple times. The video farthest to the right, for example, had 1,103 quarter segment watches for only 40 people.
That’s the equivalent of each person watching the whole video nearly seven times during the course of the month. The video that is most popular in December is still exceedingly popular in January but is no longer the most popular. In February it is even less popular and is no longer one of the exceedingly popular videos.
Again, this is a good video to explore in more detail to understand its change in popularity over time. Is there something that could be replicated for future videos?
For instance, because the video corresponded to a new CAT product launched in November 2018, could the additional watches have been a result of sales reps showing the video to customers? If so, what made reps use this video for that purpose?
Recommended Reading
Rate of Change
Separating the reports by month gives Alfonso an additional dimension to explore the data. He can look at which videos are the outlier successes in a given month and then compare where these videos appear in previous and subsequent months.
Like a radio DJ keeping track of the chart positions of pop songs, Alfonso can look at newly popular videos and see how they rise and fall compared to others.
Actionable Insights
A scatter report configured like the one above doesn’t directly give actionable insight. In fact, it may leave you with more questions than when you arrived.
However, a scatter plot report is ideal for highlighting exceptional outlying videos and other learning content that are performing especially well. Following Brinkerhoff's Success Case Model, you can then further explore those successes to gain actionable insights.
Further analysis of these exceptionally successful videos might include:
- Which parts of the video are people watching most often? This might give insights around the optimum length of videos or types of content that either keep people watching or put them off.
- Who is watching the video? Is it a few people watching many times or many people watching a few times? This might give insights into how to better target videos for specific audiences.
- How often are people watching the video over time? This might give insights into the expected lifecycle of your videos.
- How often are people liking or commenting on the video? This might give insights into how social interactions affect a video’s popularity.
Use Scatter Reports to Highlight Popular Learning Content [GUIDE]
Use the Brinkerhoff Success Case Method to explore the outstanding content and resources in your organization. Start by creating a scatter report to highlight those outliers. Here are the resources you’ll need:
Or, if you’re interested in finding out more about reporting on data from Kaltura MediaSpace, check out the Kaltura Data Source Guide.
About the author
As a co-author of xAPI, Andrew has been instrumental in revolutionizing the way we approach data-driven learning design. With his extensive background in instructional design and development, he’s an expert in crafting engaging learning experiences and a master at building robust learning platforms in both corporate and academic environments. Andrew’s journey began with a simple belief: learning should be meaningful, measurable, and, most importantly, enjoyable. This belief has led him to work with some of the industry’s most innovative organizations and thought leaders, helping them unlock the true potential of their learning strategies. Andrew has also shared his insights at conferences and workshops across the globe, empowering others to harness the power of data in their own learning initiatives.
Subscribe to our blog
