The Latest Tools & Tech Trends in Learning Analytics [Survey Results]

The State of Learning Analytics: Our Approach and Methodology

This blog series explores the State of Learning Analytics, combining the results from two recent surveys, which provide insights from more than 1,000 L&D professionals on applying learning analytics in their organizations.

1. Measuring the Business Impact of Learning. Since 2016, Watershed has partnered with LEO Learning to conduct the annual Measuring the Business Impact of Learning (MBIL) survey.
2. Adopting Learning Analytics: Closing the C-Suite/L&D Language Gap. Watershed teamed up with Chief Learning Officer magazine to survey its readers on topics relating to L&D and learning analytics.

Having the Right Learning Analytics Tools and Technology Matters

Our last blog post explored the importance of having the right people in place to turn your raw data into actionable insights. But this is just part of the equation; to achieve scalable, reliable and cost-efficient analytics over time, you need the right technology.

So what tech stacks are organizations currently using? Our CLO survey found:

• 60% use Excel
• 59% used built-in reporting (e.g. LMS reports)
• 26% used BI tools, and
• 13 % use a learning analytics platform

Note that respondents could select “all that apply,” hence the combined totals more than 100%.

Respondents who selected ‘None of the above’ were encouraged to leave commentary. There were no real consistent trends here, but a few made reference to their organization building their own dashboards.

The build versus buy debate is an interesting one for organizations, especially if you struggle to get stakeholder buy-in to invest in a dedicated Learning Analytics Platform. Maintaining reliable data governance and manually handling ecosystem data over time can be an expensive operation.

French food-products giant Danone employs more than 100,000 people across 57 countries and connected 7 major data sources into their Watershed account. They benefit from automated, real-time data connections that feed their analytics, saving an estimated €120,000 per year compared to the cost of manually handling this data.

Likewise, Nebraska Medicine uses automated data connections in their hand-hygiene program, saving them an estimated $4050 per month.

Do large organizations use different technology for Learning Analytics?

We were also curious to see how this varied by organization size:

Segmenting these results by organization size is interesting, as often the larger the organization, the greater the need to handle data efficiently (due to larger ecosystems and learner numbers). 

There is also more potential for having a dedicated analytics budget. We’ll revisit the budget conversation in our next blog post.

Points to note:

• Excel usage drops in large organizations (20,000+)
• Built-in reports are consistently used in organizations with more than 2,500 employees
• Learning Analytics Platforms are more common in large organizations
• BI tool usage is consistently higher in organizations with more than 2,500 employees.
• But a surprising number of smaller organizations (<2,500) use them too

Excel is widely used, but many organizations want to make the switch.

The high levels (60%) of Excel usage for learning analytics are no surprise due to the availability and familiarity with the program.

We segmented the data to cross-reference those who answered “Are you planning to change” [your technology stack] to those who had selected Excel.

These organizations are planning on:

• switching from Excel to other tools (45%)
• changing their tool sets but keeping Excel (15%), or
• not making any changes to their learning analytics toolkits (40%).

It is equally unsurprising that many organizations want to move away from using spreadsheets for learning analytics. Spreadsheet software is not explicitly designed for learning analytics and is best suited for one-off report generation rather than ongoing data monitoring and reporting.

That’s because building reports in Excel is often arduous, and the work needs to be repeated when new data is added. This process can result in a delay between when data is generated and when reports are available. And in some cases, the data is no longer useful by the time it is ready to be used.

Part of Nebraska Medicine’s use case to invest in automated data feeds was to enable real-time reporting that would allow for immediate interventions by learning leaders. Prior to this, data was manually recorded from pen and paper observations and could take between a week to a month to be manually uploaded.

Built-In Reports Are Useful But Don’t Tell the Whole Story

Built-in reports featured prominently (59%) in organizations of all sizes, notably so in organizations with over 2,500 employees. And it’s no surprise - LMS and other platforms often contain out-of-the-box reporting options, which are convenient and offer a certain level of instant insights.

Alongside built-in reports, 60% of these organizations use Excel, 32% use BI tools, and 14% use a learning analytics platform.

30% are planning to stop using built-in reports in favor of other tools, 21% are planning to change their toolkit but keep built-in reports, and 50% are not planning to change the tools they use for learning analytics.

Like spreadsheets, built-in reports are common for learning analytics. But unlike spreadsheets, built-in reports still feature in the future plans of many organizations. That’s because built-in reports play a vital role in learning analytics—but their limitations mean that on their own, they can’t provide everything you need. Specifically:

• A siloed view: Built-in reports only provide data about one platform—the platform that they are built into. So to see the whole picture, you also need reports from your other systems or a way to combine the data from all your systems in one place.
• Built-in reports provide views into your data, but they generally are not as flexible in terms of how data is organized and presented compared to learning analytics platforms, BI tools, or even Excel.
• Built-in reports often do not incorporate HRIS data and, therefore, cannot be segmented or filtered by department or used by line managers to keep track of their people.

For all these reasons, built-in reports work best in conjunction with other tools.

Learning Analytics Platforms: The ideal solution, but are they for everyone?

A Learning Analytic Platform (LAP) offers a way to collate all the data you could want in one place. This covers all your learning data sources (LMS, LXP, content libraries etc.), HRIS data (providing organizational hierarchy views) and data sources containing Business KPIs (to enable you to measure the business impact of learning).

The LAP itself presents your data in reports and dashboards that make it easy to spot trends at all levels of organizational hierarchy. It is essentially a reporting suite that sits on top of a Learning Record Store. See "What is a Learning Analytics Platform" for a deeper dive.

So what did the survey results show us?

Only 13% of the organizations we surveyed currently use learning analytics platforms, which is the lowest percentage of the four options presented (BI tools were next lowest at 26%).

However, there is significant interest in learning analytics platforms. For instance, 52% of organizations planning to change their learning analytics technology intend to use a learning analytics platform—the highest of the four options presented (followed by BI tools at 47%).

Alongside their learning analytics platform, 55% of these organizations used built-in reports, 52% used Excel, and 30% used BI Tools. A whopping 92% of organizations using learning analytics platforms plan to continue doing so, though 31% plan to make other changes to their toolkit.

These survey results indicate that many organizations not yet using learning analytics platforms want to start using them, and the organizations currently using them want to keep doing so. To put it another way, organizations like what they see when it comes to learning analytics platforms!

Learning analytics platforms address the challenges and limitations of using Excel or built-in reports. Specifically:

• Unlike spreadsheets, learning analytics platforms set up automated data flows, so dashboards always show the latest data without having to be reconfigured. This approach ensures report stakeholders can always find up-to-date, relevant data regardless of when they want to view it.
• Unlike built-in reports, learning analytics platforms combine all the learning data from multiple learning systems. This means that you can get a complete view of the learning in your organization all in one place.
• Finally, learning analytics platforms integrate with HRIS systems so learning data can be filtered and segmented based on organizational hierarchies, manager information, and other HR data. Senior leaders can compare departments, and managers can report on their teams.

Up Next: Did organizations with a learning analytics budget answer differently to those without?

Next up, we explore the stories of those who have learning measurement budget versus those that wish they had one. How does their approach vary? And how do you go about attaining a budget in the first place?