Data collected about student behaviour doesn't help improve teaching or learning

Universities and schools around the world face constant pressure to find measures that demonstrate their impact on student learning. Most recently, they are devoting immense amounts of time, money and other resources to a new measurement approach called learning analytics.

Learning analytics captures data about student and teacher behaviour, most frequently from the learning management software systems (LMS) used by schools and universities to design, manage and deliver their programs and courses.

The data includes tweets, file uploads, downloads, logons and participation in LMS blogs and chat rooms. This data is frequently augmented with information from other systems like student records and administration.

The goal of learning analytics is to establish what works to improve learning and teaching in the same way other fields use analytic approaches to predict the behaviour of shoppers, banking fraudsters and stock traders.

The idea of using data to understand and predict better learning and teaching makes complete sense, so what’s the problem?

The problem is that the learning analytics data gathered is not much about learning or teaching.

When analytics systems monitor the behaviour of shoppers, or fraudsters, or stock traders, they are watching what people do when they shop, commit frauds or make trades; the behaviour the analysts are interested in predicting.

In learning analytics, the concept is the same – to predict whether students learn well and teachers teach effectively.

However, the quality of learning and teaching cannot be determined from the behaviours being watched and counted by an LMS or related systems.

Why? Because knowing how many times a student tweeted or used a chat room has little to do with how teachers teach or the ways students learn.

The current learning analytics approach is like deciding whether a medical practice is successful by counting whether people attend their appointments or pick up their prescriptions, instead of focusing on doctors interacting with patients and the quality of what happens when they do.

No data to show it works

Not surprisingly, there is no body of evidence showing that LMS and other system data improve student learning or teaching.

It is not the case that current learning analytic data is irrelevant. It is correlated with student engagement and participation and may offer general indicators of student needs. It just does not inform teachers or learners what they need to do better or differently to make learning happen.

Focusing on things that do not make an important difference to student learning means we are not paying attention to the things that do.

Over 50 years of research on learning and teaching has told us about what makes a lecture active, how students work best in groups, the strategies that help students learn most effectively, and what makes for quality assessment.

These things among many others are well known, while data about them can be gathered from the interaction among teachers and students face-to-face or online.

Most importantly, they are powerful predictors of student learning. The issue is worthy of serious concern because the things we measure focus our attention, shape our priorities and can ultimately determine what we understand and how we behave. This is a big problem if you are not gathering the right data.

Learning analytics data and the systems that gather it have become proxies, surrogates for what we should be measuring to improve student learning.

Three ways to solve the problem

1. Pay attention to the over 50 years of research about learning and teaching that show visible effects on student achievement, including what makes co-operative and teacher-led learning most effective.

2. Build technology tools that help teachers to design, deliver and evaluate what they do in ways that include effective learning and teaching approaches. A growing body of research is showing that technology can be used in a different way to assist teachers design and deliver more effective learning experiences. This approach offers the potential of a new kind of learning analytics data that focuses on what learners and teachers do.

3. Gather data directly from the people involved – the students and teachers. Ask them to give feedback when they are designing, delivering and participating in learning and teaching, instead of surveilling them. This feedback emerges all the time from the day-to-day interaction among students and teachers.

We know that doing something about these things can make a big difference in student learning. Implementing the three solutions means focusing on the evidence we need instead of the data we have.

We can gather data on how well programs and courses are designed, whether effective practices are being employed, whether assignments line up with what is taught, and whether all those efforts are improving student learning outcomes.

The problem with learning analytics is not simply another example of education wastefully impersonating other fields for little benefit.

These types of data and the systems that gather them are defining what learning and teaching mean. At present, quality is being defined by the data that is available instead of the feedback we need about learning and teaching in schools and universities.