We can learn more about motivation by sampling data through a digital learning tool

Sara Bengtsson Flexit researcher 2016-2017 Karolinska Institutet and Invigos

Aim
This project sets out to understand what motivates pupils to write, by continuously measure motivation with self-assessment ratings and questions, via a digital writing and learning tool, and quantitatively evaluate if motivation covary with actions. The pupils are writing in the learning tool Invigos (www.invigos.se). The aim is to get a systematic view of how the pupils’ thoughts regarding their motivation influence, or are influenced, by actions during the writing process from the perspective of McGuire’s motivation theory (McGuire, 1983). In short, this theory stipulates that actions differ if the motivation has its origin in emotional or cognitive aspects, and whether the motivation is caused by internal or external factors.
We sample motivation level as well as responses to the following reported thoughts; My motivation depends on: my thought process, my emotions, my surroundings, and my writing task. Actions we have analysed are the number of text reviews the pupils have made on their own texts, on their peers texts, and how many peer-reviews they have received.
 
Short on methods
The motivation question pop-ups consist of three windows. The first window requests the student to indicate their motivation level, at the time being, on a staircase of four levels numbered 1-4. Each step is associated with an emoticon that in wide terms illustrates the state of mind, going from very sleepy (1) to happy thumbs up (4). On the second pop-up the pupil is asked to indicate what their previous choice depends on. There are two clickable options; ‘My feelings’, or ‘My thought process’. The third pop-up asks the pupil to indicate with two clickable options if their motivation level depends on their writing assignment/performance or other people/things in their surroundings.  The questions are presented on the computer screen while the pupils are working in Invigos with a frequency of 20 active minutes. The pupil has the possibility to turn off the question definitely with a button click.
We have studied the relationship between motivation questions and text reviews. The review function in Invigos consists of a checklist of questions on the quality of the written text, such as ‘is the text coherent with regard to the purpose it is aimed to describe?’ or ‘is the environment well described?’. Each question can be graded ‘yes’, ‘partly’, and ‘not yet’.  The student can perform a self-review, or ask for a peer-review.  When calling for a peer-review through the digital tool, the request is sent out anonymously to peers in the class. Several peers can review a text but the request disappears when a first review is returned. The remaining reviews which have been initiated can be completed and returned to the sender.  
Data was collected between February and April 2017 from Swedish high-school students (13-15 years of age), who’s teachers have voluntarily chosen to use Invigos in their teaching. Only students who have responded to at least two sets of pop-ups are included in the data analysis. 988 responses are analysed. The data is analyzed in QuickSight (2017 Amazon web services, Inc), and in SPSS (IBM SPSS, NY, USA) or Matlab (Mathworks Inc, USA).

Three main results
We suggest that a digital learning tool can be useful in sampling motivation levels and thoughts regarding one’s motivation. 22% of the pupils chose to turn off the motivation questions immediately, which implies that remaining 78% of the pupils find the questions worthwhile to answer. We found statistical significant relationships between the ratings and reported thoughts and actions, which reflect non-random responses. There is a relationship between motivation and pro-social behaviour, and between motivation and how much the environment impacts on the pupil. When the student is highly motivated s/he more often gives peer-feedback. When motivation is very low, the pupil is more likely to infer the cause to the surroundings rather than their own performance. In addition, when the pupils perceive the motivation to depend on their emotions they are less likely to interact with peers than if they attribute their motivation to their thought process, even when the motivation is very high. When they attribute their motivation to their thought process they are more likely to blame the surroundings when motivation is low.
The distribution of motivation levels was as follows: 27% very high(4);  26% high(3); 19% low(2); 28% very low(1). Very low motivation tends to be coupled to the perception that the motivation depends on other people, or things, in the environment (p<0.05), and very high motivation is often seen in pupils who engage in peer-reviews (p<0.05). We didn’t find any relationship between motivation level and the number of self-reviews.
There is no significant relationship between motivation level and whether the motivation is reported to depend on emotions (E) or thought processes (T) (p=0.4). We find however that T to a greater extent blames the surrounding when motivation is very low (p<0.035) and that T to a greater extent than E feedback on peers’ texts when motivation is very high (p<0.014). There is no significant difference in how often E and T requests feedback in the digital tool. Note that these results are preliminary and may change as more data is collected.
We don’t know of any previous study where motivation has been continuously sampled via a digital learning tool. The method enables collection of a large sample size from a wide population, and the studies of dynamic changes. The downside however is the lack of insight into the action which are not registered on-line, such as social interactions between pupils and teachers in the physical classroom.
An interesting observation is that the pupil’s motivation can be either emotional or cognitive regardless of motivation level and that behaviour differs depending on type. This is an observation that is relevant to understand on a deeper level. Teachers all over the world question what they can do to motivate their students and the insight that motivation is a multifaceted concept will enable the development of fruitful interventions. McGuire’s contextualist theory of motivation is not particularly well researched nor well cited in current motivation literature.   

New research questions generated from this project
The connection between motivation and the quality of the text the pupils write will be evaluated from the checklist scores obtained. The main question addressed is whether high motivation leads to better academic outcome and whether the type of motivation matters (Emotion/Cognition).
I also aim to study which type of reward that appeals to a person when they view their motivation as being emotional in nature and when they view it as being cognitive. The methods for this could be to analyse the text content in regard to chosen words, and brain imaging studies that investigate which neural motivational processes that drive goal oriented behaviour for the different types of motivation.
 
The international aspects of the project
One or more papers will be published in international peer-reviewed journals. The project is of international interest since digital teaching is increasingly used in Schools all around and motivation is a topic that concerns most teachers.
During the project period I have also published a study in an international journal which concern processes that take place after an error has been made during learning. I have previously found that motivation plays a particular role during error monitoring (Bengtsson et al., 2008). The current study (Schiffler et al., 2017) shows that several processes are in operation post-error, some which are functional and others which are non-functional, and alterations of these processes can be seen for several trials post-error. The level of each of these processes predict performance success.  These results are important since it was previously thought that only one process, caution, played a role post-error and thus the new result widens the understanding of learning.

Interaction between academia and organization outside academia
The Flexit project was terminated after a year since the researcher got a position at a UK university. Therefore, we did not arrange the workshops between academia and the company that is part of Flexit year 2. We have however arranged three workshops where the researcher gave talks regarding the research to teachers and heads of schools. The project also generated a talk at the conference SETT 2017 (Scandinavian Education Technology Transformation), where the researcher described results from the motivation project with Invigos. The project group has also initiated collaborations with Hermods and VLM-institutet to develop the motivation research and deepen discussions. The researcher has been communicating science every other week on the company’s facebook page.

Acknowledgement
S Bengtsson, flexitforskare Karolinska Institutet, responsible for the hypothesis testing, design of the motivation pop-ups, analysis of data and writing reports. M Tiberg, CEO and F Marchal, head of software infrastructure have provided access to Invigos and input on process and solutions. R Koski, Invigos has been in charge for implementation of pop-ups and its features, with the assistance of A Grelias, Invigos. R Ali and M Goncalves, Invigos has enabled data access for the researcher.
 
References
McGuire, W. J. (1983). A contextualist theory of knowledge: Its implications for innovation and reform in psychological research. Advances in experimental social psychology, 16, 1-47.
Bengtsson, S. L., et al. (2008). Motivation to do well enhances responses to errors and self-monitoring. Cerebral Cortex 19(4), 797-804.