Case study: Scotland on Screen Evaluation
Scotland on Screen is a website featuring films from the National Library of Scotland’s Moving Image Archive, curated with learning resources for teachers and other learning practitioners. It is funded and supported by Screen Scotland.
The Education Department at Screen Scotland and the National Library of Scotland identified a possible need to refresh the site and wanted teacher feedback to inform its redevelopment.
The purpose of the project was to capture feedback from teachers using the resources in a real life classroom setting. The target audience for this project was defined by the client as Primary School Teachers, and Secondary School Teachers of social sciences and English.
The project took place over the summer of 2021, a period of significant challenge for teachers across Scotland as they continued to deal with the implications of the Covid-19 pandemic. It was therefore vitally important that the methodology be flexible and adjusted where necessary to provide teachers with the maximum opportunity to take part.
An invitation to participate was issued on the Screen Scotland Education Teams page and other social media channels, as well as via the Scotinform Teacher Panel. Participants were given the option of completing the project in June or August.
Scotinform designed an online feedback platform that took participants through a series of tasks, from forming initial impressions of the home page, through searching for particular films and resources, and finally creating and teaching a lesson using one of the films on the site. Participants provided a range of quantitative and qualitative feedback to these tasks and also made a short video reflecting on their experiences once they had finished. All participants were paid an incentive for taking part and Scotinform provided full support.
Nine ‘mystery shops’ were completed in total from across Scotland. Screen Scotland was able to view the feedback as it was being collated through access to the online feedback platform. Scotinform provided individual reports and videos on an ongoing basis throughout the project and then produced a summary report covering all nine participants. As with our other mystery visit programmes, core elements were scored on a red-amber-green matrix to identify strengths as well as areas for development.
The findings identified a number of specific areas for improvement, particularly around the way in which the site is organised and teachers can search for suitable clips. This is now being used as the basis for redevelopment of the website.
Participants were keen to be involved further with the project and to provide further feedback to Screen Scotland in the future.
As a result of taking part in the project, several participants had already recommended Scotland on Screen to their colleagues.
“Scotinform’s stellar work has been vital in connecting us with a wide group of teachers and gaining us their valuable perspective of our website. The findings have given us a crystal clear overview of their thoughts and will allow us to redevelop and improve Scotland on Screen so that it meets the needs and wants of teachers across the country”
Gail Robertson, Film Education Officer, Screen Scotland
We are mindful of the requirements of different target audiences – and design flexibility into our research methodologies and timescales as standard.
We work extensively with teachers across Scotland and have our own Teacher Panel, which can be used to assist with recruitment.
Our feedback mechanisms allow clients to see the results as they come in, rather than waiting for a final report.
We provide full technical (and sometimes moral!) support to all research participants, allowing us to gain a very rich level of feedback even in an extensive task like this.
The inclusion of video feedback helps to bring the research to life and we have found that it is very well received by organisational colleagues and board members.