Dr Jeunese Payne is a Psychologist and Researcher at Africa’ Voices, responsible for evaluating bespoke research tools and software, and training researchers how to use them. She explains the importance of evaluating these tools systematically by showcasing the newest version of Scouta — our interface for exploring large volumes of short message data.
Africa’s Voices Foundation (AVF) aims to gather rich insights into the opinions and beliefs of citizens based on the content of SMS and social media messages. This involves exploring large amounts of complex, textual data in local languages — a task that becomes near impossible to do manually with the vast amount of data we analyse.
To help us manage this, we create tools for data exploration and analyses. We invest time and effort into the planned evaluation of these tools. The evaluation of ‘Scouta’, an interface created by the AVF Research and Innovation (R&I) team, is an example of such an investment.
Scouta helps researchers explore textual data — for example, text messages gathered from interactive discussions over radio. This data exploration can be a starting point to thematic analysis (to reveal key topics in the audience’s responses), or it can be used to create word lists (lexica) that represent themes based on associations between words. Scouta has four main elements that assist AVF researchers in these goals:
- a section for filtering the data by words and time frame, and for exploring associated words;
- a section for filtering the data based on county, gender, and age;
- a section that displays the messages based on filters;
- a section that allows the researcher to visualise that data.
Before we created Scouta, we used “TextVis” (Figure 1) — a proof-of-concept visualisation tool, created by Giles Barton-Owen. It was designed to help AVF researchers visualise SMS data based on associations between words and according to key filters, such as demographics. Relative to Scouta, TextVis performed a very specific function. In TextVis, existing lexica were uploaded to update a word network visualisation so that researchers could actually see the relationships between words. However, it lacked practical message reading tools, and it wasn’t very flexible or easy to understand on first use.
Figure 1. TextVis: The proof of concept prototype that shows how word networks can be created based on a large corpus of short message data. The message section is blurred for data protection reasons.
After creating this prototype, it became clear that it was important to see more of the text messages, as well as the actual lists of associated words, and to create an interface that was more streamlined and versatile in general. This is how the first version of Scouta came to be, created by Dr Sebastian Ahnert. Figure 2 shows how the interface looked in its very early development, before it had a name and before we embarked on systematic usability testing. In this version, rather than creating a lexicon to upload to the interface, we included a text box that users could add words or lists of words to, making the process much more flexible.
Figure 2. The initial version of Scouta, before it had its name and additional features. The data is from Well Told Story`s young audience on the topic of contraception and relationships. The message section is blurred for data protection.
It can be tempting to design products only for the organisation (or even just the team) you work in, with its own processes, and based on specific knowledge and training. Scouta was originally created based on an internal need — in this case, an interface that would allow us to interactively and visually explore big databases of Swahili and Sheng SMS and demographic data based on networks of words. However, it was always understood that, in the long-term, the interface could be used by very different users, such as researchers at one of our partner organisations, Well Told Story (WTS) — a media and communications research and production company in Kenya reaching and engaging with millions of young people. The idea was that WTS should be able to use the interface to first explore the data and develop hypotheses, which could be tested in subsequent analyses.
Development began with initial iterated drafts of an interface, intended for creating multilingual lexica of terms relevant to particular topics (such as contraception or government), filtering the data for later analysis (statistical and thematic), and linking this to location, age, and gender. Ways of navigating, selecting, and visualising data using the interface were re-imagined to create a version of Scouta (Figure 3) that is closer to how it looks today. This was created by Giles with an aim to improve the user experience, and to go beyond the immediate needs of the AVF R&I team.
Figure 3. The version of Scouta shown to our partners for initial feedback. The data here is from our interactive radio project in Uganda, which explored maternal health issues, and pre-eclampsia in particular. Messages are blurred for data protection reasons.
Following the creation of this more user-friendly version, the idea was to modify the interface over time, making incremental improvements based on user needs and the context of use (who will use it, what they will use it for, and under what conditions) — to evolve rather than to present a final “grand design” that doesn’t take the needs of the likely users into account.
Since its development, one of our primary aims has been to improve the performance of existing data exploration and analysis activities using Scouta, to identify opportunities for integration with our other systems, and to remove inefficiencies. We also wanted to transform how we conduct research collaboratively, allowing partners such as WTS to explore their own data using this tool.
Before we can do this, we need to properly evaluate and modify the interface using a user-centred design approach. To evaluate and improve Scouta, we planned user studies progressing over phases. These phases cover key aspects of usability testing (Figure 4). After identifying needs and the context of use, user-centred design typically involves: gathering user requirements based on identified user needs and context, creating design solutions, and then evaluating the designs.
Figure 4. The stages typically involved in user-centred design. Note that design solutions themselves are also often created in stages, from rough concepts to fully functioning prototypes; this is an iterative process, which means that researchers will likely have to go back to design solutions or even the requirements gathering stages after the design has been evaluated and conduct the process again until a final version is created.
We thus embarked on a “requirements gathering” phase, with an aim to determine the qualities that the interface should have in order to make it more attractive to researchers at AVF and at WTS, and the extent to which the most recent version did or did not meet expectations. This phase involved two tasks: a cognitive walkthrough (CW) and recruitment of WTS researchers for a focus group discussion.
A CW is a practical evaluation technique used to discover what can go wrong when users have no prior knowledge of a system, i.e., whether it supports “exploratory learning”, or first-time use, without formal training (Reiman, Fanzke, & Redmiles, 1995). The CW described the types, the number, the percentages, and the severity of predicted errors. Many predictions were confirmed by a discussion with researchers at WTS in our focus group, who highlighted key usability issues with Scouta, as well as other functionalities they would like to see in the interface. This included confusion caused by the layout and labelling in the interface, and a desire for more control over what functions were visible and when.
Following the focus group discussion, we identified ways that the interface could be made more intuitive and efficient. More than this, its purpose needed to be better communicated within the interface itself, since the researchers we talked to appeared to be placing emphasis on being able to do things that Scouta was not designed to do, such as downloading report-ready results and seeing full English translations. We needed to make changes that showed how Scouta helps to provide the groundwork for more systematic evaluation of messages – the beginning rather than the end point of analysis.
We thus made changes to Scouta (Figure 5) to make the main functions more intuitive, and to modify the content to communicate the overall purpose of the interface as a tool for exploring messages rather than producing results. Specific modifications included: changes to label names and salience, addition of buttons as shortcuts to command lines, changes in how the associated words are ordered and explained, minimising the number of times Scouta reloads by making changes to the search process, having certain elements minimised until opened to avoid clutter, and changing the presentation of graphics so that users could better understand what they represent and mean. Other changes were made iteratively, such as the addition of another word network that was not split by gender, as was originally available in TextVis (Figure 1), and the ability to select individual messages to download.
WTS were then re-introduced to Scouta, and given a walkthrough of the changes that were made. Since then, the interface has also been used by research assistants for a project with UNICEF Somalia to aid them in the thematic analysis of Somali language SMS data.
Figure 5. The current version of Scouta with added buttons, collapsed demographic filters, a larger results section, and a word network without gender splitting. The data here is from an interactive radio project with UNICEF Somalia on polio (filtering messages by ‘debeyl’ or ‘dabeysha’). Messages blurred for data protection reasons.
The next step is to test Scouta with actual tasks performed by potential users as part of a usability study with researchers and research assistants at WTS and UNICEF. These participants will be asked to carry out three to five defined tasks and to talk through what they are thinking and doing – a Think-Aloud Protocol. We will collect whether actions were completed (alone, with help, with errors, or not completed), the number and types of errors that were made, observed confusion or frustration, and general comments. We will also give participants a survey with usability questions to answer.
The final phase will test usability after formal training. We have already created a video tutorial for Scouta in its current form for the purpose of training research assistants. Following exposure to the tutorial and other instructional guides, including advice on how to use regular expressions in the interface, relevant users will be given a beta version of Scouta and a feedback mechanism, perhaps online diaries, to report their experiences to us (completion rates, number of errors, overall impressions, and perceived usability of the interface). The aim is to engage potential users in the design process so as to effectively co-ordinate the research and design phases.
Evaluation and usability testing will help us to reach specific goals, not only improving our research quality and performance, but increasing our credibility and alleviating usability burdens. The road to these goals may seem like a lot of time effort and resources, but digital projects such as this are an investment. AVF is a forward-thinking organisation that sees this investment as an opportunity to design more progressive digital research tools, based first on the behaviours, needs, and goals of the intended users, and followed by iterative research to help refine prototypes and concepts.