The jungle of evaluation: beyond right or wrong
Ask 10 people to mark the same writing task and you are likely to end up with 10 rather divergent scores. Evaluating a complex skill – like writing, but also presenting, drawing, etc. – is not a black-and-white matter. That’s why research on assessment validity and reliability has long been an important research topic in the educational field.
Today, the most popular method is to evaluate complex competencies by using rubrics or criteria, i.e. to assess several subskills separately. For instance, for writing, you could assess criteria like grammar, structure, fluency, originality, etc.
The D-PAC spin-off originated as a joint research project involving the University of Antwerp, Ghent University, and imec. Roos Van Gasse was one of the PhD students involved in the initial project and still plays an important role in D-PAC today.
Roos Van Gasse: “The popular rubrics method poses a number of problems. First of all, it is time-consuming – as you have to score each subskill separately. In addition, assessors do not necessarily interpret the assessment criteria in the same way. In the D-PAC research project, we therefore decided to explore another evaluation method, i.e. comparative judgment. With this method you evaluate complex competencies by comparing them with each other (or with an agreed-upon point of reference). We also decided to use a holistic approach, so evaluators assess the task as a whole, which is more intuitive and time-efficient.”
How does it work?
The idea behind D-PAC is simple. When evaluating complex skills, people automatically tend to compare and rank them, sometimes even physically organizing them from weak to strong. This kind of intuitive comparative judgment works if you have to assess 10 assignments, but what if you have to evaluate more than 500 tasks? And what if you are not the only evaluator?
D-PAC is a digital platform that enables comparative judgment on a large scale by using the principle of pairwise comparison. Evaluators are presented with two tasks (e.g. two texts, two drawings, etc.) and have to decide which of the two is best.
They are then presented with the next pair and have to make the same decision. The algorithms behind D-PAC make sure that each evaluator gets to see the same tasks, but in different combinations. In the end, the tool then uses this input to create a complete ranking of all tasks. It is also built in such a way that evaluators can easily give feedback to the participants.
Sophie Mols (lecturer Pedagogy of the Young Child’ at the Karel De Grote University College): “At the Karel De Grote University College we use D-PAC to evaluate assignments from students in the Pedagogy of the Young Child program. We use it for tasks that are very similar to what they will need to do in their future workplace, so that’s why we also wanted them to be evaluated by professionals in the field. The D-PAC platform makes it easy to have several evaluators assess the same tasks in a fair way and to give valuable feedback.”
From academia to spin-off with imec’s support
The D-PAC platform is based on extensive research involving more than 40 institutes, including the University of Antwerp, Ghent University, and imec. Imec also played role in bridging the gap between academia on the one hand and business on the other hand.
During the research project, the team collaborated with imec’s Application Prototyping team, which developed a prototype of the platform and wrote the first version of the underlying algorithms.
Tanguy Coenen (Application Prototyping team manager): “In a nutshell, our job is to make ideas tangible. People come to us with an initial idea and we then guide them through the entire design process.
The aim is to develop a minimum viable product, a prototype that is ready to be used by early customers. We help our clients think about the functionalities they want, which features take priority and which can be added later, what kind of interface is most user-friendly, etc. It is always an iterative process: we make a design, test it with end users (through imec.livinglabs), and finally we adapt the design to users’ needs. For D-PAC, for instance, we made 6 different versions before creating the final prototype.”
Because the software – initially developed as part of the research project - proved to be really successful, the D-PAC team decided to continue as a spin-off after the research project ended. Roos Van Gasse: “At the end of 2017, we participated in a 12-week coaching program set up by imec to explore the commercial potential of academic technology research. This really helped us to take the leap from academia to business. Imec showed us how to look at our project from a different, more commercial angle and helped us to determine our focus and target market.”
The latter was not an easy decision as the tool is actually quite multifunctional. For instance, it was also tested at the recruitment company Hudson to evaluate CVs. In fact, it could play a supporting role in any context where complex decisions need to be made collaboratively. However, as the D-PAC team has extensive expertise in higher education and they already had a number of satisfied users in this field, they decided to choose education as their primary target market. Customers pay a fee depending on the number of users. In addition to the platform itself, D-PAC also offers workshops in which they share the insights and expertise they gathered during their 4-year research project.
Roos Van Gasse explains where she hopes to see the spin-off in 5 years: “At the moment, the company has a growing number of paying customers in Flanders and the Netherlands, including the Karel De Grote University College and Avans University of Applied Sciences. But in the near future, we also plan to expand across Europe, first tackling Scandinavia – known for its innovative approach in education. Our aim is to really become an established brand and authority in the field of high-stake assessment of complex competencies, also among examination centers. What sets us apart from our competitors is that we offer a very complete tool – backed up by years of robust research. Unlike competing solutions, we do not only focus on scoring tasks, but also on providing valuable feedback that is presented to assessees in an insightful way. In this way, we are not just a tool for assessment, but also for learning.”
Want to know more?
- D-PAC’s platform was developed in collaboration with imec’s Application Prototyping team. For more information on imec’s innovation services, check out the website. For specific questions regarding Application Prototyping, please contact Tanguy Coenen.
- Interested in using D-PAC? Please contact Roos Van Gasse.
Roos Van Gasse has been a researcher at Antwerp University for over 7 years. During the D-PAC research project, she was responsible for investigating how D-PAC data can contribute to the learning of assessors and organizations. Roos Van Gasse is one of the co-founders of the spin-off.
Copyright picture: David Jagersma
Tanguy Coenen has a Master and PhD in Business Administration from the Vrije Universiteit Brussels. He is currently the Program Manager of the Application Prototyping Team, which is part of imec's Innovation Services. The team is composed of more than 25 developers, building tools and prototypes that help clients innovate in ICT by leveraging imec's R&D IP or state-of-the art commercial building blocks.
Published on:
5 November 2018
