A Day with a Test Developer
How are the tests of the Vienna Test System actually developed? In this interview, we take a look behind the scenes of a day as a test developer at SCHUHFRIED.
Who are the masters behind the scenes at SCHUHFRIED?
We asked two of our experts from the "Test and Training Development" team for you!
Georg and Felix are test developers at SCHUHFRIED. Georg specialized in psychological assessment and methodology while studying psychology. In his free time, he enjoys running, reading, and listening to music. Felix studied philosophy and psychology and was already fascinated by the question of how to measure properties of the human mind during his studies. When he is not developing tests, he plays the saxophone or rides his mountain bike through the Vienna Woods.
What is a test developer?
Georg: As the name suggests, we develop psychological tests. Primarily, of course, this involves the content such as instructions, test items and scoring. But a digital test doesn't stop there, of course. For the implementation, we work very closely with other teams.
Which skills do you need as a test developer?
Felix: Classically, you need a degree in psychology and have a particular enthusiasm for assessment, methodological issues and working with data, as is the case with us. Good language skills and a certain affinity for software development doesn't hurt either.
Georg: Finally, it also requires stamina and patience. The development of a complex test battery such as the Intelligence Structure Battery - 2 (INSBAT-2) can take a few years.
What do test developers at SCHUHFRIED mostly do?
Felix: Our development teams are highly interdisciplinary. That means we work closely with programmers, software testers and translators. After all, a good test requires experts from many different fields.
Georg: At SCHUHFRIED, we only develop digital tests. While we as test developers are primarily concerned with literature research, data analysis and writing test items, experienced programmers are needed for the technical implementation on various platforms. For some time, our tests have not only been used on computers, but also, increasingly on cell phones, or fully integrated into our clients' software environments.
Felix: One of the things I'm currently working on is the implementation of a comparatively new answer format for personality tests. Instead of the Likert scale commonly used in questionnaires, we will implement a forced-choice response format. This requires writing several dozen items. You might imagine formulating these items would be easy but it can be a long haul and be quite complicated. After all, the desired facet should be spotted. It is important that the formulations sound good, are easy to understand and are unambiguous. We also must ensure good translatability for our international customers.
Of course, it's not just a matter of developing the items; the test items and the scoring also must be programmed. For the forced-choice answer format, we are at the forefront when it comes to the practical application of new scoring algorithms. I am currently working intensively with a programmer to anchor the new algorithm for our customers in the Vienna Test System (VTS) so that our users have a reliable and valid test to utilize.
Georg: SCHUHFRIED is certified according to ISO standard 13485 and employs a comprehensive quality management system. Therefore, an independent team of software testers gives the test a run-through. Their feedback helps us to finalize the test for the release.
What does a typical working day look like for you?
Georg: At the beginning - after our morning coffee or tea of course, we first have a meeting with the other teams from our area. We call this our "stand-up" because we stand together like in a story telling circle at school. Everyone briefly explains what they did the day before and what is planned for today so that we can all stay well-coordinated. Possible problems or crises are addressed and solved together, and surely accomplishments are celebrated.
Felix: After that, the actual project work starts. We work in an open office, together with our programmers so that we can communicate well with the other teams at any time, even outside of scheduled meetings. Sometimes improvised and extremely creative brainstorming sessions arise quite spontaneously when there is a tough nut to crack. We have a very flexible home office policy - not least because of Covid - so these are nowadays often virtual meetings. Therefore we enjoy it even more when we have meetings in person, and then usually meet for a big team lunch.
Georg: Despite the creative and spontaneous atmosphere, we work in a very structured way. We are guided by the principles of agile project management and organize our work in sprint cycles so that we can always focus on the essentials.
That sounds very exciting. Are there also routine tasks?
Felix: What job doesn't have that? We spend a lot of time on data analysis. The data, that is the test results, come in part from our Test and Research Center or are provided to us by our clients directly from practical test applications. We are always interested in collecting as much data as possible from our tests and in incorporating the findings and practical experience derived from this into our tests. Sometimes you really must be careful not to get lost in analyses, and stick to the essentials.
Georg: We work with several well-known statistical and analysis software products, such as SPSS and R, which gives us a lot of possibilities to interpret and understand data.
Do you also work internationally?
Felix: Very much so! Our tests are currently used in 67 countries and are translated into up to 30 languages, for example Arabic, Greek, or Vietnamese, and new languages are added all the time. That's why two translation managers are a part of our team. They do more than just manage translations.
Georg: Psychological tests sometimes must be adapted very extensively in other languages and for other cultures. Only rarely does a direct translation reflect the cultural connotations. That's why we often work in several translation steps and with several translators to ensure the correct meaning. This is an exciting exercise, and you learn a lot about other cultures in this process.
Do you have contact with the users of the tests you develop?
Felix: Yes, of course. As experts on our tests, we are the central point of contact, especially for detailed questions. Currently, one of our colleagues is dealing with an inquiry about the Inventory for testing cognitive abilities (INT). One of our Spanish customers has questions about adaptive testing with INT and would like to get some additional information about the comparability between the presentation of the test on a computer versus on a cell phone. My colleague will compile a clear presentation of the development steps and the results of the test reliability evaluation for our client.
Georg: We also regularly attend specialist conferences where we meet users and our scientific peers.
Do you continue education and training, or have you already learned everything you need to know at the university?
Felix: Basically, of course, we still learn a lot during our work. For example, we've recently been using the programming language R more and more for many of our data analyses, and you usually only learn the basics – if at all – at university. Here, for example, a colleague is currently taking various online courses to expand her skills in this area. In addition to more technical topics, we have the opportunity to participate in congresses and training courses, where we can exchange information about current results from psychological research and discover many new things.
Georg: We also organize regular training sessions in the form of workshops, where experts, for example from universities, are invited to share their knowledge with us on various topics, such as special scoring methods.
What do you like most about your job?
Felix: I could hardly imagine another job with a similarly broad, exciting and, above all, varied field of activity. That makes the work challenging every day, but also very rewarding.
Georg: It's also nice to know that our digital psychological tests have a high social value and benefit.
Thank you for the interview and keep up the good work!