#01: Know the diagnostic question
The quality of a test must always be seen in the context of the question. Not every good test is automatically suitable for your intended application. Therefore, the question and objective of the test should be clarified and preferably explicitly stated before the assessment.
#02: Know the target group
In most cases, test results are interpreted in the context of a comparison group. This comparison group should therefore correspond as closely as possible to the target group. A comparison group, also known as a norm sample, should therefore be available for the test used, representing the population from which a person comes or from which a person will come. Depending on the question, these can be representative, education-specific, or even job-specific norms.
#03: Detailed and transparent documentation
Good tests have nothing to hide, so transparent documentation is a must. The more concretely the test manual describes the development process and the more transparently characteristic values of the quality criteria such as reliability and validity are stated, the better.
Did you know? Trustworthy test authors often fill out such checklists in advance and enclose them with the test manual to make it easier for users to read the relevant points. These can then usually be found in the appendix.
#04: Well-cited theoretical background
Tests need a theoretical framework that is supported by empirical results. Therefore, for good tests, an extensive literature search is conducted, resulting in a precise description of the theoretical background in the test manual. Pay particular attention to the citations in the manual. These should come predominantly from independent specialist literature.
#05: Established scientific model
For the transition from theory to practical test to work well, there should already be empirical evidence that the model used actually measures the dimension targeted in the test. If the test is to be used internationally, it should also be based on an internationally established model.
#06: Transparent construction rationale
Construction rationales should also be derived from existing literature and described transparently and comprehensibly in the test manual. Caution is advised with construction rationales that have not yet been empirically investigated or that show argumentation gaps in the transition from theory to operationalization.
Did you know? Test construction increasingly relies on automated item generators (AIGs). Especially then, transparently described construction rationales are essential. Self-learning systems can do a lot, but their functioning should be well documented.
#07: Clear description of limit values for construct validity
Often, more complex statistical procedures are used to calculate validity. Test manuals should therefore always indicate how the quality criteria are to be assessed, for example by naming cut-off values or quality categories. These should be taken from independent specialist literature or guidelines and cited accordingly, so that users can judge how well a construct has been validated even without studying statistics.
#08: Good external criterion for criterion validations
A high criterion validity speaks for the quality of the test, but attention should be paid to the quality of the study design with which the value was obtained. Not only tests must meet quality criteria, but also the external criterion used, e.g. annual sales for sales employees, must be sufficiently reliable, objective and valid.
Here, too, the study design for criterion validation should be described transparently in the test manual.
#09: Sometimes it doesn’t pay to save time
For tests on the results of which major financial or safety-related decisions depend, so-called “high stakes testings”, it often pays not to save time in the test evaluation. Detailed test reviews are available for many tests, for example at Pub Psych
