Your selected text method is:
To document validation steps using the checklist, we recommend to download the Word Template below
Download (Word)tba
ValiText - A Validation Framework for Computational Text-based Measures of Social Constructs
Beta Version 0.9
About
ValiText is a novel validation framework for computational text-based measures of social constructs (Birkenmaier et al., 2024). The framework offers practical guidance for researchers and users to conduct and document validation for computational text analysis.
Overview
Validation is a critical task in text analysis and natural language processing. As its core, validation requires different activities to demonstrate that a method measures what it purports to measure (Cureton, 1951). However, validating text-based measures can be challenging (Krippendorf, 2018).
Therefore, any empirical measure needs to be validated. One crucial problem in the validation of text-based measures, however, is the lack of concepual clarity on how to conduct validation. To provide practical guidance for researchers and users to conduct and communicate validation, ValiText provides a flexible and consistent appraoch to validation.
At its core, ValiText requires three types of validation evidence: substantive, structural, and external evidence
-
Substantive Evidence:Requires to outline the theoretical underpinning of the measure -
Structural Evidence:Requires to examine and evaluate properties of the model and its measures -
External Evidence:Requires to test for how the measure relates to other independent information or criteria
The framework is complemented by a checklist that defines and outlines empirical validation steps available to collect validity evidence for different use cases.
If you want to learn more about the framework and the checklist, please click on the respective section below or have a look at our Working Paper.
Framework
The framework is rooted in the well-established principles of measurement theory found within the psychometric literature which offers the most comprehensive and cohesive conception of validity for social science research. The complete framework is visually depicted in Figure 1.
Figure 1: Conceptual Model (click to expand)
Checklist
The checklist provides a comprehensive list of validation steps for each phase within the framework. Each validation step comes with a set of additional , such as
Validation Step
|
Name of the validation step |
Considerations
|
Detailed Description of the validation step |
Performance Criteria
|
Information on how to conduct the Validation step |
Source / References
|
Additional Literature |
To use the checklist in your research just click on Get Started in the navigation bar. There, you can generate your own checklist depending on the use case applied. Furthermore, you will be able to download a template which you can fill out on your own and attach to your analysis.
Figure 2: Screenshot Checklist (click to expand)
User Instructions
This application generates an adaptable checklist that you can use to validate your text-based measures. Each row within the table corresponds to one validation step (i.e., a single reported and clearly demarcated validation activity). Validation steps can be either context-independent or context-dependent.
ValiText accounts for different use cases in validation practices across text-based methods and research contexts. At present, ValiTex differentiates between four use cases, which are highlighted in the Table below (click on it for a detailed view)
Table: Use Cases (click to expand)
Please start by choosing a method from the drop-down menu below. The complete list of validation steps is also available on Github.
Feedback
Please note that this application is currently under construction and evolution. Please feel free to reach out to us with any feedback you have. We are eager to hear from you and will take your suggestions into consideration as we continue to develop and enhance our website. To reach us, please refer to lukas.birkenmaier@gesis.org
Citation
When refering to the site or the corresponding workin paper in publications, please cite the following references:
Birkenmaier, Lukas; Lechner, Clemens; Wagner, Claudia. (2024). ValiTex - a uniform validation framework for computational text-based measures of social constructs. https://doi.org/10.48550/arXiv.2307.02863