Percepções dos usuários no uso indevido da avaliação: um estudo com surveys experimentais

Abstract

This thesis explores the area that deals with the use of evaluation. In other words, it is concerned with assisting the main stakeholders (users) in the program or public policy in the decision-making process. Evaluation users are individuals, personnel involved in the program and project, decision makers and policy makers or anyone who uses an evaluation in some way. As a main objective, this thesis aims to investigate whether malicious and erroneous misuses are perceived as different by the users of an evaluation. The specific objective was considered to be the perception of regularity, frequency, inadequacy and severity. To achieve the objectives, two studies with experimental vignettes were structured. In the first study, the experimental model “between subjects” was used, while in the second study “within subjects” was used. The sample consisted of higher education teachers who are part of a postgraduate program (PPG) that receives the evaluation result of the Foundation Coordination for the Improvement of Higher Education Personnel (CAPES). The vignettes were created according to the systematic review carried out, allocating for each type of use (instrumental, conceptual and symbolic). In addition to the perceptions of the teachers mentioned above, the variables gender, age, type of institution and experience were explored in order to verify if they have any impact on the dependent variables. The results indicated that PPG teachers perceive that there are significant differences between malicious use and erroneous use in instrumental, conceptual and symbolic use. Regarding the other independent variables, the results were isolated and not consistent, considering the two experimental studies. This thesis contributes to the literature in three aspects (theoretical, empirical and methodological). In the theoretical area, the findings reinforce the need for an equitable understanding between malicious and erroneous misuse, since there is no concern in the literature about the two being equally relevant. In the empirical field, exploring users, such as teachers, open space for new investigations in other evaluations, because the literature normally explores the perception of the evaluators. Regarding the methodology, the experimental vignettes were not explored in the literature on the use of evaluation. Thus, it allows reproducibility in other evaluations or with other samples. Regarding the limitations of the thesis, it was not possible to apply a probabilistic sampling. Furthermore, it was not possible to measure the order of the questions in each vignette in order to mitigate biases in the responses, due to the impact on the significant increase in the number of questionnaires that would meet this demand. As a suggestion for future research, exploring other evaluations with the same variables will allow the consolidation of the theme in the literature.