Proposal of a rubric to evaluate the quality of doctoral theses: A formative evaluation approach

Abstract

The objective of this research was to determine the psychometric properties of a rubric designed, with formatives purposes, to evaluate the quality of doctoral theses in PhD programs in Education. The rubric is made up of six dimensions, and its scores range is from 1 to 4. To determine its degree of validity, the Aiken V coefficient was used, and to determine its level of reliability, Cohen's Kappa concordance coefficient and the coefficient of Kappa de Fleiss were used. In both procedures, 6 researchers with a PhD in Education participated as judges and evaluators. In addition, a thesis was selected from each of the 31 best universities in the world in the area of education to perform inter-observer reliability analysis. After of the respective analyzes, it was concluded that each dimension of the rubric has a high level of validity when obtaining scores higher than 0.83; and that the rubric is highly reliable since the strength of agreement between one and another observer, as well as a whole, turned out to be considerable or almost perfect.