In the traditional plan for assessing the reliability of a measurement system, a number of raters each measure the same group of subjects. If the system has a large number of raters, we recommend a new set of plans that have two advantages over the traditional plan. First, the proposed plans provide greater precision for estimating the intraclass correlation coefficient with the same total number of measurements. Second, the plans are flexible and can be adapted to constraints on the number of times any subject can be assessed or the number of times any rater can make an assessment. We provide a simple tool for planning a reliability study, access to the software for the planning in the case where there are constraints and an example to demonstrate the analysis of data from the proposed plans.
Steiner, S.H., Stevens, N.T., Browne, R., & MacKay, R.J. (2011). Planning and analysis of measurement reliability studies. Canadian Journal of Statistics, 39(2), 344–355. http://dx.doi.org/10.1002/cjs.10103