File:DEVELOPING A SCALED PERFORMANCE EVALUATION MEASUREMENT SYSTEM TO EVALUATE MARINE PERFORMANCE (IA developingascale1094562718).pdf

Training developers lack methods for determining the benefits of integrating live, virtual, and constructive training. This study defined and tested a scaled performance evaluation measurement system (SPEMS) to be used across tasks. We used the buddy rush task to test SPEMS and compare it to the current “Go/No Go” performance evaluation checklist (PECL). We developed SPEMS in three steps: we convened focus groups to establish 5-level behaviorally anchored rating scales (BARS); confirmed SPEMS reliability using subject-matter expert (SME) virtual video analysis; and empirically tested SPEMS’ predictive capability in an operational environment. Suitable inter-rater reliability was found for BARS (87% agreement) and SPEMS (Cronbach’s Alpha 0.93 to 0.98). Percent exposure was selected by SMEs as the objective measure of buddy rush performance. Fifty-two trainees (26 pairs) were evaluated using a PECL and SPEMS at three time points. The results showed that SPEMS has a moderate, negative, linear relationship with percent exposure at an R2 = 0.41/0.40. Conversely, PECL has a weak, slightly negative linear relationship with percent exposure at an R2 = 0.03/0.2. We reject the null hypotheses and conclude that SPEMS scores are significantly related to percent exposure and have more predictive strength than PECL scores. These findings demonstrate a verifiable, repeatable, and reliable potential solution to the problem of measuring military task performance across training solutions.

Subjects: evaluation; human performance; LVC; metrics; operational environment; proficiency; readiness; return on investment; training