Test Validity

As educational technology practitioners and researchers engage in research in the field, a wide array of research methodologies are available to them. One such methodology is the Delphi Technique. Use of the Delphi Technique offers many benefits, including the ability to obtain expert opinion, build consensus, determine the suitability of the application of instructional interventions, forecast trends, and interact with

This paper represents one outcome from the Invitational Research Symposium on Technology-Enabled and Universally Designed Assessments, which examined technology-enabled assessments (TEA) and universal design (UD) as they relate to students with disabilities (SWD). It was developed to stimulate research into TEAs designed to make tests appropriate for the full range of the student population through enhanced accessibility. Four themes are explored: (a) a construct-centered approach to developing accessible assessments; (b) how technology and UD can provide access to targeted knowledge, skills, and abilities by embedding access and interactive features directly into systems that deliver TEAs; (c) the possibility of incorporating scaffolding directly into innovative assessment items; and (d) the importance of investigating the validity of inferences from TEAs that incorporate accessibility features designed to maximize validity. Through the paper, symposium participants and contributin...

The Shedler-Westen Assessment Procedure (SWAP) is a personality assessment instrument designed for use by expert clinical assessors. Critics have raised questions about its psychometrics, most notably its validity across observers and situations, the impact of its fixed score distribution on research findings, and its test-retest reliability. We review empirical data addressing its validity, emphasizing the multitrait-multimethod approach to evaluating test validity. To evaluate the hypothesis that the fixed, asymmetric score distribution artifactually inflates correlations between SWAP profiles, we conducted Monte Carlo simulations and also presented empirical data from a large patient sample. We observed a mean correlation of zero between simulated SWAP profiles, indicating that the score distribution does not impact the correlation coefficients. Empirical correlations between SWAP profiles of actual patients were small and similar to those obtained using Diagnostic and Statistica...

Abstract  Diagnosis of dementia in adults with intellectual disabilities requires documentation of clinically significant declines in memory and other cognitive skills, as well as changes in everyday and emotional functioning. To improve diagnostic accuracy in adults with Down syndrome, the authors examined conditions often associated with dementia, as well as tests useful for documentation of decline. Specific aims were to identify psychiatric disorders or medical conditions that increased the odds of a dementia diagnosis; to evaluate the sensitivity and specificity of widely used dementia scales; and to determine which tests, used singly or in combination, most accurately supported the presence of dementia. Participants were 78 adults with Down syndrome. Two methods based on a large test battery and one method based on clinical judgment were used to diagnose dementia. It was found that combinations of tests lead to increased levels of diagnostic sensitivity compared with single tests. When taken in combination with other investigations, our results suggest that assessment for psychiatric disorders, delayed memory decline, adaptive behavior decline, and the presence of seizures would be useful for the diagnosis of dementia and that dementia scales would provide additional useful information. The authors conclude that combinations of tests and scales will be most useful for diagnosing dementia in adults with intellectual disabilities. The authors suggest that further research is needed to promote rapid progress, with studies that focus on common diagnostic methodology, identification of screening instruments, and amounts of decline indicative of dementia.

AUTHOR Hwalek, Melanie A.; Sengstock, Mary C. TITLE A Screening Instrument for Identifying Elderly at Risk of Abuse and Neglect. SPONS AGENCY Department of Health and Human Services, Washington, D.C. PUB DATE Nov 85 GRANT 90-AR-0040/2 NOTE 18p.; Paper presented at the Annual Scientific Meeting of the Gerontological Society (38th, New Orleans, LA, November 22-26, 1985). PUB TYPE Speeches/Conference Papers (150) -Tests/Evaluation Instruments (160) -Reports Research/Technical (143)

The psychometric properties of the Hopelessness Scale for Children (HSC) were evaluated with a sample of 834 normal adolescents and 93 adolescent suicide attempters. Factor analyses studies revealed two factors with both groups. Internal consistency with item-total score correlations were acceptable, while moderate test-retest reliability was found over a 10-week period with the normal sample. Support for the validity of the HSC was provided via studies with the suicide attempter group in which positive correlations with depression and depressive attributional style were found. Predicted differences between the suicide attempters, an outpatient psychiatric sample, and normal controls were also found. However, some items did not appear to discriminate suicide attempters from controls. Results are discussed in terms of the utility of the HSC with adolescents and adolescent suicide attempters and with regard to differences between child and adult samples in hopelessness.

The Community of Inquiry model provides a framework for recognizing and evaluating interpersonal behaviors in online educational settings. One of its three components, teaching presence, describes behaviors that are under the auspices of the online instructor. By examining behaviors through the theoretical lens provided by teaching presence, and by measuring them with the Teaching Presence Scale (TPS), it may be possible to better understand the most effectively online instruction practices. The purpose of the study was to confirm the factor composition of TPS in an online professional development course and to determine the relationship between teaching presence and student satisfaction. Participants (n = 718) were in-service educators enrolled in online professional development. Confirmatory factor analysis results provided strongest support for a three-factor TPA model using 17 of the 28 original TPS items and evidence of a strong relationship between TPS and student satisfaction...

Teachers' competence to estimate the effectiveness of learning materials is important and often neglected in programmes for teacher education. In this lecture I will try to explore the possibilities of designing scaffolding instruments for a priori assessment of language learning tasks, based on insights from SLA and cognitive psychology, more specifically connectionist theory. I will subsequently outline the development and evaluation of a ‘yardstick’ to judge complex, integrated, life-like tasks, such as WebQuests. The possibilities will be explored of performing in-depth a priori task analyses as a learning task for teachers in order to enhance their competence in making ‘educated guesses’ about task effectiveness. Finally, an experiment will be described to determine the reliability and validity of an instrument for in-depth analysis of language learning tasks based on the theoretical framework previously described.

This study used classical test theory to assess the psychometric properties of the Environmental Rating Scale (ERS), a measure specifically designed to assess the treatment programs in residential settings that serve individuals with autism. Results of the confirmatory factor analysis support the presence of a single factor represented by the total score. The reliability of the measure was demonstrated by assessments of the internal consistency, stability, and interrater reliability. Preliminary analysis of the validity of the ERS indicates that this measure discriminates between treatment settings designed specifically for individuals with autism and those designed for other populations of developmentally handicapped clients and family homes. The ERS was also significantly positively correlated with a measure of the caregiver's knowledge about autism and a visitor's global impression of the desirability of the setting as a place to live.

Objective: The study examined the concordance among rating sources on attention problems of elementary school–aged children. Method: A randomly selected sample ( N = 841) of children was rated by the physical educators, the teachers, and the parents, using the Attention Scales of the Motor Behavior Checklist (MBC), the Teacher Report Form, the Child Behavior Checklist, and the ADHD–Rating Scale–IV (ADHD-RS-IV ). Results: Convergent validity of the Lack of Attention Scale of the MBC with the corresponding subscales was supported. Correlations were higher between teachers’ ratings and between physical educators’ and teachers’ ratings than between physical educators’ and parents’ ratings or between teachers’ and parents’ ratings. Conclusion: Findings underscore the importance of taking the child’s settings and observer influences into account and suggest that MBC is a new promising instrument for screening attention problems in school settings.

Read-aloud accommodations have been proposed as a way to help remove barriers faced by students with disabilities in reading comprehension. Many empirical studies have examined the effects of read-aloud accommodations; however, the results are mixed. With a variance-known hierarchical linear modeling approach, based on 114 effect sizes from 23 studies, a meta-analysis was conducted to examine the effects of read-aloud accommodations for students with and without disabilities. In general, both students with disabilities and students without disabilities benefited from the read-aloud accommodations, and the accommodation effect size for students with disabilities was significantly larger than the effect size for students without disabilities. Further, this meta-analysis reveals important factors that influence the effects of read-aloud accommodations. For instance, the accommodation effect was significantly stronger when the subject area was reading than when the subject area was math. The effect of read-aloud accommodations was also significantly stronger when the test was read by human proctors than when it was read by video/audio players or computers. Finally, the implications, limitations, and directions for future research are discussed.

ABSTRACT Due to the scarcity of cross-cultural comparative studies in exploring students' self-efficacy in science learning, this study attempted to develop a multi-dimensional science learning self-efficacy (SLSE) instrument to measure 316 Singaporean and 303 Taiwanese eighth graders' SLSE and further to examine the differences between the two student groups. Moreover, within-culture comparisons were made in terms of gender. The results showed that, first, the SLSE instrument was valid and reliable for measuring the Singaporean and Taiwanese students' SLSE. Second, through a two-way multivariate analysis of variance analysis (nationality by gender), the main result indicated that the SLSE held by the Singaporean eighth graders was significantly higher than that of their Taiwanese counterparts in all dimensions, including ‘conceptual understanding and higher-order cognitive skills’, ‘practical work (PW)’, ‘everyday application’, and ‘science communication’. In addition, the within-culture gender comparisons indicated that the male Singaporean students tended to possess higher SLSE than the female students did in all SLSE dimensions except for the ‘PW’ dimension. However, no gender differences were found in the Taiwanese sample. The findings unraveled in this study were interpreted from a socio-cultural perspective in terms of the curriculum differences, societal expectations of science education, and educational policies in Singapore and Taiwan.