7
observer accuracy based on observer agreement with a preestablished criterion, such as a standard protocol, is one form of establishing validity. Evidence of criterion-related validity is also important and can be demonstrated by relating the observations to data collected by alternate methods either at the same point in time (i.e., concurrent validity) or at some point in the future (i.e., predictive validity). Convergent validity describes the correlation between the observed data and measures that are expected to relate, whereas discriminant validity describes the correlation between the observed data and measures that are not expected to relate. Discriminantive validity, in contrast, identifies differences between groups of individuals whose dissimilarities are established by other criteria.
Other forms of validity related to the construction of the coding system also need to be considered. Content validity applies to the adequacy of the sample of behavioral codes. If the sample of behavior is incomplete, does not reflect the intended use of the data, or is not theoretically based, then the data's relation to an alternative measure of that behavior is minimized. Construct validity is a broad concept that refers to the extent to which a measure reflects the construct of interest. Construct validity becomes an issue when individual codes are collapsed into larger clusters. Codes can be combined that (a) are assumed a priori to be part of a specific response class, (b) are demonstrated to have similarities in their functional relations, or (c) are demonstrated by factor analysis (Hops, Davis, & Longoria, 1995).
Factors Affecting the Reliability and Validity of Observational Systems
Variables related to the design of the coding system, the coding procedure, and the training of observers affect the reliability and validity of the data collected (Johnson &