A Comparison Of Three Methods Of Rating Scale Construction

Four job evaluation factors were used as the basis of rating 10 Air Force specialties. For each factor three different methods were used in constructing the scale: (1) each scale division was defined and illustrated; (2) neither scale division definitions nor examples were used; and (3) definitions were used but the examples were omitted. Ratings by samples of aviation cadets were analyzed for effects of method on mean ratings. For three of the four factors, the mean ratings obtained were not different as a function of the method of scale construction. Methods 1 and 3 were about equally reliable, both yielding more reliable means than method 2. Method 3 is suggested as being the most effective because the task of the rater is somewhat simpler than for method 1 and the reliability is higher than for method 2.