Reading...
Play button
Play button
Use LEFT and RIGHT arrow keys to navigate between flashcards;
Use UP and DOWN arrow keys to flip the card;
H to show hint;
A reads text to speech;
75 Cards in this Set
- Front
- Back
|
reliability
|
the degree of dependability, consistency, or stability of scores on a measure used in selection research
|
|
|
Predictions concerning selection must be based on the results of “true measures” of differences that have:
|
Consistency in results
Dependability of measurement Stability over time |
|
|
Errors of Measurement
|
Factors that affect obtained scores but are not related to the characteristic, trait, or attribute being measured
|
|
|
Obtained Score Components:
|
Xobtained = Xtrue + Xerror
|
|
|
Xobtained =
|
obtained score for a person on a measure
|
|
|
Xtrue =
|
true score on the measure, that is, actual amount of the attribute measured that a person really possesses
|
|
|
Xerror =
|
error score on the measure that is assumed to represent random fluctuations or chance factors.
|
|
|
True Score
|
The mean or average score made by an individual on many different administrations of a measure if external and inter conditions were perfect
|
|
|
Error Score
|
The score made by individuals that is influenced by factors present at the time of measurement that distort individuals’ scores either over or under what they would have been on another measurement occasion
|
|
|
Reliability Coefficient
|
An index that summarizes the estimated relationship between two sets of measures
The higher the coefficient, the higher the reliability estimate and the lower the error of measureme |
|
|
Principle Methods of Estimating Reliability
|
Test-retest or stability
Parallel or equivalent forms Internal consistency Interrater reliability estimates |
|
|
Test-Retest Reliability
|
The same measure is used to collect data from the same respondents at two different points in time
|
|
|
Factors that may affect test-retest reliability
|
Memory (overestimate)
Learning (underestimate) |
|
|
Use test-retest reliability when
|
-the length of time between the two administrations is long enough to offset the effects of memory or practice
-there is little reason to believe that memory will affect responses to a measure -it can be determined that nothing has occurred between the two testings that will affect responses -information is available on only a single item measure |
|
|
Internal Consistency Reliability Estimate
|
An index of a measure that shows the extent to which all parts of a measure are similar in what they measure
|
|
|
Internal Consistency Procedures
|
Split-half reliability
Kuder-Richardson reliability Cronbach’s coefficient alpha (α) reliability |
|
|
Sources of Measurement Error
|
What is being rated (e.g., employee behavior)
Who is doing the rating (rater characteristics) |
|
|
Interrater Reliability Estimates
|
Sources of Measurement Error
Consistency of the implementation of a rating system Procedures for Calculating Interrater Reliability |
|
|
Interrater Agreement
|
Percentage of rater agreement
Kendall’s coefficient of concordance (W) Cohen’s kappa (K) |
|
|
Reliability Analyses
|
help determine the dependability of data we will use in selection decision making
help in estimating the amount of error included in scores on any measure we choose to study |
|
|
What Does a Reliability Coefficient Mean?
|
The extent (in percentage terms) to which individual differences in scores on a measure are due to “true” differences in the attribute measured and the extent to which they are due to chance errors
|
|
|
Characteristics of Reliability Coefficients
|
They are specific to the reliability estimation method and group on which they are calculated
They are a necessary but not a sufficient condition for validity They are based on responses from a group of individuals They are expressed by degree, and are determined ultimately by judgment |
|
|
Factors Affecting Estimated Reliability
|
Method of estimating reliability
Stability Sample Individual differences among respondents Length of a measure Test question difficulty Homogeneity of content response format Administration and scoring |
|
|
Validity: A Definition
|
The degree to which available evidence supports inferences made from scores on selection measures.
|
|
|
Importance of Validity in Selection
|
Shows how well a predictor (such as a test) is related to important job performance criteria
Can indicate what types of inferences may be made from scores obtained on the measurement device |
|
|
Validation Study
|
Provides evidence for determining the accuracy of judgments made from scores on a predictor about important job behaviors as represented by a criterion.
|
|
|
Types of Validation Strategies
|
Content validation
Criterion-related validation Concurrent and predictive strategies Construct validation |
|
|
Content Validity
|
Is shown when the content (items, questions, etc.) of a selection measure representatively samples the content of the job for which the measure will be used.
|
|
|
Why Content Validation?
|
Is applicable to hiring situations involving a small number of applicants for a position
Focuses on job content, not job success criteria Increases applicant perception of the fairness of selection procedures |
|
|
“Content of the job”
|
Job behaviors, the associated knowledge, skills, abilities, and other personal characteristics (KSAs) that are necessary for effective job performance
|
|
|
Job Content Domain
|
The behaviors and KSAs that compose the content of the job to be assessed
Content of the measure must be representative of the job content domain for the measure to possess content validity |
|
|
Content Validation Strategy
|
Involves construction of a new measure rather than the validation of an existing one.
Emphasizes the role of expert judgment in determining the validity of a measure rather than relying on statistical methods |
|
|
Face Validity
|
Concerns the appearance to job applicants of whether a measure is measuring what is intended—the appearance to applicants taking the test that the test is related to the job.
Increases acceptability of a measure |
|
|
Major Aspects of Content Validation
|
Conduct a Comprehensive Job Analysis
Selection of Experts Participating in Study Specify Selection Measure Content Assessment of Selection Measure and Job Content Relevance |
|
|
Content validation is not appropriate when:
|
Mental processes, psychological constructs, or personality traits are not directly observable but inferred from the selection device.
The selection procedure involves KSAs an employee is expected to learn on the job. The content of the selection device does not resemble a work behavior; when the setting and administration of the selection procedure does not resemble the work setting |
|
|
Content validity
|
Focuses on the selection measure itself
Is based on a broader base of data and inference Is generally characterized as using broader, more judgmental descriptors (description) |
|
|
Criterion-related validity
|
Focuses on an external variable
Is narrowly based on a specific set of data Is couched in terms of precise quantitative indices (prediction) |
|
|
Concurrent Validation Strategy
|
Both predictor and criterion data is obtained on a current group of employees, and statistical procedures are used to test for a statistically significant relationship (correlation coefficient) between these two sources of data
Sometimes referred to as the “present employee method” because data is collected for a current group of employees |
|
|
Concurrent Validation
|
Conduct analyses of the job.
Determine relevant KSAs and other characteristics required to perform the job successfully. Choose or develop the experimental predictors of these KSAs. Select criteria of job success. Administer predictors to current employees and collect criterion data. Analyze predictor and criterion data relationships |
|
|
Predictive Validation
|
Conduct analyses of the job.
Determine relevant KSAs and other characteristics required to perform the job successfully. Choose or develop the experimental predictors of these KSAs. Select criteria of job success. Administer predictors to job applicants and file results. After passage of a suitable period of time, collect criterion data. Analyze predictor and criterion data relationships. |
|
|
Construct
|
A postulated concept, attribute, characteristic, or quality thought to be assessed by a measure.
|
|
|
Construct Validation
|
A research process involving the collection of evidence used to test hypotheses about relationships between measures and their constructs.
|
|
|
What precisely does a statistically significant coefficient mean
|
Coefficient of determination
Expectancy tables and charts Utility analysis |
|
|
Coefficient of determination
|
The percentage of variance in the criterion that can be explained by variance associated with the predictor.
|
|
|
Utility analysis
|
An economic interpretation to the meaning of a validity coefficient
|
|
|
Linear Regression
|
The determination of how changes in criterion scores are functionally related to changes in predictor scores
|
|
|
Regression Equation
|
Mathematically describes the functional relationship between the predictor and criterion
-Simple regression assumes only one predictor -Multiple regression assumes two or more predictors |
|
|
Utility Analysis: A Definition
|
Shows the degree to which use of a selection measure improves the quality of individuals selected versus what would have happened if the measure had not been used.
|
|
|
Validity Generalization
|
Uses evidence accumulated from multiple validation studies to show the extent to which a predictor that is valid in one setting is valid in other, similar settings
|
|
|
Situational Specificity Hypothesis (Ghiselli)
|
The validity of a test is specific to the job or situation where the validation study was completed.
|
|
|
Deficiencies Affecting Validity
|
The use of small sample sizes (sampling error)
Differences in test or predictor reliability Differences in criterion reliability Differences in the restriction of range of scores Criterion contamination and deficiency Computational and typographical errors Slight differences among tests thought to be measuring the same attributes or constructs |
|
|
Conclusions
|
It is not necessary to conduct validity studies within each organization for every job
Mental ability tests can be expected to predict job performance in most, if not all, employment situations |
|
|
job performance measures
|
to serve as criteria measures in validation studies
|
|
|
production data
hr personnel data training proficiency judgmental data |
types of job performance measures
|
|
|
Types of Production Data
|
Quantity—usually expressed as the number of units produced within a specified time period.
Quality—the goodness of the product indirectly measured by defects, errors, or mistakes identified either per number of units or per unit of time. |
|
|
The assumption that promotions are the result of high performance rather than other factors such as job tenure and organization change/growth.
Limitations imposed by short career ladders. Distinguishing lateral from vertical promotions. Accounting for promotions voluntarily declined |
Issues in Measuring Promotions
|
|
|
Training Proficiency
|
A measure of employees’ performance immediately after completing a training program.
|
|
|
Promotion
|
A measure of career progress, based on the number of vertical job changes for an individual that result in increased responsibility and authority
|
|
|
Measuring Proficiency
|
Standardization of training programs—instructional materials, techniques, training mastery measures
Linking of training effects to increases in worker KSAs |
|
|
Measures of Training Proficiency
|
Training instructor judgments about trainees
Scores on paper-and-pencil tests Scores on work sample tests |
|
|
problem of bias
|
avoiding supervisor biases in making judgment
training supervisors to avoid common errors systematically collecting data to ensure complete data |
|
|
Halo—
|
rating the subordinate equally on different performance scales because of a rater’s general impression of the worker.
|
|
|
Leniency/Severity—
|
rating a disproportionate number of workers either high or low respectively.
|
|
|
Central tendency—
|
rating a large number of subordinates in the middle of the scale.
|
|
|
Signs
|
Selection tests that are indirect indicators of an individual’s predisposition to behave in certain ways.
|
|
|
Samples
|
Selection tests that gather information about behaviors that are consistent with the job behaviors being predicted.
|
|
|
Motor Tests
|
Require the physical manipulation of things—operating a machine, installing a piece of equipment, or making a product.
|
|
|
Verbal Tests
|
Require the use of spoken or written messages or interpersonal interaction if the problem situation is primarily language or people oriented.
|
|
|
Trainability Tests
|
Are most often used for jobs that do not presently exist and specialized jobs for which extensive training is necessary.
|
|
|
Establish Independent Test Sections
|
Develop the test such that an applicant’s performance on one part of the test is not closely tied to another part of the test.
|
|
|
Assessment Center (AC)
|
A procedure for measuring KSAs in groups of individuals (usually 12 to 24) that uses a series of devices (exercises), many of which are verbal performance tests.
|
|
|
Uses of ACs
|
Selection—identifying participants who demonstrate behaviors necessary for the position considered.
Career development—determining those behaviors each participant does well and those in which each is deficient. |
|
|
In-Basket
|
A paper-and-pencil test designed to replicate administrative tasks of the job under consideration.
|
|
|
Leaderless Group Discussion (LGD)
|
A small group of applicants are each assigned roles and charged with resolving either a competitive or a cooperative scenario.
|
|
|
Situation Judgment Tests (SJTs)
|
“Low-fidelity” simulations that present a series of written descriptions of work situations and multiple responses to each situation.
|
