Psychometrics and Personality Assessment

Test Theory

All scientific theories require measurement of the constructs underlying the field. Personality theories are no different. Whether we are developing theories of species typical behavior, of individual differences in behavior, or unique patterns of thoughts and feelings, we need to be able to measure the responses in question. The fields of psychometrics and personality assessment are devoted to the study of the measurement of pscyhological constructs associated with personality.

Consider the case of differences in vocabulary in a particular language (e.g., English). Although it is logically possible to organize people in terms of the specific words they know in English, the more than 2^(500,000) possible response patterns that could be found by quizzing people on each of the more than 500,000 words in English introduces more complexity rather than less. Classical Test Theory (CTT) ignores individual response patterns and estimates an individual's total vocabulary size by measuring performance on small samples of words. Words are seen as random replicates of each other and thus individual differences in total vocabulary size are estimated from observed differences on these smaller samples. The Pearson Product Moment Correlation Coefficient (r) compares the degree of covariance between these samples with the variance within samples. As the number of words sampled increases, the correlation of the individual differences within each sample and with those in the total domain increases accordingly.

Estimates of ability based upon Item Response Theory (IRT) take into account parameters of the words themselves (i.e., the difficulty and discriminability of each word) and estimate a single ability parameter for each individual. Although CTT and IRT estimates are highly correlated, CTT statistics are based on decomposing the sources of variance within and between individuals while IRT statistics focus on the precision of an individual estimate without requiring differences between individuals. CTT estimates of reliability of ability measures are assessed across similar items (internal consistency), across alternate forms, and across different forms of assessment as well as over time (stability). Tests are reliable to the extent that differences within individuals are small compared to those between individuals when generalizing across items, forms, or occasions. CTT reliability thus requires between subject variability. IRT estimates, on the other hand, are concerned with the precision of measurement for a particular person in terms of a metric defined by item difficulty.

The test theory developed to account for sampling differences within domains can be generalized to account for differences between domains. Just as different samples of words will yield somewhat different estimates of vocabulary, different cognitive tasks (e.g., vocabulary and arithmetic performance) will yield different estimates of performance. Using multivariate procedures such as Principal Components Analysis or Factor Analysis, it is possible to decompose the total variation into between domain covariance, within domain covariance, and within domain variance. One of the most replicable observations in the study of individual differences is that almost all tests thought to assess cognitive ability have a general factor (g) that is shared with other tests of ability. That is, although each test has specific variance associated with content (e.g., linguistic, spatial), form of administration (e.g., auditory, visual), or operations involved (e.g., perceptual speed, memory storage, memory retrieval, abstract reasoning), there is general variance that is common to all tests of cognitive ability.

Statistical programs

Statistical techniques in personality measurement are available in a very powerful (and open source) package, R. Although somewhat intimidating for the casual user, R is a must have for all serious personality researchers. The R project, based upon the S and S+ stats packages, has developed an extremely powerful set of "packages" that operate within one program. Although described as merely "an effective data handling and storage facility [with] a suite of operators for calculations on arrays, in particular, matrices" R is, in fact, a very useful interactive package for data analysis. When compared to most other stats packages used by psychologists, R has at least three compelling advantages: it is free, it runs on multiple platforms (e.g., Windows, Unix, Linux, and Mac OS X and Classic), and combines many of the most useful statistical programs into one quasi integrated program. (R is free software as part of the GNU Project. That is, users are free to use, modify, and distribute the program, within the limits of the GNU non-license). The program itself and detailed installation instructions for Linux, Unix, Windows, and Macs are available through CRAN (Comprehensive R Archive Network).

A guide to R for the personality researcher as well as a package of functions particularly suited for personality measurement is now part of the personality project. The R suite of programs includes many useful for the personality researcher, including factor analysis, structural equation modeling, and multidimensional scaling. The psych package includes basic tools for scale construction and analysis, including finding basic descriptive statistics, using the Very Simple Structure (VSS) criterion for determing the optimal number of factors , cluster analysis of items using the ICLUST algorithm, hierarchical factor analysis with Schmid Leiman tranformations, and procedures for estimating alternative measures of test reliablility (i.e., alpha, beta, and omega.) All of these functions are available in the psych package which may be downloaded from CRAN. First install R and then install.packages("psych").

Statistical packages for personality research are also commercially available in such programs as SPSS, SYSTAT, or SAS. Bob Muenchen has developed a comparison of the features of R with SAS and SPSS.

In addition, a short course on R was offered at Northwestern in 2009. The readings for that provide more information about R for the psychology student, with a particular emphasis upon those interested in personality.

General research methods

B. Robins, C. Fraley, and R. Krueger (2007), Personality Research Methods, Guilford. is a handbook on research in personality. Chapters include:
Revelle, W. (2007) Experimental Approaches to the Study of Personality, to appear in B. Robins, C. Fraley, and R. Krueger, Personality Research Methods, Guilford. (includes an online appendix simulating experimental data with a discussion of how to analyze it.)

Psychometric Theory

The measurement of individual differences is of primary concern to personality psychologists. Issues in measurement include fundamentals of scaling, measures of association, determining the relationship between observed and latent variables, reliable measurement of variables, and validity of these measures. These topics are addressed in as yet unfinshed text on psychometric theory with applications in R (Revelle, in prep). Chapters currently available include:

Chapter 1: Introduction
Chapter 2: A theory of Data
Chapter 3: The problem of scale
Chapter 4: Correlation and Covariance
Chapter 5: Multiple Correlation and Multiple Regression
Chapter 6: Constructs, components, and factor models
Chapter 7: Reliability
Chapter 8: Item Response Theory (incomplete)
Appendix A Getting Started (with R)
Appendix B: R Commands (preliminary)
Appendix E: Review of Matrix Algebra including using R for matrix operations

Readings in Measurement

Brief biographies of important figures in measurement are available from the University of Indiana Intelligence project:
- Binet,
- Galton,
- James,
- Pearson,
- Spearman, and
- Yerkes.
Much of psychological measurement has been concerned with the measurement of ability and intelligence. Relevant readings are discussed on the intelligence page.
ERIC has a series of useful online tutorials in test construction and measurement: "This is a set of on-line, full-text books and booklets addressing practical evaluation, research, measurement, and statistical issues. The series started as a joint project of ERIC/AE and Bruce Thompson, Texas A&M University. The ERIC Clearinghouse on Assessment and Evaluation, and Bill Trochim's Center for Social Research Methods". This later reference is a very good tutorial for basic multivariate concepts.

ERIC "How to series" including:
- Research Methods
- Measurement
- Statistics
For an introduction to recent advances in Item Response Theory see:
- Lowman, R. L. (1996) What Every Psychologist Should Know about Assessment. Psychological Assessment, 8, 339-340.
- Embretson, S.E. (1996) The New Rules of Measurement. Psychological Assessment, 8, 341-349.
- Reckase, M.D. (1996) Test Construction in the 1990s: Recent Approaches Every Psychologist Should Know. Psychological Assessment, 8, 354-359.
- See also an example of how IRT techniques differ from conventional Factor Analytic results when developing scales from dichotomous items.
  Waller, N. G., Tellegen, A., McDonald, R. P., & Lykken, D. T. (1996). Exploring Nonlinear Models in Personality Assessment: Development and Preliminary Validation of a Negative Emotionality Scale. Journal of Personality, 64, 545-576.
- Software to do IRT is available as part of the ltm package in R.
The SEMNET FAQ(The Structural Equation Net-- Frequently Asked Questions) includes
- What is Structural Equation Modeling
- The Form of Structural Equation Models
- Alternatives to Structural Equation Modeling and Confirmatory Factor Analysis
- Programs for doing Structural Equation Modeling include John Fox's sem package in R.
- Courses or readings in Structural Equation Modeling include
  - Bollen, K.A. (2002) Latent variables in psychology and the social sciences. Annual Review of Psychology.
  - John Fox: York University: Structural Equation Modeling
  - Kenny, D. University of Connecticut Structural Equation Modeling
  - Revelle, W., Northwestern University Structural Equation Modeling
  - An important discussion of the criteria used for evaluating SEM adequacy is found in the May, 2007 issue of Personality and Individual Differences with a lead article by Paul Barrett and commentaries by Peter Bentler, Roger Millsap, Stanly Mulaik, James Steiger, and others. (For readers with access to the Science Direct journals, the URL is http://www.sciencedirect.com/science/journal/01918869 and then search for the May, 2007 issue.)
Classic or well cited readings in personality assessment include:
- Rorer, LG. (1990). Personality Assessment: A Conceptual Survey. In LA Pervin (Ed.), Handbook of Personality: Theory and Research (pp.693-720). New York: Guilford.
- Wiggins, J.S. (1973) Personality and Prediction: Principles of Personality Assessment. Reading, Ma: Addison-Wesley. Reprinted (1988) by Krieger Publishing, Malabar, FL.
- John, OP. (1990). The "Big Five" factor taxonomy: Dimensions of Personality in the Natural Language and in Questionnaires. In LA Pervin (Eds.), Handbook of Personality: Theory and Research . New York: Guilford. (See also Big 5 references)
- An introduction to Personality Measurement suitable for advanced undergraduates is Briggs, S. R. Personality Measurement in In Derlega, V., Winstead, B.A., Jones, W.H. Personality: Contemporary Theory and Research. (2nd Edition). Nelson Hall, 1999, Chicago.
Other Important Books in Psychometrics and Personality Assessment include:
- Collins, L. M. & Horn, J. L. (1991) Best Methods for the Analysis of Change: Recent Advances, Unanswered Questions, Future Directions. Washington,D.C.: American Psychological Association.
- Costa Jr., PT., & Widiger, TA. (Ed.). (1994). Personality Disorders and the Five-factor Model of Personality. Washington, D.C.: American Psychological Association.
- Nunnally, Jum & Bernstein, Ira (1994)Psychometric Theory New York: McGraw Hill, 3rd ed.
- Loehlin, John (1987/1994) Latent Variable Models: an Introduction to Factor, Path, and Structural Analysis. Hillsdale, N.J.: LEA.
- Shrout, P. E. and Fiske, S. T. (Eds.) (1995) Advances in Personality Research, Methods and Theory: A Festschrift Honoring Donald W. Fiske. Hillsdale, NJ: L. Erlbaum Associates.
- Reviews and information about tests of personality and ability from ERIC.
- The APA answers some Frequently Asked Questions (FAQ) on Psychological Tests
A very useful resource for listings of research measures in personality is
- Robinson, J. P., Shaver, P. R., and Wrightsman, L.S. (1991) Measures of Personality and Social Psychological Attitudes. Academic Press, San Diego.
- See also Lew Goldberg's International Personality Item Pool or IPIP.
- The Measurement Group, provides a source of excellent information about personality tests and statistical techniques as well as current results in HIV/AIDS research.

Psychological and Educational Statistics

For overviews, first consult the ERIC series of useful online tutorials in test construction and measurement:
- Research Methods
- Measurement
- Statistics
Bentler, P.M. & Dudgeon, P. Covariance Structure Analysis: Statistical Practice, Theory, and Directions, Annual Review of Psychology, 46, 1996.
Multi-level models are particularly appropriate for studying personality dynamics and changes over time.
Arizona State University's resources for learning about data analysis are worth visiting for an on line education in statistics.
Larry Lyon's guide to Meta-analysis was a very helpful guide to the process of combining results from multiple studies. An excellent tutorial!
Gary McClelland's course materials for his class on data analysis using the text Judd, C. M., & McClelland, G. H. (1989). Data analysis: A Model Comparison Approach. New York: Harcourt, Brace Jovanovich.
Additional statistical information is available on the web from Statistics on the Web or A Guide to Statistical Computing Resources on the Internet.
See also the Statistics Textbook being developed at UCLA.
The University of Texas has a helpful "FAQ" on frequently asked questions about statistics.

Recent Review Articles on Personality and Measurement

From the Annual Review of Psychology

These reviews are excellent starting points for background in a particular area. Note that for a limited time, the abstracts and full texts of the most recent several volumes are available on line.

Bollen, K.A. Latent variables in psychology and the social sciences 2002
Ann Robinson and Pamela R. Clinkenbeard GIFTEDNESS: An Exceptionality Examined Annu. Rev. Psychol. 1998, Vol. 49: 117-139. (abstract)
Walter Mischel and Yuichi Shoda RECONCILING PROCESSING DYNAMICS AND PERSONALITY DISPOSITIONS Annu. Rev. Psychol. 1998, Vol. 49: 229-258. [Abstract]
Susan Mineka, David Watson, and Lee Anna Clark COMORBIDITY OF ANXIETY AND UNIPOLAR MOOD DISORDERS Annu. Rev. Psychol. 1998, Vol. 49: 377-412. [Abstract]
Walter C. Borman, Mary Ann Hanson, and Jerry W. Hedge PERSONNEL SELECTION Annu. Rev. Psychol. 1997, Vol. 48: 299-337. [Abstract]
James N. Butcher and Steven V. Rouse PERSONALITY: Individual Differences and Clinical Assessment Annu. Rev. Psychol. 1996, Vol. 47: 87-111. [Abstract]
Bentler, P.M. & Dudgeon, P. Covariance Structure Analysis: Statistical Practice, Theory, and Directions, 46, 1996.
Hartup, W. W. & van Lieshhout, C.F.M. Personality Development in Social Context, 46, 1995.
Revelle, W. Personality Processes , 46, 1995.
Rose, R.J. Genes and Human Behavior, 46, 1995.
Ozer, D. J. & Reise, S. P. Personality Assessment, 45, 1994.
Mathews, A. & MacLeod, C. Cognitive Approaches to Emotion and Emotional Disorders. 45, 1994.
Banaji, M.R. & Prentice, D.A., The Self in Social Context. 45, 1994.
Sechrest, L. & Figueredo, A. J., Program Evaluation, 44, 1993.
Arabie, P. & Hubert, L. J. Combinatorial Data Analysis, 43, 1992.
Wiggins, J. S. & Pincus, A. L. Personality: Structure and Assessment, 43, 1992.
Schmidt, F. L., Ones, D. S., & Hunter, J. E. Personnel Selection, 43, 1992.
Plomin, R. & Rende, R. Human Behavioral Genetics, 42, 1991.
Widiger, T. A. & Trull, T. J., Diagnosis and Clinical Assessment, 42, 1991.
Digman, J. M. Personality Structure: Emergence of the Five-factor Model, 41, 1990.
Carson, R.C. Personality, 40, 1989.
Jones, L. V. & Appelbaum, M. Psychometric Methods, 40, 1989.
Gescheider, G. A. Psychophysical Scaling, 39, 1988.
Cook, T. D. & Shadish, W. R. Jr. Program Evaluation: The Worldly Science, 37, 1986.
Anastasi, A. Evolving Concepts of Test Validation, 37, 1986.
Hakel, M. D. Personnel Selection and Placement, 37, 1986.
Robins, L.N. & Helzer, J.E. Diagnosis and Clinical Assessment: The Current State of Psychiatric Diagnosis, 37, 1986.
Pervin, L.A. Personality: Current Controversies, Issues, and Directions, 36, 1985.
Traub, R.E. & Lam, Y.R. Latent Structure and Item Sampling Models for Testing. 36, 1985.
Young, F. W. Scaling, 35, 1984.
Lanyon, R.I. Personality Assessment. 35, 1984.
Zedeck, S. and Cascio, W.F. Psychological Issues in Personnel Decisions. 35, 1984.
Rorer, L.G. & Widiger, T.A. Personality Structure and Assessment. 34,1983.
Eysenck, H.J., Wakefield, J.A., & Friedman, A.F. Diagnosis and Clinical Assessment. 34, 1983.
Weiss, D.J. & Davison, M.L. Test Theory and Methods. 33, 1982.
Wainer, H. and Thissen, D. Graphical Data Analysis, 32, 1981.
Weiss, D.J. and Davison, M.L. Test Theory and Methods. 32, 1981.
Bentler, P.M. Multivariate Analysis with Latent Variables: Causal Modeling. 31, 1980.
Carroll, J.D. and Arabie, P. Multidimensional Scaling. 31, 1980.
Jackson, D.N. and Paunonen, S.V. Personality Structure and Assessment. 31, 1980.
Carroll, J.B. and Maxwell, S.E. Individual Differences in Cognitive Abilities. 30, 1979.
Lumsden, J. Test Theory. 27, 1976.
Goldberg, L. Objective Diagnostic Tests and Measures. 25, 1974.
Edwards, A. Personality: Theory and Techniques of Assessment. 24, 1973
Molish, B. Projective Methodologies. 23, 1932.
Bock, R.D. Test Theory. 22, 1971.

Some useful publications from the APA are available online:

The Personality Project