Professor George Michaelides, Norwich Business School, University of East Anglia
Professor Duncan J. R. Jackson, King’s Business School, King’s College London
Chair: George Michaelides (University of East Anglia) & Duncan J.R. Jackson (King’s College London)
Date: May 21st, 2025
Time: 9:00 – 12:00 (Half-day session)
Price: 40 EUR
Capacity: 20 People
MAIN LEARNING OBJECTIVES
In this workshop, we aim to:
1. Familiarize attendees with core G theory concepts and how G theory conceptualizes reliability as a generalization process;
2. Describe how a G study is constructed and why G theory is often preferable to traditional psychometric approaches to analysing MMDs;
3. Discuss the advantages and challenges of adopting Bayesian inference for G theory and how it assists with meaningful estimation and flexibility;
4. Engage with assessees on a series of worked examples of univariate (single-score) and multivariate (multi-score) G studies;
TARGET AUDIENCE
Our target audience includes postgraduate students, researchers, and practitioners with an interest in measurement who:
- Have no familiarity with G theory or Bayesian estimation and who wish to expand their knowledge on these topics
- Are familiar with G theory and who are interested in finding out more about Bayesian estimation in this context (or vice versa)
WORKSHOP SUMMARY AND PURPOSE
Generalizability (G) theory is one of the most comprehensive psychometric frameworks available for the analysis of data generated by complex evaluation procedures often applied in organizational research and practice (e.g., performance ratings, Brennan, 2001; Jackson et al., 2020; Jackson et al., 2016). G theory is applicable to multifaceted measurement designs (MMDs), which are characterized by multiple design features that could impact reliability (e.g., raters, rating items, simulation exercises, Cronbach et al., 1972). MMDs are widely applied as a basis for decision-making and as a data source in organizational research and practice.
Because MMDs involve many different design elements that could interrelate in ways that are difficult or impossible to anticipate (Cronbach et al., 1972), the application of traditional approaches towards measurement structure and reliability to MMDs (e.g., confirmatory factor analysis, coefficient alpha,) often raise concerns about statistical confounding (Jackson et al., 2016; Putka & Hoffman, 2013). In contrast, G theory was specifically developed to handle complex MMD configurations (Brennan, 2001; Shavelson & Webb, 1991).
Although occasionally applied in organizational settings (e.g., DeShon, 2002; Highhouse et al., 2009; Lievens, 2001), G theory has not (yet) become a mainstream analytical approach. The purpose of the current session is to facilitate the adoption of G theory-based analyses by providing introductory insights about the Bayesian G theory perspective on MMD data. We will cover core concepts in and applications of G theory. We will discuss statistical estimation and the substantial advantages of adopting Bayesian inference in the G theory context over traditional statistical estimation. Bayesian analyses allow for statistical estimation with large data sets where traditional estimates often tend to fail. Bayesian estimators moreover avoid negative or zero-fenced variance estimates associated with traditional approaches. From this point in the workshop, we will move to examples of G theory analyses (G studies) from both univariate and multivariate perspectives. On this latter point,
despite holding much potential to inform on organizational evaluations, multivariate G theory has remained almost completely unutilized in this context. Our workshop will offer a guided approach to the interpretation of G theory analyses and their underlying estimation engine.
WHAT ATTENDEES ARE EXPECTED TO GAIN FROM WORKSHOP ATTENDANCE
- A new or enhanced understanding about the G theory perspective on reliability
- An appreciation for how G theory analyses are constructed
- An awareness about how G theory is often preferable to traditional psychometrics in the study of MMDs
- An understanding about the role Bayesian analysis has in providing enhanced flexibility and sophistication compared to traditional estimation in the G theory context
- Knowledge about how to conduct a basic univariate Bayesian G theory analysis using the brms package
- Knowledge about how to interpret a multivariate G theory analysis (adopting the format presented in Brennan (2001)
REFERENCES
Brennan, R. L. (2001). Generalizability theory. Springer Verlag.
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. John Wiley.
DeShon, R. P. (2002). Generalizability theory. In F. Drasgow & N. Schmitt (Eds.), Measuring and analyzing behavior in organizations (pp. 189-220). Jossey-Bass.
Highhouse, S., Broadfoot, A., Yugo, J. E., & Devendorf, S. A. (2009). Examining corporate reputation judgments with generalizability theory. Journal of Applied Psychology, 94(3), 782-789. https://doi.org/10.1037/a0013934
Jackson, D. J. R., Michaelides, G., Dewberry, C., Schwencke, B., & Toms, S. (2020). The implications of unconfounding multisource performance ratings. Journal of Applied Psychology, 105(3), 312-329. https://doi.org/10.1037/apl0000434
Jackson, D. J. R., Michaelides, M., Dewberry, C., & Kim, Y. (2016). Everything that you have ever been told about assessment center ratings is confounded. Journal of Applied Psychology, 101(7), 976-994. https://doi.org/10.1037/apl0000102
Lievens, F. (2001). Assessors and use of assessment centre dimensions: A fresh look at a troubling issue. Journal of Organizational Behavior, 22(3), 203-221. https://doi.org/10.1002/job.65
Putka, D. J., & Hoffman, B. J. (2013). Clarifying the contribution of assessee-, dimension-, exercise-, and assessor-related effects to reliable and unreliable variance in assessment center ratings. Journal of Applied Psychology, 98(1), 114-133. https://doi.org/10.1037/a0030887
Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Sage.