Skip to main content
SpringerLink
Log in

What Reviewers Should Expect from Authors Regarding Common Method Bias in Organizational Research

  • Published:
Journal of Business and Psychology Aims and scope Submit manuscript

Abstract

We believe that journal reviewers (as well as editors and dissertation or thesis committee members) have to some extent perpetuated misconceptions about common method bias in self-report measures, including (a) that relationships between self-reported variables are necessarily and routinely upwardly biased, (b) other-reports (or other methods) are superior to self-reports, and (c) rating sources (e.g., self, other) constitute measurement methods. We argue against these misconceptions and make recommendations for what reviewers (and others) should reasonably expect from authors regarding common method bias. We believe it is reasonable to expect (a) an argument for why self-reports are appropriate, (b) construct validity evidence, (c) lack of overlap in items for different constructs, and (d) evidence that authors took proactive design steps to mitigate threats of method effects. We specifically do not recommend post hoc statistical control strategies; while some statistical strategies are promising, all have significant drawbacks and some have shown poor empirical results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Arora, R. (1982). Validation of an S-O-R model for situation, enduring and response components of involvement. Journal of Marketing Research, 19, 505–516.

    Article  Google Scholar 

  • Bagozzi, R. P., & Yi, Y. (1990). Assessing method variance in multitrait-multimethod matrices: The case of self-reported affect and perceptions at work. Journal of Applied Psychology, 75, 547–560.

    Article  Google Scholar 

  • Becker, T. E., & Cote, J. A. (1994). Additive and multiplicative method effects in applied psychological research: An empirical assessment of three models. Journal of Management, 20, 625–641.

    Article  Google Scholar 

  • Bozeman, D., & Perrewé, P. (2001). The effect of item content overlap on Organizational Commitment Questionnaire–turnover cognitions relationships. Journal of Applied Psychology, 86, 161–173.

    Article  PubMed  Google Scholar 

  • Brannick, M. T., Chan, D., Conway, J. M., Lance, C. E., & Spector, D. (in press). What is method variance and how can we cope with it? A panel discussion. Organizational Research Methods.

  • Campbell, J. (1982). Editorial: Some remarks from the outgoing editor. Journal of Applied Psychology, 67, 691–700.

    Article  Google Scholar 

  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–106.

    Article  PubMed  Google Scholar 

  • Chan, D. (2001). Method effects of positive affectivity, negative affectivity, and impression management in self-reports of work attitudes. Human Performance, 14, 77–96.

    Article  Google Scholar 

  • Chan, D. (2009). So why ask me? Are self-report data really that bad? In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and metholodogical myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 311–338). New York: Routledge.

    Google Scholar 

  • Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.

    Article  Google Scholar 

  • Conway, J. M. (2002). Method variance and method bias in I/O psychology. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 344–365). Oxford: Blackwell Publishers.

    Google Scholar 

  • Cote, J. A., & Buckley, M. R. (1987). Estimating trait, method and error variance: Generalizing across 70 construct validation studies. Journal of Marketing Research, 26, 315–318.

    Article  Google Scholar 

  • Crampton, S., & Wagner, J. (1994). Percept-percept inflation in microorganizational research: An investigation of prevalence and effect. Journal of Applied Psychology, 79, 67–76.

    Article  Google Scholar 

  • Dooley, W. K., Shaffer, D. R., Lance, C. E., & Williamson, G. M. (2007). Informal care can be better than adequate: Development and evaluation of the exemplary care scale. Rehabilitation Psychology, 52, 359–369.

    Article  Google Scholar 

  • Doty, D. H., & Glick, W. H. (1998). Common method bias: Does common methods variance really bias results? Organizational Research Methods, 1, 374–406.

    Article  Google Scholar 

  • Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 105–146). Washington, DC: American Council on Education and National Council on Measurement in Education.

    Google Scholar 

  • Heisenberg, W. (1927). Über den anschaulichen Inhalt der quantentheoretischen Kinematik und Mechanik. [About the graphic content of quantum theoretic kinematics and mechanics]. Zeitschrift für Physik, 43, 172–198.

    Article  Google Scholar 

  • Hoffman, B. J., Lance, C. E., Bynum, B. H., & Gentry, W. A. (2010). Rater source effects are alive and well after all. Personnel Psychology, 63, 119–151.

    Article  Google Scholar 

  • Hoffman, B. J., & Woehr, D. J. (2009). Disentangling the meaning of multisource performance rating source and dimension factors. Personnel Psychology, 62, 735–765.

    Article  Google Scholar 

  • Hunter, J. (1997). Needed: A ban on the significance test. Psychological Science, 8, 3–7.

    Article  Google Scholar 

  • International Personality Item Pool. (2010). International Personality Item Pool: A scientific collaboratory for the development of advanced measures of personality traits and other individual differences (http://ipip.ori.org/). http://ipip.ori.org/newNEOKey.htm#Conscientiousness. February 5, 2010.

  • James, L. A., & James, L. R. (1989). Integrating work environment perceptions: Explorations into the measurement of meaning. Journal of Applied Psychology, 74, 739–751.

    Article  Google Scholar 

  • Judge, T., Bono, J., & Locke, E. (2000). Personality and job satisfaction: The mediating role of job characteristics. Journal of Applied Psychology, 85, 237–249.

    Article  PubMed  Google Scholar 

  • Lance, C. E., Baranik, L. E., Lau, A. R., & Scharlau, E. A. (2009). If it ain’t trait it must be method: (Mis)application of the multitrait-multimethod design in organizational research. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and metholodogical myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 339–362). New York: Routledge.

    Google Scholar 

  • Lance, C. E., Baxter, D., & Mahan, R. P. (2006). Multi-source performance measurement: A reconceptualization. In W. Bennett, C. E. Lance, & D. J. Woehr (Eds.), Performance measurement: Current perspectives and future challenges (pp. 49–76). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Lance, C. E., Dawson, B., Birklebach, D., & Hoffman, B. J. (in press). Method effects, measurement error, and substantive conclusions. Organizational Research Methods.

  • Lance, C. E., Hoffman, B. J., Baranik, L. E., & Gentry, W. A. (2008). Rater source factors represent important subcomponents of the criterion construct space, not rater bias. Human Resource Management Review, 18, 223–232.

    Article  Google Scholar 

  • Lance, C. E., Teachout, M. S., & Donnelly, T. M. (1992). Specification of the criterion construct space: An application of hierarchical confirmatory factor analysis. Journal of Applied Psychology, 77, 437–452.

    Article  Google Scholar 

  • Lance, C. E., & Vandenberg, R. J. (2009). Introduction. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and metholodogical myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 1–4). New York: Routledge.

    Google Scholar 

  • Lindell, M. K., & Whitney, D. J. (2001). Accounting for common method variance in cross-sectional research designs. Journal of Applied Psychology, 86, 114–121.

    Article  PubMed  Google Scholar 

  • Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley Publishing Company.

    Google Scholar 

  • MacKenzie, S. B., Podsakoff, P. M., & Ahearne, M. (1998). Some possible antecedents and consequences of in-role and extra-role salesperson performance. Journal of Marketing, 62, 87–98.

    Article  Google Scholar 

  • Mallard, A. G. C., & Lance, C. E. (1998). Development and evaluation of a parent–employee interrole conflict scale. Social Indicators Research, 45, 343–370.

    Article  Google Scholar 

  • Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13–103). Washington, DC: American Council on Education and National Council on Measurement in Education.

    Google Scholar 

  • Motowidlo, S., Borman, W., & Schmit, M. (1997). A theory of individual differences in task and contextual performance. Human Performance, 10, 71–83.

    Article  Google Scholar 

  • Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., & Hezlett, S. A. (1998). Trait, rater, and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557–576.

    Article  Google Scholar 

  • Mowday, R. T., Steers, R. M., & Porter, L. W. (1979). The measurement of organizational commitment. Journal of Vocational Behavior, 14, 224–247.

    Article  Google Scholar 

  • Netemeyer, R. G., Boles, J. S., & McMurrian, R. (1996). Development and validation of work-family conflict and family-work conflict scales. Journal of Applied Psychology, 81, 400–410.

    Article  Google Scholar 

  • Ones, D., Viswesvaran, C., & Reiss, A. (1996). Role of social desirability in personality testing for personnel selection: The red herring. Journal of Applied Psychology, 81, 660–679.

    Article  Google Scholar 

  • Organ, D., & Ryan, K. (1995). A meta-analytic review of attitudinal and dispositional predictors of organizational citizenship behavior. Personnel Psychology, 48, 775–802.

    Article  Google Scholar 

  • Podsakoff, P., MacKenzie, S., Lee, J., & Podsakoff, N. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903.

    Article  PubMed  Google Scholar 

  • Podsakoff, P., & Todor, W. (1985). Relationships between leader reward and punishment behavior and group processes and productivity. Journal of Management, 11, 55–73.

    Article  Google Scholar 

  • Putka, D. M., & Sackett, P. R. (2010). Reliability and validity. In J. L. Farr & N. T. Tippins (Eds.), Handbook of employee selection (pp. 9–49). New York: Routledge.

    Google Scholar 

  • Richardson, H., Simmering, M., & Sturman, M. (2009). A tale of three perspectives: Examining post hoc statistical techniques for detection and correction of common method variance. Organizational Research Methods, 12, 762–800.

    Article  Google Scholar 

  • Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956–970.

    Article  PubMed  Google Scholar 

  • Scullen, S. E., Mount, M. K., & Judge, T. A. (2003). Evidence of the construct validity of developmental ratings of managerial performance. Journal of Applied Psychology, 88, 50–66.

    Article  PubMed  Google Scholar 

  • Shalley, C., Gilson, L., & Blum, T. (2009). Interactive effects of growth need strength, work context, and job complexity on self-reported creative performance. Academy of Management Journal, 52, 489–505.

    Google Scholar 

  • Skinner, B. F. (1957). Verbal behavior. New York: Appleton-Century-Crofts.

    Book  Google Scholar 

  • Spector, P. (2006). Method variance in organizational research: Truth or urban legend? Organizational Research Methods, 9, 221–232.

    Article  Google Scholar 

  • Tornow, W. W. (1993). Perceptions or reality: Is multi-perspective measurement a means or an end? Human Resource Management, 32, 221–229.

    Article  Google Scholar 

  • Van Scotter, J. R., & Motowidlo, S. J. (1996). Interpersonal facilitation and job dedication as separate facets of contextual performance. Journal of Applied Psychology, 81, 525–531.

    Article  Google Scholar 

  • Vandenberg, R. J. (2006). Statistical and methodological myths and urban legends: Where, pray tell, did they get this idea? Organizational Research Methods, 9, 194–201.

    Article  Google Scholar 

  • Westaby, J., & Braithwaite, K. (2003). Specific factors underlying reemployment self-efficacy: Comparing control belief and motivational reason methods for the recently unemployed. Journal of Applied Behavioral Science, 39, 415–437.

    Article  Google Scholar 

  • Widaman, K. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1–26.

    Article  Google Scholar 

  • Williams, L. J., & Anderson, S. E. (1994). An alternative approach to method effects by using latent-variable models: Applications in organizational behavior research. Journal of Applied Psychology, 79, 323–331.

    Article  Google Scholar 

  • Williams, L. J., Cote, J. A., & Buckley, M. R. (1989). Lack of method variance in self-reported affect and perceptions at work: Reality or artifact? Journal of Applied Psychology, 74, 462–468.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Central Connecticut State University, 1615 Stanley Street, New Britain, CT, 06050, USA

    James M. Conway

  2. University of Georgia, Athens, GA, USA

    Charles E. Lance

Authors
  1. James M. Conway
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Charles E. Lance
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to James M. Conway.

Appendix: Attenuation of Correlations by Unshared Method Variance

Appendix: Attenuation of Correlations by Unshared Method Variance

Method variance is usually assumed to inflate correlations, but in a fairly common research situation, method variance will actually attenuate correlations as compared to the no-method-variance situation. First consider a situation with no method variance in measures of two traits (T X and T Y ) which are measured by two different, uncorrelated methods (M j and M j′). The equation for the correlation coefficient, expressed in terms of variances and covariance, when no method variance is present, is:

$$ r_{xy} = {\frac{{{\text{COV}}(T_{X},T_{Y} )}}{{\sqrt {{\text{VAR}}(T_{X} ) + {\text{VAR}}(E_{X} )} \sqrt {{\text{VAR}}(T_{Y} ) + {\text{VAR}}(E_{Y} )} }}} $$
(A1)

Now consider what happens when each measure contains method variance that is unshared with the other measure. The variance terms for each measure’s method effect, VAR(M j ) and VAR(M j′), are added to the denominator:

$$ r_{xy} = {\frac{{{\text{COV}}(T_{X} ,T_{Y} )}}{{\sqrt {{\text{VAR}}(T_{X} ) + {\text{VAR}}(M_{j} ) + {\text{VAR}}(E_{X} )} \sqrt {{\text{VAR}}(T_{Y} ) + {\text{VAR}}(M_{{j^{\prime}}} ) + {\text{VAR}}(E_{Y} )} }}} $$
(A2)

Because additional method variance appears in the denominator when there are method effects (Eq. 8), but not when there are no method effects (Eq. 7), Eq. 8 denominator will be larger while the numerators are the same. The correlation value in Eq. 8 will therefore be attenuated.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Conway, J.M., Lance, C.E. What Reviewers Should Expect from Authors Regarding Common Method Bias in Organizational Research. J Bus Psychol 25, 325–334 (2010). https://doi.org/10.1007/s10869-010-9181-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10869-010-9181-6

Keywords

Search

Navigation