A solution to the problem of separation in logistic regression
Corresponding Author
Georg Heinze
Section of Clinical Biometrics, Department of Medical Computer Sciences, University of Vienna, Spitalgasse 23, A-1090 Vienna, Austria
Department of Medical Computer Sciences, University of Vienna, Spitalgasse 23, A-1090 Vienna, AustriaSearch for more papers by this authorMichael Schemper
Section of Clinical Biometrics, Department of Medical Computer Sciences, University of Vienna, Spitalgasse 23, A-1090 Vienna, Austria
Search for more papers by this authorCorresponding Author
Georg Heinze
Section of Clinical Biometrics, Department of Medical Computer Sciences, University of Vienna, Spitalgasse 23, A-1090 Vienna, Austria
Department of Medical Computer Sciences, University of Vienna, Spitalgasse 23, A-1090 Vienna, AustriaSearch for more papers by this authorMichael Schemper
Section of Clinical Biometrics, Department of Medical Computer Sciences, University of Vienna, Spitalgasse 23, A-1090 Vienna, Austria
Search for more papers by this authorAbstract
The phenomenon of separation or monotone likelihood is observed in the fitting process of a logistic model if the likelihood converges while at least one parameter estimate diverges to ± infinity. Separation primarily occurs in small samples with several unbalanced and highly predictive risk factors. A procedure by Firth originally developed to reduce the bias of maximum likelihood estimates is shown to provide an ideal solution to separation. It produces finite parameter estimates by means of penalized maximum likelihood estimation. Corresponding Wald tests and confidence intervals are available but it is shown that penalized likelihood ratio tests and profile penalized likelihood confidence intervals are often preferable. The clear advantage of the procedure over previous options of analysis is impressively demonstrated by the statistical analysis of two cancer studies. Copyright © 2002 John Wiley & Sons, Ltd.
REFERENCES
- 1 Albert A, Anderson JA. On the existence of maximum likelihood estimates in logistic regression models. Biometrika 1984; 71: 1–10.
- 2 Santner TJ, Duffy DE. A note on A. Albert and J.A. Anderson's conditions for the existence of maximum likelihood estimates in logistic regression models. Biometrika 1986; 73: 755–758.
- 3 Lesaffre E, Albert A. Partial separation in logistic discrimination. Journal of the Royal Statistical Society, Series B 1989; 51: 109–116.
- 4 Hirji KF, Tsiatis AA, Mehta CR. Median unbiased estimation for binary data. American Statistician 1989; 43: 7–11.
- 5 Clarkson DB, Jennrich RI. Computing extended maximum likelihood estimates for linear parameter models. Journal of the Royal Statistical Society, Series B 1991; 53: 417–426.
- 6 Lesaffre E, Marx BD. Collinearity in generalized linear regression. Communications in Statistics—Theory and Methods 1993; 22: 1933–1952.
- 7 Kolassa JE. Infinite parameter estimates in logistic regression, with application to approximate conditional inference. Scandinavian Journal of Statistics 1997; 24: 523–530.
- 8 Bryson MC, Johnson ME. The incidence of monotone likelihood in the Cox model. Technometrics 1981; 23: 381–383.
- 9 Jacobsen M. Existence and unicity of MLEs in discrete exponential family distributions. Scandinavian Journal of Statistics 1989; 16: 335–349.
- 10 Hauck WW, Donner A. Wald's test as applied to hypotheses in logit analysis. Journal of the American Statistical Association 1977; 72: 851–853.
- 11 Væth M. On the use of Wald's test in exponential families. International Statistical Review 1985; 53: 199–214.
- 12 SAS Institute Inc. SAS/STAT User's Guide,Version 8. SAS Institute Inc.: Cary, NC, 1999.
- 13 Mehta CR, Patel NR. Exact logistic regression: theory and examples. Statistics in Medicine 1995; 14: 2143–2160.
- 14 Agresti A, Yang M-C. An empirical investigation of some effects of sparseness in contingency tables. Computational Statistics & Data Analysis 1987; 5: 9–21.
- 15 Clogg CC, Rubin DB, Schenker N, Schultz B, Weidman L. Multiple imputation of industry and occupation codes in census public-use samples using Bayesian logistic regression. Journal of the American Statistical Association 1991; 86: 68–78.
- 16 Cytel Software Corporation. LogXact 2.0. Cytel Software Corporation: Cambridge, MA, 1996.
- 17 Firth D. Bias reduction, the Jeffreys prior and GLIM. In Advances in GLIM and Statistical Modelling, L Fahrmeir, B Francis, R Gilchrist, G Tutz (eds). Springer-Verlag: New York, 1992; 91–100.
10.1007/978-1-4612-2952-0_15 Google Scholar
- 18 Firth D. Generalized linear models and Jeffreys priors: an iterative weighted least-squares approach. In Computational Statistics, Vol. 1, Y Dodge, J Whittaker (eds). Physica-Verlag: Heidelberg, 1992; 553–557.
10.1007/978-3-662-26811-7_76 Google Scholar
- 19 Firth D. Bias reduction of maximum likelihood estimates. Biometrika 1993; 80: 27–38.
- 20 Schaefer RL. Bias correction in maximum likelihood logistic regression. Statistics in Medicine 1983; 2: 71–78.
- 21 Cordeiro GM, McCullagh P. Bias correction in generalized linear models. Journal of the Royal Statistical Society, Series B 1991; 53: 629–643.
- 22 Bull SB, Greenwood CMT, Hauck WW. Jackknife bias reduction for polychotomous logistic regression. Statistics in Medicine 1997; 16: 545–560.
10.1002/(SICI)1097-0258(19970315)16:5<545::AID-SIM421>3.0.CO;2-3 CASPubMedWeb of Science®Google Scholar
- 23 Cordeiro GM, Cribari-Neto F. On bias reduction in exponential and non-exponential family regression models. Communications in Statistics—Simulation and Computation 1998; 27: 485–500.
- 24 Leung DH-Y, Wang YG. Bias reduction using stochastic approximation. Australian & New Zealand Journal of Statistics 1998; 40: 43–52.
- 25 Anderson JA, Blair V. Penalized maximum likelihood estimation in logistic regression and discrimination. Biometrika 1982; 69: 123–136.
- 26 le Cessie S, van Houwelingen JC. Ridge estimators in logistic regression. Applied Statistics 1992; 41: 191–201.
- 27 Collett D. Modelling Survival Data in Medical Research. Chapman and Hall: London, 1994.
10.1007/978-1-4899-3115-3 Google Scholar
- 28 Belsley DA, Kuh E, Welsch RW. Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. Wiley: New York, 1980.
10.1002/0471725153 Google Scholar
- 29 Hirji KF, Mehta CR, Patel NR. Computing distributions for exact logistic regression. Journal of the American Statistical Association 1987; 82: 1110–1117.
- 30 SAS Institute Inc. SAS Language Reference, Version 8. SAS Institute Inc.: Cary, NC, 1999.
- 31 Heinze G. Technical Report 10: The application of Firth's procedure to Cox and logistic regression. Department of Medical Computer Sciences, Section of Clinical Biometrics, Vienna University, Vienna, 1999.
- 32 Mehta CR, Patel NR. LogXact-Turbo User Manual. Cytel Software Corporation: Cambridge, MA, 1993.
- 33 Breslow NE, Day NE. Statistical Methods in Cancer Research. Volume 1—The Analysis of Case-control Studies. IARC Scientific Publications: Lyon, 1980.
- 34 Hirji KF, Mehta CR, Patel NR. Exact inference for matched case-control studies. Biometrics 1988; 44: 803–814.
- 35 Cox DR, Hinkley DV. Theoretical Statistics. Chapman and Hall: London, 1974.
10.1007/978-1-4899-2887-0 Google Scholar
Citing Literature
Recommended
Metrics
Details
Copyright © 2002 John Wiley & Sons, Ltd.
Keywords
Publication History
- 26 July 2002
- 26 July 2002
- April 2001
- July 2000
