Comparing Groups: Randomization and Bootstrap Methods Using R [Rilegato]

My view of this book is somewhat in between the tremendous enthusiasm shown by George Cobb and the luke warm evaluation by the other amazon reviewer Dmitri Shvorob. What I really like about the book is the detailed introduction to R in Chapters 1 and 2 and the gentle exposition to the subject of resampling as the book is aimed at graduate students in the social and behavioral sciences.

Frankly, I think the coverage of R is better than what LaBudde and I provide in our new book "An Introduction to Bootstrap Methods with Applications to R." The coverage of resampling is focused on group comparisons so the messy issues regarding the general use of bootstrap and permutation methods is avoided. Best of all, it is high time that graduate students outside of statistics get an introduction to and appreciation for resampling methods.

The authors do not shy away from the methodological and philosophical issues of statistical inference. They provide a whole chapter on this topic (Chapter 8). They give a brief introduction to Fisher's approach to hypothesis testing and then go on to explain in a very clear manner the Neyman-Pearson theory of hypothesis testing. The authors carefully avoid discussion of fiducial inference and the heated controversy between Fisher and Neyman on this topic. It is perhaps wise to avoid discussion of theses complicated issues for the intended readers. But, it does not give a clear portrayal of the conflict. In fact fiducial theory is perhaps the only area of Fisher's theory that many statisticians believe has been discredited.

Chapters 3-5 give a nice introduction to graphical and inferential methods in statistics before introducing resampling techniques. Chapter 6 is the first chapter to introduce resampling as permutation tests are introduced. Chapter 7 introduces bootstrap tests. In both chapters discussion is focused on group comparisons where the distribution of the difference between two group means is what is approximated through these techniques. The coverage is clear and sound but perhaps as Dimitri Shvorob expresses in his review, a little repetitive.

The meat of the remaining chapters is on bootstrap confidence intervals and adjustment of p-values in repeated testing. I have a minor quibble with the authors when they discuss whether to prefer permutation methods or bootstrap. They consider random assignment and random sampling as the source of distinction (i.e. use permutation tests for random assignment and bootstrap tests for random sampling). I find this a little too simplistic as there are situations where treatments can be randomly assigned to patients and the patients can be viewed as a random sample from a population of potential patients. In such cases the two methods do compete. for examples such as p-value adjustment, the choice should be made based on performance (e.g. power of the test) to the extent that it assessed.

Another point that I wish to quibble about is the characterization of the nonparametric bootstrap as "another Monte Carlo simulation method for approximating the variation in the distribution of a test statistic." This statement is false. The bootstrap approximates the sampling distribution for the test statistic (under the null hypothesis) by mimicking it through sampling with replacement from the original data set. This has nothing to do with Monte Carlo. This was pointed out by Efron and emphasized in my book, Chernick (2007). The bootstrap distribution in some instances can be derived without resorting to Monte Carlo. Monte Carlo methods enter the picture only as a practical way to approximate the bootstrap distribution. This distinction often is missed because it is so common to need to apply the Monte Carlo approximation when using the bootstrap for inference. However, conceptually it is important to get things right and the authors failed on this point.

In the Foreward Cobb points out 8 distinctive features of the book. In this review I would like to mention these features and describe whether I agree or disagree with Cobb.

"1. The exposition is visual/intuitive/computational rather than haunted by the old headless horseman of abstract derivations."

I agree that the presentation is for the most part very lucid with nice graphical illustrations. But it can be overly simplistic as in the case of avoiding mention of fiducial inference and when equating bootstrapping with Monte Carlo.

"2. The writing is strikingly good-the exposition reflects the authors' careful attention to word choice."

I generally agree with this assessment but not universally. As Dimitri mentions there are times when the explanations can be awkward and overly wordy.

"3. The references are a gold mine."

While I agree that there are a number of useful references in the book, I would not call it a gold mine. If you want to see a thorough list of references on resampling take a look at Chernick (2007) or Chernick and LaBudde (2011). In fact some of the major texts on resampling, Hall (1992), Chernick (1999), Chernick (2007), Lunneborg (2000), Manly (1997, 2006) and Shao and Tu (1995) are not even listed as references.

"4. The use of modern graphics is unusual for a book at this level. I refer, in particular, to panel plots and kernel density estimates."

This is very true.

"5. Prerequisites are minimal. No calculus is needed. Although it helps to know about vectors and matrices, one does not need a linear algebra course."

This is true and probably warranted. But avoiding mathematics is not always a virtue. Graduate students in the social and behavioral sciences should have taken a first course in calculus (although it may not have been required for their degree).

"6. The emphasis is practical throughout."

This is the case and is as it should be.

"7. Content reflects the current research literature. For example, the exposition recognizes the importance of effect sizes, and the treatment of multiple testing addresses the recent research on false discovery rates."

It is certainly a distinctive feature for a book like this to address false discovery rates which is a new approach to multiple testing based on the fact that when a large number of hypotheses are tested simultaneously familywise error rate becomes an untenable criterion. But given the omission of some major texts on the bootstrap the book could be omitting some of imprtant aspects of the recent literature as well.

"8. Overall, the emphasis is on statistics with a purpose, statistics for deep interpretive understanding."

While I agree with this statement, I find it hard to see this as a distinctive feature. There are numerous books in applied statistics that successfully emphasize the importance of statistics in understanding and interpreting data. All this said I do consider this to be a good book for a first course in resampling for social and behavioral science majors and graduate students and I hope that with its entry will come a number of such university courses.

My references:
1. Chernick, M. R. (1999). "Bootstrap Methods: A Practitioner's Guide". Wiley, New York.
2. Chernick, M. R. (2007). "Bootstrap Methods: A Guide for Practitioners and Researchers". 2nd Edition, Wiley, Hoboken.
3. Chernick M. R. and LaBudde, R.A. (2011). "An Introduction to Bootstrap Methods with Applications to R." Wiley, Hoboken.
4. Hall, P. (1992). "The Bootstrap and Edgeworth Expansion." Springer-Verlag, New York.
5. Lunneborg, C. E. (2000). "Data Analysis by Resampling: Concepts and Applications." Brooks/Cole, Pacific Grove.
6. Manly, B. F. J. (1997). "Randomization, Bootstrap and Monte Carlo Methods in Biology." 2nd Edition, Chapman & Hall, London.
7. Manly, B. F. J. (2006). "Randomization, Bootstrap and Monte Carlo Methods in Biology." 3rd Edition, Chapman & Hall/CRC, Boca Raton.
8. Shao, J. and Tu, D. (1995). The Jackknife and Bootstrap. Springer-Verlag, New York.

I feel bad about docking a star from a thoughtful and helpful book, but my appreciation of "Comparing groups" does not reach the "I love it" level. Standing in the way are occasional difficult-to-understand or could-be-presented-better passages, and a feeling that, were the writing "tightened up", one would be left with a fairly thin introductory book, priced at $90. (Note that the titular randomization and bootstrap methods only begin in Chapter 6: the first 100+ pages are taken up by introduction to R and basic EDA). One cannot claim competence in the subject based on "Comparing groups" alone, and needs to go further, assisted by the book's references. Phillip Good's books are a sensible first port of call; selected pages of "Data analysis using regression and multilevel/hierarchical models" by Andrew Gelman and Jennifer Hill are likely to provide an important complementary perspective. Unlike those, "Comparing groups" is better checked out from the library, rather than bought.