Treatment, Volume 1, Comment 2, posted September 22,
1997
Copyright 1997 by the American Psychological Association and the
American Psychiatric Association
Comment on Control Groups in Pharmacotherapy and Psychotherapy Evaluations
Richard G. Heimberg
Department of Psychology, Temple University
ABSTRACT
Klein (1997) asserts that pill-placebo conditions are a necessary part of the evaluation of psychotherapy, regardless of whether a pharmacotherapy is evaluated as a part of the study in question. Although Klein raises a number of worthwhile points about treatment evaluation, this conclusion is open to question from multiple perspectives. In this response, Klein's assertion is considered in the light of (a) Klein's insistence that medically responsive patients constitute the appropriate population for study, (b) the idealization of double-blind pill placebo as a control strategy, and (c) issues in the development of psychological placebo control conditions.
Preparation of this article was supported in part by Grant 44119 from the National Institute of Mental Health.
Correspondence concerning this article may be sent to Richard G. Heimberg, Department of Psychology, Temple University, 1701 North 13th Street, Philadelphia, PA 19122-6085. Electronic mail may be sent to rheimber@nimbus.ocis.temple.edu .
Two recent exchanges have appeared in the literature examining aspects of comparative studies of psychotherapy and pharmacotherapy as treatments for the emotional disorders. The first of these exchanges (Elkin, Gibbons, Shea, & Shaw, 1996; Jacobson & Hollon, 1996a, 1996b; Klein, 1996c; McNally, 1996a) centered around the famous and infamous Treatment of Depression Collaborative Research Program (TDCRP). The second exchange featured Klein’s (1996b) response to McNally’s article from the prior series, McNally’s (1996b) reply, and Klein’s (1996a) "final" critique.
These papers have covered a wide range of topics, and there appears to be little common ground between the players. The Klein–McNally exchange was a heated debate on the question of whether or not double-blind pill placebo control conditions are appropriate, or even necessary, for the proper evaluation of studies involving the comparison of psychotherapy and medication treatments. To no one’s great surprise, Klein, the physician strongly grounded in that tradition, said they are essential, and McNally, the psychologist grounded in the empiricism of the experimentalist, said they are not necessary although they may be used to achieve specific purposes. At times, this debate has bordered on the cynical ("The purpose of my reply is to provide a critique of Klein’s critique…of my critique…of his critique…", McNally, 1996b, p. 855; "meta-analysis/schmeta-analysis", Klein, 1996a, p.860) and the acrimonious ("This is not a fantasy…", Klein, 1996b, p. 849; "This is not a fantasy…", McNally, 1996b, p. 856).
I will not recount the specifics of the previous arguments here, as they have been summarized by Klein (1997) and other contributors to the current exchange. However, it is important to note an important shift in emphasis in Klein’s position relative to his previous writings on the topic. In the previous exchanges, Klein maintained that a pill-placebo condition was necessary to the evaluation of studies comparing psychotherapies and pharmacotherapies. In his current contribution, Klein maintains that a pill-placebo condition is a necessary part of the evaluation of psychotherapy, regardless of whether or not a pharmacotherapy is a part of the study in question. Although he also argues that pharmacotherapy conditions should be included in studies of psychotherapy evaluation, he opens his article with "Here we focus on a narrower issue: Does a pill-placebo case management group provide a useful pragmatic comparison in the experimental evaluation of psychotherapy?" (Klein, 1997, ¶ 7). This is a major shift in the nature of Klein’s argument and requires closer evaluation.
Klein (1997) presents some history of the development of the TDCRP design and how a pill-placebo condition came to be included, an apparently controversial step. The TDCRP was originally intended to compare the efficacy of interpersonal psychotherapy and cognitive therapy for depression, but Klein suggests that a design directly involving only these two treatments would not control for placebo effects or spontaneous remission. It certainly would not; nor would it control for the passage of time, maturation effects, effects of repeated assessments, or any number of threats to the internal validity of a study. It would simply be an inadequate research design and one that should never have been seriously considered (see Kazdin, 1993, for a more thorough review of the threats to validity in studies of psychological treatment, and Heimberg & Liebowitz, 1996, for an explication of these problems in the context of the treatment of social phobia.). The designers of the TDCRP chose to include imipramine (plus clinical management) as a standard reference treatment, and pill placebo (plus clinical management) was then included to determine whether or not imipramine was effective in the study sample. It is not my purpose to evaluate the design of the TDCRP, but it must be said that the choice to include imipramine as a standard reference treatment and pill placebo as a means to gauge its performance in the study sample was only one of a number of choices available to that research group. The choice of control conditions is an outgrowth of the determination of the variables that require control, and this is an outgrowth of the experimental hypothesis posed by a research study. Although Klein’s comments about the inadequacies of waiting list control groups or psychological placebos suggest he might disagree, either one of those choices would have been defensible, depending on the hypotheses that the TDCRP group put forth.
A large part of Klein’s argument for the inclusion of pill placebo conditions in psychotherapy studies is his strong assertion (1996a, 1996b, 1996c, 1997) that the population to which one would wish to generalize the results of a psychotherapy study is the group of patients who are medication responsive. A number of points need to be made here.
First, the concept of "medication responsiveness" should be defined independently of the specific research design in question. Any patient who improves with medication is medication responsive. This does not automatically address the reasons for the person’s response to the medication. Is it because of the active pharmacological ingredient of the medication or one of the many active psychological ingredients of the ritual of pill-taking or of the doctor–patient relationship? However, depending on the investigator’s goal (e.g., examining efficacy versus mechanism of change), double-blind pill placebo may or may not be necessary to determine medication responsiveness. Klein (1997) would apparently reject this notion, as he states that "the claim that placebos are only necessary to delineate particular specific mechanisms is untrue" (¶ 32).
Second, the concept of medication responsiveness is entirely confounded with the specific medication and disorder under consideration. If one medication is less effective than another, then the population of medication-responsive patients is smaller for the first medication than the second. Is the population to which psychotherapy findings may be generalized to change depending on the drug to which the psychotherapy is compared or which drug is considered the best pharmacological treatment for a disorder at a particular time in history? It may be legitimate to compare the efficacy of a psychotherapy to the most effective pharmacotherapy in a particular epoch, but this is not the same thing as restricting the population to which psychotherapy results may be generalized.
Third, if a psychotherapy is effective with both medication-responsive and medication-nonresponsive patients, this should be considered a good thing because it means that we can treat more patients effectively. The percentage of patients with a disorder who are responsive to a particular treatment is an index of the effectiveness of the treatment. If more people are psychotherapy responsive than medication responsive, the psychotherapy is more effective. Although it is unlikely to occur in the extreme, it is also a good thing if a psychotherapy and a drug are equally effective but exert their effectiveness with different patients. In this case, the efficacy of a psychotherapy with medication-responsive patients is nil, but the range of patients successfully treated is maximized. Is this not why we decided to get into this game in the first place?
Klein’s (1997) arguments appear to rest on several implicit assumptions about double-blind pill placebo conditions. These assumptions might be summarized as (a) all pill-placebo conditions are created equal, (b) all pill-placebo conditions are implemented perfectly, and (c) double-blind pill placebo adequately controls all threats to the validity of an experiment. None of these assumptions appears tenable. These assumptions are called into question by my own experiences in conducting double-blind placebo-controlled trials of the monoamine oxidase inhibitor (MAOI) phenelzine and the tricyclic antidepressant nortriptyline as well as the writings of other investigators (Fisher & Greenberg, 1993; Margraf et al., 1991).
Pill-placebo conditions and their effects are inextricably linked to the medication conditions they control for. It is certainly unlikely that all are the same because different medications are associated with such different therapeutic and side-effect profiles and because they place different demands on patients’ resources. A placebo condition in a study of the efficacy of an MAOI will include instructions on the side effects of MAOIs and the dietary restrictions that must be followed for the patient to avoid a hypertensive crisis or its sequelae. Patients who agree to take MAOIs often do so only after deep reflection on the consequences of this action and may do so with considerable trepidation. It is not so unreasonable to conjecture that anyone who is willing to enter a study of MAOI pharmacotherapy does so seriously and with strong motivation to adhere to both medication and dietary regimens. Clearly, patients who participate in controlled trials of benzodiazepines, tricyclic antidepressants, or other psychotropic medications do so with concern about side effects, and so forth., but few must invest so much in the expected treatment as the patient entering an MAOI trial.
It is also common for patients who receive MAOIs to make small but frequent violations of the MAOI dietary restrictions (Sweet et al., 1995), and this fact makes the interpretation of placebo controls problematic in these studies. When patients consume even a small amount of a food, beverage, or medication from the restricted list but do not suffer the symptoms supposedly associated with such transgressions, they may begin to wonder whether they are receiving the active medication or the placebo. With each further dietary violation, the likelihood that they are receiving placebo is increased. Before long, the patient is no longer "blind." I have often speculated (but never tested the hypothesis) that patients who receive placebo but who are nevertheless classified as treatment responders are the patients who are the most likely to comply with the MAOI dietary restrictions and the least likely to test the limits.
In responsible pharmacotherapy trials, therapeutic effects and side effects are carefully monitored. Medication dosages are often shifted upward slowly to ease side effects or are brought down because of difficulties in handling side effects. It is no surprise to anyone who has ever been involved in pharmacotherapy studies that these side effects often differ systematically between patients receiving active medication versus placebo (see Fisher & Greenberg, 1993). Although there certainly are patients in the placebo conditions who report side effects consistent with expectation for active medication, they are not the norm. Physicians who administer active medication and placebo may be "blind" in terms of official knowledge of the patient’s assignment, but I have often wondered if their vision is 20/20 when it comes to "knowing" which medication a specific patient has received.
Several studies have been conducted which suggest that double-blind procedures may be less effective than we would like, that is, that some participants in double-blind studies may be able to determine whether active drug or placebo was administered (Hughes & Krahn, 1985; Margraf et al., 1991; Marini, Sheard, Bridges, & Wagner, 1976; Rabkin et al., 1986; Rickels, Hesbacher, Weise, Gray, & Feldman, 1970; Stallone, Mendlewicz, & Fieve, 1975). For example, Margraf et al. reported on an 8-week double-blind trial of alprazolam, imipramine, and placebo in panic disorder. After 4 weeks of treatment, patients and physicians were asked to state whether they thought the patient was receiving active drug or placebo, and if active drug, which one. Both patients and physicians were able to correctly determine whether the patient received active drug or placebo, and physicians were also able to accurately determine whether patients receiving an active drug were given alprazolam or imipramine. Physicians and patients reported greater confidence in their judgments if they were correct. Furthermore, both patients and physicians gave lower ratings of Week 8 treatment success and side effects when they correctly determined that placebo had been administered than when they correctly determined that an active drug was given.
Carroll, Rounsaville, and Nich (1994) examined the effectiveness of blinding procedures in a comparison of psychotherapy and pharmacotherapy for ambulatory cocaine abusers. In this study, however, the emphasis was on the adequacy of efforts to keep the independent evaluator (rather than the physician or patient) unaware of the treatment condition. Independent evaluators were able to correctly guess whether a patient had been assigned to active versus control psychotherapy and active versus placebo medication at rates much greater than chance, although there appeared to be little difference in the efficacy of the psychotherapy versus pharmacotherapy blinding procedures. As in Margraf et al.'s (1991) study, correct guesses regarding treatment assignment significantly predicted evaluators’ judgments about patients’ progress, especially if the rating index was based on clinical judgment rather than behavioral observation.
Findings such as these should motivate us to find other means of experimental control in pharmacotherapy (and psychotherapy) trials rather than simply accepting the efficacy of blinding procedures. Fisher and Greenberg (1993) provided a thorough review of the problems that may arise in the conduct of a double-blind study of pharmacotherapy treatment.
One of Klein’s (1997) reasons for favoring the use of pill placebo in the evaluation of psychotherapy is the difficulty that may be experienced in the mounting of a psychological placebo condition. Of course, this is a daunting task because we know a great deal less about what causes change in human behavior than we would like to admit. However, this is not a reason to avoid the challenge. The general approach to the creation of an attention-placebo control is to examine the experimental treatment and to generate a procedure that is like it in as many ways as possible with the notable exception of the ingredient(s) that are hypothesized to account for its effect. This strategy has been used in a number of studies with varying degrees of success. However, it is possible to determine the extent to which an attention-placebo condition is successful in its "mission", and it is incumbent on the researcher to do so. Klein quite rightly states that the "demand characteristics associated with treatment may well differ from the demand characteristics associated with the placebo treatment, especially if the treatment is of poor credibility" (¶ 34). Treatment credibility, the extent to which the treatment and control conditions are equally believable to the patient and whether or not these procedures inspire relatively equal expectations for a positive treatment outcome among patients (Borkovec & Nau, 1972), has received much attention in this regard. Failure to demonstrate equivalent credibility has caused interpretive troubles for more than one study. Consider the fate of an otherwise elegantly designed study by Butler, Cullington, Munby, Amies, and Gelder (1984) on the treatment of social phobia with exposure combined with anxiety management training. This package was compared with exposure combined with filler material designed to serve as an attention-placebo control for anxiety management training. After treatment, the two exposure conditions were equally effective on most measures, surpassing a waiting-list control on most measures. In addition, exposure plus anxiety management training was more effective than exposure plus the control material on two self-report scales. At 6-month follow-up, the superiority of exposure plus anxiety management training was increased, and patients in this condition were less likely to seek additional treatment in the following year. Butler et al. quite appropriately measured patients’ outcome expectations before treatment and after 4 weeks of treatment. Unfortunately, at the fourth session, exposure plus anxiety management training was associated with greater outcome expectations than exposure plus the control material. Although this pattern is open to several interpretations, it is difficult to rule out the possibility that patients with the greater belief in the effectiveness of their treatment invested a greater effort in their treatment and that this effort rather than the hypothesized mechanism of action behind exposure plus anxiety management training accounted for this outcome (Heimberg & Liebowitz, 1996).
The results of the credibility analysis by Butler et al. (1984) suggest that they were less than successful in their attempts to develop an attention-placebo control for their study. However, in another study of the treatment of social phobia, a more successful attempt was made. Heimberg et al. (1990) examined the efficacy of cognitive–behavioral group therapy (CBGT) for social phobia relative to an attention-placebo treatment thaat they dubbed educational supportive group psychotherapy (ES for short). This treatment was similar to CBGT in terms of number and duration of sessions, therapist contact, and so forth. The first hour of each 2-hr group session was devoted to lectures, demonstrations, and discussions on topics of relevance to persons with social phobia. During the second hour, patients participated in a therapist-facilitated discussion about upcoming anxiety-evoking events, how they might attempt to cope with them, and what other group members had done in similar situations in the past. This component of ES was modeled on support groups that were common in the local community at the time of the study. Before the initiation of the study, attempts were made to determine whether the credibility of ES would be equal to that generated by CBGT and other treatments for social phobia that were common in the literature at that time (social skills training, systematic desensization). Kennedy and Heimberg (1986) administered written versions of the rationales for ES and the active treatments to a panel of judges who rated them equally and equally superior to the rationale of another control condition hypothesized to be less credible.
ES and CBGT were both administered according to detailed manuals by Heimberg et al. (1990). After the first and fourth sessions, ES was rated as highly credible by patients assigned to that condition, and their ratings were equivalent to those given by patients assigned to CBGT. After 12 weeks of treatment, CBGT patients were more likely to be classified as responders to treatment by clinical assessors than patients receiving ES and also reported less anxiety in an individualized behavioral test. Similar data were reported in an independent study by Lucas and Telch (1993). Furthermore, differences between patients assigned to CBGT and ES remained apparent after follow-ups of 6 months (Heimberg et al., 1990) and 5 years (Heimberg et al., 1993).
In our recently completed collaborative study of the efficacy of CBGT and phenelzine as treatments for social phobia (Heimberg et al., 1997), we included both attention-placebo (ES) and pill placebo conditions in the design. This was not done because we believed that both controls are essential for the evaluation of psychotherapy (or pharmacotherapy) but because previous studies had examined either CBGT and ES or phenelzine and pill placebo. Neither active treatment had been compared to the other control condition, nor had there been any direct comparison of the control conditions. To best evaluate the efficacy of CBGT and phenelzine, and to shed the greatest light on the past research on these two treatments, the inclusion of both control conditions was deemed necessary. Furthermore, an important mission of our collaboration has been to conduct studies that would be meaningful to practitioners and researchers from both the medical and psychological communities, and inclusion of controls familiar to both was also desirable.
The relative performance of the two control conditions is important to the present discussion. The ES and pill placebo conditions were virtually equivalent to each other. Pill placebo was associated with better outcomes on a few independent assessor ratings while ES was associated with better outcomes assessed by self-report, but there were few consistent differences. Interestingly, CBGT, phenelzine, and the two control conditions did not differ in patient-rated credibility in this study. The successful implementation of a psychological placebo control in our studies certainly suggests that it is possible to do so. The equivalent outcomes of ES and pill placebo would appear to undermine Klein’s arguments about the absolute need for a pill-placebo condition in the evaluation of psychotherapy studies, although it in no way undermines some of his other points. He is absolutely correct when he states that good psychological placebos are hard to find and that control groups of lesser credibility than the experimental treatment are associated with different demand characteristics.
Klein (1997) cites Basham’s (1986, p. 90) assertion that "It is crucial in terms of the experiment’s internal validity that the treatment factors contained in the placebo group are strictly a subset of the factors in the total treatment. If such a component control condition is not used, the placebo group is no longer a formal control group, and valid statements about the causal role of specific treatment factors can no longer be made." According to Klein, this is entirely correct in explanatory trials, as defined by Schwartz and Lellouch (1967). Therefore, both Klein (1997; Klein & Rabkin, 1984) and Basham (1986) suggest that it may be better to test active treatments against each other. Klein (1997, ¶ 38) states, "Given the difficulty in constructing a psychological placebo, one should first find a difference between two credible, putatively active therapies and then pursue dismantling studies in an attempt to arrive at causal mechanisms."
The difficulties in these arguments are numerous. First, it may be difficult to test two "credible, putatively active treatments" for the many disorders for which two of these do not yet exist. Second, we can only make educated guesses about the active ingredients of most psychotherapies (and most medications) at this point, so the quote from Basham (1986) represents a theoretical goal rather than a rule that may be concretely applied. In fact, it would seem to define alternative credible treatments as poor controls for each other! One might also wonder how a pill-placebo condition might measure up to this criterion when serving as a control for a psychotherapy. There may be a number of different factors operating in the medication consultation that are not necessarily the same in the psychotherapy office. For instance, because most psychotherapies in the clinic and the research setting are administered by nonmedical therapists, there is a vast difference in the potential status and influence of the treatment provider in the eyes of the patient. Second, the activities required of the patient in and between sessions are very different. Third, the patient’s attributions for change in response to medication versus psychotherapy (and responsibility for that change and the meaning of it all after treatment discontinuation) are likely to be very different. And the list goes on.
Klein’s (1997) arguments about the necessity for inclusion of a pill placebo condition in the evaluation of psychotherapies does not appear warranted. While it may certainly be useful to include pill placebo in studies comparing psychotherapies and pharmacotherapies, the inclusion of other types of control procedures may also be warranted. Similarly, there is no automatic requirement, as stipulated by Klein, that there be a pharmacotherapy wing to every psychotherapy study. This requirement is unduly restrictive on research design and creativity and leaves little latitude for the differences in research traditions. Control conditions for any research study should be selected with a clear vision of the up and down sides of all alternative procedures and the goals that are to be accomplished in a specific research study.
Basham, R. B. (1986). Scientific and practical advantages of comparative designs in psychotherapy outcome research. Journal of Consulting and Clinical Psychology, 54, 88–94.
Borkovec, T. D., & Nau, S. D. (1972). Credibility of analogue therapy rationales. Journal of Behavior Therapy and Experimental Psychiatry, 3, 257–260.
Butler, G., Cullington, A., Munby, M., Amies, P., & Gelder, M. (1984). Exposure and anxiety management in the treatment of social phobia. Journal of Consulting and Clinical Psychology, 52, 642–650.
Carroll, K. M., Rounsaville, B. J., & Nich, C. (1994). Blind man’s bluff: Effectiveness and significance of psychotherapy and pharmacotherapy blinding procedures in a clinical trial. Journal of Consulting and Clinical Psychology, 62, 276–280.
Elkin, I., Gibbons, R. D., Shea, M. T., & Shaw, B. F. (1996). Science is not a trial (but it can sometimes be a tribulation). Journal of Consulting and Clinical Psychology, 64, 92–103.
Fisher, S., & Greenberg, R. P. (1993). How sound is the double-blind design for evaluating psychotropic drugs? Journal of Nervous and Mental Disease, 181, 345–350.
Heimberg, R. G., Dodge, C. S., Hope, D. A., Kennedy, C. R., Zollo, L., & Becker, R. E. (1990). Cognitive–behavioral group treatment of social phobia: Comparison to a credible placebo control. Cognitive Therapy and Research, 14, 1–23.
Heimberg, R. G., & Liebowitz, M. R. (1996). Issues in the design of trials for the evaluation of psychosocial treatments for social phobia. International Clinical Psychopharmacology, 11(Supplement 3), 55–64.
Heimberg, R. G., Liebowitz. M. R., Hope, D. A., Schneier, F. R., Holt, C. S., Welkowitz, L., Juster, H. R., Campeas, R., Bruch, M. A., Cloitre, M., Fallon, B., & Klein, D. F. (1997). Cognitive–behavioral group therapy versus phenelzine in social phobia: I. 12-week outcome. Manuscript submitted for publication.
Heimberg, R. G., Salzman, D. G., Holt, C. S., & Blendell, K. A. (1993). Cognitive–behavioral group treatment for social phobia: Effectiveness at five-year followup. Cognitive Therapy and Research, 17, 325–339.
Hughes, J., & Krahn, D. (1985). Blindness and the validity of the double-blind procedure. Journal of Clinical Psychopharmacology, 5, 138–143.
Jacobson, N. S., & Hollon, S. D. (1996a). Cognitive–behavior therapy versus pharmacotherapy: Now that the jury’s returned its verdict, it’s time to present the rest of the evidence. Journal of Consulting and Clinical Psychology, 64, 74–80.
Jacobson, N. S., & Hollon, S. D. (1996b). Prospects for future comparisons between drugs and psychotherapy: Lessons from the CBT-versus-pharmacotherapy exchange. Journal of Consulting and Clinical Psychology, 64, 104–108.
Kazdin, A. E. (1993). Research design in clinical psychology (2nd ed.). New York: Harper & Row.
Kennedy, C. R., & Heimberg, R. G. (1986, November). Treatment credibility and client outcome expectancy: An evaluation of five treatment rationales. Poster presented at the annual meeting of the Association for the Advancement of Behavior Therapy, Chicago, IL.
Klein, D. F. (1996a). Critiquing McNally’s reply. Behaviour Research and Therapy, 34, 859–863.
Klein, D. F. (1996b). Discussion of "Methodological controversies in the treatment of panic disorder." Behaviour Research and Therapy, 34, 849–853.
Klein, D. F. (1996c). Preventing hung juries about psychotherapy studies. Journal of Consulting and Clinical Psychology, 64, 81–87.
Klein, D. F. (1997, September 22). Control groups in pharmacotherapy and psychotherapy evaluations. Treatment, 1(Article 1). From World Wide Web: http://journals.apa.org/treatment/vol1/97_a1.html
Klein, D. F., & Rabkin, J. G. (1984). Specificity and strategy in psychotherapy research and practice. In J. B. W. Williams & R. L. Spitzer (Eds.), Psychotherapy research: Where are we and where should we go? (pp. 306–337). New York: Guilford Press.
Lucas, R. A., & Telch, M. J. (1993, November). Group versus individual treatment of social phobia. Paper presented at the annual meeting of the Association for Advancement of Behavior Therapy, Atlanta, GA.
Margraf, J., Ehlers, A., Roth, W. T., Clark, D. B., Sheikh, J., Agras, W. S., & Taylor, C. B. (1991). How "blind" are double-blind studies? Journal of Consulting and Clinical Psychology, 59, 184–187.
Marini, J. L., Sheard, M., Bridges, C., & Wagner, E. (1976). An evaluation of the double-blind design in a study comparing lithium carbonate with placebo. Acta Psychiatrica Scandinavica, 53, 343–349.
McNally, R. J. (1996a). Methodological controversies in the treatment of panic disorder. Journal of Consulting and Clinical Psychology, 64, 88–91.
McNally, R. J. (1996b). More controversies about panic disorder: A reply to Klein. Behaviour Research and Therapy, 34, 855–858.
Rabkin, J. G., Markowitz, J. S., Stewart, J., McGrath, P., Harrison, W., Quitkin, F. M., & Klein, D. F. (1986). How blind is blind? Assessment of patient and doctor medication guesses in a placebo-controlled trial of imipramine and phenelzine. Psychiatry Research, 19, 75–86.
Rickels, K., Hesbacher, P. T., Weise, C. C., Gray, B., & Feldman, H. S. (1970). Pills and improvement: A study of placebo response in psychoneurotic outpatients. Psychopharmacologia, 16, 318–328.
Schwartz, D., & Lellouch, J. (1967). Explanatory and pragmatic attitudes in therapeutic trials. Journal of Chronic Diseases, 20, 637–648.
Stallone, F., Mendlewicz, J., & Fieve, R. (1975). Double-blind procedure: An assessment in a study of lithium prophylaxis. Psychological Medicine, 5, 78–84.
Sweet, R. A., Brown, E. J., Heimberg, R. G., Ciafre, L., Scanga, D., Cornelius, J., Dube, S., Forsyth, K. M., & Holt, C. S. (1995). Monoamine oxidase inhibitor dietary restrictions: What are we asking patients to give up? Journal of Clinical Psychiatry, 56, 196–201.