If I could pick the one experience in my medical career that contributed the most to my development as a physician, it would be the three terms I served on the Program Committee and New Research subcommittees of the American Academy of Child and Adolescent Psychiatry.

The Academy hosts the largest (and most prestigious) scientific meeting in the world every October in the field of children’s mental health, attracting 5,000+ professionals from fifty or more countries. The Academy members who serve on the Committee are responsible for peer reviewing all of the research proposals submitted for the scientific program at the meeting, based upon the quality and importance of the work.

In a typical year, we’d literally receive hundreds upon hundreds of submissions, and the committee would break up into teams to review each study/project submitted. Five members would review each study individually, with one member assigned as the “principal reviewer.” The “fun” began when the full committee assembled for a two-day meeting each Spring, where the principal reviewer would present their analysis of the studies they reviewed in front of everyone, and anyone on the committee could comment on each study reviewed.

I wasn’t a “typical” committee member in that my role was to represent the perspective of the mental health professionals practicing in the community. Most committee members were in the upper echelon of the elite in our field. Several worked full-time for the National Institute of Mental Health. A couple were future presidents of the Academy. Nearly all were tenured professors at elite institutions who were instrumental in the pivotal studies that drive the treatment in our field. It was fascinating to observe them critically analyze data and question the methods used in conducting a study. The first year was like drinking out of a firehose! I recognized how little I learned in med school, residency and fellowship in terms of evaluating research. I took great pride by the end of my tenure on the committee in the ability to hold my own with some very prestigious colleagues in the room. Our Committee leaders spoke often about setting a standard for excellence in research in our field that might be a model for the rest of the mental health profession.

The thing that is gnawing at me about the Focus/Lifeway mental illness study is this…As Christians, part of our witness is to reflect excellence wherever we’ve been placed by God. I see that as a part of our worship, because excellence is a true reflection of the Savior we as Christians claim to represent. Because of my past experiences, I think I can recognize excellence in research when I see it, and my gut is telling me this study doesn’t reflect excellence.

In examining and interpreting the study for our readers, I thought it would be most helpful going forward to put on my “old hat” as a reviewer and offer insight into how the folks who sit where I sat might think about the Focus/Lifeway study…

First question…What were the hypotheses/objectives about mental illness, the church and faith that the study designers sought to examine?

In this case, the project objectives were 1) to equip family members and churches to care for loved ones suffering from acute mental illness, by expanding the understanding of what these persons experience, applying what Scripture says about the essentials of faith, and sharing positive contributions that can be made in these individuals’ spiritual lives; 2) to help family members and churches discern the spiritual state of loved ones suffering from mental illness.

The first take on the objectives is they’re extremely broad. Understanding of what aspect of their experience with mental illness? How will they discern the spiritual state of persons with mental illness? What do we know about this topic? How will this study build upon what we know?

Next question…Is the study population appropriate for the study objectives?

I would have MAJOR concerns here, given the information available to us by the study authors. The study authors chose adults with what was labeled as moderate to severe depression, bipolar disorder and schizophrenia. Why those three conditions? Would the experience of an adult with schizophrenia be in any way comparable to the experience of a person in the midst of a major depressive episode? Some people with bipolar disorder who have been stabilized on medication may be relatively symptom-free for extended periods of time…how might their experience be different?

Quoting from the report:

The focus of the theological research was on those in a state of psychosis, people with a severe mental disorder in which thought and emotions are so impaired that contact is lost with external reality.

How does this work with a sample in which 79% of subjects describe themselves as “very stable” or “somewhat stable?”

One of the imperatives that led to the creation of our current diagnostic classification is the need for common definitions and common terminology in order to conduct meaningful research. Looking at the study objectives, one would need a sample that reflects the breadth of conditions characterized as mental illness that result in significant functional impairment. Collectively, anxiety disorders are our most common mental illnesses in adults, followed by depression. Next comes ADHD, followed by PTSD. Where do they fall in the sample? The authors lumped together three conditions to represent “mental illness”…two of which are closely related, while one isn’t.

BIG PROBLEM: The authors are exceptionally vague as to how the individuals surveyed for the study were selected. The lack of transparency as to precisely how the sample was obtained would be utterly unacceptable if this work were submitted as a paper. How do we know that the subjects selected for this study actually had the conditions described in the study? How were they prescreened? Who diagnosed them with depression? Who was the “large, national panel” doing the prescreening? 200 of 355 sample participants attended Christian churches once a month or more as adults? Isn’t that percentage significantly higher than the rates of church attendance reported in the general population?

I can’t tell from the information disclosed by the authors that their sample is valid. How do we know their data is valid? Would someone seeking to test or replicate FOTF/Lifeway’s findings have enough information to do it?

Next question…Who are we comparing them to? If we’re going to look at measures of spiritual practices…Do they (or their family members) attend church? How frequently do they pray? Attend small groups? Serve? How do they stack up compared against others selected from the same population who don’t have mental illness? There is no comparison population here. That’s not necessarily a fatal flaw…but data that is significantly divergent from norms for the general population is a big red flag without a comparison (control) group selected from the same pool. Lifeway may have data on spiritual practices from the same sample pool. If so, it’s not presented here.

Are there confounding variables that might impact the data that are unaccounted  for by the study methodology? Here’s one…we know from research out of the Barna group that the vast majority of Christians make their initial profession of faith by their early teen years. I’d speculate that mental illnesses that impact children and teens would have a more profound impact on faith or spiritual maturity than the postpartum depression that lands a 30 year old mother in the hospital who has been active with church or campus ministry for the last fifteen years. The age of onset of the conditions selected here typically occurs after the time most Christians have professed their faith. How do the results look different if you selected Asperger’s Disorder, ADHD and adults treated for depression prior to the age of 18? I can think of others…lots of adults with schizophrenia are homeless or in assisted living situations. Their family’s church experiences might be very different than those families caring for a son or daughter or parent with schizophrenia who is still living at home. We don’t know that from looking at the information presented.

What instruments are used to measure the variables the authors seek to study? When we’re involved with a study in our practice, our outcome measures are typically normed and validated rating instruments. For example, we used the ADHD Rating Scale as an outcome measure in comparing long-acting ADHD medications, or the Brief Psychiatric Rating Scale in a study of kids with early onset schizophrenia. If I were reviewing a drug-company sponsored study of XYZ medication and their primary outcome measure is something I’ve never seen used in a paper before, my index of suspicion goes way up. What instrument or interview was used here? It’s not clear. Investigators at Baylor have developed standardized questionnaires to examine spiritual practices. Were they used? How did the authors decide on the questions they asked and why?

Looking at the study objectives, I’d expect to see questions about how often one attends church, prays, reads the Bible, participates in a small group or Bible study or serves in a ministry associated with a church. Did the authors intentionally oversample for church attenders? How many of the non-attenders became non-attenders because of their mental illness or a family member’s mental illness? When was the onset of their illness, and is there an association between the onset of illness and a decrease/discontinuation in church attendance? Are there specific church activities that are more difficult for the person with mental illness or their family members? Then again, nearly 80% of their sample was self characterized as “very stable” or “somewhat stable.” Why would one expect to see much of a difference at all in studying a group that may not be all that different from the other people attending church?

Next…we look at the data and data analysis. Is the analysis clear? Are the statistical measures appropriate? Is there any data that appears grossly inconsistent with our understanding of the topic? If some very unusual results exist, it doesn’t mean they’re not valid, but they’ll draw closer scrutiny.

In the FOTF/Lifeway study, here are some of the findings that made me look carefully…

• 37% of evangelical pastors and 55% of mainline pastors have taken seminary courses on care for the mentally ill. They may have taken a course in pastoral counseling, but the conversations I’ve had (and members of our Key Ministry team) had with pastors and seminary staff suggests this type of training is very uncommon. The percentages of pastors who claim to have taken continuing education classes on counseling or psychology seem very high also.

• 94% of individuals with acute mental illness say they have been prescribed medication to treat their mental illness. It’s very difficult to know what to make of this claim without a knowledge of how the sample was obtained. Most persons who received a formal diagnosis of bipolar disorder or schizophrenia have likely been treated with medication. This figure seems far above what one would expect for the general population of adults with depression, given the availability of non-medical treatments (cognitive-behavioral therapy) for depression.

• 85% of participants reported medication was very effective or somewhat effective. Given that the majority (94%) of the sample was identified as moderately to severely depressed at one time (depression is also common among persons with schizophrenia and bipolar disorder), this number appears wildly optimistic given the research on the effectiveness of antidepressants.

Another observation about the data…the authors point out item by item where findings are deemed “significant” statistically, but never indicate whether some findings were more robust (statistically significant) than others.

Conclusions…When I’m looking at a paper, I’m interested in how the authors interpret the data in light of their findings and what we previously knew about their area of research. In this case, the authors didn’t include any such summary in the download of the study results. We’re given lots of information without context. If I’m a pastor or a church board member, what should I do with this information? To me, the survey felt like a big data dump without any attempt to make sense of the data for the reader.

So…If I were given this study as the lead reviewer, it would have been turned down. It’s possible that portions of the study could be salvaged. The pastor survey had very large numbers with lots of statistical strength, even though I suspect a survey bias in which pastors tended to give the “right” answers when questioned. My sense from the remainder of the study is that the authors didn’t fully understand their subject material, and that’s too bad. There are critically important questions we as the church need to answer if we’re going to more effectively minister with a large segment of the population.

***********************************************************************************************************

Key Ministry has assembled resources to help churches more effectively minister to children and adults with ADHD, anxiety disorders, Asperger’s Disorder, Bipolar Disorder, depression and trauma. Please share our resources with any pastors, church staff, volunteers or families looking to learn more about the influence these conditions can exert upon spiritual development in kids, and what churches can do to help!