Risk of Bias Assessment in the PTSD Repository

All research studies are at risk of being biased. This may come from how the study was designed, how the study was carried out, or both. A study with high (versus low or medium) bias is one in which there are other factors besides the treatment being tested that may influence the findings in a meaningful way. Studies that are at high risk of bias may not be:
  • Reliable, meaning that if the study was repeated, investigators may not obtain the same results
  • Valid, meaning that the treatment may not be the reason for changes in symptoms
  • Generalizable, meaning that the outcomes found in the study may not apply to people who are not in the study
All studies in the PTSD Repository are randomized controlled trials (RCTs) in which participants were randomly assigned to treatment groups. Random assignment reduces a lot of bias. Randomization helps to assure that participants in treatment group A are similar to those in treatment group B. But even RCTs are at risk of being biased for reasons described below.
The PTSD Repository provides risk of bias (RoB) ratings for each RCT using Cochrane’s Risk of Bias 2 rating system.  Cochrane—a global, non-profit network of researchers and experts—is the gold standard in risk of bias ratings. When studies include more than one PTSD outcome measure, RoB ratings are included for both the primary and secondary measures.

Domain Ratings

Five domains were assessed as part of the overall RoB rating:
  1. Bias due to randomization
  2. Bias due to deviation from intended treatment
  3. Bias due to missing data
  4. Bias in measurement of the outcome
  5. Bias in selection of the intended result
Each domain is assessed by a set of questions—signaling questions—to help determine bias.  The questions are rated separately as Yes, Probably Yes, Probably No, No, or Not Applicable / No Information.
  •  Yes / Probably Yes: A rating of Yes or Probably Yes indicates that the study design and/or conduct took all appropriate measures to reduce the risk of bias.
  • No / Probably No: A rating of No or Probably No means that the study design and/or conduct clearly did not meet standards for methods used to reduce the risk of bias.
  • Not Applicable / No Information: A rating of Not Applicable or No Information most often reflects a lack of adequate reporting on the elements of the domain due to either not reporting on the element at all or unclear wording and is rated only when it is unreasonable to respond Probably Yes or Probably No.

Overall Rating


The answers to the signaling questions are then used to rate the overall domain as either Low, Some Concerns, or High.  In the prior rating system, the author determined an overall risk of bias rating using their own judgment. Cochrane's Risk of Bias 2 rating system (used in the PTSD-Repository) includes an algorithm to determine overall risk of bias.

Domain Details

Domain 1: Bias Due to Randomization
This domain focuses on whether the treatment groups were equally representative of the population being studied.   
1.1 Was the allocation sequence random?
1.2 Was the allocation sequence concealed until participants were enrolled and assigned to treatment?
1.3 Did baseline differences between intervention groups suggest a problem with the randomization process?
Domain 2: Bias Due to Deviations From Intended Interventions
This domain focuses on whether the treatment was delivered as intended and if there were deviations based on the participants’ or treatment providers’ knowledge of which treatment is assigned.
2.1 Were participants aware of their assigned intervention during the trial?
2.2 Were carers and people delivering the interventions aware of participants’ assigned intervention during the trial?
2.3 If Y/PY/NI to 2.1 or 2.2: Were there deviations from the intended intervention that arose because of the trial context?
2.4 If Y/PY/NI to 2.3: Were these deviations likely to have affected the outcome?
2.5 If Y/PY to 2.4: Were these deviations from intended intervention balanced between groups?
2.6 Was an appropriate analysis used to estimate the effect of assignment to intervention?
2.7 If N/PN/NI to 2.6: Was there potential for a substantial impact (on the result) of the failure to analyze participants in the group to which they were randomized?

Domain 3: Bias Due to Missing Data
This domain focuses on whether participants provided data as intended in the study or if instead they were unavailable or dropped out from measurement,
3.1 Were data for this outcome available for all, or nearly all, participants randomized?
3.2 If N/PN/NI to 3.1: Is there evidence that the result was not biased by missing outcome data?
3.3 If N/PN to 3.2: Could missingness in the outcome depend on its true value?
3.4 If Y/PY/NI to 3.3: Is it likely that missingness in the outcome depended on its true value?

Domain 4: Bias in the Measurement of the Outcome
This domain focuses on whether the measured values equal the true values.  Measurement error is a particular problem when there are differences in measurement between the treatment and control groups.
4.1 Was the method of measuring the outcome inappropriate?
4.2 Could measurement or ascertainment of the outcome have differed between intervention groups?
4.3 If N/PN/NI to 4.1 and 4.2: Were outcome assessors aware of the intervention received by study participants?
4.4 If Y/PY/NI to 4.3: Could assessment of the outcome have been influenced by knowledge of intervention received?
4.5 If Y/PY/NI to 4.4: Is it likely that assessment of the outcome was influenced by knowledge of intervention received?

Domain 5: Bias in the Selection of the Reported Result
This domain focuses on bias due to the selecting a primary outcome based on the reported results, rather than selecting the outcome in advance of the results.
5.1 Were the data that produced this result analyzed in accordance with a prespecified analysis plan that was finalized before unblinded outcome data were available for analysis?
Is the numerical result being assessed likely to have been selected, on the basis of the results, from:
5.2 ... multiple eligible outcome measurements (eg, scales, definitions, time points) within the outcome domain?
5.3 ... multiple eligible analyses of the data?

Assigning Overall Risk of Bias Rating

The overall RoB rating is assigned based on the 5 domain ratings. As with the domains, the overall ratings can be Low, Some Concerns or High. The overall rating is typically based on the lowest domain rating, However, if multiple domains were rated as Some Concerns, a lower overall rating can be assigned. 
  • High: A study with either serious flaws in 1 domain or moderate flaws in multiple domains. An example of a serious flaw would be lack of clarity on methods of randomization and how group assignment was concealed combined with large differences between groups on important clinical characteristics at baseline.
  • Low: A study with only minor flaws, generally limited to 1 domain. An example of a minor flaw would be attrition that is just outside of the threshold set for the review (e.g. 22% attrition when the threshold was set at 20%).
  • Some Concerns: A study that is neither high nor low risk of bias.

Overall Risk of Bias Ratings for RCTs in the PTSD-Repository

Of the 550 RCTs in the PTSD Repository, only a few studies are rated as low risk of bias (18%) where there is the greatest confidence that results are valid.  The majority are rated as being high risk of bias (54%).  These are studies in which there is the least confidence in the results. Twenty seven percent of the studies are rated as having some concerns.
Risk of Bias Ratings for Different Types of Treatment
When you look at RoB by type of treatment delivered (called Study Class), the proportion of studies that are rated as low, some concerns, and high vary by treatment type (see below).  The treatments that have been studied most (psychotherapy and pharmacotherapy) are roughly equivalent with just over half of the studies evaluated as high RoB,  18% as low risk of bias, and about 25% as having some concerns. Studies that combine psychotherapy and medications, other mixed studies, and nonpharmacologic cognitive are rated as having higher quality overall.
Risk of Bias Ratings for Trauma-Focused and Non-Trauma-Focused Psychotherapy RCTs
When looking specifically at trauma-focused psychotherapy studies, which include the most effective PTSD treatments, the proportion of studies that are high, some concerns, and low RoB is roughly equivalent to the non-trauma-focused treatments.* This suggests that as a treatment class trauma-focused psychotherapies are roughly equivalent to non-trauma-focused treatments with respect to their study design and implementation. Where they differ is in their effectiveness. Read about Trauma-Focused Psychotherapy for PTSD. 
*Note: This visualization presents data at the intervention arm level, not the study level (as studies may include both trauma-focused and non-trauma-focused arms). 
Risk of Bias Ratings for Studies of Recommended Medications for PTSD
The 2 most effective medication classes for PTSD are 2 types of antidepressants: selective serotonin reuptake inhibitors (SSRIs) and serotonin-norepinephrine reuptake inhibitors (SNRIs). When comparing SSRI/SNRI studies (below left) to all other medication RCTs (below right)*, the SSRI/SNRI studies have a larger proportion of rated high risk of bias (55% vs. 45%) and a lower proportion rated as low risk of bias (9% vs. 26%)  compared to the non-SSRI/SNRI medication studies.
*Note: The visualizations below present data at the intervention arm level, not the study level (as studies may include both SSRI/SNRI and non-SSRI/SNRI arms). 
Risk of Bias Ratings for RCTs of PTSD+SUD or PTSD Alone
The PTSD Repository includes studies that focus on treating PTSD alone as well as those that treat both PTSD and substance use disorder (SUD). A comparison of RoB in studies that target PTSD as compared to PTSD+SUD shows minimal variability between the RoB ratings (see below).

Summary

When considering the effectiveness of a treatment, or class of treatments, it is important to pay attention to RoB ratings. Regardless of study class, the majority of RCTs in the PTSD Repository are high risk of bias. 

Finding Risk of Bias Data in the PTSD Repository

Users of the PTSD Repository will find the overall and domain ratings in the Risk of Bias dataset. The overall ROB rating is also included as a repeated variable in the other datasets. This allows users to look at the relationship between RoB and treatment variables that are found in the Study Interventions dataset.  
If users are interested in knowing the RoB of a specific study:
  • Go to the primer page for the dataset of interest
  • Locate that study (by StudyID) in “Table Preview”
  • Scroll right to see the variable "Risk of Bias Rating (Study level)"