Risk of Bias Assessment in the PTSD-Repository

All research studies are at risk of being biased. This may come from how the study was designed, how the study was carried out, or both. A study with high (versus low or medium) bias is one in which there are other factors besides the treatment being tested that may influence the findings in a meaningful way. Studies that are at high risk of bias may not be:
  • Reliable, meaning that if the study was repeated, investigators may not obtain the same results
  • Valid, meaning that the treatment may not be the reason for changes in symptoms
  • Generalizable, meaning that the outcomes found in the study may not apply to people who are not in the study
All studies in the PTSD-Repository are randomized controlled trials (RCTs) in which participants were randomly assigned to treatment groups. Random assignment reduces a lot of bias. Randomization helps to assure that participants in treatment group A are similar to those in treatment group B. (Meaning, participants in any arm of the study are similar at the start.) But even RCTs are at risk of being biased for reasons described below. All RCTs in the PTSD-Repository include a risk of bias (ROB) rating to help users decide how much confidence to have in the findings. 
There are many different ROB rating systems. Most assess bias in specific domains of the study design/execution and then, based on those ratings, assign an overall ROB rating. The most well-known ROB system was created by Cochrane—a global, non-profit network of researchers and experts. The RCTs in the PTSD-Repository were all assigned a ROB rating of low, medium or high bias using a modified Cochrane rating, a system that uses a 2-step approach.  To learn more about the specific approach used to determine ROB, see: Comparative Effectiveness Review [CER] No. 207, Psychological and Pharmacologic Treatments for Adults with Posttraumatic Stress Disorder.

Domain Ratings

Four domains were assessed as part of the overall ROB rating: bias in selection, performance, detection and attrition. Domains contain 1 or more elements, each of which are rated separately.  Every element was rated as Yes, No or Unclear:
  •  Yes: A rating of Yes indicates that the study design and/or conduct took all appropriate measures to reduce the risk of bias.
  • No: A rating of No means that the study design and/or conduct clearly did not meet standards for methods used to reduce the risk of bias.
  • Unclear: Unclear most often reflects a lack of adequate reporting on the elements of the domain due to either not reporting on the element at all or unclear wording.
Selection Bias Domain
Selection bias is focused on whether the 2 or more treatment groups are equally representative of the population being studied.   
  • Randomization adequate: There was no selection bias due to the method in which study participants were assigned to treatment groups. Every participant had an equal chance of being in each treatment group.
  • Allocation concealment adequate: The researcher assigning participants to a study group is unaware as to which group the next participant will be assigned.
  • Groups similar at baseline: There were no meaningful differences on clinically important baseline characteristics between the treatment groups (or those differences were statistically controlled).
  • Intention to treat analysis were reported: Data from all participants were analyzed according to the groups they were initially assigned to even if participants dropped out of the study or switched groups. 
Performance Bias Domain
Performance bias is focused on whether either the participants’ or treatment providers’ knowledge of which treatment a participant is assigned to impacts results.
  • Masking (or blinding) of participants and providers: Potential impact on results due to knowledge of treatment assigned was eliminated; patients and providers were unaware of the treatment assigned.
Detection Bias Domain
Detection bias is focused on whether knowledge of treatment condition of assessors (i.e., research staff who assess clinical symptoms) impacts results.
  • Masking (or blinding) of assessors: Potential impact on measurement of outcomes due to knowledge of treatment assigned was eliminated; outcome assessors were unaware of the treatment assigned. Knowledge of treatment condition did not impact assessors.
Attrition Bias Domain
Attrition bias is focused on whether the amount, nature/pattern, or handling of missing data impacts results.
  • Overall attrition less than or equal to 20%: Overall loss of outcome measurement across groups in the study is small enough to not impact results meaningfully.
  • Differential attrition less than or equal to 15%: The difference in outcome measurement between treatment groups is small enough to not distill results meaningfully.

Assigning Overall Risk of Bias Rating

The final rating for risk of bias (high, medium, low) reflects the risk of bias across the 4 domains, and not only the individual elements. Ratings also require consensus (agreement) between 2 reviewers. There is not a strict formula, but generally:
  • High Risk of Bias: A study with either serious flaws in 1 domain or moderate flaws in multiple domains. An example of a serious flaw would be lack of clarity on methods of randomization and how group assignment was concealed combined with large differences between groups on important clinical characteristics at baseline.
  • Low Risk of Bias: A study with only minor flaws, generally limited to 1 domain. An example of a minor flaw would be attrition that is just outside of the threshold set for the review (e.g. 22% attrition when the threshold was set at 20%).
  • Moderate Risk of Bias: A study that is neither high nor low risk of bias.

Overall Risk of Bias Ratings for RCTs in the PTSD-Repository

Of the 389 RCTs in the PTSD-Repository, the majority are medium risk of bias (60%).  These studies are susceptible to bias but probably not enough to invalidate the results. Only a few studies are low risk of bias (6%) where there can be the greatest confidence that results are valid.  About a third are high risk of bias (34%); these are studies in which there is the least confidence in the results.

Risk of Bias Ratings for Different Types of Treatment
When you look at ROB by study class (treatment type), the proportion of studies that are high, medium and low remains mostly the same (see below). The majority of studies are about 2/3 medium ROB and 1/3 high ROB. Nonpharmacologic cognitive studies are all medium ROB. There are few low risk of bias in most treatment types and none among the nonpharmacologic biological studies (or the nonpharmacologic cognitive studies). 

Risk of Bias Ratings for Trauma-Focused and Non-Trauma-Focused Pharmacotherapy RCTs
When looking specifically at trauma-focused psychotherapy studies, the most effective type of psychotherapy, the proportion of studies that are high, medium and low ROB is roughly equivalent to the non-trauma-focused treatments.* This suggests that as a treatment class they are roughly equivalent to non-trauma-focused treatments with respect to their study design and implementation. Where they differ is in their effectiveness. Read about Trauma-Focused Psychotherapy for PTSD
*Note: This visualization presents data at the intervention arm level, not the study level (as studies may include both TF and non-TF arms). 

Risk of Bias Ratings for Studies of Recommended Medications for PTSD
The 2 most effective medication classes for PTSD are 2 types of antidepressants: selective serotonin reuptake inhibitors (SSRIs) and serotonin-norepinephrine reuptake inhibitors (SNRIs). When comparing SSRI/SNRI studies (below left) to all other medication RCTs (below right)*, there is more variability in risk of bias associated with other medications. In particular, non-SSRI/SNRI studies are more likely to have high risk of bias (38%) than RCTs of SSRIs/SNRIs (33%). 
*Note: The visualizations below present data at the intervention arm level, not the study level (as studies may include both SSRI/SNRI and non-SSRI/SNRI arms). 


Risk of Bias Ratings for RCTs of PTSD+SUD or PTSD Alone
In 2021, studies that focus on treating both PTSD and substance use disorder (SUD) were added to the PTSD-Repository. RCTs were coded as focused on either PTSD or PTSD+SUD. A comparison of ROB in studies that target PTSD as compared to PTSD+SUD shows that there is less variability in the ROB ratings of PTSD+SUD studies. While there are no low risk of bias studies among the PTSD+SUD studies, there are also fewer high risk of bias studies (17% as compared to 35%).
For more information on these studies, see our story, Treatment of Co-Occurring PTSD and Substance Use Disorder

Summary

When considering the effectiveness of a treatment, or class of treatments, it is important to consider ROB. Regardless of study class, the majority of RCTs in the PTSD-Repository are medium risk of bias. 

Finding Risk of Bias Data in the PTSD-Repository

Users of the PTSD-Repository will find the overall and domain ratings in the Risk of Bias dataset. The overall ROB rating is also included as a repeated variable in the other datasets. This allows users to look at the relationship between ROB and other variables such as different treatment variables that are found in the Study Interventions dataset.  
If users are interested in knowing the ROB of a specific study:
  • Go to the primer page for the dataset of interest
  • Locate that study (by StudyID) in “Table Preview”
  • Scroll right to see the risk of bias rating variable