Aims and Background: Research into prognostic factors (PFs) is important in spinal care and across healthcare. Knowledge of PFs is used to provide a basis for overall prognosis, to develop prognostic models, and to place individuals into stratified treatment plans. There is an increasing number of systematic reviews of PF studies, however no quality appraisal instrument specifically designed for them. We have conducted a multi-phase process to develop such an instrument. Here we present agreement and time of use evaluations of user experience feedback on iterative versions of that tool.
Methods: Ten individuals were recruited to test the appraisal instrument, and seven were able to complete the testing procedure as planned. Four reviews were appraised, before appraisers met to compare interpretations and decisions, and then another seven or eight articles were appraised. Eight articles were appraised by all seven appraisers, and agreement for each question and the article overall was compared using Gwet’s Agreement Coefficient (AC). Time of use was averaged across all completed appraisals.
Results: Interrater agreement was moderate for the overall appraisal (Gwet’s AC 0.46, 95%CI 0.34-0.58), and on average was also moderate across the questions (Gwet’s AC 0.57). When answering options were collapsed (Yes and Probably Yes together, No and Probably No together), Gwet’s AC averaged 0.67; substantial agreement. Time of use to appraise a systematic review averaged 29 minutes. Usability and appropriateness were considered high; further guidance was recommended in certain areas including overall judgement.
Conclusions: Pilot testing of a new quality appraisal instrument for systematic reviews of PFs revealed moderate agreement and acceptable time of use, comparable to results obtained on testing of appraisal instruments in other areas such as interventional research, and provided clear recommendations for refinement before presentation of the final tool to the field.