Expected difference questionnaire

What difference between treatments do you think a trial such as this, with several thousand patients, would show? Here clinicians are being asked not to predict the results of the planned trial, but those of a very large and impeccably conducted trial addressing the same question. When asked this question in a structured way, individual clinicians will rarely give a point estimate, it is much more natural to give different weights to various differences. The questionnaire asks the clinician to distribute 100 per cent across various differences as shown in Fig. 5.5.

In the hypothetical example, the clinician is somewhat cautious, giving most weight to there being a modest benefit to the experimental treatment, for example a 25 per cent chance of there being a 0-5 per cent absolute improvement and a 30 per cent chance of there being a 5-10 per cent absolute improvement. She also considers that there is only a 15 per cent chance that the experimental treatment will produce an absolute improvement of more than 10 per cent. The clinician also has some concerns (5 per cent probability) that it could actually be as much as 10-15 per cent worse than control treatment.

 Control treatment (C) better Experimental treatment (E) better Absolute difference between arms (per cent) 20+ 15-20 10-15 5-10 0-5 0-5 5-10 10-15 15-20 20+ Total Hypothetical example 0 0 5 10 15 25 30 10 5 0 100 Your entry 100

Fig. 5.5 Example of 'expected difference' questionnaire.

Ask as many relevant individuals as possible to complete these questionnaires to provide a representative spread of clinical opinion. An overall distribution of anticipated results can be obtained by averaging the percentages allocated to each of the effect-size categories; often this will form a symmetrical distribution of which the mean or median is a reasonable guess at the typical likely difference. Figure 5.6 shows the summary distribution of responses to the question posed with respect to the CHART bronchus trial; here the median difference was approximately 10 per cent.

Fig. 5.6 Overall distribution of expected differences - CHART bronchus trial. B. Worthwhile difference questionnaire

Here a different question is posed; ignoring what differences may be likely, we aim to elucidate the types of differences that, if observed, would be likely to change that clinician's practice. Using the CHART bronchus trial as an example again, we ask the following questions:

Suppose the result of the trial is known with absolute certainty. What is the largest difference in 2-year survival that would lead you to continue with conventional radiotherapy (CRT) as standard treatment? What is the smallest difference that would lead you to adopt CHART?

The clinician is then asked to mark these two points on a diagram, as shown in Fig. 5.7, beginning with a hypothetical example.

"I- ,i,.i.ï,.i.ii.i1iiïl,i1,1,,1,ï,.,.i,.i.i.fi1ii1i1,i..|ni.i1.i1.1,i|..1..1.ii.i.|-1

—15 to -10-10 to -5 -5 to 0 0 to 5 5 to 10 10 to 15 15 to 20 20 to 25 25 to 30 30 to 35 <-- benefit to conventional benefit to CHART--> Range of likely differences

Fig. 5.6 Overall distribution of expected differences - CHART bronchus trial. B. Worthwhile difference questionnaire

Here a different question is posed; ignoring what differences may be likely, we aim to elucidate the types of differences that, if observed, would be likely to change that clinician's practice. Using the CHART bronchus trial as an example again, we ask the following questions:

Suppose the result of the trial is known with absolute certainty. What is the largest difference in 2-year survival that would lead you to continue with conventional radiotherapy (CRT) as standard treatment? What is the smallest difference that would lead you to adopt CHART?

The clinician is then asked to mark these two points on a diagram, as shown in Fig. 5.7, beginning with a hypothetical example.

If a clinician would treat patients with CHART if the 2-year survival rate was more than 10 per cent higher than with CRT, and would use CRT if the survival rate was better, or no more than 5 per cent worse with CRT, they would answer as below:

Absolute difference in 2-year survival