Dear Participant,

 

 

 

Thank you for participating. I am studying how people think about confidence intervals (CIs). I will be asking you to read a few brief passages, look at some diagrams, and answer some simple questions. This is not a test of you! It is the materials that are under test. It should all take no more than 15 minutes.

 

 

 

 

 

 

 

Thanks again,

 

Pav Kalinowski

PhD Candidate.

p.kalinowski@latrobe.edu.au

 

 

 

 

 

Ethics information and contact details for researchers are available at the end of the survey.

 

 

 


Question 1.

 

A Sydney research team is interested in the duration of migraines. They used a sample of 25 participants to estimate the average duration of a migraine in the population. They use 50% confidence intervals (CIs) to estimate the population mean. The figure below shows the mean duration of a migraine for this sample marked by the black dot (11 hours) and a 50% CI around this mean.

Question 1a. Consider the values indicated by X1 and X2. Please click on whichever of the 13 buttons, on the scale below, best represents your intuitions.

 

Note: The unlabelled options represent a likelihood between the options just above it and just below it.

 

 

X1 is much more likely to be the pop. mean than X2.

X1 is moderately more likely to be the pop. mean than X2.

X1 is slightly more likely to be the pop. mean than X2.

X1 and X2 are about equally likely to be the pop. mean.

X1 is slightly less likely to be the pop. mean than X2.

X1 is moderately less likely to be the pop. mean than X2.

X1 is much less likely to be the pop. mean than X2.

Question 1b. Consider the values indicated by X2 and X3. Please click on whichever of the 13 buttons, on the scale below, best represents your intuitions.

 

 

 

 

 

X2 is much more likely to be the pop. mean than X3...

X2 is moderately more likely to be the pop. mean than X3.

X2 is slightly more likely to be the pop. mean than X3.

X2 and X3 are about equally likely to be the pop. mean.

X2 is slightly less likely to be the pop. mean than X3.

X2 is moderately less likely to be the pop. mean than X3.

X2 is much less likely to be the pop. mean than X3.

 

Question 1c. Consider the values indicated by X3 and X4. Please click on whichever of the 13 buttons, on the scale below, best represents your intuitions.

 

 

 

 

 

X3 is much more likely to be the pop. mean than X4.

X3 is moderately more likely to be the pop. mean than X4.

X3 is slightly more likely to be the pop. mean than X4.

X3 and X4 are about equally likely to be the pop. mean.

X3 is slightly less likely to be the pop. mean than X4.

X3 is moderately less likely to be the pop. mean than X4.

X3 is much less likely to be the pop. mean than X4.

 


Question 2.

 

A Brisbane research team is also interested in migraine duration. They have also used a sample of 25 participants to estimate the average duration of a migraine in the population. They commonly use 95% CIs to estimate the population mean. The figure below shows the mean duration of a migraine for their sample marked by the black dot (22 hours), the figure below gives a 95% CI around this mean.

Question 2a. Consider values indicated by Y1 and Y2. Please click on whichever of the 13 buttons, on the scale below, best represents your intuitions.

 

Y1 is much more likely to be the pop. mean than Y2..

Y1 is moderately more likely to be the pop. mean than Y2.

Y1 is slightly more likely to be the pop. mean than Y2.

Y1 and Y2 are about equally likely to be the pop. mean.

Y1 is slightly less likely to be the pop. mean than Y2.

Y1 is moderately less likely to be the pop. mean than Y2.

Y1 is much less likely to be the pop. mean than Y2.

Question 2b. Consider values indicated by Y2 and Y3. Please click on whichever of the 13 buttons, on the scale below, best represents your intuitions.

 

Y2 is much more likely to be the pop. mean than Y3...

Y2 is moderately more likely to be the pop. mean than Y3.

Y2 is slightly more likely to be the pop. mean than Y3.

Y2 and Y3 are about equally likely to be the pop. mean.

Y2 is slightly less likely to be the pop. mean than Y3.

Y2 is moderately less likely to be the pop. mean than Y3.

Y2 is much less likely to be the pop. mean than Y3.

Question 2c. Consider values indicated by Y3 and Y4. Please click on whicever of the 13 buttons, on the scale below, best represents your intuitions.

 

Y3 is much more likely to be the pop. mean than Y4..

Y3 is moderately more likely to be the pop. mean than Y4.

Y3 is slightly more likely to be the pop. mean than Y4.

Y3 and Y4 are about equally likely to be the pop. mean.

Y3 is slightly less likely to be the pop. mean than Y4.

Y3 is moderately less likely to be the pop. mean than Y4.

Y3 is much less likely to be the pop. mean than Y4.


Question 3.

 

Dr Smyth is a researcher interested in smoking cessation using a new type of Nicotine Replacement Therapy (NRT). After conducting a study on 30 participants, Dr Smyth reported that the sample mean amount of nicotine required to help smokers quit using the new therapy was 22mg.

In the figure above M is the sample mean, with a 95% CI. Dr Smyth wishes to estimate the population mean. Consider M and the point labelled L1. Compared with M, do you feel L1 is more or less likely to be the population mean? Please use the first column (below) to rate how likely L1 is, relative to M to be the population mean . Use the second column to rate L2 relative to M, and so on for points L3 to L9.

 

 L1 L2 L3 L4 L5 L6 L7 L8 L9  
More likely (to be the pop mean) than M
Slightly more likely (...) than M
About as likely (...) as M
 
Very slightly less likely (...) than M
 

Slightly less likely (...) than M
 
 
Moderately less likely(...) than M
 
 
 
Much less likely (...) than M
 
 
Very much less likely (...) than M
 
Almost zero likelihood


Question 4.

 

Dr Smyth has decided to explore CIs at a number of different confidence levels and would like your help. Below you will find a picture of a 95% CI. Please click on the slider to the right of the screen to set the lower interval to what you feel is the right length for an 80% CI for the same data.

 

 

95%CI
 
80%CI 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72
 

Now please click on the slider to the right of the screen to set the lower interval to what you feel is the right length for a 50% CI for the same data.

 
  95%CI  
50%CI 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72


Question 5.

 

Dr Kuresh has been working on an intervention to improve long term memory in patients with acquired brain injury using random number sequences of 30 numbers. Empirical evidence suggests that after intervention her participants could remember a sample mean (M) of 16 consecutive numbers.

In the figure above M is the sample mean, with a 50% CI. Dr Kuresh wants to estimate the population mean. Consider M and the point on the labelled L1. Compared with M do you feel L1 is more or less likely to be the population mean? Please use the first column (below) to rate how likely L1 is, relative to M to be the population mean. Use the second column to rate L2 relative to M, and so on for points L3 to L9.
  L1 L2 L3 L4 L5 L6 L7 L8 L9  
More likely (to be the pop mean) than M
Slightly more likely (...) than M
About as likely (...) as M
 
Very slightly less likely (...) than M
 

Slightly less likely (...) than M
 
 
Moderately less likely(...) than M
 
 
 
Much less likely (...) than M
 
 
Very much less likely (...) than M
 
Almost zero likelihood


Question 6.

 

Dr Kuresh has decided to explore CIs at a number of different confidence levels and would like your help. Below you will find a picture of a 50% CI. Please click on the slider to the right of the screen to set the lower interval to what you feel is the right length for an 80% CI for the same data

 

  50%CI
 
80%CI 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72
 

Now please click on the slider to the right of the screen to set the lower interval to what you feel is the right length for a 95% CI for the same data.

 
  50%CI  
95%CI 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72


Question 7.

 

Please think of a CI. Remember a CI is a set of plausible values for a parameter (for example the population mean), that you are trying to estimate from your sample.

 

Would you expect that all the points in the CI are equally likely to be the population mean? If so, then the distribution of the CI would have a flat top (e.g, diagram A or B below). If, on the other hand, you expect points in the middle of the interval to be much more likely than the points at the end, then the distribution might be like diagrams C or D.

 

There are many other distribution shapes, depending on if and how you expect the likelihood changes as a function of interval position.

 

The following pictures show 95% CIs. The dotted lines represent a distribution of likelihood that various points are the population mean. Please click to select which is closest to the way you think about a 95% CI.

High Likelihood

 

 

 

 

 

 

 

 

 

Low Likelihood

                          A

High Likelihood

 

 

 

 

 

 

 

 

 

Low Likelihood

                          B

High Likelihood

 

 

 

 

 

 

 

 

 

Low Likelihood

                          C

High Likelihood

 

 

 

 

 

 

 

 

 

Low Likelihood

                          D

High Likelihood

 

 

 

 

 

 

 

 

 

Low Likelihood

                          E

High Likelihood

 

 

 

 

 

 

 

 

 

Low Likelihood

                          F



How did you go about answering these questions, for example, did you have any particular model in mind that helped guide you? Did you make your decisions using any rules of thumb?

 
Question 7 referred to a 'likelihood distribution'. Is this concept familiar to you? If so, where did you come across this concept?

 
How familiar are you with the use of CIs? (Please tick one)

Not at all.

I've seen them before but I'm not sure what they mean.

I have seen CIs in research and have a rough idea of what they tell me.

I have seen CIs in research and know what they tell me.

I am familiar with CIs and I have sometimes used them in my own research.

I am very familiar with CIs and have often used them in my own research.

Please type in your your area (discipline) of study (for example, Psychology, Ecology, Medicine)?

 

How many years have you been studying at university level?

If you have studied part time please choose the equivalent number of full years.
What is your current level of education? If you are currently completing a degree, please choose the closest answer.
Please describe any other relevant research experience?



Thank you very much!

Would it be possible to contact you some time in the future for a short follow up interview? It would be very valuable to my research. This interview would focus on statistical intuitions, and again will not be a test of your abilities.

  Yes, I would prefer to be contacted by phone. Preferred contact number
  Yes, I would prefer to be contacted by email. Preferred email
  No, I don't wish to be contacted.  

 

 

 

 

 

 

 

This research is supported by the Australian Research Council. This project has been approved by the Human Ethics Committee, La Trobe University, Melbourne, Australia 3086. Data may be published; however, no information that could identify any respondent will be published. If you have any comment to make about this survey please contact Professor Geoff Cumming, School of Psychological Science, La Trobe University (Bundoora Campus) on +613 9479 2820 or g.cumming@latrobe.edu.au.

In the event you have a complaint or a query he is unable to answer, please contact the secretary of the HEC at +613 9479 1443 or humanethics@latrobe.edu.au.

 

 

Pav Kalinowski PhD candidate p.kalinowski@latrobe.edu.au

Professor Geoff Cumming, supervisor g.cumming@latrobe.edu.au

Dr Fiona Fidler, co-supervisor f.fidler@latrobe.edu.au