Computer Graphics and Visualization › Publications › From a User Study to a Valid Claim: How to Test Your Hypothesis and Avoid Common Pitfalls

In Proceedings of EuroRV³@EuroVis

From a User Study to a Valid Claim: How to Test Your Hypothesis and Avoid Common Pitfalls

Niels de Hoon, Elmar Eisemann, and Anna Vilanova

The 95% confidence interval highlighted in green for the t-distribution with 4 degrees of freedom (N = 5, N = 3 and α = 0.05). The left most image shows the two-sided confidence interval, while the middle and right image show the one-sided confidence intervals. The areas highlighted in red fall outside the interval.

Publisher Paper Cite More

The evaluation of visualization methods or designs often relies on user studies. Apart from the difficulties involved in the design of the study itself, the existing mechanisms to obtain sound conclusions are often unclear. In this work, we review and summarize some of the common statistical techniques that can be used to validate a claim in the scenarios that are commonly present in user studies in visualization, i.e., hypothesis testing. Usually, the number of participants is small and the mean and variance of the distribution are not known. Therefore, we will focus on the techniques that are adequate within these limitations. Our aim for this paper is to clarify the goals and limitations of hypothesis testing from a user study perspective, that can be interesting for the visualization community. We provide an overview of the most common mistakes made when testing a hypothesis that can lead to erroneous claims. We also present strategies to avoid those.

More Information

Publisher Paper preprint Presentation Presentation

Citation

Niels de Hoon, Elmar Eisemann, and Anna Vilanova, From a User Study to a Valid Claim: How to Test Your Hypothesis and Avoid Common Pitfalls, In Proceedings of EuroRV³@EuroVis, pp. 25–28, 2017.

BibTex 

@inproceedings{bib:de hoon:2017,
    author       = { de Hoon, Niels and Eisemann, Elmar and Vilanova, Anna },    
    title        = { From a User Study to a Valid Claim: How to Test Your Hypothesis and Avoid Common Pitfalls },
    booktitle    = { In Proceedings of EuroRV³@EuroVis },
    year         = { 2017 },
    pages        = { 25--28 },
    doi          = { 10.2312/eurorv3.20171110 },
    dblp         = { conf/eurorvvv-ws/HoonEV17 },
    url          = { https://publications.graphics.tudelft.nl/papers/237 },
}