As part of an otherwise frustrating dialogue with the NAS, Carol Povey, Director of its Centre for Autism, shared some of the processed anonymised data from this data as used internally by the NAS. A partial set of the cleaned data as processed for the article was later obtained from Liz Pellicano and Lorcan Kenny. The data for the article had been cleaned by the paper's authors in their initial submission and following review comments. Unfortunately, these data sets were shared in confidence and neither have been published.
In trying to better understand these datasets, I noticed some discrepancies between them. A concern is that the process of cleaning the data before analysis may affect survey-based articles on autism in a specific way. When both autistics and non-autistics are surveyed, the cleaning process appears to disproportionately affect the autistic cohort. Here's the text of the letter to the editor of the Autism Journal in which I detail this concern:
Does data cleaning disproportionately affect autistics?
In Kenny et al.’s paper (2016), titled ‘Which terms should be used to describe autism? Perspectives from the UK autism community’, the authors analysed data from the UK’s National Autistic Society’s (NAS) survey on terminology. In the paper, they detail how they removed a significant number of participants prior to data analysis. They state,
In all, 4622 people responded to the survey. Participants who (a) did not specify any connection with autism (n=19), (b) did not complete all four key questions on describing autism (n=453), (c) were under 18 years or preferred not to state their age (n = 284) and (d) were not resident in the United Kingdom or preferred not to state their place of residence (n=396) were excluded from the data set prior to analysis. Subsequent analysis was therefore based on complete responses from 3470 participants.
I believe that comparing results from the raw data and that of the data with these participants removed shows that this removal has disproportionately affected the processing for autistics compared to that for the other categories of respondents such as families and professionals.
For example, in Table 2, the modes for four rows for the ‘autistic’ column are different for the processed data compared to the raw results. Only one row is different for the ‘Parent’ and ‘Family/friend’ columns and none for the ‘Professional’ column.
I would suggest there is a reason for this. Anecdotally, autistics who have issues with the wording of survey questions or the possible set of answers often either object to continue filling in the survey or skip the offending questions (and often attempt to get in touch with the researchers for corrections and clarifications). There are 453 incomplete such entries in the NAS survey by autistics – which have been removed from analysis and, ultimately, were not taken into account for the processing in the final article.
This very limited comparison raises the hypotheses, supported by anecdotes, that cleaning of data in surveys targeted at both an autistic and non-autistic cohort may introduce a bias disproportionately affecting the responses from autistics. Further work on whole sets of data before and after the cleaning for several surveys is required to reach any conclusion.
Kenny L, Hattersley C, Molins B, et al. (2016) Which terms should be used to describe autism? Perspectives from the UK autism community. Autism 20(4): 442–462.
Looking solely at responses by autistics in the cleaned data, i.e., the autistic perspective, there are a few interesting outcomes (see tables below).
The preference for identity-first terminology (e.g., I am autistic) by autistics is clear:
- when communicating about autism, 61% of autistics prefer ‘autistic’ vs 35% for ‘has autism’;
- when describing themselves, 43% of autistics prefer ‘autistic’ vs 28% for ‘has autism’; and
- although the mode when rating terms by autistics is 4 for both ‘autistic’ and ‘has Asperger’s or autism’, the former is liked or strongly liked by 63% of autistics while the latter only by 57%.
Also ‘autistic person’ is always rated lower, by autistics, than ‘autistic' or 'is autistic’. This seem to imply that adding 'person' is redundant, though it may of course be context dependent.
Lastly, the second table below shows at least 53% of the autistic respondents were aspie, so the autistic respondents may not represent the full diversity of autistics.
Hopefully some researchers will investigate further whether the cleaning of data in surveys targeted at both an autistic and non-autistic cohort does introduce a bias disproportionately affecting the responses from autistics.
Several questions in the survey had very similar, but different, sets of answers and I wonder how many respondents analysed the subtlety of the different questions when answering them. (It also means that one cannot directly compare the different questions as the sets of answers are not identical.)
This important research paper raises many issues about accessibility at all stages of research, when surveys questions could be confusing and the way the data was analysed may have created bias disproportionately affecting autistic respondents, who may have felt unable to answer one or more of the questions. The solution to improve autism research is to involve autistics from the design to the analysis of research projects.
This post was simultaneously published on the Calm, almost too calm blog.