A report commissioned by the Joseph Rowntree Reform Trust, entitled ‘Database State’ (Anderson et al 2009), has captured significant media coverage. Reuters headlines its coverage of the report with ‘Quarter of state databases “should be scrapped”. The claims are indeed alarming, and it is not surprising that they have gained media attention. Moreover, the report has been written by respected academics, including colleagues of mine, who have real expertise in security issues.
In this case, however, I question my colleagues’ findings. As the Reuters story notes: ‘The Ministry of Justice said the report had no real evidence to base its assessment’ (Tim Castle 23 March 2009). On this one, I must side with the Ministry of Justice, until the authors can convince me otherwise.
The report does not explain its methodology or the nature of the evidence on which the authors draw their conclusions. The report notes that the UK public sector has ‘an enormous number of databases’ (p. 11). One agency alone, the Serious and Organised Crime Agency is said to have over 500 databases (p. 11). However, the report focuses on 46 databases across the major departments of the entire UK government (p. 4), and provides no sense of how these 46 were chosen. So there is a serious sampling issue. Because journalists are drawing conclusions that suggest this sample is somehow representative of all databases, it is very important to spell this out. Even the Chair of the Rowntree Trust writes of only six given a ‘green light’ (p. 2), as if this was a representative sample.
Even if we disregard the sampling methodology, there are other issues of measurement. The 46 datasets were graded by a ‘traffic light system’ with red indicating that that a system is not compliant with the European Convention on Human Rights (ECHR) and that the design is such that it could not be made compliant without substantial redesign. The reader should not be in a position that requires us to trust the judgment of the authors, based on their authority, but there is no evidence provided to substantiate these ratings. Nor is there a methodology spelled out for applying or operationally defining this rating. Could someone replicate this?
In fact, from the references, it is not clear whether the authors went beyond desk or screen-based research. The acknowledgments indicate that various colleagues fed them ‘market intelligence’, but that is a problematic source for a systematic study. There do not appear to be personal interviews or field visits to meet with those managing these datasets or examine the systems. The risk is that we are reading weak journalistic coverage of research based on journalism and the input of pundits with similar views.
Their study is on a key topic at an important time. It seeks to build on a long-term debate over the role of computerization on privacy and data protection. Alan Westin and Michael Baker’s (1972) study was seminal in this area, but based on major field studies, and survey research, indicating the scale of research required in order to generate evidence. Undoubtedly available resources limit the rigour of the present study, but the limits of the study need to be clearly spelled out.
Debate over privacy and data protection is critical, but it could be undermined unless we know something more authoritative about the problem. When the Guardian (Travis 2009) reports that the ‘Right to privacy broken by a quarter of UK’s public databases, says report’, it is important to know more about the evidence on which these recommendations are based, and not rely on the authority of the authors.
Anderson, R., Brown, I., Dowty, T., Heath, W., and Sasse, A. (2009), Database State: A Report Commissioned by the Joseph Roundtree Reform Trust Ltd. York, UK: The Joseph Roundtree Reform Trust Ltd. http://www.jrrt.org.uk/uploads/Database%20State.pdf
Castle, T. (2009), ‘Quarter of State Databases “should be scrapped”’. Reuters, 23 March at: http://uk.reuters.com/article/domesticNews/idUKTRE52M04N20090323
Travis, A. (2009), ‘Right to Privacy Broken by a Quarter of UK’s Public Databases, Says Report’, Guardian, 23 March: http://www.guardian.co.uk/politics/2009/mar/23/dna-database-idcards-children-index
Westin, A. and Baker, M. A. (1972), Databanks in a Free Society: Computers, Record-Keeping and Privacy. New York: Quadrangle Books.