The lack of reliability testing is casting a shadow over the entire field of digital forensics. Digital forensics is very reliant on software tools which often act like blackboxes impervious to verification. So how do we know that the software’s interpretation of the data is correct?
Graeme Horsman points out in the journal Digital Investigation that while the field of digital forensics has boomed, research into tool testing and verification has stagnated.
Errors present at any stage of an examination can undermine an entire investigation, compromising any potentially evidential results. Despite a clear dependence on digital forensic tools, arguably, the field currently lacks sufficient testing standards and procedures to effectively validate their usage during an investigation.
Digital forensics is a discipline which provides decision-makers with a reliable understanding of digital traces on any device under investigation, however, it cannot say with 100% certainty that the tools used to undertake this process produce factually accurate results in all cases. This is an increasing concern given the push for digital forensic organisations to now acquire ISO 17025 accreditation.
[…]
In regards to tool testing, the advantages and disadvantages of both federated and centralized approaches should be considered. A centralized approach places a significant burden on an identified entity to design, implement and carry out tool testing approaches. This is then followed by the challenge of establishing methods for the effective dissemination of findings. Suggestions for the use of such an overarching governing body responsible for the regulation and implementation of tool-testing are easy to make, despite associated resourcing costs and the difficulty in establishing the vast expertise required all under one umbrella organization. The feasibility of developing and maintaining an organisation of this type may also be unrealistic, given associated costs both at setup and for long-term continued operation. Further, centralized approaches may not allow testing processes and results to be exposed to the levels of scrutiny required to achieve the levels of reliability required given the limited number on individuals involved in the initial testing processes.
[…]
In comparison, federated approaches offer a lesser level of centralized oversight, granting autonomy to those individuals involved in any procedure. This is seen with the CFTT Federated Testing Project introduced by NIST (2018), where test materials are centrally developed but disseminated to individual labs to carry out their own testing and subsequent sharing of results. This approach potentially sees more practitioners involved in tool testing and in doing so, governance of the application of test procedures is left to each individual entity.
