Lawyer Challenge

Case Crunch scored an accuracy of 86.6%. The lawyers scored an accuracy of 62.3%.

Over 100 commercial London lawyers signed up for the competition and made over 750 predictions over the course of a week in an unsupervised environment.

The problems were real complaints about PPI mis-selling decided and published by the Financial Ombudsman Service under the FOIA.

The results received international acclaim. For more details see the BBC article.

The main reason for the large winning margin seems to be that the network had a better grasp of the importance of non-legal factors than lawyers.

Evaluating these results is tricky. These results do not mean that machines are generally better at predicting outcomes than human lawyers. These results show that if the question is defined precisely (such as - was this complaint about PPI mis-selling upheld or rejected by the FOS?), machines are able to compete with and sometimes outperform human lawyers.

This experiment also suggests that there may be factors other than legal factors contributing to the outcome of cases. Further research is necessary to establish this proposition beyond the specific parameters of this experiment.

The use case for systems like Case Crunch is clear. Legal decision prediction systems like ours can solve legal prediction tasks permanently and reliably.

 We would like to thank our technical judge, Premonition, and our legal judge Dr. Steffek, for their involvment.