Even lawyers that are not currently using artificial intelligence (AI) in their document review practice undoubtedly know what it is and are familiar with the benefits that technology assisted review (TAR) solutions offer. TAR programs use predictive coding algorithms to sort through large data sets and weed out irrelevant documents. After users train the program on what is relevant, it can identify data patterns and relevant concepts in future reviews. Currently, litigation uses data that is digital, repetitive, and voluminous. Compared to the time-consuming aspect of manual review, using TAR software makes document review more manageable, accurate, and consistent. However, even with all these known benefits, many lawyers retain steep reservations and are hesitant to rely on this technology.
 

eDiscovery and AI Expert Maura Grossman 

Enter Maura Grossman. She is a licensed New York attorney who is a well-known eDiscovery and AI expert. She joined the Computer Science faculty at University of Waterloo in Canada to further her research on integration of AI into legal practice. Her primary goal is to have computer scientists and lawyers work together to learn from one another when it comes to deploying AI in legal practice. She believes that a collaborative approach will yield the best results and help squash the fears that many lawyers still have about utilizing TAR.

Human Intervention vs TAR

In 2011, Grossman and her colleague Gordon V. Cormack published a groundbreaking study on the effectiveness of TAR software.  The study set out to disprove the belief among many lawyers that manual document review creates more accurate results than TAR. The study provided illustrated that TAR offers results with higher recall and precision while using far less human intervention and completes the assignments at a fraction of the time. Research prior to this study suggested that even though manual review is not perfect and had a lower recall/precision rate; many lawyers still maintained a deference to manual review practices. Now, Grossman and Cormack’s study concretely demonstrates that TAR can provide better results that manual review.

How Large is the AI Advantage?

To reach their conclusion, Grossman and Cormack compared a set of data that different participants reviewed against the same standard, using both TAR and manual review methods. While the two review methods had similar recall rates, TAR far exceeded the manual review on precision, which overall made it much more accurate and effective. The results showed that the manual reviewers missed 20% - 75% of all relevant documents. The study attributed the majority of this range to inarguable error or data misinterpretation, with only 4% of missed documents resulting from error where there was room to argue about the relevance. These results are significant because if a lawyer fails to produce responsive documents in discovery, they could face litigation delays and even sanctions. The study concluded: “by all measures, the average efficiency and effectiveness of the five technology-assisted reviews surpasses that of the five manual reviews. The technology-assisted reviews require, on average, human review of only 1.9% of the documents [which is] a fifty-fold savings over exhaustive manual review.” The study noted that future research should analyze what types of TAR programs can help improve manual review efforts.

Conclusion 

Grossman continues to research this topic and advocate for lawyers to explore how TAR can help with document review during litigation. For example, she currently teaches a class entitled “Artificial Intelligence: Law, Ethics and Policy” which is open to both law students and computer science students. When asked about this hybrid class offering, she commented: “I’m trying to raise the consciousness of both sets of students. The law students learn about the technologies they will be responsible for regulating, and the technologists learn about the ethical and policy imperatives surrounding the technologies they develop…more and more, law firms are hiring data scientists, and both the creators and users of technology need to be sensitive to the legal implications, such as bias in data.”  If other universities adopt this innovative and cooperative approach, it will greatly prepare law students for the realities of legal practice and potentially open up new job opportunities as the legal technology industry grows. Lawyers regularly dealing with eDiscovery should stay tuned for any future studies or work Grossman and others do on this topic. If you found this blog informative, you may enjoy reading Judge the Judge by Using AI to Win Every Argument or The Epiq Angle Blog