client need

This litigation involved a dispute about the use of specialized software in the residential and commercial valuation industry. Our client was responding to a lawsuit alleging that its employees did not sufficiently account for usage of the software it licensed from one of their partners and that the fees charged were not adequate as a result.

After basic filtering workflows (deduplication, keyword search, etc.) were applied, outside counsel was left with a volume of 149K documents that needed to be reviewed and quickly produced to opposing counsel. A sampling exercise showed that less than three percent of the document population was relevant, but due to the nature of the documents and the production request criteria, additional filtering would be difficult.

why Epiq?

The firm approached Epiq with the idea of using a combination of technology and contract reviewers to facilitate a continuous active learning-based review. Continuous active learning is a variation of predictive coding that puts review first and seamlessly recommends the most interesting documents to the review team. Powered by sophisticated machine learning algorithms, the system learns while review is underway. This is a change from the more traditional practice of predictive coding that requires up-front training of the system before true review begins.

Epiq solution

Epiq organized a team of ten contract attorneys and used a combination of NexLP Story Engine and Relativity for the review. The law firm provided a small set of 140 seed documents to start NexLP’s classifier. These were a mix of documents similar to those they hoped to find, as well as examples of documents that were not relevant. Having a small set of seed documents allowed the system to begin prioritizing documents from the very start of the review. 

On day one, a Monday, the lead attorney spent half the day training the ten-member contract review team. The second half of that day and the following day, were spent reviewing the top scoring documents although less than 3% of the overall document population was relevant, the initial review of the system suggested documents were highly effective. On average seven out of ten of the documents reviewed on Monday and Tuesday were relevant. Once the lead attorney was confident the review team was coding consistently, all the review calls were fed to the continuous active learning system (NexLP Cosmic).

The model immediately stabilized, and the set of documents recommended for human review dropped in half again. The review team continued working through the documents, batching them out by relevance score. The third day, Wednesday, the percent of relevant documents in the high scoring range remained high. By the end of the day, the team had completed review of all the highest scoring documents and started review on the lower scored documents.

On Thursday, the prevalence of relevant material in the remaining population dropped to 38% and then on Friday down to 19%. At that point, the case team met with Epiq and decided to take a random sample of the remaining documents to validate that important documents weren’t being missed. The validation sample showed that the scoring correlated well with relevance and that the lowest scoring documents were obviously not relevant.

Sampling of the unreviewed set of 134,000 document resulted with no significant relevant content being uncovered. While review continued for several more days with some additional sets of documents added, some redaction of sensitive information, and review of specific population subsets, the effort to identify the relevant documents in the population was substantially complete after only one week of effort.

results and benefits

Ultimately, the team reviewed 33,000 documents, including 3,000 documents that were not suitable for the Continuous Active Learning workflow (excel spreadsheets and documents missing text). Thus, the discovery review effort took less time and money, and yet was more effective than many comparably sized efforts. The firm not only met their timeline objectives, but did so with a controlled and measured approach ensuring defensibility throughout the project lifecycle.