More and more often, parties are taking advantage of advanced document review technologies to reduce the burden of discovery. Search terms and technology-assisted review (TAR) are two ways parties typically reduce the volume for review and target the most relevant data. However, the opposing party may question whether the search terms and/or TAR process failed to identify potentially responsive documents.
To address this concern, a party can perform a statistically significant random sample of the documents that did not hit on terms or were deemed not relevant by the TAR process. The random sample can help determine elusion, or the ratio of documents identified as non-relevant that are, in fact, relevant. Parties can use this information to determine whether additional searches are warranted.
In City of Rockford v Mallinckrodt ARD Inc., pending in the U.S. District Court for the Northern Division of Illinois, the parties agreed to use search terms and TAR, but ultimately disagreed on how to validate the completeness of document production. The defendants wanted to meet and confer to discuss whether additional terms were necessary if the plaintiffs believed categories of requested documents were absent from production. The plaintiffs wanted to validate the documents produced by sampling the documents that did not hit on search terms (the null set). The court analyzed the plaintiffs’ proposal under Rule 26(g) reasonableness and under Rule 26(b)(1) proportionality standards. The court determined that a random sample of the null set was both reasonable and proportional, weighing the factors below during its analysis.
- Issues at stake: This was a “bet the company” type of case.
- Amount in controversy: There could be a huge damages award.
- Asymmetrical discovery: The defendants had access to the vast majority of relevant information.
- Resources: The defendant is a large company with substantial resources.
- Importance of electronically stored information (ESI): ESI will play a key role in resolving issues in the case.
- Burden and expense: In this instance, they did not outweigh the likely benefit.
To take full advantage of e-discovery technology with the goal of maximizing efficiency and accuracy, parties should consider validating the culling method used. One way to assuage an opposing party’s fears of incomplete production is to agree to conduct a limited sampling of the null set for the documents that do not hit on the search terms. Statistical random sampling of the null set can validate the process and provide reasonable assurance that the production is complete. It can also provide good evidence for the court that the culling method is valid and reasonable in cases where the method is challenged.