Summary

LexGLUE's contribution to the state of the art are two-fold. First, it combines and refines seven, huge document datasets into one, easy-to-access corpus. Second, thanks to the authors' thoughtful packaging of their datasets, they can be effortlessly integrated into Hugging Face's state-of-the-art transformer library for training and evaluation purposes with just a couple lines of codes.

According to the authors:

By unifying and facilitating the access to a set of law-related datasets and tasks, we hope to attract not only more NLP experts, but also more interdisciplinary researchers (e.g., law doctoral students willing to take NLP courses). More broadly, we hope LexGLUE will speed up the adoption and transparent evaluation of new legal NLP methods and approaches in the commercial sector too. Indeed, there have been many commercial press releases in the legal-tech industry on high-performing systems, but almost no independent evaluation of the performance of machine learning and NLP-based tools. A standard publicly available benchmark would also allay concerns of undue influence in predictive models, including the use of metadata which the relevant law expressly disregards.>

LexGLUE's seven, constituent datasets contain over 100,000 training instances total, primarily for multi-label or multi-class text classification tasks:

The seven included datasets and tasks are:

1. ECtHR Tasks A: includes over 11,000 cases from the European Court of Human Rights mapped to specific violation of human rights provisions of the European Convention of Human Rights (ECHR), permitting trained models to match fact patterns to alleged violations of the ECHR. Task A takes a list of facts and outputs a list of violated articles.
2. ECtHR Tasks B: Tasks B uses the same training data as ECtHR Tasks A and the same inputs (a list of case facts), except the outputs are allegedly violated articles.
3. SCOTUS: this dataset maps thousands of U.S. Supreme Court opinions to one of 14 different issue areas, such as Criminal Procedure, Civil Rights, Economic Activity, etc.. This permits the training and evaluation of models that can do the same.
4. EUR-LEX: includes over 65,000 EU Laws mapped to corresponding legal concepts assigned by the EU’s Publications Office.
5. LEDGAR: includes an excerpt of the 100 most common contractual provision labels found in a dataset containing 850,000 contract provisions labelled with 12,500 different categories.
6. UNFAIR-ToS: includes thousands of terms of service sentences labeled with any of 8 applicable types of unfair contractual terms (sentences) under European consumer law.
7. CaseHOLD: is comprised of around 53,000 multiple choice questions on the holdings of U.S. court cases.