LegalBench is a collection of benchmark tasks for evaluating legal reasoning in large language models.

Dataset Card for Dataset Name

Homepage: https://hazyresearch.stanford.edu/legalbench/ Repository: https://github.com/HazyResearch/legalbench/ Paper: https://arxiv.org/abs/2308.11462

  Dataset Description







  Dataset Summary

The LegalBench project is an ongoing open science effort to collaboratively curate tasks for evaluating legal reasoning in English large language models (LLMs). The benchmark currently consists of 162 tasks gathered from 40… See the full description on the dataset page: https://huggingface.co/datasets/omkar334/legalbench.

prithviraj-maurya/legalbench-entire dataset hosted on Hugging Face and contributed by the HF Datasets community

LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

  Dataset Description

LegalBench is an open science effort to collaboratively curate tasks for evaluating legal reasoning in English large language models (LLMs). The benchmark consists of 162 tasks gathered from 40 contributors, covering a wide range of legal domains, task structures, and difficulty levels.

  Homepage

Website:… See the full description on the dataset page: https://huggingface.co/datasets/sarthak-wiz01/legalbench.

amentaphd/legalbench-qa-contractnli dataset hosted on Hugging Face and contributed by the HF Datasets community

LegalBenchCorporateLobbying An MTEB dataset Massive Text Embedding Benchmark

The dataset includes bill titles and bill summaries related to corporate lobbying.

Task category t2t

Domains Legal, Written

Reference https://huggingface.co/datasets/nguha/legalbench/viewer/corporate_lobbying

  How to evaluate on this task

You can evaluate an embedding model on this dataset using the following code: import mteb

task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/legalbench_corporate_lobbying.

Dataset Card for "legalbench-function_of_decision_section"

More Information needed

ContractNLIInclusionOfVerballyConveyedInformationLegalBenchClassification An MTEB dataset Massive Text Embedding Benchmark

This task is a subset of ContractNLI, and consists of determining whether a clause from an NDA clause provides that Confidential Information may include verbally conveyed information.

Task category t2c

Domains Legal, Written

Reference https://huggingface.co/datasets/nguha/legalbench

  How to evaluate on this task

You can evaluate an… See the full description on the dataset page: https://huggingface.co/datasets/mteb/ContractNLIInclusionOfVerballyConveyedInformationLegalBenchClassification.

Equall/legalbench_instruct dataset hosted on Hugging Face and contributed by the HF Datasets community

CUADGoverningLawLegalBenchClassification An MTEB dataset Massive Text Embedding Benchmark

This task was constructed from the CUAD dataset. It consists of determining if the clause specifies which state/country’s law governs the contract.

Task category t2c

Domains Legal, Written

Reference https://huggingface.co/datasets/nguha/legalbench

  How to evaluate on this task

You can evaluate an embedding model on this dataset using the following code: import mteb… See the full description on the dataset page: https://huggingface.co/datasets/mteb/CUADGoverningLawLegalBenchClassification.

CUADNoSolicitOfEmployeesLegalBenchClassification An MTEB dataset Massive Text Embedding Benchmark

This task was constructed from the CUAD dataset. It consists of determining if the clause restricts a party's soliciting or hiring employees and/or contractors from the counterparty, whether during the contract or after the contract ends (or both).

Task category t2c

Domains Legal, Written

Reference https://huggingface.co/datasets/nguha/legalbench

  How to evaluate… See the full description on the dataset page: https://huggingface.co/datasets/mteb/CUADNoSolicitOfEmployeesLegalBenchClassification.

SCDBPAuditsLegalBenchClassification An MTEB dataset Massive Text Embedding Benchmark

This is a binary classification task in which the LLM must determine if a supply chain disclosure meets the following coding criteria: 'Does the above statement disclose whether the retail seller or manufacturer performs any type of audit, or reserves the right to audit?'

Task category t2c

Domains Legal, Written

Reference https://huggingface.co/datasets/nguha/legalbench… See the full description on the dataset page: https://huggingface.co/datasets/mteb/SCDBPAuditsLegalBenchClassification.

CUADLicenseGrantLegalBenchClassification An MTEB dataset Massive Text Embedding Benchmark

This task was constructed from the CUAD dataset. It consists of determining if the clause contains a license granted by one party to its counterparty.

Task category t2c

Domains Legal, Written

Reference https://huggingface.co/datasets/nguha/legalbench

  How to evaluate on this task

You can evaluate an embedding model on this dataset using the following code: import… See the full description on the dataset page: https://huggingface.co/datasets/mteb/CUADLicenseGrantLegalBenchClassification.

ContractNLISharingWithEmployeesLegalBenchClassification An MTEB dataset Massive Text Embedding Benchmark

This task is a subset of ContractNLI, and consists of determining whether a clause from an NDA clause provides that the Receiving Party may share some Confidential Information with some of Receiving Party's employees.

Task category t2c

Domains Legal, Written

Reference https://huggingface.co/datasets/nguha/legalbench

  How to evaluate on this task

You… See the full description on the dataset page: https://huggingface.co/datasets/mteb/ContractNLISharingWithEmployeesLegalBenchClassification.

PersonalJurisdictionLegalBenchClassification An MTEB dataset Massive Text Embedding Benchmark

Given a fact pattern describing the set of contacts between a plaintiff, defendant, and forum, determine if a court in that forum could excercise personal jurisdiction over the defendant.

Task category t2c

Domains Legal, Written

Reference https://huggingface.co/datasets/nguha/legalbench

  How to evaluate on this task

You can evaluate an embedding model on this… See the full description on the dataset page: https://huggingface.co/datasets/mteb/PersonalJurisdictionLegalBenchClassification.

LegalReasoningCausalityLegalBenchClassification An MTEB dataset Massive Text Embedding Benchmark

Given an excerpt from a district court opinion, classify if it relies on statistical evidence in its reasoning.

Task category t2c

Domains Legal, Written Reference https://huggingface.co/datasets/nguha/legalbench

  How to evaluate on this task

You can evaluate an embedding model on this dataset using the following code: import mteb

task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/LegalReasoningCausalityLegalBenchClassification.

Diversity2LegalBenchClassification An MTEB dataset Massive Text Embedding Benchmark

Given a set of facts about the citizenships of plaintiffs and defendants and the amounts associated with claims, determine if the criteria for diversity jurisdiction have been met (variant 2).

Task categoryt2c

Domains Legal, Written

Reference https://huggingface.co/datasets/nguha/legalbench

  How to evaluate on this task

You can evaluate an embedding model on this dataset… See the full description on the dataset page: https://huggingface.co/datasets/mteb/Diversity2LegalBenchClassification.

DefinitionClassificationLegalBenchClassification An MTEB dataset Massive Text Embedding Benchmark

This task consists of determining whether or not a sentence from a Supreme Court opinion offers a definition of a term.

Task category t2c

Domains Legal, Written

Reference https://huggingface.co/datasets/nguha/legalbench

  How to evaluate on this task

You can evaluate an embedding model on this dataset using the following code: import mteb

task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/DefinitionClassificationLegalBenchClassification.

ContractNLIPermissiblePostAgreementPossessionLegalBenchClassification An MTEB dataset Massive Text Embedding Benchmark

This task is a subset of ContractNLI, and consists of determining whether a clause from an NDA clause provides that the Receiving Party may retain some Confidential Information even after the return or destruction of Confidential Information.

Task category t2c

Domains Legal, Written

Reference https://huggingface.co/datasets/nguha/legalbench… See the full description on the dataset page: https://huggingface.co/datasets/mteb/ContractNLIPermissiblePostAgreementPossessionLegalBenchClassification.

CUADEffectiveDateLegalBenchClassification An MTEB dataset Massive Text Embedding Benchmark

This task was constructed from the CUAD dataset. It consists of determining if the clause specifies the date upon which the agreement becomes effective.

Task category t2c

Domains Legal, Written

Reference https://huggingface.co/datasets/nguha/legalbench

  How to evaluate on this task

You can evaluate an embedding model on this dataset using the following code: import… See the full description on the dataset page: https://huggingface.co/datasets/mteb/CUADEffectiveDateLegalBenchClassification.