Lean Copilot allows large language models (LLMs) to be used in Lean for proof automation, e.g., suggesting tactics/premises and searching for proofs. You can use our built-in models from LeanDojo or bring your own models that run either locally (w/ or w/o GPUs) or on the cloud.
Lean.Copilot.Demo.mp4
- Requirements
- Using Lean Copilot in Your Project
- Advanced Usage
- Caveats
- Getting in Touch
- Acknowledgements
- Citation
- Supported platforms: Linux, macOS, and Windows WSL;
⚠️ Native Windows currently not supported. - Git LFS
- Optional (recommended if you have a CUDA-enabled GPU): CUDA and cuDNN
- Required for building Lean Copilot itself (rather than a downstream package): CMake >= 3.7 and a C++17 compatible compiler
lean4:v4.3.0-rc2
.
- Add the package configuration option
moreLinkArgs := #["-L./.lake/packages/LeanCopilot/.lake/build/lib", "-lctranslate2"]
to lakefile.lean. For example,
package «my-package» {
moreLinkArgs := #[
"-L./.lake/packages/LeanCopilot/.lake/build/lib",
"-lctranslate2"
]
}
- Add the following line to lakefile.lean, including the quotation marks:
require LeanCopilot from git "https://github.com/lean-dojo/LeanCopilot.git" @ "LEAN_COPILOT_VERSION"
LEAN_COPILOT_VERSION
depends on your lean-toolchain:
lean-toolchain | Recommended Lean Copilot version |
---|---|
v4.10.0 |
v1.5.0 |
v4.10.0-rc2 |
v1.4.2 |
v4.10.0-rc1 |
v1.4.1 |
v4.9.0 |
v1.4.0 |
v4.9.0-rc3 |
v1.3.3 |
v4.9.0-rc2 |
v1.3.2 |
v4.9.0-rc1 |
v1.3.1 |
v4.8.0 |
v1.3.0 |
v4.8.0-rc2 |
v1.2.2 |
v4.8.0-rc1 |
v1.2.1 |
v4.7.0 |
v1.2.0 |
v4.7.0-rc2 |
v1.1.2 |
v4.6.0-rc1 |
v1.1.1 |
v4.5.0 |
v1.1.0 |
v4.5.0-rc1 |
v1.1.0 |
- Run
lake update LeanCopilot
- Run
lake exe LeanCopilot/download
to download the built-in models from Hugging Face to~/.cache/lean_copilot/
- Run
lake build
Here is an example of a Lean package depending on Lean Copilot. If you have problems building the project, our Dockerfile, build.sh or build_example.sh may be helpful.
After import LeanCopilot
, you can use the tactic suggest_tactics
to generate tactic suggestions. You can click on any of the suggested tactics to use it in the proof.
![suggest_tactics](https://private-user-images.githubusercontent.com/114432581/319074489-f06865b6-58be-4938-a75c-2a23484384b4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM2NDU5MzMsIm5iZiI6MTcyMzY0NTYzMywicGF0aCI6Ii8xMTQ0MzI1ODEvMzE5MDc0NDg5LWYwNjg2NWI2LTU4YmUtNDkzOC1hNzVjLTJhMjM0ODQzODRiNC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxNFQxNDI3MTNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05ZDE5MDc2MDNlYWM3NTQ1YTQxNDBmZDc3OWIxNGQ4YjIxM2IyOWY3ZTUyYzM5NmY5OWNkM2U1N2U5NzA4Yjg2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.3jiV7sxHD7ProyhQrOFgDHR-xwxqg43V8aBD9JTZ8Co)
You can provide a prefix (e.g., simp
) to constrain the generated tactics:
![suggest_tactics_simp](https://private-user-images.githubusercontent.com/114432581/319074493-95dcae31-41cb-451c-9fdf-d73522addb6e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM2NDU5MzMsIm5iZiI6MTcyMzY0NTYzMywicGF0aCI6Ii8xMTQ0MzI1ODEvMzE5MDc0NDkzLTk1ZGNhZTMxLTQxY2ItNDUxYy05ZmRmLWQ3MzUyMmFkZGI2ZS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxNFQxNDI3MTNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02YzkwZWM4ZWRlYmZiMDk3NjkxN2YzZTUwYjgxZWI0M2FmM2IyNjdmODc2MGIwYmRmZDQ2YjhlOTdhZGMzODgxJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.4qOeRZOqaA7H3AsbzxHD6Cv28cLW9qnF6iWLZ-RxdCw)
The tactic search_proof
combines LLM-generated tactics with aesop to search for multi-tactic proofs. When a proof is found, you can click on it to insert it into the editor.
![search_proof](https://private-user-images.githubusercontent.com/114432581/334678388-26381fca-da4e-43d9-84b5-7e27b0612626.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM2NDU5MzMsIm5iZiI6MTcyMzY0NTYzMywicGF0aCI6Ii8xMTQ0MzI1ODEvMzM0Njc4Mzg4LTI2MzgxZmNhLWRhNGUtNDNkOS04NGI1LTdlMjdiMDYxMjYyNi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxNFQxNDI3MTNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05YTU1YTdjNGNhMjhiNzJhYzc2OGY1MjhmOTk4ZDRiN2JiODhkODExMWIyNjhiYTBiNzhlNGRlNjQwMGFiNmI2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.cC5v_a1ldT4PZm4A8N5taa8Amr_57QgsOop_6nLTBAs)
The select_premises
tactic retrieves a list of potentially useful premises. Currently, it uses the retriever in LeanDojo to select premises from a fixed snapshot of Lean and mathlib4.
You can also run the inference of any LLMs in Lean, which can be used to build customized proof automation or other LLM-based applications (not limited to theorem proving). It's possible to run arbitrary models either locally or remotely (see Bring Your Own Model).
![run_llms](https://private-user-images.githubusercontent.com/5431913/288648043-a4e5b84b-a797-4216-a416-2958448aeb07.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM2NDU5MzMsIm5iZiI6MTcyMzY0NTYzMywicGF0aCI6Ii81NDMxOTEzLzI4ODY0ODA0My1hNGU1Yjg0Yi1hNzk3LTQyMTYtYTQxNi0yOTU4NDQ4YWViMDcucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MTRUMTQyNzEzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OTU5MjJhMmQxZWM2ZTc5MDlkNjQ2MjFiYTUxYmM2MzAxM2E1NTg5NmM5YTFiNmU1MjI4NjNhYjRkOTQxZGIxNiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.PThw76KGvW0vIbv4T8eWoGV9l1EKzga8-QmgEZNO5SU)
This section is only for advanced users who would like to change the default behavior of suggest_tactics
, search_proof
, or select_premises
, e.g., to use different models or hyperparameters.
- Examples in TacticSuggestion.lean showcase how to configure
suggest_tactics
, e.g., to use different models or generate different numbers of tactics. - Examples in ProofSearch.lean showcase how to configure
search_proof
using options provided by aesop. - Examples in PremiseSelection.lean showcase how to set the number of retrieved premises for
select_premises
.
Examples in ModelAPIs.lean showcase how to run the inference of different models and configure their parameters (temperature, beam size, etc.).
Lean Copilot supports two kinds of models: generators and encoders. Generators must implement the TextToText
interface:
class TextToText (τ : Type) where
generate (model : τ) (input : String) (targetPrefix : String) : IO $ Array (String × Float)
input
is the input stringtargetPrefix
is used to constrain the generator's output.""
means no constraint.generate
should return an array ofString × Float
. EachString
is an output from the model, andFloat
is the corresponding score.
We provide three types of Generators:
NativeGenerator
runs locally powered by CTranslate2 and is linked to Lean using Foreign Function Interface (FFI).ExternalGenerator
is hosted either locally or remotely. See Bring Your Own Model for details.GenericGenerator
can be anything that implements thegenerate
function in theTextToText
typeclass.
Encoders must implement TextToVec
:
class TextToVec (τ : Type) where
encode : τ → String → IO FloatArray
input
is the input stringencode
should return a vector embedding produced by the model.
Similar to generators, we have NativeEncoder
, ExternalEncoder
, and GenericEncoder
.
In principle, it is possible to run any model using Lean Copilot through ExternalGenerator
or ExternalEncoder
(examples in ModelAPIs.lean). To use a model, you need to wrap it properly to expose the APIs in external_model_api.yaml. As an example, we provide a Python API server and use it to run a few models, including llmstep-mathlib4-pythia2.8b.
- Lean may occasionally crash when restarting or editing a file. Restarting the file again should fix the problem.
select_premises
always retrieves the original form of a premise. For example,Nat.add_left_comm
is a result of the theorem below. In this case,select_premises
retrievesNat.mul_left_comm
instead ofNat.add_left_comm
.
@[to_additive]
theorem mul_left_comm : ∀ a b c : G, a * (b * c) = b * (a * c)
- In some cases,
search_proof
produces an erroneous proof with error messages likefail to show termination for ...
. A temporary workaround is changing the theorem's name before applyingsearch_proof
. You can change it back aftersearch_proof
completes.
- For general questions and discussions, please use GitHub Discussions.
- To report a potential bug, please open an issue. In the issue, please include your OS information and the exact steps to reproduce the error. The more details you provide, the better we will be able to help you.
- Feature requests and other suggestions are extremely welcome. Please feel free to start a discussion!
- We thank Scott Morrison for suggestions on simplifying Lean Copilot's installation and Mac Malone for helping implement it. Both Scott and Mac work for the Lean FRO.
- We thank Jannis Limperg for supporting our LLM-generated tactics in Aesop (leanprover-community/aesop#70).
If you find our work useful, please consider citing our paper:
@misc{song2024largelanguagemodelscopilots,
title={Towards Large Language Models as Copilots for Theorem Proving in Lean},
author={Peiyang Song and Kaiyu Yang and Anima Anandkumar},
year={2024},
eprint={2404.12534},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2404.12534},
}