Inicio Explorar Axuda
Rexistro Iniciar sesión
ai
/
OpenHands
2
0
Fork 0
Ficheiros Incidencias 0 Pull Requests 0 Wiki
Árbore: 0.9.5
Ramas Etiquetas
OpenHands
/
evaluation
/
logic_reasoning
tobitege 52c5abccbf (enh) Dockerfile.j2: improve env vars for bash and activate in .bashrc (#3871) hai 1 ano
..
.cache_program 243cb492aa Run pre-commit on all files (#3884) hai 1 ano
scripts 31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230) hai 1 ano
Dockerfile 83f36c1d66 test: build and run runtime tests on different custom docker images (#3324) hai 1 ano
README.md 31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230) hai 1 ano
__init__.py a9823491e6 Support Logic Reasoning Benchmark (#1973) hai 1 ano
instruction.txt 31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230) hai 1 ano
logic_inference.py 31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230) hai 1 ano
run_infer.py 52c5abccbf (enh) Dockerfile.j2: improve env vars for bash and activate in .bashrc (#3871) hai 1 ano

README.md

Logic Reasoning Evaluation

This folder contains evaluation harness for evaluating agents on the logic reasoning benchmark ProntoQA and ProofWriter.

Setup Environment and LLM Configuration

Please follow instruction here to setup your local development environment and LLM.

Run Inference on logic_reasoning

The following code will run inference on the first example of the ProofWriter dataset,

./evaluation/logic_reasoning/scripts/run_infer.sh eval_gpt4_1106_preview_llm ProofWriter