Etusivu Tutki Apua
Rekisteröidy Kirjaudu sisään
ai
/
OpenHands
2
0
Haarauta 0
Tiedostot Esitykset 0 Vetopyynnöt 0 Wiki
Puu: 27716171bf
Haarat Tunnisteet
OpenHands
/
evaluation
/
logic_reasoning
Engel Nyst eeb2342509 Refactor history/event stream (#3808) 1 vuosi sitten
..
.cache_program 243cb492aa Run pre-commit on all files (#3884) 1 vuosi sitten
scripts 50c13aad98 [Eval] Improve SWE-Bench Eval harness: multi-run support & entry script simplification (#4396) 1 vuosi sitten
Dockerfile 152f99c64f Chore Bump python version (#3545) 1 vuosi sitten
README.md 31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230) 1 vuosi sitten
__init__.py a9823491e6 Support Logic Reasoning Benchmark (#1973) 1 vuosi sitten
instruction.txt 31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230) 1 vuosi sitten
logic_inference.py 31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230) 1 vuosi sitten
run_infer.py eeb2342509 Refactor history/event stream (#3808) 1 vuosi sitten

README.md

Logic Reasoning Evaluation

This folder contains evaluation harness for evaluating agents on the logic reasoning benchmark ProntoQA and ProofWriter.

Setup Environment and LLM Configuration

Please follow instruction here to setup your local development environment and LLM.

Run Inference on logic_reasoning

The following code will run inference on the first example of the ProofWriter dataset,

./evaluation/logic_reasoning/scripts/run_infer.sh eval_gpt4_1106_preview_llm ProofWriter