Kezdőlap Felfedezés Súgó
Regisztráció Bejelentkezés
ai
/
OpenHands
2
0
Másolás 0
Fájlok Problémák 0 Beolvasztási kérések 0 Wiki
Fa: b4a7e27bfd
Branch-ok Tag-ek
OpenHands
/
evaluation
/
logic_reasoning
Xingyao Wang b30a2dd87a completely remove update_source_code (#3280) 1 éve
..
.cache_program a9823491e6 Support Logic Reasoning Benchmark (#1973) 1 éve
scripts 31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230) 1 éve
Dockerfile 31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230) 1 éve
README.md 31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230) 1 éve
__init__.py a9823491e6 Support Logic Reasoning Benchmark (#1973) 1 éve
instruction.txt 31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230) 1 éve
logic_inference.py 31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230) 1 éve
run_infer.py b30a2dd87a completely remove update_source_code (#3280) 1 éve

README.md

Logic Reasoning Evaluation

This folder contains evaluation harness for evaluating agents on the logic reasoning benchmark ProntoQA and ProofWriter.

Setup Environment and LLM Configuration

Please follow instruction here to setup your local development environment and LLM.

Run Inference on logic_reasoning

The following code will run inference on the first example of the ProofWriter dataset,

./evaluation/logic_reasoning/scripts/run_infer.sh eval_gpt4_1106_preview_llm ProofWriter