Engel Nyst b295f5775c Revert "Fix issue #5609: Use litellm's modify_params with default True" (#5631)		11 달 전
..
.cache_program	678436da30 Fix issue #5222: [Refactor]: Refactor the evaluation directory (#5223)	1 년 전
scripts	9908e1b285 [Evaluation]: Log openhands version in eval output folder, instead of agent version (#5394)	1 년 전
Dockerfile	678436da30 Fix issue #5222: [Refactor]: Refactor the evaluation directory (#5223)	1 년 전
README.md	8f47547b08 docs: fix markdown linting and broken links (#5401)	1 년 전
__init__.py	678436da30 Fix issue #5222: [Refactor]: Refactor the evaluation directory (#5223)	1 년 전
instruction.txt	678436da30 Fix issue #5222: [Refactor]: Refactor the evaluation directory (#5223)	1 년 전
logic_inference.py	678436da30 Fix issue #5222: [Refactor]: Refactor the evaluation directory (#5223)	1 년 전
run_infer.py	b295f5775c Revert "Fix issue #5609: Use litellm's modify_params with default True" (#5631)	11 달 전

Logic Reasoning Evaluation

This folder contains evaluation harness for evaluating agents on the logic reasoning benchmark ProntoQA and ProofWriter.

Setup Environment and LLM Configuration

Please follow instruction here to setup your local development environment and LLM.

The following code will run inference on the first example of the ProofWriter dataset,

./evaluation/benchmarks/logic_reasoning/scripts/run_infer.sh eval_gpt4_1106_preview_llm ProofWriter