Graham Neubig f9088766e8 Allow setting of runtime container image (#3573)		1 سال پیش
..
.cache_program	a9823491e6 Support Logic Reasoning Benchmark (#1973)	1 سال پیش
scripts	31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230)	1 سال پیش
Dockerfile	83f36c1d66 test: build and run runtime tests on different custom docker images (#3324)	1 سال پیش
README.md	31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230)	1 سال پیش
__init__.py	a9823491e6 Support Logic Reasoning Benchmark (#1973)	1 سال پیش
instruction.txt	31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230)	1 سال پیش
logic_inference.py	31b244f95e [Refactor, Evaluation] Refactor and clean up evaluation harness to remove global config and use EventStreamRuntime (#3230)	1 سال پیش
run_infer.py	f9088766e8 Allow setting of runtime container image (#3573)	1 سال پیش

Logic Reasoning Evaluation

This folder contains evaluation harness for evaluating agents on the logic reasoning benchmark ProntoQA and ProofWriter.

Setup Environment and LLM Configuration

Please follow instruction here to setup your local development environment and LLM.

The following code will run inference on the first example of the ProofWriter dataset,

./evaluation/logic_reasoning/scripts/run_infer.sh eval_gpt4_1106_preview_llm ProofWriter