Boxuan Li
4add8a5595
SWE-bench: Allow selection of tasks (#1935)
|
1 年之前 | |
|---|---|---|
| .. | ||
| regression | 1 年之前 | |
| static | 1 年之前 | |
| swe_bench | 1 年之前 | |
| README.md | 1 年之前 | |
| TUTORIAL.md | 1 年之前 | |
| __init__.py | 1 年之前 | |
README.md
Evaluation
This folder contains code and resources to run experiments and evaluations.
Logistics
To better organize the evaluation folder, we should follow the rules below:
- Each subfolder contains a specific benchmark or experiment. For example,
evaluation/swe_benchshould contain all the preprocessing/evaluation/analysis scripts. - Raw data and experimental records should not be stored within this repo.
- For model outputs, they should be stored at this huggingface space for visualization.
- Important data files of manageable size and analysis scripts (e.g., jupyter notebooks) can be directly uploaded to this repo.
Supported Benchmarks
- SWE-Bench:
evaluation/swe_bench
Result Visualization
Check this huggingface space for visualization of existing experimental results.
Upload your results
You can start your own fork of our huggingface evaluation outputs and submit a PR of your evaluation results to our hosted huggingface repo via PR following the guide here.