|
|
@@ -0,0 +1,107 @@
|
|
|
+# CPU Benchmark (ONNX)
|
|
|
+
|
|
|
+## Configuration
|
|
|
+### Data set:
|
|
|
+Aishell1 [test set](https://www.openslr.org/33/) , the total audio duration is 36108.919 seconds.
|
|
|
+
|
|
|
+### Tools
|
|
|
+#### Install [modelscope and funasr](https://github.com/alibaba-damo-academy/FunASR#installation)
|
|
|
+
|
|
|
+```shell
|
|
|
+pip3 install torch torchaudio
|
|
|
+pip install -U modelscope
|
|
|
+pip install -U funasr
|
|
|
+```
|
|
|
+
|
|
|
+#### Export [onnx model](https://github.com/alibaba-damo-academy/FunASR/tree/main/funasr/export)
|
|
|
+
|
|
|
+```shell
|
|
|
+python -m funasr.export.export_model --model-name damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch --export-dir ./export --type onnx --quantize True
|
|
|
+```
|
|
|
+
|
|
|
+#### Building for Linux/Unix
|
|
|
+
|
|
|
+Download onnxruntime
|
|
|
+```shell
|
|
|
+# download an appropriate onnxruntime from https://github.com/microsoft/onnxruntime/releases/tag/v1.14.0
|
|
|
+# here we get a copy of onnxruntime for linux 64
|
|
|
+wget https://github.com/microsoft/onnxruntime/releases/download/v1.14.0/onnxruntime-linux-x64-1.14.0.tgz
|
|
|
+tar -zxvf onnxruntime-linux-x64-1.14.0.tgz
|
|
|
+```
|
|
|
+
|
|
|
+Install openblas
|
|
|
+```shell
|
|
|
+sudo apt-get install libopenblas-dev #ubuntu
|
|
|
+# sudo yum -y install openblas-devel #centos
|
|
|
+```
|
|
|
+
|
|
|
+Build runtime
|
|
|
+```shell
|
|
|
+git clone https://github.com/alibaba-damo-academy/FunASR.git && cd funasr/runtime/onnxruntime
|
|
|
+mkdir build && cd build
|
|
|
+cmake -DCMAKE_BUILD_TYPE=release .. -DONNXRUNTIME_DIR=/path/to/onnxruntime-linux-x64-1.14.0
|
|
|
+make
|
|
|
+```
|
|
|
+
|
|
|
+#### Recipe
|
|
|
+
|
|
|
+set the model, data path and output_dir
|
|
|
+
|
|
|
+```shell
|
|
|
+./bin/funasr-onnx-offline-rtf /path/to/model_dir /path/to/wav.scp quantize(true or false) thread_num
|
|
|
+```
|
|
|
+
|
|
|
+The structure of /path/to/models_dir
|
|
|
+```
|
|
|
+config.yaml, am.mvn, model.onnx(or model_quant.onnx)
|
|
|
+```
|
|
|
+
|
|
|
+## [Paraformer-large](https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary)
|
|
|
+
|
|
|
+Number of Parameter: 220M
|
|
|
+
|
|
|
+Storage size: 880MB
|
|
|
+
|
|
|
+Storage size after int8-quant: 237MB
|
|
|
+
|
|
|
+CER: 1.95%
|
|
|
+
|
|
|
+CER after int8-quant: 1.95%
|
|
|
+
|
|
|
+ ### Intel(R) Xeon(R) Platinum 8369B CPU @ 2.90GHz 16core-32processor with avx512_vnni
|
|
|
+
|
|
|
+| concurrent-tasks | processing time(s) | RTF | Speedup Rate |
|
|
|
+|---------------------|:------------------:|:--------:|:------------:|
|
|
|
+| 1 (onnx fp32) | 2129s | 0.058974 | 17 |
|
|
|
+| 1 (onnx int8) | 1020s | 0.02826 | 35 |
|
|
|
+| 8 (onnx fp32) | 273s | 0.007553 | 132 |
|
|
|
+| 8 (onnx int8) | 128s | 0.003558 | 281 |
|
|
|
+| 16 (onnx fp32) | 146s | 0.00403 | 248 |
|
|
|
+| 16 (onnx int8) | 67s | 0.001868 | 535 |
|
|
|
+| 32 (onnx fp32) | 133s | 0.003672 | 272 |
|
|
|
+| 32 (onnx int8) | 64s | 0.001778 | 562 |
|
|
|
+| 64 (onnx fp32) | 136s | 0.003771 | 265 |
|
|
|
+| 64 (onnx int8) | 67s | 0.001846 | 541 |
|
|
|
+| 96 (onnx fp32) | 137s | 0.003788 | 264 |
|
|
|
+| 96 (onnx int8) | 68s | 0.001875 | 533 |
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+### Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz 32core-64processor without avx512_vnni
|
|
|
+
|
|
|
+| concurrent-tasks | processing time(s) | RTF | Speedup Rate |
|
|
|
+|---------------------|--------------------|----------|--------------|
|
|
|
+| 1 (onnx fp32) | 2903s | 0.080404 | 12 |
|
|
|
+| 1 (onnx int8) | 2714s | 0.075168 | 13 |
|
|
|
+| 8 (onnx fp32) | 373s | 0.010329 | 97 |
|
|
|
+| 8 (onnx int8) | 340s | 0.009428 | 106 |
|
|
|
+| 16 (onnx fp32) | 189s | 0.005252 | 190 |
|
|
|
+| 16 (onnx int8) | 174s | 0.004817 | 207 |
|
|
|
+| 32 (onnx fp32) | 109s | 0.00301 | 332 |
|
|
|
+| 32 (onnx int8) | 88s | 0.00245 | 408 |
|
|
|
+| 64 (onnx fp32) | 113s | 0.003129 | 320 |
|
|
|
+| 64 (onnx int8) | 79s | 0.002201 | 454 |
|
|
|
+| 96 (onnx fp32) | 115s | 0.003183 | 314 |
|
|
|
+| 96 (onnx int8) | 80s | 0.002222 | 450 |
|
|
|
+
|
|
|
+
|
wucong.lyb