FunASR/fun_text_processing

Fundamental Text Processing (FunTextProcessing)

Introduction

FunTextProcessing is a Python toolkit for fundamental text processing in ASR including text processing , inverse text processing, which is included in the FunASR.

Highlights

• FunTextProcessing supports inverse text processing (ITN), text processing (TN).
• FunTextProcessing supports multilingual, 10+ languages for ITN, 5 languages for TN.

Installation

Fun Text Processing, specifically (Inverse) Text Normalization, requires Pynini to be installed.

bash fun_text_processing/install_pynini.sh

Example

Inverse Text Processing (ITN)

Given text inputs, such as speech recognition results, use fun_text_processing/inverse_text_normalization/inverse_normalize.py to output ITN results. You may refer to the following example scripts.

• ITN example for English

  python fun_text_processing/inverse_text_normalization/inverse_normalize.py --text="one hundred twenty three" --language=en
  

• ITN example for Indonesian

  python fun_text_processing/inverse_text_normalization/inverse_normalize.py --text="ratus dua puluh tiga" --language=id --cache_dir ./model/ --output_file output.txt
  cat output.txt
  

Arguments:

• text - Input text. Should not exceed 500 words.
• input_file - Input file with lines of input text. Only one of text or input_file is accepted.
• output_file - Output file to save normalizations. Needed if input_file is specified.
• language - language id.
• input_case - Only for text normalization. lower_cased or cased.
• verbose - Outputs intermediate information.
• cache_dir - Specifies a cache directory for compiled grammars. If grammars exist, this significantly improves speed.
• overwrite_cache - Updates grammars in cache.
• whitelist - TSV file with custom mappings of written text to spoken form.

Acknowledge

1. We borrowed a lot of codes from NeMo.
2. We refered the codes from WeTextProcessing for Chinese inverse text normalization.

License

This project is licensed under the Apache-2.0 license. FunTextProcessing also contains various third-party components and some code modified from other repos under other open source licenses.