Colab & PPT:

https://colab.research.google.com/drive/1MfnvRjF-nCgaPh-Oa9z98HRrz5pmOzKu?usp=sharing

https://colab.research.google.com/drive/1MfnvRjF-nCgaPh-Oa9z98HRrz5pmOzKu?usp=sharing

https://drive.google.com/drive/folders/1JEAssw1tw0bfyHUepdX1E69v1v9hypPT?usp=drive_link

https://drive.google.com/drive/folders/1JEAssw1tw0bfyHUepdX1E69v1v9hypPT?usp=drive_link

Words vs Term

LLM uses words but using words to train model requires a massive amount of data, and with insufficient data will lead to weird and inconsistent outcome. But in NLP we uses term to train model for consistence and readable output.

Import Jieba

https://github.com/fxsjy/jieba

https://github.com/fxsjy/jieba

use: 精确模式