⚡LLM Zoo is a project that provides data, models, and evaluation benchmark for large language models.⚡ [Tech Report]

• [07/12/2023]: More instruction-following data of different languages is available here.
• [05/05/2023]: Release the training code. Now, you can replicate a multilingual instruction-following LLM by yourself. :-)
• [04/24/2023]: Add more results (e.g., MOSS) in the evaluation benchmark.
• [04/08/2023]: Release the Phoenix (for all languages) and Chimera (for Latin languages) models.

• Break "AI supremacy" and democratize ChatGPT

"AI supremacy" is understood as a company's absolute leadership and monopoly position in an AI field, which may even include exclusive capabilities beyond general artificial intelligence. This is unacceptable for AI community and may even lead to individual influence on the direction of the human future, thus bringing various hazards to human society.

• Make ChatGPT-like LLM accessible across countries and languages
• Make AI open again. Every person, regardless of their skin color or place of birth, should have equal access to the technology gifted by the creator. For example, many pioneers have made great efforts to spread the use of light bulbs and vaccines to developing countries. Similarly, ChatGPT, one of the greatest technological advancements in modern history, should also be made available to all.

Run the following command to install the required packages:

pip install -r requirements.txt

python -m llmzoo.deploy.cli --model-path /path/to/weights/

For example, for Phoenix, run

python -m llmzoo.deploy.cli --model-path FreedomIntelligence/phoenix-inst-chat-7b

and it will download the model from Hugging Face automatically. For Chimera, please follow this instruction to prepare the weights.

Check here for deploying a web application.

We used the following two types of data for training Phoenix and Chimera:

+ Self-Instructed / Translated (Instruction, Input) in Language A
- ---(Step 1) Translation --->
+ (Instruction, Input) in Language B (B is randomly sampled w.r.t. the probability distribution of realistic languages)
- ---(Step 2) Generate--->
+ Output in Language B

+ (Role, Instruction, Input) seeds
- ---(Step 1) Self Instruct--->
+ (Role, Instruction, Input) samples
- ---(Step 2) generate output Instruct--->
+ (Role, Instruction, Input) ---> Output

+ ChatGPT conversations shared on the Internet
- ---(Step 1) Crawl--->
+ Multi-round conversation data

Check InstructionZoo for the collection of instruction datasets.

Check GPT-API-Accelerate Tool for faster data generation using ChatGPT.

• phoenix-sft-data-v1: The data used for training Phoenix and Chimera.

Due to LLaMA's license restrictions, we follow FastChat to release our delta weights. To use Chimera, download the original LLaMA weights and run the script:

python tools/apply_delta.py \
 --base /path/to/llama-13b \
 --target /output/path/to/chimera-inst-chat-13b \
 --delta FreedomIntelligence/chimera-inst-chat-13b-delta

Check our models in HuatuoGPT or try our demo . Similar biomedical models could be seen in biomedical LLMs.

We provide a bilingual, multidimensional comparison across different open-source models with ours.

• Automatic Evaluation Using GPT-4:

Observation: It shows that Phoenix-chat-7b achieves 85.2% performance of ChatGPT in Chinese. It slightly underperforms Baidu-Wenxin (96.8%) and ChatGLM-6b (94.6 %), both are not fully open-source; ChatGLM-6b only provides model weights without training data and details. Although Phoenix is a multilingual LLM, it achieves SOTA performance among all open-source Chinese LLMs.

• Human Evaluation:

Observation: It shows that the human evaluation results show the same trend as the automatic evaluation results.

• Automatic Evaluation Using GPT-4:

We offer int8 and int4 quantizations, which will largely reduce the GPU memory consumption, e.g., from ~28GB to ~7GB for phoenix.

You can directly obatin int8 version of phoenix by passing --load-8bit when using cli inference. E.g.,

python -m llmzoo.deploy.cli --model-path FreedomIntelligence/phoenix-inst-chat-7b --load-8bit

For int4 version, we take advantage of GPTQ. You can directly obatin int4 version of Phoenix by passing int4 version model and --load-4bit when using cli inference. This would require package AutoGPTQ be installed. E.g.,

python -m llmzoo.deploy.cli --model-path FreedomIntelligence/phoenix-inst-chat-7b-int4 --load-4bit

We use AutoGPTQ to support Phoenix via,

BUILD_CUDA_EXT=0 pip install auto-gptq[triton]

For Chimera, we can not share the int4 version parameters due to restrictions. And you can follow the example in our patched AutoGPTQ to conduct quantization by yourselves.

Thank yhyu13, please check the merged weight and GPTQ quantized weight for chimera in chimera-inst-chat-13b-hf and chimera-inst-chat-13b-gptq-4bit.

Inference in pure C/C++: You can refer to this link to run Chimera or Phoenix on your PC.

python -m llmzoo.deploy.webapp.controller

python -m llmzoo.deploy.webapp.model_worker --model-path /path/to/weights/

python -m llmzoo.deploy.webapp.gradio_web_server

Now, you can open your browser and chat with a model.

You can either download the phoenix-sft-data-v1 data or prepare your own data. Put your data on the path data/data.json.

For Phoenix, run

bash scripts/train_phoenix_7b.sh

For Chimera, prepare the LLaMA weights following this instruction and run

bash scripts/train_chimera_7b.sh
bash scripts/train_chimera_13b.sh

Our goal in releasing our models is to assist our community in better replicating ChatGPT/GPT4. We are not targeting competition with other competitors, as benchmarking models is a challenging task. Our models face similar models to those of ChatGPT/GPT4, which include:

• Lack of common sense: our models may not always have the ability to apply common sense knowledge to situations, which can lead to nonsensical or inappropriate responses.

• Limited knowledge domain: our models' knowledge is based on the data it was trained on, and it may not have the ability to provide accurate or relevant responses outside that domain.

• Biases: our models may have biases that reflect the biases in the data it was trained on, which can result in unintended consequences or unfair treatment.

• Inability to understand emotions: While our models can understand language, it may not always be able to understand the emotional tone behind it, which can lead to inappropriate or insensitive responses.

• Misunderstandings due to context: our models may misunderstand the context of a conversation, leading to misinterpretation and incorrect responses.

LLM Zoo is mainly contributed by:

• Data and Model: Zhihong Chen, Junying Chen, Hongbo Zhang, Feng Jiang , Chen Zhang, Benyou Wang (Advisor)
• Evaluation: Fei Yu, Tiannan Wang, Guiming Chen
• Others: Zhiyi Zhang, Jianquan Li and Xiang Wan

As an open-source project, we are open to contributions. Feel free to contribute if you have any ideas or find any issue.

We are aware that our works are inspired by the following works, including but not limited to

• LLaMA: https://github.com/facebookresearch/llama
• Bloom: https://huggingface.co/bigscience/bloom
• Self-instruct: https://github.com/yizhongw/self-instruct
• Alpaca: https://github.com/tatsu-lab/stanford_alpaca
• Vicuna: https://github.com/lm-sys/FastChat

Without these, nothing could happen in this repository.

@article{phoenix-2023,
  title={Phoenix: Democratizing ChatGPT across Languages},
  author={Zhihong Chen and Feng Jiang and Junying Chen and Tiannan Wang and Fei Yu and Guiming Chen and Hongbo Zhang and Juhao Liang and Chen Zhang and Zhiyi Zhang and Jianquan Li and Xiang Wan and Benyou Wang and Haizhou Li},
  journal={arXiv preprint arXiv:2304.10453},
  year={2023}
}

@misc{llm-zoo-2023,
  title={LLM Zoo: democratizing ChatGPT},
  author={Zhihong Chen and Junying Chen and Hongbo Zhang and Feng Jiang and Guiming Chen and Fei Yu and Tiannan Wang and Juhao Liang and Chen Zhang and Zhiyi Zhang and Jianquan Li and Xiang Wan and Haizhou Li and Benyou Wang},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/FreedomIntelligence/LLMZoo}},
}

We are from the School of Data Science, the Chinese University of Hong Kong, Shenzhen (CUHKSZ) and the Shenzhen Rsearch Institute of Big Data (SRIBD).