Falcon vs. RedPajama-INCITE

Overview

Falcon LLM is the flagship LLM of the Technology Innovation Institute in Abu Dhabi.

Falcon LLM is a powerful LLM developed by the Technology Innovation Institute (https://www.tii.ae). Unlike other popular LLMs, Falcon was not built off of LLaMA, but instead using a custom data pipeline and distributed training system. The dataset is the RefinedWeb dataset (available on Hugging Face), and the initial models are available in 7B and 40B forms. Instruct and chat-finetuned models are available as well. Perhaps most excitingly, Falcon outperforms LLaMA, the previous de facto pretrained LLM.

Initial release: 2023-05-23

Reference

https://falconllm.tii.ae/

Further Reading

• https://huggingface.co/tiiuae
• https://twitter.com/Thom_Wolf/status/1662085371826405377
• https://www.businesswire.com/news/home/20230531005608/en/UAE%E2%80%99s-Falcon-40B-World%E2%80%99s-Top-Ranked-AI-Model-from-Technology-Innovation-Institute-is-Now-Royalty-Free

Overview

RedPajama-INCITE is the first family of models trained on the RedPajama base dataset. The goal of the RedPajama-INCITE models is to replicate the LLaMA recipe but make the model fully open source under the Apache license. As of the initial release, the 3B parameter model is best-in-class, with the 7B parameter model in progress. Update as of June 6, 2023: the 7B parameter model was made available, outperforming other models of the same size.

Initial release: 2023-05-05

Reference

https://www.together.xyz/blog/redpajama-models-v1

Further Reading

• https://www.together.xyz/blog/redpajama-7b