Gemma 2 vs. RedPajama-INCITE

Overview

Gemma 2 succeeds the Gemma family of lightweight open models from Google built using the same processes used for the the larger Gemini models.

Gemma 2 is the successor to the Gemma family of open models, including larer models (9B and 27B parameters) with outsized performance across benchmarks. Using a combination of techniques such as training on twice as much data, knowledge distillation, and architectural improvements such as sliding window attention, logit soft-capping, and model merging, Gemma 2 outperforms models of similar size (such as Llama 3), with the 27B parameter model being competitive with models more than twice its size (such as Llama 70B).

Initial release: 2024-06-27

Reference

https://blog.google/technology/developers/google-gemma-2/

Further Reading

• https://ai.google.dev/gemma/docs
• https://huggingface.co/blog/gemma2

Overview

RedPajama-INCITE is the first family of models trained on the RedPajama base dataset. The goal of the RedPajama-INCITE models is to replicate the LLaMA recipe but make the model fully open source under the Apache license. As of the initial release, the 3B parameter model is best-in-class, with the 7B parameter model in progress. Update as of June 6, 2023: the 7B parameter model was made available, outperforming other models of the same size.

Initial release: 2023-05-05

Reference

https://www.together.xyz/blog/redpajama-models-v1

Further Reading

• https://www.together.xyz/blog/redpajama-7b