There's a new big language model (LLM) in town - two, to be exact - and '90s kids will immediately recognize their names: FreeWilly1 and FreeWilly2.
See also: OpenAI launches custom instructions for ChatGPT
Stability AI, the company behind Stable Diffusion image generation AI and founded by former UK financier Emad Mostaque, has unveiled two new LLMs based on versions of Meta’s open-source LLaMA and LLaMA 2 models. Both models are notable for their sophisticated reasoning, linguistic detail, and ability to answer complex questions related to specialized fields such as law and mathematics. Stability’s subsidiary, CarperAI, has released FreeWillys under a “non-commercial license” aimed at fostering research and promoting open access in the AI.
Smaller whales, more environmentally friendly
The model names are a play on the “ Orca ” AI training methodology developed by Microsoft researchers, which allows “smaller” models (exposed to more limited data) to achieve the performance of large fundamental models exposed to more massive data sets. (This is not a reference to IRL orcas sinking boats.)
Specifically, FreeWilly1 and FreeWilly2 were trained on 600,000 data points – just 10% of the size of the original Orca dataset – using instructions from four datasets created by Enrico Shippole, meaning they were much less expensive and much more environmentally friendly (using less energy and having a smaller carbon footprint) than the original Orca model and most leading LLMs. The models still produced excellent performance, comparable to ChatGPT on GPT-3.5 and in some cases even better.
Proposal: Google: Introduced Genesis AI for journalists
Training on synthetic data holds great promise
One issue with the proliferation of LLMs is the potential for “model collapse,” where LLMs trained on increasing amounts of AI-generated data perform worse than their predecessors trained on human-generated data. However, when training FreeWillys, Stability AI used two other LLMs to generate synthetic examples and found that FreeWillys still performed well, showing that synthetic data may be an answer to model collapse and avoiding the use of copyrighted or proprietary data.
Swimming into the future with Stability AI
Stability AI envisions that these models will set new standards in the field of open access LLMs, enhancing natural language understanding and enabling complex tasks.
“We are excited about the endless possibilities these models will bring to the AI community and the new applications they will inspire,” said the Stability AI team. They expressed their gratitude to the researchers, engineers, and collaborators whose dedication made this milestone possible.
Researchers and developers can access the FreeWilly2 weights as is, while the FreeWilly1 weights are published as deltas relative to the original model.
Read also: OpenAI may release an open-source AI model
information source:venturebeat.com
