Meta AI is sharing Open Pretrained Transformer (OPT-175B) With the World
We will remember Facebook democratizing access to all (sarcasm)
Hey Guys,
Personal Note
My Mom is in the hospital with terminal cancer, so my cadence of articles might be significantly lower during this period.
Inspite of the major exodus of talent we are seeing from Meta AI and even the likes of Google Brain, it appears BigTech companies are trying harder than ever to win some “AI for Good” brownie points. Yann LeCun is not uninteresting to follow on LinkedIn, even as Facebook’s drama in the Metaverse continues.
Google and increasingly Microsoft care about the grandest headlines in the hype of the A.I. story., even if that means firing employees who question their papers. It is a grand time for A.I. where monopolies attract the best talent and Transformers allow for new gigantic language models of scale.
MIT Review was praising Meta for “giving it away for free” and one wonders what is real journalism in the field. What have we come to on this internet of dumb headlines? Meta's AI lab has created a massive new language model that shares both the remarkable abilities and the harmful flaws of OpenAI's pioneering neural network GPT-3. So in this incredibly generous and unprecedented move for Big Tech, Facebook, (I mean Meta) is giving it away to researchers—together with details about how it was built and trained.
How desperate the latest loser of the internet has become.
Meta’s PR minions say it’s giving it away, but Meta itself says no, it’s “democratizing” it for everyone.
Democratizing access to large-scale language models with OPT-175B
So what is OPT-175B? In a world where Elon Musk wants to open Twitter’s algorithm to the world, I present to you, that Yann LecCun wants to open Meta’s vast treasure for us all to envy or enjoy. This as ByteDance, the rival of Facebook that’s eating its lunch each day, has announced something actually meaningful.
Meta AI @MetaAI
Today Meta AI is sharing OPT-175B, the first 175-billion-parameter language model to be made available to the broader AI research community. OPT-175B can generate creative text on a vast range of topics. Learn more & request access: https://t.co/3rTMPms1vq https://t.co/DzjAdbPDLxHere here the Knights of the Dystopia of our future, brave Elon and generous Yann.
All the Meta AI lords are stacked in my home town, of Montreal.
Meta did unveil the codebase, development process logbook, data, research paper and other information associated with Open Pretrained Transformer, or OPT-175B, its new 175-billion-parameter open-source large language model.
The company called the effort an exercise in transparency that is part of its commitment to open science.
Referring to GPT-3, Joelle Pineau, managing director of Meta AI, said recently “Of course others have come before us in terms of training large language models, and in some cases have provided an API to run inference. But the code and trained parameters for these models have not been released to the broader research community.”
“With the release of OPT-175B, we are opening up direct access to the large scale models to this community for the first time, so that scientific discourse on LLMs can be conducted on reproducible results,” she said.
Meta that shining community of talented Ad-tech people who stole Dfinity’s logo. I knew I saw it somewhere else before! Like about 10 times.
When the Internet Gods Give you Gifts
Still we must not let this opportunity pass us by, it’s rare Facebook is in a generous mood. Please study along with me the secrets of this marvel.
OPT-175b:
Open Pre-Trained language model with 175 billion parameters is now available to the research community.
Blog post: https://lnkd.in/esZZf5R6
Paper: https://lnkd.in/eARwzjt3
Code + small pre-trained models: https://lnkd.in/eusfCqmq
Article at SiliconAngle: https://lnkd.in/e_WSueR6
Article at Technology Review: https://lnkd.in/emYxdFSF
Using the full OPT-175b requires a registration: https://lnkd.in/evWAuwDt
(Your data may be collected and shared with third parties, again!)
Montreal is a hot-bed of A.I. talent and even has its own Montreal A.I. Ethics Institute. They also have a Substack here.
"We strongly believe that the ability for others to scrutinize your work is an important part of research. We really invite that collaboration," says Joelle Pineau, a longtime advocate for transparency in the development of technology, who is now managing director at Meta AI.
Large language models — natural language processing (NLP) systems with more than 100 billion parameters — have transformed NLP and AI research over the last few years.
Consider a Cat has 900 million neurons and a person has 86 billion neurons.
Trained on a massive and varied volume of text, they show surprising new capabilities to generate creative text, solve basic math problems, answer reading comprehension questions, and more.
Meta the Open Science Company
With Meta high-level AI’s commitment to open science, they have come to the decision to share the Open Pretrained Transformer (OPT-175B), a language model with 175 billion parameters trained on publicly available data sets, to allow for more community engagement in understanding this foundational new technology.
This is indeed a grand moment in the history of A.I. for Good.
Yann is not one to suffer fools in his LinkedIn posts.
The AI Exodus begins
At least four prominent members of Meta AI have departed in recent months. Many on the London side. Back in April, 2022 we saw a quote: “Meta’s London office just collapsed and they lost most of their [top] researchers in the span of six weeks,” Karl Hermann, an AI entrepreneur who used to work at rival lab DeepMind, said.
The employee exodus at Facebook since its stock drop is legendary in recent months. Meta’s Chief AI Scientist Yann LeCun, who co-founded the firm’s AI lab in 2013 after a dinner with founder Mark Zuckerberg keeps upping the hype on what Meta A.I. will be able to do (it’s nearly as bad as Tesla’s claims) from an academic standpoint. We already were used to the boastful claims of OpenAI (basically acquired by Microsoft) and Google AI.
As the metaverse takes centre stage, multiple Meta AI scientists are departing from the company in an alleged exodus. Mark Zuckerberg does not rule in a democracy, by the way. So saying that Meta is democratizing anything, is nonsense.
Dfinity, creator of the Internet Computer blockchain, says the logo of Facebook parent company Meta is too similar to its own. It has slapped them with a logo lawsuit. But fines and lawsuits are not problem for Facebook, they are that rich.
Those who have left the company include Edward Grefenstette, a research scientist that led Meta’s efforts on a branch of AI known as reinforcement learning, who departed in February.
Heinrich Kuttler, one of Meta AI’s research engineering managers, left in recent weeks to join Inflection AI, a start-up set up by DeepMind co-founder Mustafa Suleyman and LinkedIn billionaire Reid Hoffman. I covered this news here.
Another recent departure is Ahmad Beirami who left his research scientist position at Meta in January and joined Google in the same role.
And last year, in December, Douwe Kiela left his research scientist role at Meta after spending five years at the company.
I’m sure by now there are others. Meta is among the most cushy company in the world to work for with very high mean salaries, that there is an exodus there says a lot.
Facebook is also saying that its model: OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. They are also releasing their logbook detailing the infrastructure challenges they faced, along with code for experimenting with all of the released models.
Generous and environmentally friendly! Here to save the environment! With the Metaverse!
One Happy Metaverse Family
Facebook is however clear that they believe the entire AI community — academic researchers, civil society, policymakers, and industry — must work together to develop clear guidelines around responsible AI in general and responsible large language models in particular, given their centrality in many downstream language applications.
According to Yann, it’s even more than open-science.
Mike Schroepfer @schrep
Behind the scenes on what it takes to train a state of the art language model - 100 pages of raw notes released with the 175B parameter large language model OPT-175. There's a lot more trial and error than people like to admit in modern AI https://t.co/gcJt7x35fOMeta is releasing all our notes documenting the development process, including the full logbook detailing the day-to-day training process, so other researchers can more easily build on our work.
Access the open source code and small-scale pretrained models here, request access to OPT-175B here, and read the paper here.
Pretrained models are all licensed under the OPT-175B License Agreement.
Personal Note
My Mom is in the hospital with terminal cancer, so my cadence of articles might be significantly lower during this period.
If you want access to more articles join the 66 other paying subscribers and support the author. If you want to appear as a Sponsor in my Newsletter go here.
Create your profile
Only paid subscribers can comment on this post
Check your email
For your security, we need to re-authenticate you.
Click the link we sent to , or click here to sign in.