AI2 is developing a large language model optimized for science
Most of these models are walled behind APIs, making it impossible for researchers to see exactly what makes them tick. But increasingly, community efforts are yielding open source AI that’s as sophisticated, if not more so, than their commercial counterparts.
The latest of these efforts is the Open Language Model, a large language model set to be released by the nonprofit Allen Institute for AI Research (AI2) sometime in 2024. Open Language Model, or OLMo for short, is being developed in collaboration with AMD and the Large Unified Modern Infrastructure consortium, which provides supercomputing power for training and education, as well as Surge AI and MosaicML (which are providing data and training code).
“The research and technology communities need access to open language models to advance this science,” Hanna Hajishirzi, the senior director of NLP research at AI2, told TechCrunch in an email interview. “With OLMo, we are working to close the gap between public and private research capabilities and knowledge by building a competitive language model.”
One might wonder — including this reporter — why AI2 felt the need to develop an open language model when there’s already several to choose from (see Bloom, Meta’s LLaMA, etc.). The way Hajishirzi sees it, while the open source releases to date have been valuable and even boundary-pushing, they’ve missed the mark in various ways.
AI2 sees OLMo as a platform, not just a model — one that’ll allow the research community to take each component AI2 creates and either use it themselves or seek to improve it. Everything AI2 makes for OLMo will be openly available, Hajishirzi says, including a public demo, training data set and API, and documented with “very limited” exceptions under “suitable” licensing.
“We’re building OLMo to create greater access for the AI research community to work directly on language models,” Hajishirzi said. “We believe the broad availability of all aspects of OLMo will enable the research community to take what we are creating and work to improve it. Our ultimate goal is to collaboratively build the best open language model in the world.”
OLMo’s other differentiator, according to Noah Smith, senior director of NLP research at AI2, is a focus on enabling the model to better leverage and understand textbooks and academic papers as opposed to, say, code. There’s been other attempts at this, like Meta’s infamous Galactica model. But Hajishirzi believes that AI2’s work in academia and the tools it’s developed for research, like Semantic Scholar, will help make OLMo “uniquely suited” for scientific and academic applications.
“We believe OLMo has the potential to be something really special in the field, especially in a landscape where many are rushing to cash in on interest in generative AI models,” Smith said. “AI2’s unique ability to act as third party experts gives us an opportunity to work not only with our own world-class expertise but collaborate with the strongest minds in the industry. As a result, we think our rigorous, documented approach will set the stage for building the next generation of safe, effective AI technologies.”
That’s a nice sentiment, to be sure. But what about the thorny ethical and legal issues around training — and releasing — generative AI? The debate’s raging around the rights of content owners (among other affected stakeholders), and countless nagging issues have yet to be settled in the courts.
To allay concerns, the OLMo team plans to work with AI2’s legal department and to-be-determined outside experts, stopping at “checkpoints” in the model-building process to reassess privacy and intellectual property rights issues.
“We hope that through an open and transparent dialogue about the model and its intended use, we can better understand how to mitigate bias, toxicity, and shine a light on outstanding research questions within the community, ultimately resulting in one of the strongest models available,” Smith said.
What about the potential for misuse? Models, which are often toxic and biased to begin with, are ripe for bad actors intent on spreading disinformation and generating malicious code.
Hajishirzi said that AI2 will use a combination of licensing, model design and selective access to the underlying components to “maximize the scientific benefits while reducing the risk of harmful use.” To guide policy, OLMo has an ethics review committee with internal and external advisors (AI2 wouldn’t say who, exactly) that’ll provide feedback throughout the model creation process.
We’ll see to what extent that makes a difference. For now, a lot’s up in the air — including most of the model’s technical specs. (AI2 did reveal that it’ll have around 70 billion parameters, parameters being the parts of the model learned from historical training data.) Training’s set to begin on LUMI’s supercomputer in Finland — the fastest supercomputer in Europe, as of January — in the coming months.
AI2 is inviting collaborators to help contribute to — and critique — the model development process.
SoftBank Says Goodbye to Alibaba, Hello to More AI Investments
Japanese tech investor praises ChatGPT, says time is ripe to go on offense again
TOKYO—Japanese technology investor SoftBank Group 9984 -0.85%decrease; red down pointing triangle said it was getting ready to go on the offensive again with its eye on investing in artificial-intelligence companies, a buzzy area it has long targeted but failed to capitalize on.
One of the world’s most influential tech investors, SoftBank has been in a defensive crouch for over a year, slashing its once profligate spending after many of its investments in startups went sour during the recent tech downturn.
The company on Thursday reported a smaller net loss in the quarter ended March of 57.6 billion yen, equivalent to around $430 million. That is far less than the $15.7 billion loss a year earlier. The company also signaled a shift in direction in its aggressiveness.
“We are getting ready to go on the offensive with the AI revolution on the horizon,” said SoftBank Chief Financial Officer Yoshimitsu Goto. Shares in some SoftBank-backed tech companies have begun to rebound, he said.
Giving it fresh financial firepower, SoftBank cashed out almost its entire stake in Chinese e-commerce company Alibaba Group in recent months.
The promise of AI to transform businesses has gained extraordinary attention in boardrooms and marketing departments alike thanks to the introduction of ChatGPT, the artificial-intelligence chatbot developed by Microsoft-backed research company OpenAI.
Mr. Goto spent a good portion of his presentation praising ChatGPT. He said SoftBank Chief Executive Masayoshi Son—who has stopped attending his company’s quarterly financial presentations—has been dedicating time to studying SoftBank’s approach to artificial intelligence and working on plans for the listing of its chip-design unit, Arm.
“AI has finally come, especially generative AI,” Mr. Goto said. SoftBank’s chief executive “is as excited today as when he launched the company,” he said.
The renewed embrace of the technology comes after Mr. Son has spent the better part of a decade pledging that SoftBank was fully focused on the forthcoming “AI revolution.” He said AI was the unifying theme of his $100 billion Vision Fund startup investment vehicle launched in 2017.
But those investments largely went to companies without obvious grounding in AI such as office-space company WeWork, hotel company Oyo and an array of ride-hailing apps. Meanwhile, SoftBank didn’t invest in OpenAI. Mr. Goto declined to comment to reporters why it didn’t.
SoftBank is turning to AI as it says goodbye to Alibaba, the most successful investment in Mr. Son’s more than four-decade career. He backed Alibaba in its infancy in 2000, and as recently as mid-2021 SoftBank’s stake in the Chinese company was worth the equivalent of nearly $100 billion.
In recent years, SoftBank has cashed in its Alibaba holdings through financing deals that handed it money upfront while leaving it the option of keeping the shares later. Ultimately it has given up many of the shares outright, and Mr. Goto said virtually all of SoftBank’s remaining Alibaba shares have been used to procure funds.
Over the past year, SoftBank’s stake in Alibaba has helped ride out losses in its two Vision Funds. After posting a record loss equivalent to more than $20 billion in the April-June quarter, SoftBank reported an equally large profit in the quarter ended in September thanks to profits from Alibaba transactions.
Overall, for the year ended March 31, SoftBank lost $7.2 billion, smaller than its $15.6 billion loss a year earlier.
Those losses are due largely to pain in the company’s Vision Fund unit, as the company continues to gradually write down the value of the hundreds of privately held startups it owns stakes in. While public tech stocks have recently rebounded some, SoftBank reported $3.9 billion in losses in the private holdings of the unit for the quarter.
Its $50 billion Vision Fund 2—which invested in scores of companies including online lender Klarna and now bankrupt cryptocurrency company FTX— has lost over $18 billion, based on those lowered valuations. Some analysts expect the valuations to fall further.
In its latest results out Thursday, the company said it had cut new investments in startups to about $3 billion in the year ended in March, less than a tenth of its investments the previous fiscal year.
Mr. Goto said SoftBank’s unwinding of its stake in Alibaba was part of broader efforts to diversify its investments across companies and regions of the world. Mr. Goto said the company has put more priority on investing in the U.S. and Europe as it looks to minimize geopolitical risks related to U.S.-China frictions and the war in Ukraine.
China accounted for 14% of the equity value of SoftBank’s holdings as of March, down from 50% two years earlier, driven by the Alibaba unwinding.
Mr. Goto said Mr. Son was excited at how ChatGPT achieved 100 million monthly average users in just two months.
“I spend time with him every day,” Mr. Goto said of his boss, “and I try not to be overwhelmed by him too much.”
Earlier this week, Junichi Miyakawa, president of SoftBank’s mobile unit, said he had set up a team to work on launching a Japanese version of ChatGPT. Mr. Miyakawa said Mr. Son had recently given a presentation to group engineers on ways to use the AI chatbot.
Germany sets stage for G7 clash with push to endorse gas
Berlin at odds with UK and France over support for public investment in fossil fuel despite climate impact
Germany is pushing for G7 leaders to endorse public investment in the gas sector at their summit next week, setting up a clash with countries that argue such support is incompatible with global climate goals.
Tensions have flared in pre-summit discussions as nations including the UK and France reject Germany’s demands to include support for public investments in gas in the meeting’s final document, said several people briefed on the talks.
Environmentalists argue G7 countries should lead a global shift away from fossil fuels at the Japan summit on May 19-21 and avoid weakening commitments reached last month among G7 environment ministers.
Russia’s full-scale invasion of Ukraine last year and its subsequent move to drastically reduce gas supplies to Europe plunged Germany and other big consumers of Russian hydrocarbons into an energy crisis from which they are only now recovering.
Berlin’s response was to build liquefied natural gas terminals on its northern coast, supported by huge public subsidies, and to scour the world to lock up alternative supplies of gas.
Germany insisted on wording in the statement from last year’s G7 summit in Schloss Elmau, Bavaria, that conceded the necessity of public investment in gas — a move that led environmental groups to accuse the G7 of “backsliding” on its climate goals.
The 2022 statement said that in the “exceptional circumstances” created by Russia’s war and its halt to energy supplies, “publicly supported investment in the gas sector can be appropriate as a temporary response”.
“Investment in this sector is necessary in response to the current crisis,” it added.
German officials want this year’s communiqué from Hiroshima to include similar wording. “We need to use gas as a transitional source of energy,” one said. “We can’t overlook the fact that things have changed [with the war in Ukraine] and the overall supply of gas is scarce.”
But countries including the UK and France have argued that last year’s support for gas was meant to be temporary, saying Germany has already built the LNG terminals it needs, according to people familiar with the discussions.
Germany has insisted its gas investments are compatible with its climate goals because the LNG terminals it is building can be repurposed to receive hydrogen, which is cleaner than gas when burnt, but whose production typically involves significant use of fossil fuels. So-called green hydrogen, made using renewable energy, has not been developed at scale.
Petter Lydén, head of international climate policy at Germanwatch, a non-profit group, called on G7 leaders “to show their commitment to phasing out fossil fuels”.
“Countries like Germany have already successfully found ways of reducing their dependence on gas, and any signal that more gas is needed goes directly against the known needs,” he said, adding that any reversal on last month’s G7 meeting was “unacceptable”.
In March, a coalition of industry bodies, including the American Petroleum Institute, the Asia Natural Gas & Energy Association, Eurogas and the US Chamber of Commerce, wrote to the leaders of the G7, urging them to continue to back investments in LNG.
At the same time, climate activists have been lobbying heavily for the G7 to roll back its support for gas, arguing that it harms the environment and communities that live near fracking and drilling sites.
The EU’s decision to label gas as “green” under its financial taxonomy rules after German lobbying has sparked legal challenges from environmental groups and member state governments.
Gapping down
In reaction to earnings/guidance:
- SONO -23.2%, CDNA -21.4% (also files mixed shelf securities offering), AMRC -18.6%, MXCT -10.8%, DM -10% (also files for 444,793 share offering by selling shareholders), VUZI -6.5%, NTR -5.9%, DIS -5.5%, BLFS -3.7% (also exploring strategic alternatives for CBS and Stirling), JBI -3.7%, RBA -3.4%, CYBR -3.2%, MARA -2.7%, BYND -2.6% (also files mixed shelf securities offering), GDEN -2.5%, COOK -2.3%, PAR -2.2%, HPK -2.1%, NDLS -2.1%, MFC -1.8%, TAK -1.7%, DOX -1.4%, CDE -1.4%, HAE -1.3%, SOVO -1.1%, UTZ -1.1%, RPAY -1%, HIMX -1%, NUVL -1%, PKI -0.9%, RIOT -0.8%
Other news:
- WW -12.7% (reports Artal Group S.A. has sold its remaining stake of WW common stock)
- HLMN -6.6% (prices offering of 22.455 mln shares of common stock by selling shareholders)
- BE -6.4% (proposed offering of $500 mln in green convertible senior notes due 2028)
- TSLX -5.6% (priced a public offering of 4500000 shares of its common stock for total gross proceeds of ~$79.2 mln)
- REXR -3.1% (prices offering of 13.5 mln shares of common stock at $55.60 per share)
- BELFB -1.7% (files $250 mln mixed shelf securities offering)
- IRT -1.5% (increases dividend)
- CDZI -1.3% (stock offering by selling shareholders)
- HIMX -1% (declares cash dividend of $0.48 per ADS equivalent to $0.24 per ordinary share for FY22)
- SST -0.9% (to delay 10-Q filing)
Analyst comments:
- LXU -2.8% (downgraded to Neutral from Overweight at Piper Sandler)
- MOS -2.1% (downgraded to Neutral from Overweight at Piper Sandler)
- CF -1.9% (downgraded to Neutral from Overweight at Piper Sandler)
- NOC -1% (downgraded to Equal Weight from Overweight at Barclays)
Gapping up
In reaction to earnings/guidance:
- APP +16%, MGNI +15.8%, PRPL +14.8%, MAXN +12.1%, FLNC +10.5%, FLWS +10%, TPR +9.3%, U +9.1%, NXT +8.3%, ALGM +7.4%, HLLY +6.9%, TNK +6.5%, FVRR +5.6%, OSUR +5.5%, OLK +4.5%, DV +4.3%, NICE +4.1%, CPA +4%, TTD +3.7% (also CFO to step down, names new CFO), HOOD +3.6%, DXPE +3.6%, USFD +3.4%, YETI +3.2%, MATV +3.1%, JD +3%, DCBO +3%, NVMI +2.9%, ALRM +2.8%, ING +2.8%, STE +2.7%, DNUT +2.7%, ACVA +2.5%, ENTG +2.4%, CPRX +2.3%, KELYA +2.3%, DYN +2%, ORCC +1.8%, DDS +1.4%, CR +1.3%, HMC +1.1%, EYE +1.1%, BN +0.9%
Other news:
- HMPT +15.7% (COOP to acquire HMPT)
- VVV +3% (commences tender offer to repurchase up to $1.0 billion in value of its common stock at a price not greater than $40.00/share nor less than $35.00/share)
- COOP +2.3% (COOP to acquire HMPT)
- ADEA +2.1% (TVStorm renews license for Adeia's media IP portfolio)
- INTT +1.5% (initiated an at-the-market equity offering program for the sale of up to $20 mln of the Company's common stock)
Analyst comments:
- ARCT +2.7% (upgraded to Buy from Neutral at H.C. Wainwright)
- AA +1.6% (upgraded to Outperform from Neutral at Credit Suisse)
- IGT +1.5% (upgraded to Positive from Neutral at Susquehanna)
- LTHM +1.5% (upgraded to Overweight from Sector Weight at KeyBanc Capital Markets)
