TheChek News Flow

This week, Google took the wraps off of Gemini, its new flagship generative AI model meant to power a range of products and services including Bard, Google’s ChatGPT competitor. In blog posts and press materials, Google touted Gemini’s superior architecture and capabilities, claiming that the model meets or exceeds the performance of other leading gen AI models like OpenAI’s GPT-4.

But the anecdotal evidence suggests otherwise.

A “lite” version of Gemini, Gemini Pro, began rolling out to Bard yesterday, and it didn’t take long before users began voicing their frustrations with it on X (formerly Twitter).

The model fails to get basic facts right, like 2023 Oscar winners:

Note that Gemini Pro claims incorrectly that Brendan Gleeson won Best Actor last year, not Brendan Fraser — the actual winner.

I tried asking the model the same question and, bizarrely, it gave a different wrong answer:

“Navalny,” not “All the Beauty and the Bloodshed,” won Best Documentary Feature last year; “All Quiet on the Western Front” won Best International Film; “Women Talking” won Best Adapted Screenplay; and “Pinocchio” won Best Animated Feature Film. That’s a lot of mistakes.

Science fiction author Charlie Stross found many more examples of confabulation in a recent blog post. (Among other mistruths, Gemini Pro said that Stross contributed to the Linux kernel; he never has.)

Translation doesn’t appear to be Gemini Pro’s strong suit, either. It struggles to give a six-letter word in French:

When I ran the same prompt through Bard (“Can you give me a 6-letters word in French?”), Gemini Pro responded with a seven-letter word instead of a five-letter one — which gives some credence to the reports about Gemini’s poor multilingual performance.

What about summarizing news? Surely Gemini Pro, with Google Search and Google News at its disposal, can give a recap of something topical? Not necessarily.

It seems Gemini Pro is loathe to comment on potentially controversial news topics, instead telling users to… Google it themselves.

I tried the same prompt and got a very similar response. ChatGPT, by contrast, gives a bullet-list summary with citations to news articles:

Interestingly, Gemini Pro did provide a summary of updates on the war in Ukraine when I asked it for one. However, the information was over a month out of date:

Google emphasized Gemini’s enhanced coding skills in a briefing earlier this week. Perhaps it’s genuinely improved in some areas — posts on X suggest as much. But it also appears that Gemini Pro struggles with basic coding functions like this one in Python:

And these:

And, as with all generative AI models, Gemini Pro isn’t immune to “jailbreaks” — i.e. prompts that get around the safety filters in place to attempt to prevent it from discussing controversial topics.

Using an automated method to algorithmically change the context of prompts until Gemini Pro’s guardrails failed, AI security researchers at Robust Intelligence, a startup selling model-auditing tools, managed to get Gemini Pro to suggest ways to steal from a charity and assassinate a high-profile individual (albeit with “nanobots” — admittedly not the most realistic weapon of choice).

Now, Gemini Pro isn’t the most capable version of Gemini — that model, Gemini Ultra, is set to launch sometime next year in Bard and other products. Google compared the performance of Gemini Pro to GPT-4’s predecessor, GPT-3.5, a model that’s around a year old.

But Google nevertheless promised improvements in reasoning, planning and understanding with Gemini Pro over the previous model powering Bard, claiming Gemini Pro was better at summarizing content, brainstorming and writing. Clearly, it has some work to do in those departments.

After months of build-up, developers were excited to finally see the long-awaited release of Google’s Gemini models yesterday. But their enthusiasm was tempered by the fact that the only model available right now is Gemini Nano, which you can guess means it’s small—so small it can run on Google’s Pixel phones. Developers have to wait until next week’s release of Gemini Pro, Google’s equivalent of GPT-3.5, to have a model worthy of testing, they told Jon Victor and me.

And that’s not the only reason to be cautious about coming to any conclusion about Gemini. While many developers were impressed by the multimodal capabilities Gemini showcased in a splashy video demo, replicating that sort of performance is likely to be harder than the video made it seem. (Leave aside the fact that those multimodal capabilities aren’t available yet.) In making the video, Google researchers didn’t prompt Gemini with normal speech and video visuals like hand gestures. Instead, they seem to have used a carefully crafted combination of images and text, according to this blog post. Some of the actual prompts were more specific and detailed than the dialogue in the video.

Google staffers must be feeling the pressure because they also took some additional steps to make Gemini look extra good compared to its rivals. Front and center on Gemini’s release site is a chart claiming that Gemini’s most-advanced model, Ultra, outperforms OpenAI’s GPT-4 by a large margin on a commonly-used evaluation, MMLU.

But that’s not an apples-to-apples comparison. GPT-4’s score of 86.4% was based on the industry standard for evaluation, called “5-shot.” However, Gemini Ultra’s result of 90% was based on a different method developed by Google researchers, one based on “chain-of-thought with 32 samples.” In simple terms, using this approach means that, for the same question, Gemini Ultra generates 32 answers and the reasoning for those answers. Then, the model chooses the most-commonly provided answer as its final one.

It could be that this new approach allows Gemini to “reason” through questions better. But using the industry standard—5-shot MMLU—for both models, GPT-4’s 86.4% outranked Gemini Ultra’s 83.7%.

Still, there’s been lots of discussion around how these model evaluations can be hacked, so we’ll be waiting for the release of Gemini Pro and Ultra to make our final verdict. Until then, take things with a grain of salt.—Stephanie Palazzolo and Jon Victor

Here’s what else is going on…

Watch Out for Model Hijacking

Over the past year, we’ve seen plenty of examples of large language models spitting out worrisome content, from encouraging users to harm themselves to providing them with instructions on how to build bombs. (Just yesterday, we featured a new security threat to LLMs.) These incidents have caught the eye of security experts and raised concerns around the safety of generative AI models.

As headline-grabbing as these examples are, though, there’s a much less sexy but equally dangerous security threat in AI: supply chain attacks. In plain English, supply chain attacks refer to a type of cyber attack that targets a vendor who sells services or software used by other companies to build their products.

Cybersecurity startup Legit Security recently uncovered a supply chain weakness in the popular open-source model repository Hugging Face, highlighting some of the often-overlooked downsides of using open-source AI.

Commonly, developers will use open-source models or datasets from the Hugging Face platform when building AI-powered products. When the owners of these models or datasets update or rename them, any products using the models or datasets automatically begin using the updated versions to prevent disruptions, said Liav Caspi, co-founder of Legit Security.

The problem arises when a developer registers a new model or dataset under the old name. When that happens, products can start accidentally using that model or dataset instead, Caspi explained. That allows attackers to create new malicious models that borrow names from legitimate ones in order to feed poisoned data or malware into users’ apps, he said. Legit Security used the hack to prove that hundreds of thousands of developer apps that relied on the “ai-forever” models could quickly be corrupted, he said.

Legit Security dubbed this bug “AIJacking.” And when Caspi’s team alerted Hugging Face to the bug, Hugging Face said it had a mechanism that automatically retired old names of the most popular models and datasets once developers renamed them. That leaves Hugging Face to manually retire less-popular models, which Caspi worried isn’t a scalable solution. A spokesperson for Hugging Face declined to comment.

The incident shows that while regulators and doomers worry about AI ending humanity, or terrorists creating bioweapons with open-source models, a variety of more immediate security threats looms right now.

Developers can guard against AIJacking attacks by referencing a specific version of a model hosted by Hugging Face, and preventing their apps from automatically executing code from third-party vendors like Hugging Face, Caspi said.

The AIJacking flaw is also a stark reminder of the dangers of building software on top of young and relatively untested startups, as we saw with the controversy surrounding OpenAI in recent weeks. That debacle prompted a number of customers to consider switching to OpenAI models offered by a tech veteran like Microsoft instead.

In his quarterly letter to designers and brands, Saks CEO Marc Metrick looks to dispel murmurings of stalled payments to vendors.

Amid reports of delayed payments, Saks chief executive officer Marc Metrick is telling vendors not to fret.

“Saks, which is managed separately from the SFA Stores and under its own credit group, has sufficient liquidity and a sound balance sheet,” Metrick wrote in his quarterly letter to vendors updating the Saks business.

“As we carefully manage our business against the industrywide softness in the U.S. luxury market, we are confident in our ability to continue meeting our financial obligations,” Metrick wrote.

Metrick’s letter underscored that Saks’ parent company, HBC, recently revealed the completion of transactions raising about $340 million in cash through planned real estate monetizations in the U.S. and Canada, as previously reported by WWD. The transactions, Metrick wrote, “significantly enhances HBC’s liquidity position.” HBC wholly owns SFA, the company that operates the Saks Fifth Avenue stores, and Hudson’s Bay in Canada. HBC is also the majority owner of Saks, the company that operates saks.com.

In addition, as Metrick wrote in his letter, “HBC is the owner and developer of an approximately $7 billion portfolio of real estate throughout North America, which consistently generates $300 million to $500 million in cash per year. This valuable asset base and the incremental liquidity it generates strengthens HBC’s operating businesses, including the SFA stores.”

Reports of SFA and Saks.com delaying payments to vendors started circulating weeks ago, as have media reports that HBC this season has been negotiating hard to purchase the Neiman Marcus Group. It’s believed that HBC is getting closer to striking a deal, though there is no guarantee, and talks have been on and off for years.

According to one source close to HBC, “SFA and Hudson’s Bay are being managed as one credit group and managing through a difficult moment. It’s one credit profile,” the source said. “Store payables remain tied to HBC. Managing that cash position has a residual impact on SFA stores. There are actions being taken in Canada. Business is very difficult in Canada.” Hudson’s Bay, which operates more than 80 stores, has not recovered from the pandemic as well as stores in the U.S.

With regards to saks.com, “there is some derivative impact on the U.S. causing a lot of noise, but that’s all it is,” the source said. Payment terms vary depending on the brand. Asked if any brands decided to drop Saks, the source said, “Saks has not experienced any kind of flight.”

“HBC is getting a lot more thoughtful on how they want to conserve cash,” said another source, who suggested HBC has been seeking to narrow the time between when payables are due and when goods are received. “Ninety-five percent of the vendors in Canada are fine. There’s another five percent where the model is broken,” said the source.

In an exclusive interview, Metrick told WWD, “We have done a really good job of managing inventory. We are in a healthy inventory position. We expect margins for the year to be better.” He declined comment on the speculation regarding payables.

He did say that saks.com “really put emphasis in 2023 moving into that ‘second horizon.’ In 2021 business was all about the top line, customer acquisition and really about scale and driving the business. As we moved into the ‘second horizon’ in the summer of 2022, we started enhancing our cash flow and profitability profile, and did work around costs, efficiencies, inventory management, while not giving back the scale, but getting that free cash flow level and profitability level vastly improved, even with the top line down. We feel very good that our profitability levels are vastly improved on a percent of sales basis and on an absolute basis. We are seeing gains versus 2022 and 2021.”

Traffic on the Saks website, based on 16 million visits that occurred from Thanksgiving through the Tuesday after, was “flattish with last year, maybe slightly above, which is a good indication of what’s happening. It was sort of a nice outcome,” Metrick said.

Citing the latest Saks Luxury Pulse survey, Metrick said 38 percent of respondents indicated they will shop after Thanksgiving, compared to 28 percent last year. “People are shopping later. We’re not a big Black Friday business.” He said saks.com has been no more promotional than last year.

He also cited stronger average unit retail prices, which he attributed to “rationalizing” the assortment beginning last summer, looking at the unit economies of products, and targeting higher lifetime value customers. “That doesn’t necessarily mean rich people. It’s people who have a propensity to grow and spend with us over their lifetime.” Average order value was up 6 percent to last year, driven by stronger average unit retail, he said.

Business has been tough. As Metrick wrote, “The current softness across the U.S. luxury market has created a turbulent environment, and Saks is working to be nimble while staying true to our strategy. We believe that when it comes to luxury, we have to play the long game rather than react to this moment in time. With that, we continue to take a long-term view on our growth prospects.…Results have been exacerbated by the macroeconomic environment and softening in the U.S. luxury market.”

The letter indicated that in third-quarter 2023, Saks’ gross merchandising value fell 19 percent compared to the same quarter in 2022. However, on a two-year stack, third-quarter 2023 GMV fell slightly, though is still 75 percent above third-quarter 2019. “In Q3 2023, we were pleased by our ability to generate a double-digit increase in AOV [average order value] despite the consumer’s ongoing focus on value,” Metrick wrote. “Top-performing categories on saks.com in Q3 2023 were women’s apparel, jewelry and accessories.”

At the Saks Fifth Avenue stores, GMV fell 13 percent last quarter from the same quarter a year ago, but is above 2019 levels, according to the letter. Top-performing categories were women’s ready-to-wear, beauty and jewelry, and top-performing stores were the Manhattan flagship, and the Naples, Sarasota and Palm Beach locations in Florida.

No profit figures were disclosed, though Metrick wrote, “We are pleased with our cost management and gross profit, which has resulted in significantly improved profitability versus last year, despite the more challenging top line. We entered the critical holiday period with our inventory aligned with sales, which we expect to drive an ongoing improvement in gross profit and gross margin performance.”

Metrick noted that customers who have purchased from Saks at least once before represent 73 percent of the retailer’s total customer base, a 6 percentage points increase from the year-ago quarter and a 10 point gain over third-quarter 2021. Year-to-date, existing customers are up 8 percent to last year.

Among the highlights of the latest quarter, Brunello Cucinelli launched his first fragrances, licensed to Euroitalia, at Saks; the Saks Fifth Avenue flagship for holiday launched a “World of Dior” experience, marking the first time all categories of Dior are available online anywhere other than dior.com and the first time Saks Fifth Avenue collaborated with a luxury fashion house for its annual holiday initiative, and Saks’ chief merchandising officer Tracy Margolies was inducted into the Footwear News Hall of Fame. A Saks Fifth Avenue-branded collection was launched benefiting the Saks Fifth Avenue Foundation’s ongoing mission to support mental health initiatives, and saks.com introduced more visually-engaging product detail pages, accelerated load times and added an AI chatbot.

In early 2024, Saks Fifth Avenue will relocate its Beverly Hills store into the former Barneys New York on Wilshire Boulevard. SFA has invested $52 million in the project. SFA is also renovating its women’s store in Boston and the men’s floors in the Manhattan flagship and Atlanta store.

Next spring, Saks plans to launch a retail media network, using first-party data to help brands reach Saks customers with sponsored product features on Saks.com.

On the outlook for next year, Metrick told WWD, “We are taking a pretty measured approach to 2024. It’s an election year, which is certainly distracting. I don’t see the macro headwinds dissipating in the next six months. If you are watching the market, you could start to see rates get cut. There are a lot of unknowns, but the beautiful thing is we have our strategy. Strategically, we are focused on luxury. This segment is growing. It’s a customer that will be there when we need them to be there. It is a very resilient consumer. We don’t worry about next year or the year after. We are focused on the long game.”

Liz Magill says in video that calls for genocide of Jewish people would be considered harassment and intimidation in her view

University of Pennsylvania President Liz Magill backtracked from comments she made during a congressional hearing addressing antisemitism on college campuses after facing an outpouring of criticism over her remarks.

Lawmakers questioned Magill and the presidents of Harvard University and the Massachusetts Institute of Technology on Tuesday about their responses to protests against the Israel-Hamas war, sometimes featuring antisemitic rhetoric, that have led to rising tensions on their campuses. During one exchange, the presidents were asked if calling for the genocide of Jewish students would violate the policies of the schools. Magill said it depended on the context.

“In that moment, I was focused on our university’s longstanding policies aligned with the U.S. Constitution, which say that speech alone is not punishable,” Magill said in a video message released Wednesday night. “I was not focused on, but I should have been, the irrefutable fact that a call for genocide of Jewish people is a call for some of the most terrible violence human beings can perpetrate.”

Magill said in the video calls for genocide of Jewish people would be considered harassment and intimidation in her view.

“A call for genocide of Jewish people is threatening, deeply so,” Magill said. “It is intentionally meant to terrify a people who have been subjected to pogroms and hatred for centuries and were the victims of mass genocide in the Holocaust.”

Magill said Penn would immediately evaluate and clarify the university’s policies on this matter. A spokesman for Penn didn’t immediately return a request for comment.

Magill’s new statement comes after a wave of criticism of her congressional testimony from Penn’s alumni, elected officials and others. Pennsylvania Gov. Josh Shapiro, a Democrat, said Wednesday Magill’s initial testimony was “absolutely shameful” and called on Penn’s board of trustees to meet to discuss her future. Magill had already faced calls for her resignation over how Penn has handled growing antisemitism on campus following the Hamas attack in Israel on Oct. 7.