FT : Anthropic makes ‘jailbreak’ advance to stop AI models producing harmful res

Anthropic makes ‘jailbreak’ advance to stop AI models producing harmful results
Leading tech groups including Microsoft and Meta also invest in similar safety systems

Artificial intelligence start-up Anthropic has demonstrated a new technique to prevent users from eliciting harmful content from its models, as leading tech groups including Microsoft and Meta race to find ways that protect against dangers posed by the cutting-edge technology.

In a paper released on Monday, the San Francisco-based start-up outlined a new system called “constitutional classifiers”. It is a model that acts as a protective layer on top of large language models such as the one that powers Anthropic’s Claude chatbot, which can monitor both inputs and outputs for harmful content.

The development by Anthropic, which is in talks to raise $2bn at a $60bn valuation, comes amid growing industry concern over “jailbreaking” — attempts to manipulate AI models into generating illegal or dangerous information, such as producing instructions to build chemical weapons.

Other companies are also racing to deploy measures to protect against the practice, in moves that could help them avoid regulatory scrutiny while convincing businesses to adopt AI models safely. Microsoft introduced “prompt shields” last March, while Meta introduced a prompt guard model in July last year, which researchers swiftly found ways to bypass but have since been fixed.

Mrinank Sharma, a member of technical staff at Anthropic, said: “The main motivation behind the work was for severe chemical [weapon] stuff [but] the real advantage of the method is its ability to respond quickly and adapt.”

Anthropic said it would not be immediately using the system on its current Claude models but would consider implementing it if riskier models were released in future. Sharma added: “The big takeaway from this work is that we think this is a tractable problem.”

The start-up’s proposed solution is built on a so-called “constitution” of rules that define what is permitted and restricted and can be adapted to capture different types of material.

Some jailbreak attempts are well-known, such as using unusual capitalisation in the prompt or asking the model to adopt the persona of a grandmother to tell a bedside story about a nefarious topic.

To validate the system’s effectiveness, Anthropic offered “bug bounties” of up to $15,000 to individuals who attempted to bypass the security measures. These testers, known as red teamers, spent more than 3,000 hours trying to break through the defences.

Anthropic’s Claude 3.5 Sonnet model rejected more than 95 per cent of the attempts with the classifiers in place, compared to 14 per cent without safeguards.

Leading tech companies are trying to reduce the misuse of their models, while still maintaining their helpfulness. Often, when moderation measures are put in place, models can become cautious and reject benign requests, such as with early versions of Google’s Gemini image generator or Meta’s Llama 2. Anthropic said their classifiers caused “only a 0.38 per cent absolute increase in refusal rates”.

However, adding these protections also incurs extra costs for companies already paying huge sums for computing power required to train and run models. Anthropic said the classifier would amount to a nearly 24 per cent increase in “inference overhead”, the costs of running the models.


Security experts have argued that the accessible nature of such generative chatbots has enabled ordinary people with no prior knowledge to attempt to extract dangerous information.

“In 2016, the threat actor we would have in mind was a really powerful nation-state adversary,” said Ram Shankar Siva Kumar, who leads the AI red team at Microsoft. “Now literally one of my threat actors is a teenager with a potty mouth.”

FT : Thames Water faces fresh opposition to emergency fundraising plan

Thames Water faces fresh opposition to emergency fundraising plan
Creditors criticise “ransom terms” on up to £3bn in funding as the utility faces running out of cash

Thames Water faced mounting opposition in its attempt to win court approval for an expensive £3bn loan on Monday after the judge agreed to hear “public interest” objections from environmental campaigners and another group of creditors sought to block the plan.

A group of bondholders and lenders represented by the regulated utility’s parent company — Thames Water Limited — released court documents opposing the emergency funding for the first time. They argued that the “ransom terms” on the funding would allow lenders to make a “handsome return” on an “essentially risk-free” deal, which would give them “effective control” over any future restructuring.

The intervention came at the start of a crowded four-day court hearing, in which Britain’s biggest water utility is seeking approval to take out up to £3bn in loans from its so-called class A bondholders, which include US hedge funds such as Elliott Management, to stave off an imminent cash crunch.

The proposed loan carries a hefty 9.75 per cent annual interest rate, as well as further fees and onerous terms that critics argue will hand the lenders greater control over the company.

London’s High Court on Monday also allowed Charlie Maynard, a Liberal Democrat MP representing environmental campaigners in the Oxfordshire constituency of Witney, to speak for the “public interest and the consumer interest” in considering Thames Water’s restructuring proposal. 

In documents submitted to court, Maynard argued that Thames Water’s restructuring plan was a “poor, short-term fix” that “aggravates rather than mitigates the Thames Water debt doom loop”.

Without the loan, Thames Water says it will run out of cash on March 24 and risk crashing into the government’s special administration regime, a form of temporary renationalisation. This process would allow services to keep running while the debt is frozen ahead of a potential restructuring and sale of the business, or a full nationalisation. 

The loan proposal has led to an increasingly bitter spat between the company and its lower-ranking class B bondholders, who claim that the utility has not properly considered their rival offer, which they say comes at a lower cost and with less restrictive terms.

These class B lenders, who could face near-total losses if the company’s proposed restructuring went ahead, have argued that they would fare better under a special administration than under the company’s plan.

They argued in written submissions to court that the planned loan from class A creditors had “inbuilt control mechanics” that were typical of “an aggressive loan-to-own strategy adopted by distressed [debt] hedge funds”. 

Thames Water has said that the loan is a necessary bridge to a wider restructuring, which will give it time to raise equity from new investors and renegotiate its debts.

Julian Gething, Thames Water chief restructuring officer, said: “Our plan remains the only implementable solution to putting the business on a firmer financial footing, and approval will not affect customer bills, but it will unlock billions of pounds for investment.”

Maynard argued in his written submission that the class A loan “provides a bridge to nowhere” and the “better and fairer course would be a special administration”. 

A judgment is not expected for at least a week after the hearing is scheduled to end on Thursday.

>>> US Early premarket gappers

Early premarket gappers

Gapping up:
TGI +35.6%, SSYS +10.6%, SHEN +6.3%, PERI +3.8%, NVX +2.1%, WF +1.5%
Gapping down:
PLBY -6.1%, LI -4.9%, SPR -3.8%, FLR -3.5%, AVA -3.3%, ZK -2.8%, IP -2.6%, CRGY -2.4%, PCT -2.1%, NIO -2.1%, IWM -2.1%, XPEV -2%, SNY -1.9%, UTL -1.8%, QQQ -1.5%, AZN -1.4%, SPY -1.3%, DIA -1.2%, SPHR -1.2%, LMT -1.1%

Wired: The Young, Inexperienced Engineers Aiding Elon Musk’s Government Takeover

The Young, Inexperienced Engineers Aiding Elon Musk’s Government Takeover
Engineers between 19 and 24, most linked to Musk’s companies, are playing a key role as he seizes control of federal infrastructure.

Elon Musk’s takeover of federal government infrastructure is ongoing, and at the center of things is a coterie of engineers who are barely out of—and in at least one case, purportedly still in—college. Most have connections to Musk, and at least two have connections to Musk’s longtime associate Peter Thiel, a cofounder and chair of the analytics firm and government contractor Palantir who has long expressed opposition to democracy.

WIRED has identified six young men—all apparently between the ages of 19 and 24, according to public databases, their online presences, and other records—who have little to no government experience and are now playing critical roles in Musk’s so-called Department of Government Efficiency (DOGE) project, tasked by executive order with “modernizing Federal technology and software to maximize governmental efficiency and productivity.” The engineers all hold nebulous job titles within DOGE, and at least one appears to be working as a volunteer.

The engineers are Akash Bobba, Edward Coristine, Luke Farritor, Gautier Cole Killian, Gavin Kliger, and Ethan Shaotran. None have responded to requests for comment from WIRED. Representatives from OPM, GSA, and DOGE did not respond to requests for comment.

Already, Musk’s lackeys have taken control of the Office of Personnel Management (OPM) and General Services Administration (GSA), and have gained access to the Treasury Department’s payment system, potentially allowing him access to a vast range of sensitive information about tens of millions of citizens, businesses, and more. On Sunday, CNN reported that DOGE personnel attempted to improperly access classified information and security systems at the US Agency for International Development and that top USAID security officials who thwarted the attempt were subsequently put on leave. The Associated Press reported that DOGE personnel had indeed accessed classified material.

“What we're seeing is unprecedented in that you have these actors who are not really public officials gaining access to the most sensitive data in government,” says Don Moynihan, a professor of public policy at the University of Michigan. “We really have very little eyes on what's going on. Congress has no ability to really intervene and monitor what's happening because these aren't really accountable public officials. So this feels like a hostile takeover of the machinery of governments by the richest man in the world.”

Bobba has attended UC Berkeley, where he was in the prestigious Management, Entrepreneurship, and Technology program. According to a copy of his now-deleted LinkedIn obtained by WIRED, Bobba was an investment engineering intern at the Bridgewater Associates hedge fund as of last spring and was previously an intern at both Meta and Palantir. He was a featured guest on a since-deleted podcast with Aman Manazir, an engineer who interviews engineers about how they landed their dream jobs, where he talked about those experiences last June.

Coristine, as WIRED previously reported, appears to have recently graduated from high school and to have been enrolled at Northeastern University. According to a copy of his résumé obtained by WIRED, he spent three months at Neuralink, Musk’s brain-computer interface company, last summer.e

Both Bobba and Coristine are listed in internal OPM records reviewed by WIRED as “experts” at OPM, reporting directly to Amanda Scales, its new chief of staff. Scales previously worked on talent for xAI, Musk’s artificial intelligence company, and as part of Uber’s talent acquisition team, per LinkedIn. Employees at GSA tell WIRED that Coristine has appeared on calls where workers were made to go over code they had written and justify their jobs. WIRED previously reported that Coristine was added to a call with GSA staff members using a nongovernment Gmail address. Employees were not given an explanation as to who he was or why he was on the calls.

Farritor, who per sources has a working GSA email address, is a former intern at SpaceX, Musk’s space company, and currently a Thiel Fellow after, according to his LinkedIn, dropping out of the University of Nebraska—Lincoln. While in school, he was part of an award-winning team that deciphered portions of an ancient Greek scroll.

Kliger, whose LinkedIn lists him as a special adviser to the director of OPM and who is listed in internal records reviewed by WIRED as a special adviser to the director for information technology, attended UC Berkeley until 2020; most recently, according to his LinkedIn, he worked for the AI company Databricks. His Substack includes a post titled “The Curious Case of Matt Gaetz: How the Deep State Destroys Its Enemies,” as well as another titled “Pete Hegseth as Secretary of Defense: The Warrior Washington Fears.”

Killian, also known as Cole Killian, has a working email associated with DOGE, where he is currently listed as a volunteer, according to internal records reviewed by WIRED. According to a copy of his now-deleted résumé obtained by WIRED, he attended McGill University through at least 2021 and graduated high school in 2019. An archived copy of his now-deleted personal website indicates that he worked as an engineer at Jump Trading, which specializes in algorithmic and high-frequency financial trades.

Shaotran told Business Insider in September that he was a senior at Harvard studying computer science and also the founder of an OpenAI-backed startup, Energize AI. Shaotran was the runner-up in a hackathon held by xAI, Musk’s AI company. In the Business Insider article, Shaotran says he received a $100,000 grant from OpenAI to build his scheduling assistant, Spark.

Got a Tip?

Are you a current or former employee with the Office of Personnel Management or another government agency impacted by Elon Musk? We’d like to hear from you. Using a nonwork phone or computer, contact Vittoria Elliott at vittoria_elliott@wired.com or securely at velliott88.18 on Signal.

“To the extent these individuals are exercising what would otherwise be relatively significant managerial control over two very large agencies that deal with very complex topics,” says Nick Bednar, a professor at University of Minnesota’s school of law, “it is very unlikely they have the expertise to understand either the law or the administrative needs that surround these agencies.”

Sources tell WIRED that Bobba, Coristine, Farritor, and Shaotran all currently have working GSA emails and A-suite level clearance at the GSA, which means that they work out of the agency’s top floor and have access to all physical spaces and IT systems, according a source with knowledge of the GSA’s clearance protocols. The source, who spoke to WIRED on the condition of anonymity because they fear retaliation, says they worry that the new teams could bypass the regular security clearance protocols to access the agency’s sensitive compartmented information facility, as the Trump administration has already granted temporary security clearances to unvetted people.

This is in addition to Coristine and Bobba being listed as “experts” working at OPM. Bednar says that while staff can be loaned out between agencies for special projects or to work on issues that might cross agency lines, it’s not exactly common practice.

“This is consistent with the pattern of a lot of tech executives who have taken certain roles of the administration,” says Bednar. “This raises concerns about regulatory capture and whether these individuals may have preferences that don’t serve the American public or the federal government.”

>>> Europe : Brokers Upgrades & Downgrades - 2nd of February 2025 V2(+)

>>> Up
* Accesso Technology Raised to Buy at Peel Hunt; PT 600 pence
* Addnode Group AB Raised to Buy at Pareto Securities
* Atoss Software SE Raised to Buy at Hauck & Aufhaeuser (+)
* Big Yellow Group Raised to Buy at HSBC; PT 1,156 pence
* British Land Raised to Buy at HSBC; PT 450 pence
* DNO Raised to Buy at SpareBank; PT 15 kroner
* Gulf Keystone Raised to Buy at SpareBank; PT 300 pence
* Hammerson Raised to Buy at HSBC; PT 373 pence
* H&M Raised to Accumulate at Inderes; PT 160 kronor
* Informa Raised to Buy at Investec; PT 950 pence
* Knorr-Bremse Raised to Buy at Jefferies; PT 91 euros
* Northrop Grumman Raised to Peerperform at Wolfe
* Redeia Raised to Neutral at Oddo BHF; PT 17 euros
* Serabi Gold Raised to Buy at Peel Hunt; PT 160 pence (+)
* Smiths PT Raised to 2,480 pence at Kepler Cheuvreux (+)
* SSAB Raised to Buy at Jefferies; PT 64 kronor

>>> Down
* Addnode Group AB Cut to Hold at Kepler Cheuvreux (+)
* Cool Cut to Sell at Arctic Securities; PT 47 kroner (+)
* FLEX LNG Cut to Hold at Arctic Securities; PT 289 kroner (+)
* Forvia Cut to Neutral at CIC; PT 11 euros (+)
* Getinge Cut to Market Perform at Handelsbanken; PT 300 kronor
* Grieg Seafood Cut to Hold at Nordea
* Hemnet Cut to Neutral at Grupo Santander; PT 404.50 kronor
* HSBC Cut to Hold at Deutsche Bank; PT 910 pence
* Intrum Cut to Reduce at Kepler Cheuvreux (+)
* Ricardo Cut to Hold at Stifel; PT 280 pence (+)
* Saipem Cut to Hold at Stifel; PT 2.50 euros
* Sandoz Group Cut to Neutral at UBS (+)
* Segro Cut to Hold at HSBC; PT 685 pence
* Solwers Cut to Reduce at Inderes; PT 2.80 euros
* Swedbank Cut to Underweight at Barclays; PT 199 kronor
* US Steel Cut to Equal-Weight at Morgan Stanley; PT $39
* Vestas Cut to Sell at SpareBank; PT 90 kroner

>>> Initiation
* Elixirr International Rated New Buy at Peel Hunt; PT 1,100 pence
* Louis Hachette Group Rated New Buy at CIC; PT 2.20 euros (+)
* Raysearch Rated New Buy at SEB Equities; PT 335 kronor

>>> Call
* ConvaTec, Straumann Have Tariff Risk Exposure, Barclays Says (+)
* Goldman Strategists See Risk to US Earnings, Stocks From Tariffs (+)
* Sandoz Downgraded to Neutral at UBS After Year-to-Date Rally (+)
* Sainsbury, Carrefour Shares Fall as HSBC Downgrades to Hold

>>> Stoxx 600 Pre-Market Indications

  • Pernod Ricard (PER TH) -4.4%
    • Watch Europe’s Tariff-Exposed Stocks; Trump Ramps Up Threat (1)
  • Campari (58H TH) -4.7%
  • ASML (ASME TH) -5%
  • Daimler Truck (DTG TH) -5%
  • Stellantis (8TI TH) -5.1%
  • VW (VOW3 TH) -5.2%
  • Mowi (PND TH) -5.2%
  • IAG (INR TH) -6.7%
  • Siemens Energy (ENR TH) -7.7%
  • Indutrade (I1M TH) -9.9%

>>> TradeGate Pre-Market Indications

DAX:
  • Munich Re (MUV2 TH) -1.1%
  • Beiersdorf (BEI TH) -1.2%
  • Vonovia (VNA TH) -1.3%
  • Airbus (AIR TH) -3.6%
  • Adidas (ADS TH) -3.8%
  • Sartorius (SRT3 TH) -4.2%
  • VW (VOW3 TH) -4.9%
    • Watch Europe’s Tariff-Exposed Stocks; Trump Ramps Up Threat (1)
  • Siemens Energy (ENR TH) -7.6%
MDAX:
  • Bilfinger (GBF TH) -1%
  • Talanx (TLX TH) -1%
  • Freenet (FNTN TH) -1.1%
  • HelloFresh (HFG TH) -1.2%
  • Jungheinrich (JUN3 TH) -1.3%
  • Puma (PUM TH) -3%
  • Nemetschek (NEM TH) -3.4%
  • Hugo Boss (BOSS TH) -3.4%
    • Volkswagen Calls, Nokia Calls, DAX Puts: EMEA Options Snapshot
  • TeamViewer (TMV TH) -3.9%
  • Thyssenkrupp (TKA TH) -5.9%
SDAX:
  • GFT (GFT TH) -3.3%
  • Deutz (DEZ TH) -3.4%
  • Salzgitter (SZG TH) -3.9%
  • SAF-Holland SE (SFQ TH) -4.6%
  • PVA TePla (TPE TH) -5.6%
    • European Stock Futures Slide as Trump Eyes Next Round of Tariffs

>>> Europe : Brokers Upgrades & Downgrades - 2nd of February 2025

>>> Up
* Accesso Technology Raised to Buy at Peel Hunt; PT 600 pence
* Addnode Group AB Raised to Buy at Pareto Securities
* Big Yellow Group Raised to Buy at HSBC; PT 1,156 pence
* British Land Raised to Buy at HSBC; PT 450 pence
* DNO Raised to Buy at SpareBank; PT 15 kroner
* Gulf Keystone Raised to Buy at SpareBank; PT 300 pence
* Hammerson Raised to Buy at HSBC; PT 373 pence
* H&M Raised to Accumulate at Inderes; PT 160 kronor
* Informa Raised to Buy at Investec; PT 950 pence
* Knorr-Bremse Raised to Buy at Jefferies; PT 91 euros
* Northrop Grumman Raised to Peerperform at Wolfe
* Redeia Raised to Neutral at Oddo BHF; PT 17 euros
* SSAB Raised to Buy at Jefferies; PT 64 kronor

>>> Down
* Getinge Cut to Market Perform at Handelsbanken; PT 300 kronor
* Grieg Seafood Cut to Hold at Nordea
* Hemnet Cut to Neutral at Grupo Santander; PT 404.50 kronor
* HSBC Cut to Hold at Deutsche Bank; PT 910 pence
* Saipem Cut to Hold at Stifel; PT 2.50 euros
* Segro Cut to Hold at HSBC; PT 685 pence
* Solwers Cut to Reduce at Inderes; PT 2.80 euros
* Swedbank Cut to Underweight at Barclays; PT 199 kronor
* US Steel Cut to Equal-Weight at Morgan Stanley; PT $39
* Vestas Cut to Sell at SpareBank; PT 90 kroner

>>> Initiation
* Elixirr International Rated New Buy at Peel Hunt; PT 1,100 pence
* Raysearch Rated New Buy at SEB Equities; PT 335 kronor

>>> Call
* Sainsbury, Carrefour Shares Fall as HSBC Downgrades to Hold