Lmst

I wouldn't want to be among the people who now need to explain why there is a single-point-of-failure for critical infrastructure. It might have looked like an acceptable risk at the time. After the fact it looks like a foolish decision.

I'd expect that other subsystems will be scrutinized as well. This looks like a higher level problem.

#heathrow #SystemsSafety

https://www.theguardian.com/uk-news/live/2025/mar/22/heathrow-airport-closure-fire-reopens-london-travel-flight-information-latest-news-updates

As a reminder: don't let LLMs handle anything in the political sphere unless you have RLHF (Reinforcement Learning from Human Feedback) active before you show the result to anyone*. Also think of automation risks and human factors (HF). That's "Good Old Systems Safety".

*) ... or unless your goal is to damage a 3rd party's reputation (fake news style).

#llm #ai #rlhf #automationrisks #SystemsSafety

https://www.theregister.com/2024/12/20/apple_ai_headline_summaries/?td=rt-3a

Now, let’s look at the underlying #SystemsSafety part.

#Autopilot and #FSDBeta-equipped vehicles are Level 2-capable vehicles - with the exact same *limitations* as many other vehicles on the market today.

Namely, the key limitation is that the human driver must remain the fallback for any dynamic driving task or vehicle failures at all times and under all conditions.

Effectively, that means that the human has the exact same control responsibilities between the two vehicles shown below.

A photograph of a black, Ford Model T vehicle.

A photograph of a white, Tesla Model 3 vehicle.

#SystemsSafety has an odd power.

People today cannot really remember a time when everyday products would readily kill or maim. So, having lost those experiences over the decades, Silicon Valley increasingly saw a business opportunity.

But modern society is grounded on the public’s trust and, given enough critical mass, trust can be virtually lost overnight.

Quite literally.

That is Great Depression stuff right there, folks.

@CrackedWindscreen Tragic.

Really.

Indeed. These incidents require an exhaustive root cause analysis... which is, as you know, very rarely performed.

But the knee-jerk "just sprinkle a little automation on it" is a absolute Cancer of Simplicity that has zero #SystemsSafety foundation.

There are immense downsides to automation that are very vague.

Let's talk about #Mercedes vehicles equipped with #DrivePilot a bit - a Level 3-capable vehicle that has been recently "approved" in a handful of US states.

This article almost entirely focuses on the legal dynamics of consumer liability should this vehicle create a direct (or, presumably, an indirect) incident.

But, as always, I want to talk about what I feel are the #SystemsSafety realities at work here and the many foot-guns that are associated with that.

#cars

https://www.autonews.com/mobility-report/mercedes-drive-pilot-automated-system-poses-legal-questions

🧵👇

@lolgop Constantly. Constantly this is done.

It is beyond exhausting.

And it has actually proved to be extraordinarily dangerous when #Musk lies about the capabilities and availability of #Tesla's #FSDBeta product, in particular.

The press often allows Wall Street analysts, that are not competent in #SafetyCritical systems, to advance Musk's dangerous lies.

The #SystemsSafety community has been battling this for years.

I suppose that I should also note that no system can ever be "perfectly safe".

That is not possible

And the concept of "perfection" is not relevant to systems safety.

Ross submits that "at no time was anybody at any risk of crashing".

No.

There is **always** risk!

#SystemsSafety is about maintaining processes such that always-present, finite risk is continuously and exhaustively identified and managed.

It is about appreciating that risk exists - the opposite of what Ross submits.

#Tesla

A Tweet published on the TD Ameritrade Network Twitter account which features a quote from Ross Gerber describing the FSD Beta-active drive sequence mentioned in this Toot Thread.

The Tweet states:

"At no time was anybody at risk of crashing," says Ross Gerber on his viral FSD Beta test video: "Out video will show an hour of incredible full self driving."

I was at #Chicago Pride all weekend while visiting with my wife (videos and photos soon!), so I missed this #Tesla Drama concerning #FSDBeta that erupted.

Ok.

Let us, again, all put on our #SystemsSafety hats and take a look at the situation here as I understand it.

Below is the video that kicked the beehive between Tesla defenders and detractors on "what really happened?".

This clearly chaotic video was taken from a larger drive sequence in which FSD Beta was active.

🧵👇

@CrackedWindscreen Bingo. You nailed it!

**Anytime** someone or something uses the term "I think", "could be" or similar and then makes some sort of #SystemsSafety assessment... you can toss it right in the trash.

Systems safety is about constantly and exhaustively asking pointed questions and seeking quantifiable answers.

Academic and industry research can be used to be develop a safety case, but it is not at all complete by itself.

@BruceMirken This is the same Bad Faith shit that us in the #SystemsSafety community have been dealing with on the #Tesla #Autopilot and #FSDBeta wrongdoings for years.

I am not going to lecture Dr. Hotez on how to deal with this... but I can say this... the strategy on the #Musk-side and of his sycophants is to **passionately pretend** like they want to have a Good Faith debate... but they are just looking for "gotcha soundbites".

@kentindell and @CrackedWindscreen,

Let me just say this for the record and for what it is worth, #Ford's Jim Farley is worrying the hell out of me lately in terms of #SystemsSafety.

And I think something really objectionable could be brewing there.

The way he is talking... I recognize this. This talk is familiar to me. I remember it at #Boeing when I was there many moons ago.

I am watching the layoff reporting at Ford very carefully, to the extent that I can.

Efficiently and reliably extracting "safety data" from the roadway is a non-starter.

It simply cannot be done.

Why?

Because the only "safety data" that actually matters for this conversation is "data" that is **forensically** extracted from all interactions with a #FSDBeta-active vehicle - **both** direct and indirect interactions.

No one has that.

Not even close.

Not even #Tesla.

Zero chance.

Numbers on a page are not important to #SystemsSafety.

The root causes are!

I am almost starting to get the feeling that we, the public and the #SystemsSafety community, are being trolled by these #Tesla #FSDBeta YouTube titles.

Perhaps gaslighted is a better term?

Anyways, yup, "safety" on display here all right...

https://www.youtube.com/watch?v=6QKklP91ITU

@colburn Zero actual independent, oversight for the validation processes associated with these automated driving system programs.

Zero.

Not that it matter much, but the #SystemsSafety community has been pounding on this table for years.

When an Uber ATG automated vehicle killed a pedestrian in Arizona way back in 2018… absolutely nothing was done by federal regulators.

#SystemsSafety is not predicated upon statistics, data, videos on YouTube/Twitter and personal experiences.

It is **continuously** defined by a **process** of exhaustively asking questions about the robustness of the system in response to identified failure modes.

No engineered system can ever be "perfectly safe".

That is not the issue.

The issue is what does the initial and continuous (that is, never ended) validation process look like for any given system.

Really thinking of just dropping #Reddit, for what it is worth.

Dropped my profile off my Mastodon bio.

Reddit management is giving me serious #Musk vibes lately and I have been getting follow spammed 4-5 times daily for the last two weeks.

Met a lot of great technical and #SystemsSafety experts on there though - mostly through pushing back against #Tesla's #Autopilot and #FSDBeta wrongdoings.

I suppose that work continues here now...

First off, #cars are **not** smartphones.

I cannot say that enough.

And if you hear anyone describing them as such, it almost certainly means that they are (knowingly or not) hand-waving away the incomparable #SystemsSafety differences between a consumer electronic device and a #SafetyCritical system.

That makes reports like this on #Tesla's hiring preferences **very** concerning: https://www.washingtonpost.com/technology/2021/05/14/tesla-apple-tech/

A screenshot of part of the linked Washington Post article which states:

It’s no accident the companies have a lot in common, according to a half-dozen former employees who worked for both Tesla and Apple, who spoke on the condition of anonymity because of the sensitive nature of the workplace dynamics and for fear of retaliation. Tesla hired managers who brought members of their teams from Apple, importing its design language and culture. Meanwhile, those employees could be dismissive of the automotive expertise within its ranks, the former employees said.

Oh memories.

Taking a break from #Musk's Hate Train on the Hellsite to recall this series of Tweets from a few years ago.

While under-appreciated then and now, the Tweet thread by Musk posted below contains an extremely damning #SystemsSafety admission and it displays the considerable #PublicSafety blind spot associated with remotely updating #SafetyCritical systems without oversight.

Musk has no clue what he admitted to here, but systems safety experts do.

#Tesla

#systemsSafety

Client Info