Overnight Open Thread - 07/28/2024
[Roger Ball]

Airline/air space meltdowns have happened before, and they will happen again. USA Today published a timely article in May, 2023 with some helpful tips about how to survive a meltdown should be unlucky enough to get caught in one.

Another airline service meltdown is coming. It's not a question of "if," but "when" it will happen. And this one could make last holiday's airline problems look like a minor delay.

Well, buckle up Horde! Here we go!

Summer, 2011--SPIRIT:

"Spirit has canceled more than 2,000 flights since Sunday due to a combination of staffing shortages, bad weather and technology problems."

Winter, 2022--Southwest:

"Southwest Airline's cascading failures have checked a veritable bingo card of travel nightmares. And while every airline faced bad weather and cancellations last week, only Southwest fell apart."

Summer, 2023--United :

"In short, Kirby traces the issue back to bad weather at Newark Airport (EWR) that limited flight operations, causing a domino effect of disruption across the system. Let’s take a look at the specifics:"

Winter 2023—The FAA:

“Our preliminary work has traced the outage to a damaged database file. At this time, there is no evidence of a cyberattack...The FAA said Thursday evening that the data file “was damaged by personnel who failed to follow procedures.”

Summer, 2024—Delta:

"With so many planes and crews now out of position, and the system used to schedule them still languishing, it's questionable whether Delta will be a reliable airline by the weekend, Potter said."

It should be obvious that the first three involved weather among other issues. On Friday July 19, Crowd Strike pushed one bad line of C++ code to worldwide workstations using Crowd Strike's security software. The weather was not a factor. Pixy has provided some great explanations and links about this disaster.

My son is a software engineer, and he complains about how companies never seem to put enough value on the QA devs who test all code before publishing. He did note that one of the bigwigs at CS came from McAfee--remember that "essential software" that would mostly suck all your computer's resources? Also, CS laid off 200 employees earlier this year. So, accident? Purposeful, like sabotage? "Press to Test" for the literal Great Rest? We likely will never know.

What we do know is that Delta was the goat this time (Note, not as in the "Greatest of all Time" GOAT.). Other carriers recovered relatively quickly but Delta dug itself a hole within the first hours. By not cancelling enough flights soon enough, the whole dispatch system "bricked". Combine that with all the workstations locked in BSOD, nothing could be done to move any aircraft or crews anywhere.

Flights that were not cancelled stayed in the system and piled up to the point where nothing could be pushed through. Dispatchers, load planners, crew scheduling could only stare at the BSOD (Blue Screen of Death) until an IT person rebooted the system, workstation by thousands of workstations*. Gate agents were able to process some passengers if they remembered their "DeltaTerm" language, which is basically DOS commands. But, there were no crews to man the jet, if the jet was even there. And even if the jet and the crew were there, there was no way to generate the dispatch release, flight plan or load plan.
*Actually, I saw an estimated 8.5 million workstations worldwide.

Like most major airlines, and service companies in general, procedures to cope with major service disruptions are created. In defense of Delta, the blizzard of 1993 initiated, at no small investment, the creation of their Operations Control Center. If you haven’t seen it, take my word for it, it rivals NORAD. Delta has been very good at anticipating problems and irregular operations by proactively “thinning” the schedule to help protect the schedule and, therefore, the passengers. But, in defense of Delta’s ops folks, it's hard to manage airplanes, crews and passengers if you are the person responsible and you are looking at a blue screen of death. This creates the question applicable to all of us: what's your plan B when the system goes down?

Delta will recover at a huge cost and maybe some learning is occurring concurrent with the loss of revenue and customer loyalty. But in the meantime, it's not a good look. Like Delta 9994 was supposed to go to Air Venture on July 23rd. It was to show off the TEAM USA livery.

Oh well, Air Venture 2025, here we come!

The Great Political Meltdown: Now it seems the demokkkrats have openly endorsed assassination as party policy. They will strive to gain total control of the government so they may escape any accountability. If you think the "cheat" was big in 2020, you haven't seen anything yet. I'm thinking that we should just do away with elections. We'll just make whoever grifts the most money out of donors the winner. What do ya think?

Speaking of meltdowns: bye, bye, Kimmie .

[slightly edited] “I have, and will always put the [DEI] needs of this agency first. In light of recent events, it is with a heavy heart that I have made the difficult decision to step down as your Director,” Cheatle wrote in the letter,".[Besides, they have just finished my anchor desk at MSNBC.]

More melting! So much melting! Can I mention the opening ceremony? The creators literally invoked Revelation 6:8. Now of course they deny any religious connotation but we know what they are up to. And, amusingly, the "City of Light" became the City of Some Light. At least the horse seemed well illuminated.

The Military Meltdown from CDR. Salamander:
How is the Red Sea Fight Going?

"So, in essence, we’ve expended all of 2022, 2023, and part of 2021’s procurement in just nine months."
Against. A. Sub-4th Rate. Non-state. Actor.
As the Wall Street Journal reported on Friday, General Michael Kurilla, USA, recently sent a classified memo to Defense Secretary Lloyd Austin warning that the strategy to stop the Houthis’ disruption of shipping routes is “failing.” (Well, it's online so I guess it's not that classified.)

Gates of Fire

“I’m hit three times! I’m shot three times!”
"With his leg mangled, Kurilla pointed and fired his rifle into the doorway, yelling instructions to the soldiers about how to get in there. "

Here's an interview if you are so inclined: Q&A with Michael Yon.

Stay frosty, Horde! No melting!

Roger dot Ball dot OK at proton.me

posted by Open Blogger at 10:00 PM

| Access Comments