Intermarkets' Privacy Policy
Support
Donate to Ace of Spades HQ!
Contact
Ace:aceofspadeshq at gee mail.com
Buck:buck.throckmorton at protonmail.com
CBD:
cbd at cutjibnewsletter.com
joe mannix:
mannix2024 at proton.me
MisHum:
petmorons at gee mail.com
J.J. Sefton:
sefton at cutjibnewsletter.com
Recent Entries
Absent Friends
AoSHQ Writers Group
A site for members of the Horde to post their stories seeking beta readers, editing help, brainstorming, and story ideas. Also to share links to potential publishing outlets, writing help sites, and videos posting tips to get published.
Contact OrangeEnt for info: maildrop62 at proton dot me
Cutting The Cord And Email Security
Moron Meet-Ups
|
« Tuesday Overnight Open Thread (4/5/22) |
Main
| The Morning Report - 4/6/22 »
April 06, 2022
Daily Tech News 6 April 2022
Top Story
- Waiting for house news.
Update: There's a buyer for this house as well so I've upped their offer by $25k. I'll be cross if this one gets away because I've already set up a shopping list at Ikea.
Update: And now I'm probably going to be cross. The other buyers lost their house in the floods, and they've already paid the deposit, so unless they can't exchange contracts tomorrow the owners are going to take their offer.
Also, apologies are due to unknown real-estate agent #1 who I thought was playing games. That property - which had been on the market for quite a while - has sold.
- Had a report that one of my (virtual) servers was port scanning someone. I'm not sure if it was legit, and a scan of the server showed nothing untoward, but I have to take it seriously.
First step was to block unwanted outbound traffic at the firewall. The first thing I do with any server is to block unwanted inbound traffic, but everyone* does that these days so all the major exploits sneak unwanted data in on trusted connection. - like the huge Log4j mess a few weeks ago.
Easy peasy because if you screw up the firewall settings on a virtual server you can get in via the virtual console. I've done this on physical servers on the other side of the planet, with complex networking arrangements and no console, and that is nerve-wracking.
Next step was to rebuild that server entirely since all it runs is a proxy server (Caddy) and it's needed an update for a while. Before doing that I checked on the backups to make sure if anything went wrong I could easily restore and, well, f*ck.
The backup drive on the backup server is not responding at all. The syslog is full of ZFS deadman events. And the load average is a personal record, and I've worked on some pretty big servers.

That is not a happy 10-core bunny.
The backups run A->B->C where A is the active server and C is a an archive server that can collect lots of daily backups and B is broken. Which means that for, uh, ten weeks, no backups were going off site from that origin server to anywhere.
So, I arranged three off-site backups and then rebuilt the proxy server with the latest software and the new firewall rules and basically tweaked the config file at random and kept restarting it until it all worked. I don't know why that was necessary, but it was. At least I could flick the routes back and forth internally and didn't need to wait for DNS to update.
Then back to the backup server. Look at the I/O stats. It's written how much data? 18 petabytes? No wonder it's not working, the SSD must be fried.
Wait... Insert commas manually. 18 terabytes. That's not much at all.
Kill the stuck processes (all 24,000 of them). Load average goes up. Kill the parent process... Load goes up even more.
Guess it's reboot time.
And... It sits there with ping working but no other sign of life for ten minutes. (This one doesn't have a remote console either. I grabbed it during the datacenter fire last year and took what I could get.)
It does have a button to remotely power cycle the server but the page asks you not to do that if the server can ping because they'd rather have a technician look at it before the evidence of whatever the problem was disappears.
So I start writing a tech support ticket and just as I'm about to submit it the server comes back up and is working perfectly as if there was never anything wrong.
Okay.
How was your evening?
* Not everyone.
- Twitter is adding an edit button, maybe. (The Verge)
"Protecting the integrity of that public conversation" is of course Newspeak for eradicating dissent.
Tech News
Disclaimer: No, we are not having fun yet.

posted by Pixy Misa at 04:20 AM
| Access Comments
|
Recent Comments
mindful webworker - real query formulations:
"Older than Google? How did you live??
Alta Vist ..."
ShainS -- The Marshall Plan is MORTE! [/b][/i][/s][/u]:
"Atlas Shrugged in front of St. Patrick's Cathedral ..."
Itinerant Alley Butcher:
"I don't know much about men, admittedly, but I tho ..."
[/i][/i][/i][/s][/s][/s][/b][/b][/b]Christopher R Taylor:
"[i]So she’s auditioning for the role of pros ..."
Mr Aspirin Factory:
"Anyone in East Liberty who finds ammo, really shou ..."
browndog with the rule book:
"210 However, there is real bacon at the TX MoMee, ..."
St. Patrick's Cathedral:
"If memory serves, Atlas was a Titan, not a God. ..."
COMountainMarie :
"Yeah baby! Look at the Rex being on time....
Me.. ..."
mindful webworker - or was it intentional? Hmmmm:
"Seeing double on that video. Will it be fixed by t ..."
Thomas Bender:
"Dude ate an entire bloomin onion himself, who the ..."
Berserker-Dragonheads Division :
"The woman suing the blue angels should be made to ..."
Ben Had:
"TRex, well done. Of course, the "city boys" not ..."
Recent Entries
Search
Polls! Polls! Polls!
Frequently Asked Questions
The (Almost) Complete Paul Anka Integrity Kick
Top Top Tens
Greatest Hitjobs
|