2.5 Admins 225: Kinetic Response
30:06 Share

2.5 Admins, Episode 225. I'm Joe. I'm Jim. And I'm Alan. And here we are again. And before we get started, you've got another plug, Alan. Deploying PNFS file sharing with FreeBSD. Yeah, so this looks at parallel NFS, which is a way to actually have NFS basically sharded across multiple machines.

So if you need to get more bandwidth than your network can provide out of one machine, you can actually have large files be distributed across many machines or different files on different machines so that when a bunch of clients are pulling files, they can pull them from all those machines and get much better network bandwidth. It also means you can build NFS that's resilient to a single machine being down. Well, link in the show notes as usual.

FBI warns iPhone and Android users stop sending texts. They want you to use something like Signal instead. Yeah, so just in time for Apple's adoption of RCS to provide a more common way to use text messages across different phones.

The FBI and CISA and U.S. Cyber Defense Agency are warning people that Chinese hackers have breached so many U.S. telecom networks that you're better off sending fully encrypted messages and not using SMS messages. It's wild when the feds are asking you to encrypt your interpersonal communication. Yeah, the same organizations that want to put back doors...

into encryption. Yeah, I think the big message here is like, you know, they're serious about it when the same organizations that are constantly trying to whine that end-to-end encryption is bad or suddenly just straight up saying, use Signal. It's the, yeah, no, no, no, use Signal, please. It tells you a lot about how much they really mean it as opposed to it just being like,

Yet another xenophobic, oh, China bad, let's rile everybody up kind of thing. I think that that call that goes very much against their own messaging for years at trying to get themselves access to interpersonal messaging tells you a lot about whether they really mean this. Maybe stop using SMS for anything important at all to begin with.

At this point, I think we're about in the stage of telephony where we should be considering SMS a lot the same way that we consider email on the network side. And if you're a frequent listener, you've certainly heard me rail before about don't use email alerting for your monitoring system because that crap is not reliable and it will screw you over.

And that's the same thing here again with SMS. SMS is incredibly unreliable. It was never built to be a particularly secure protocol in the first place. It's been massively overtaken by spammers and grifters to the point that most people are likely getting more text messages from just random garbage ranging from crappy local sales efforts to out-and-out grifts and shai-ju-pan attempts.

It's time to move on from that, from anything that matters. You can't fix that. Yeah, although it seems the FBI can't quite help themselves. A senior official advised, quote, using a cell phone that automatically receives timely operating system updates, quote, responsibly managed encryption and phishing-resistant multi-factor authentication for email, social media, and collaborative tool accounts. So they still got to get their jive in there about back-to-end encryption, but...

It is definitely still interesting to see them pushing us towards those messaging protocols instead of away from them for once. I wonder if it'll have any effect though, because people just use iMessage, don't they, in America? Well, so that's why they mentioned the RCS thing, where suddenly you'd be able to have most of that functionality over SMS between a Google phone and an iPhone, where iMessage only worked for iPhone to iPhone before. I don't know, it's...

Definitely there are different bubbles of people on the internet. You know, there's the people that use WhatsApp, the people that use Telegram, the people that use Signal, people that use Facebook Messenger is a huge one for a lot of people. It is a huge one, yeah. Unfortunately, yeah. Whereas I prefer Threema and Signal, but I'm on lots more of them than that. If your weed man doesn't use Signal, get a better weed man. I hear that one on Telegram. Like I said, if your weed man doesn't use Signal, get a better weed man.

US senators propose mandated MFA, an encryption in healthcare. Yeah, having some bare minimum of security for healthcare seems like, you know, we're kind of tired of hearing hospitals getting held for ransom because the IT infrastructure is so bad and so on.

And so I think it's probably good to see it mandated. But knowing how the environment they have to work in, I know that it's going to probably not help as much as we would hope. I think the big news here is not that encryption and multi-factor authentication is a good idea with, you know, health care and similar protected data. I think the real news here is that even U.S. senators are actually bothering to propose it. Yeah.

My biggest concern is, especially if you get to a certain level where, you know, you need a certified solution or something, that there's only two vendors that make it and we're going to end up with a worse model culture than we have now for some of this stuff. But having at least some MFA has got to be better than what's out there now. It's okay, Alan. They'll implement it tomorrow. Just use SMS for easiness. But yeah, they had some interesting examples recently. For example, Change Healthcare took more than nine months to restore their

insurance clearinghouse service after they got hit with ransomware.

which meant a bunch of hospitals weren't getting paid properly, which for smaller ones or individual doctors and so on could have been a pretty big inconvenience for a while. All the medical services, you know, dealing with the crap you have to deal with for American health insurance. And then after the fact, they weren't paying you or, oh, sorry, the computer system we use to tell you whether you're allowed to do this for somebody or whether we'll pay for that procedure is broken. So

Hope that person doesn't need that procedure very badly. So here's a question for you guys. When do you think we're going to have the first actual no kidding boots on the ground shooting war that's fought over data excursions like this? Because I think it's inevitable. The amount of damage that you can do to modern infrastructure with nothing but hacking, nothing but screwing up the data. I mean, it's massive. And I'm honestly kind of surprised we haven't had any

Well, at least any shooting incidents that we know of yet, you know, as a military response to this kind of thing, because it's getting to the point where

nation states are doing it to other nation states. I mean, it's an attack. It gets people killed. It does billions of dollars of damage. Yeah, and we've seen possibly not nation state motivated malware hitting oil refineries and water treatment plants and so on. And it might be that those were just random, but someone who's trying very hard to target a specific water treatment plant or oil refinery or whatever could do a lot of damage very easily.

Well, and, you know, the thing with that is, yeah, in a lot of cases, that is just some random person looking for money, you know, a true criminal enterprise in the traditional sense of the word. But that's also like the most common cutout when a nation state wants to do black ops type stuff is, you know, oh, no, that was, you know, that was some rogue element. That was criminals. That wasn't us. And you have the cutouts distancing it. And we already know that a lot of nation states we know for a fact are doing exactly that. You have, you know,

on the surface supposedly criminal organizations, you know, like the APT groups in Russia, like Fancy Bear, that we already know for a fact do absolutely have ties to the nation state. And there probably is really some degree of independence there. There probably is some genuinely not directly ordered by the nation state activity, but there are also the nation state missions.

And again, this is none of this is new. This is not new to computers. The war on terror. We fought a literal global war, my country, much of it on wildly misguided basis. But nevertheless, we fought a war based on the idea that you have civilian organizations that are militarily attacking us. And if you haven't taken care of it yet, we will invade your ass and we'll fix it.

And I'm just wondering, when are we going to cross that divide? When is an electronic attack going to result in a bullets and steel response? I think it's inevitable. Well, I think we already saw some of that where before the bullets and steel attack started in the hours before that, there were cyber attacks attempting to limit the defender's response to it. Well, sure. Cyber attacks are always a component of any modern military action. I mean, you see that in like Ukraine versus Russia. There's...

between drones and hacking and you name it, it's obviously a component of a full-scale engagement underway. I'm just wondering at what point somebody acknowledges the fact that even without the physical engagement part happening yet, it's an attack serious enough that in some instances it may be worth escalating and saying, you know what? No, it turns out we're going to take this to a different playing field and you're not going to like it.

Now, I do want to make clear, I'm not like championing this. I am not some American war hawk saying, oh, y'all better watch out because we're going to come. No, no, no. Not in any way. That is not who I am. That's not what I'm suggesting. What I'm suggesting is that that is going to happen at some point, and I wonder when, and I wonder by whom. Yeah, I think it's going to end up being –

Not the same, but similar to the policy that, you know, well, if you attack us with chemical weapons, we don't have chemical weapons to retaliate with. But we do have tactical nuclear weapons to retaliate with. And that's what we will do. So don't bring your chemical weapons at us. And I think to your point that at some point a cyber attack is going to result in a kinetic response.

Okay, this episode is sponsored by Automox. Are you prepared for whatever shitstorm may hit your desk during the workday? Automox has your back. Check out the brand new Autonomous IT Podcast. Listen in as various IT experts discuss the latest Patch Tuesday releases, mitigation tips, and custom automations to help with CVE remediations. Make new work friends. Listen now to the Autonomous IT Podcast on Spotify, Apple, or wherever you tune into your podcasts.

QNAP firmware update leaves NAS owners locked out of their boxes. So yeah, recent firmware pushed as an update for QNAP meant that a number of people reported after installing it that they couldn't actually log in and access their NAS anymore. How does that happen? Like, is there no QA? No.

Come on, Joe, you live in 2024. You know damn well there's no QA. Not just at QNAP. Anywhere. It's done. QA's over. Ship it. We'll fix it later. Do they not just have one in the corner of the office? Right, but when you test something in the corner of the office, you test what you think people do with it. But it turns out a lot of people do weird shit or just have a different network configuration than you do or they use a different browser and it sends a different number of cookies to the login page and that can make a difference.

I've literally had an airline's website block me because I was sending too many cookies because they set too many cookies on my browser. And until I reset, I deleted all the cookies for that domain, I couldn't go to the airline's website. There's so much variability in how a user logs into a web app. It can be hard to actually test all of it. I've also seen problems come from the other direction where the issue is that like,

I'm not talking specifically about this QNAP outage right now. Just to be clear, I'm talking about in general. I've seen a lot of QA issues happen where what QA testing gets done is not being done on a test network that's like a reset to a known factory condition and then like worked up every time. It's the same environment that they just run the same tests on and have been for like 10 years. And that environment has actually drifted significantly from what you see in production on newly shipped units. Sorry.

So they don't uncover bugs that only are encountered by actual users who have set things up the way they're actually set up in the wild. Because turns out the QA department doesn't want to break out a new box and set it up and configure it from scratch the way the users have to. They've got some scripted thing that gets it to the condition they want it to be in, which...

again, back to Alan's point, may not match real-world conditions all that closely. Yeah, I don't know if it was maybe just a typo in the reporting, not from ours, but one of the other ones, but it also seems like when they reissued the update after they solved whatever the problem was, they used the same version number, and

And that just drives me right up the wall. Well, they didn't add a 0.1 or whatever to the end of it. Well, the version number is already 5.2.2.2950 build number 2024-11-14. The new one, I believe, is still update 5.2.2.2950, just the build number that changed. Call it 2951, come on. At least, exactly. Or 5.2.2.2950-ubuntu0.

What was interesting is when QNAP did pull it for specific models, the community thread seemed to have people with different models also having the problem. So it's a little unclear on what the exact impact was or how many people were impacted.

And, you know, it's not QNAP's first time to have these kind of problems. It turns out that they've had quite a few vulnerabilities recently. February 2023, a group called Deadbolt got them with a remote attack and they had to push emergency firmware updates to address that. And that didn't go smoothly either. QNAP seems to have been struggling with this kind of thing for a while. I really want to like QNAP because they're actually using ZFS in some versions of their NASes instead of the...

hacked together, you know, butter running on top of MD raid and LVM stuff that Synology and Netgear are doing.

But they're making it hard for me to love them despite their direct use of ZFS. Yes, although their customized version of ZFS is very interesting and very scary to me. They have some interesting features that I'd love to see the source code for and some features that they have that are the same as upstream open ZFS features, but they had them earlier and they are slightly different.

Like they have support for large blocks, but their version only goes up to one megabyte, not 16 megabytes. And it's a different feature flag because it isn't the same. Which brings me to one of my favorite rants, which is like, you know, vendors stop screwing with the upstream code. If you want new features in a file system, then...

develop the actual file system, like be a part of the upstream community and like get that stuff done. This idea that like you're just going to hack it up and make your own junkyard dog that's different from upstream. It never works out well in the long run. And I'm not just looking at QNAP for this. I'm going to stop naming names now, but some of y'all will know from previous rants who I've got in mind. Yeah, we did a whole episode of Linux Dev Time about this with Alan about how

you should just contribute to upstream. It just works out cheaper in the end. Yeah, and I get the idea of wanting market differentiation saying, oh, well, we built this feature and here's the only place you can get it.

I understand the allure of the idea. The problem is that in the real world, it always winds up being exactly the opposite. You're saying you can only get these awesome features from here, but what the rest of the world is experiencing is we only get these catastrophic bugs from you because it turns out that you're trying to compete with upstream. It's

upstream because that's more part of their day-to-day life. That's what they really do for a living. It's not just a thing that they're trying to mess with a little bit. Like that's what they do. So the odds that you're going to compete with them and do a better job than they are when it's not your primary focus are slim to none. Yeah. And you're porting your features to every new version rather than just putting that feature in upstream and letting them port it to the new versions.

or at least have compatibility with it. Precisely. Yeah, like that's been a big part of some of it with ZFS in that, oh, I want to get my feature in before this other person's because it's going to have a conflict and one of us is going to have to solve it. And if I would get there first, then it's their problem to figure out how to make these two incompatible things work together.

That's slightly selfish, but it's really the point is that once you get it there, it becomes collective responsibility. And when someone wants to change something, they have to do it in a way that doesn't break your functionality versus when you're pulling it in. And I think the biggest thing is it focuses on you being the expert on your code. Whereas if you're just trying to pull in from upstream and integrate it with your custom stuff, you now have to be the expert on everything that changed upstream.

Because you're doing it by yourself. Whereas if you push your code upstream, you're the expert on your code and the other person's expert on their code and we all make it work together. But when you're doing it all by yourself, because you're not sharing with upstream, then suddenly you have to be the expert on all of the code. And that can be a much bigger hurdle.

If you'll permit a digression here, it occurs to me that it's almost like open source software is serving a lot of the same functionality that the patent system was originally intended to serve. So you go way back in the day, we didn't have a patent office and technological advancement usually happened at like a guild level.

and new technology was kept secret. It was only within the guild. You were not allowed to let anybody else know about it. You had to be a guild member to know how to use it, to be able to produce device with it and so forth. And the whole reason that the patent system was invented was because, you know, knowledge kept getting lost that way because everybody was being super secretive about technological advancements and it wasn't efficient.

So the idea originally behind the patent system was in return for making the technology that you've created available to everybody, you get a limited term monopoly on it. You can allow other people to license your patent in return for paying you some money to use it. But

after a reasonable amount of time, patents are supposed to expire. And once the patents expire, that knowledge becomes completely public domain. And even in the meantime, before the patent expires, like the knowledge is there for everybody. It's the usage of it that is a bit limited, maybe more than you would like.

And what Alan was talking about, you know, with this idea that people were like, oh, I need to get my stuff into upstream first. That way, everybody else has to solve compatibility problems, not me. It's kind of similar. If you're a vendor who's thinking about developing a new feature that goes towards, you know, an open source application or even an entirely new open source application itself,

Getting it out there and upstreamed and open sourced first kind of gives you some of those same advantages. It means that in return for giving up on some of the secrecy, you're able to leverage a lot of additional advantages that you would not have been able to without that trade-off.

I keep seeing these issues with these off-the-shelf NASes with QNAP and Synology and whatnot. And it just kind of vindicates my choice to never have bought one of these and to just build my own NAS. If you have the technical ability to build your own NAS, yeah, there's really no doubt. You certainly should. It will cost you less money. It will perform better. It will

probably live longer. It just, it's going to be a better fit all the way around. The value proposition to buying a NAS is not having to build it or do much of the maintenance yourself. And the trade-off is, well, somebody else is maintaining it. When they do a crappy job of it, you're the one left with the crappy end of the stick. Yeah. It's just that it really seems to be a race to the bottom there. Like the NASs that you buy at a big box store, they're trying to get as much

hardware and compute and so on into that tiny box and the software is always the least amount of investment we can make in the software because we make our money off shipping units not off the long term of updates. Oh,

Oh, the hardware is usually very minimal too. Yeah. Well, it's as you're trying to get to make it as cheap as possible, right? Yeah. They're extremely wimpy hardware wise compared to even just budget stuff I would build. Yeah. And so they're also doing that same amount of keeping out on the software. And it is very, very obvious. Whereas I'm relying on Ubuntu or you might be relying on FreeBSD to basically provide the software for your NAS, which

I've got a feeling that's going to be slightly more reliable. Well, you are and you aren't. I mean, you're relying on Ubuntu, but, you know, Ubuntu then internal is, you know, relying on the Linux kernel and, you know, a whole bunch of other stuff. And those are the same dependencies behind that NAS, right?

So it's not that you have no dependency chain upstream of you. You absolutely do. But you've cut out a huge amount of like middleware dependency because you've basically done the configuration to set the middleware part up for yourself rather than relying on somebody else to do it for you. Nobody is producing any computing device these days that doesn't have upstream dependencies. They all do. If you can think of a computing device that all of the hardware and all the software and all of everything

everything is coming from one single entity, I'd like to hear about it because I'm not even sure you can find an LED light bulb that fits that criteria these days. Yeah, but my point being that say Canonical pushed a horrible update that just completely bricked my NAS and it wouldn't boot anymore.

I could just install FreeBSD on it, import my pools, and get going. Well, you can actually do that with most of these NASs as well. Like if your Synology wheels up, you can absolutely pull the drives out of it, put it in a Linux box, and you can in fact mount

Even their weird hybrid RAID that's like carved up, you know, MD RAID plus butter on top of it. You can mount all that on a standard Linux box. Yeah, but I'm going to have a much better time just plugging a flash drive into my NAS and installing FreeBSD on it. Agreed. I think the bigger point there is that...

Your dependency on Ubuntu, you're a small fish in a pond with some really, really big fish that Canonical doesn't want to piss off. And that's a good thing for you. When QNAP messes up their NASs, they screw over a whole bunch of mostly home and small business users, maybe some mid-market, a few skunkworks inside enterprises that are depending on these NASs. And that sucks for them. But if Canonical screws up Ubuntu, then, I mean...

huge chunks of critical infrastructure are affected. So Canonical has a much higher burden for quality and engineering on them in order to maintain their position where they are upstream. And basically you as this little consumer, you know that Canonical is not just trying to keep you happy. They're trying to keep enterprises happy. But it does go partly to Joe's point that monocultures are bad. We need to make sure there's always a second option.

So that if Canonical does decide to do something because their big corporate customers want it and it doesn't matter that Joe doesn't or whatever, that there is a second viable alternative. And that's why everything shouldn't be Linux. This brings us back also to something that we discussed recently on the show, talking about graceful failure and having fallback options.

And if you're a professional system administrator or system architect or, you know, whatever, you actually should have like this concept in your head of like pads that you fall back to when pieces of your infrastructure fall over and are no longer fit for purpose. You should have this idea in your head of like, well, you know, if the next version of Ubuntu that comes out is completely unfit for purpose, then you should have this idea in your head of like, well, you know, if the next version of Ubuntu that comes out is completely unfit for purpose, then you should have this idea in your head of like, well, you know, if the next version of Ubuntu that comes out is completely unfit for purpose, then you should have this idea in your head of like, well, you know, if the next version of Ubuntu that comes out is completely unfit for purpose, then you should have this idea in your head of like, well, you know, if the next version of Ubuntu that comes out is completely unfit for purpose, then you should have this idea in your head of like, well, you know, if the next version of Ubuntu that comes out is completely unfit for purpose, then you should have this idea in your head of like, well, you know, if the next version of Ubuntu that comes out is completely unfit for purpose, then you should have this idea in your head of like, well, you know, if the next version of Ubuntu that comes out is completely unfit for purpose, then you should have this idea in your head of like, well, you know, if the next version of Ubuntu that comes out is completely unfit for purpose, then you should have this idea in your head of like, well, you know, if the next version of Ubuntu that comes out is completely unfit for purpose, then you should have this idea in your head of like, well, you know, if the next version of Ubuntu that comes out is completely unfit for purpose, then you should have this idea in your head of like, well, you know, if the next version of Ubuntu

Where do I go from there? Is it going to be Debian? If so, like how much work is it going to be to get my stuff running on Debian? Well, probably not too much since Ubuntu itself is basically downstream from Debian. The relationship is a little more complicated these days, but basically. So that's probably not going to be too hard. Well, what if all Debian based distros wind up sucking for you? Where do you go from there? Do you go to Arch or maybe do you go further than that?

Maybe you do go to Arch. Well, what if the entire Linux ecosystem starts to suck for you for some reason? Well, you should already have some idea of like, how hard would it be for me to get all of my stuff, you know, stood up on FreeBSD, right?

And don't get too smug over there, Alan, because you need to be thinking the same damn thing. If FreeBSD starts sucking, you need to have some idea. Okay, what's it going to look like for me to migrate all my stuff over to Linux as much as I don't want to have to do that? All of us have to think in these terms if we want to really be good at our jobs as the folks who make decisions themselves.

about architectures of important systems and networks. Yeah, and if you think, oh, that could never happen, look at the people that were using CentOS, and it just got completely pulled out of them, out from under them in a much shorter time period than previously promised. Or expected. I don't think a promise was ever made, but it was certainly expected. Yeah.

Okay, this episode is sponsored by 1Password. Imagine your company's security like the quad of a college campus. There are nice brick paths between the buildings. Those are company-owned devices, IT-approved apps, and managed employee identities. And then there are the paths people actually use. The shortcuts worn through the grass that are the actual straightest line from point A to B. Those are unmanaged devices, shallow IT apps, and non-employee identities like contractors.

Most security tools only work on those happy brick paths, but a lot of security problems take place on the shortcuts. 1Password Extended Access Management is the first security solution that brings all these unmanaged devices, apps and identities under your control. It ensures that every user credential is strong and protected, every device is known and healthy, and every app is visible. 1Password Extended Access Management solves the problems traditional IAM and MDM can't touch.

It's security for the way we work today, and it's now generally available to companies with Okta and Microsoft Entra, and in beta for Google Workspace customers. So support the show and check it out at 1password.com slash 25A. That's 1password.com slash 25A.

Let's do some free consulting then. But first, just a quick thank you to everyone who supports us with PayPal and Patreon. We really do appreciate that. If you want to join those people, you can go to 2.5admins.com slash support. And remember that for various amounts on Patreon, you can get an advert-free RSS feed of either just this show or all the shows in the Late Night Linux family. And if you want to send in your questions for Jim and Alan or your feedback, you can email show at 2.5admins.com.

Kevin writes, there's a lot of talk about securing access to self-hosted applications revolving around needing to use a VPN or tunnel, but I don't hear much about using MTLS. Is MTLS a viable option for securing access to self-hosted applications? Think Nextcloud and such.

So I think the biggest thing there is building your own complete CA infrastructure and having access to that everywhere is probably a lot more difficult than a fairly vanilla WireGuard install that's easy to add to another device to be able to access your Nextcloud. So for a brief recap, MTLS stands for Mutual TLS.

So normally when you do TLS, which is like HTTPS, when you go to your bank or to PayPal.com, PayPal has to prove to you that they're the real PayPal, usually with verification from a certificate authority, with

With mutual TLS, in order to make the connection, both sides have to prove to each other that they are really the right client. So I helped a healthcare company set up an infrastructure where all of their computers would only talk to each other if they had valid certificates and building a whole certificate infrastructure around it and having the right revocation policy and having safeguards in place so that if a root certificate was compromised or a private key was compromised, that it wouldn't impact the entire infrastructure and so on.

It's a lot more involved. I've done a lighter version. It wasn't really called MTLS back then, but I've done HTTPS client certificates where I have a certificate installed in my browser. And so when I go to this certain website, it recognizes me and logs me in without me having to log in all the time. And just means that if you're on my machine, you can immediately go into this CRM and have access to the information.

That is kind of related, but that was much more of a HTTP-specific thing where MTLS works for every protocol. But I do think that MTLS is a lot more faff than a VPN or a tunnel and is just not as portable or easy for non-technical people to do. I think there's another problem with MTLS for this particular use case, and it's that usually when you're looking at MTLS solutions online,

It's application level security. It's not network level security. Like, you know, do you have to pass your MTLS challenge in both ways to authenticate both sides in order to get where you want to get with your HTTPS connection that you're establishing? But like you've already touched the web server at that point. Whereas if you're talking about hiding something behind WireGuard, like you don't get to poke at anything until you get past the WireGuard part.

So there's a lot more isolation involved when you're looking at isolating things behind a full VPN than just providing mutual TLS security. I would personally consider mutual TLS security as more akin to like an upgrade to password authentication rather

as a replacement for like a full-on VPN. And don't get me wrong, it's not just authentication. It is also security. You do also have encryption, but there's still a lot more attack surface exposed that way. And that would make me uncomfortable for this particular use case. Yeah, I think the biggest problem really though is with MTLS, as you say, you have to have your own certificate authority and sign certificates and distribute them and manage all those certificates in the long term. And that's...

It's a lot harder than setting up WireGuard and here's some QR codes to be able to connect and it's done. And it's also a lot more maintenance than, again, something like Tailscaler, WireGuard, or OpenVPN. Right, well, we'd better get out of here then. Remember, show at 2.5admins.com if you want to send in your questions or your feedback. You can find me at jorowest.com slash mastodon. You can find me at mercenariesysadmin.com. And I'm at Alan Jude. We'll see you next week. ♪