2.5 Admins 231: USB 3.2.2-ubuntu2
28:32 Share

Two and a half admins episode 231. I'm Joe. I'm Jim. And I'm Alan. And here we are again. Edge IO bankruptcy results in endpoint change for Microsoft.

Yeah, this one is quite amusing. Microsoft kind of surprised a bunch of .NET developers over the Christmas holidays by being like, oh, you know those URLs you've been building into all your applications forever on how to install and get your artifacts? Those all have to change now because our CDN, it turns out, it seems, owned the domain AzureEdge.net and they've declared bankruptcy and got bought and we can't find someone there to transfer the domain back to us.

So we need you to switch everything to pull from builds.net.microsoft.com. Love that. Or if it's the CI builds, you can use ci.dot.net.

Is DOT a TLD, Alan? Because if it was, we could get dot, dot, dot, dot, dot, dot, dot. It's like Abbott and Costello, man. I want dash, dot, dot. The thing that really pisses me off about this one is, so this is a pretty major impact on sysadmins who, I'm going to restrain myself and say, chose to implement Azure services because

And it's coming about because Microsoft didn't own a domain that they were using in production with their own trademarks in the domain and instead just let the CDN be the registrant for it. Because in order to lose control of the domain and have to switch to a different one the way Microsoft is right now, you have to lose control of the DNS.

And if you lose control of the DNS, the thing that you do is you take control back of the DNS, which if you're the registrant for the freaking domain, you can do at the registrar by changing what the authoritative DNS server for the domain is. So in order to say like, oh, there's nothing we can do, Egeo has gone out of business and therefore we can't use these URLs anymore means that Microsoft never owned the domain.

And I would like everybody listening to just sit there and think for a moment about being an entity, the size, the scope, and with the resources of Microsoft and accepting that and just be like, oh, yeah, whatever. We'll let the CDN buy the domain. They can own it. And it's even worse because it's not just that the CDN

was providing services to Microsoft, they were bundling this and selling it as a service out of Azure. So they were white-labeling the CDN and not owning the domain. And it turns out, so Microsoft had announced that they were going to retire the feature Azure CDN from Edge.io in November of 2025. But then it turns out they got notified that...

Because of the bankruptcy, Edge.io's platform would cease to work on January 15th. You know, Alan, I know you used to run a CDN. It strikes me that you really missed out on an opportunity because clearly Microsoft wasn't vetting things very well during this decision-making process. And I'm betting they were paying Edge.io...

A lot more than any of your customers paid you. Yes. Well, I imagine when Microsoft hooked up with them, it was still Limelight, which was a reputable CDN. But then I think they got taken over by private equity and this is what happens. Which just brings us back again to, you know, don't lose control of your domains, right?

It's not okay for a CDN to be the registrant of your domain. It's bad enough to allow them direct control of the DNS, but in some circumstances that may really kind of be necessary. You have to let them have control of the DNS, but you don't have to give them the domain for that. You keep control of the domain and you just set the authoritative name servers to their name server. And then if and when they go out of business,

And everything goes out of business sooner or later, and they may beat you there, especially if you're, you know, Microsoft. Then if you still own the domain, like I said, you can recapture the DNS and you don't leave people, you know, I mean, we're literally talking hundreds of thousands of people out there that are having to deal with this crap now because Microsoft didn't keep control of their own infrastructure. Well, in particular, this one is hitting...

The whole ecosystem of .NET applications really hurt because all the installer URLs were hard-coding these links to the assets. That's why I said hundreds of thousands. To be clear, I'm not talking users. I'm talking developers and admins directly impacted by this. If you're talking about actual users, oh, you are way into the millions.

This is a thing that we've been seeing more and more of over the last few years. They're called residential IP VPNs.

or residential proxy VPNs. And the way they work is rather than somebody who is creating a VPN app and setting up a commercial VPN service who wants to take your money and allow you to escape your own IP address and come out of an endpoint somewhere. Well, instead of them renting space in a data center and having your traffic come out of the data center, instead, they're finding ways to

Usually trick, but let's just say convince people who have residential internet connections to install an application that will then allow this company to route you, one of their customers' data through either another of their customers or even sometimes just somebody compromised or, you know, a victim of some kind. But in either way, your traffic comes out of their house, not out of your house once it hits the open internet.

This is usually something that you use for evading geofencing. Like maybe you want to watch British Netflix from America or vice versa. So you use a geolocating VPN to get you apparently in the right country in order to get your programming. So in this particular case, there's a VR game that physical location is part of the game.

And if you have a way to skip around and convince the application that your headset is plugged in in different countries, you can get a leg up on the game. The mechanics, the specific mechanics, I don't really care. It's some game nonsense. It doesn't matter. The interesting part is that

This guide got released on how to use a geolocating VPN to cheat and get ahead in this game. And the guide that was released that lots and lots of lots of people have been following to the letter specifically mentions one of these VPNs that does a residential proxy.

If one were cynical enough, suspicious enough, paranoid enough, one might actually wonder whether somebody who worked at that company that offers that service was the one who released that guide in the first place. A quick correction. According to this Ars Technica article, it's about a game called Gorilla Tag, as he said, and you've got to sneak up on other people. And...

It says, using a VPN, according to the tutorials, introduces a delay that makes it easier to sneak up and tag other players. So I don't think it is even about being in a different physical location. I think it's just adding latency. Well, this particular guide, yes. But like I said, I'm not really interested in the game part. I don't care why the game gives you an advantage if you use a VPN. The interesting thing to me is the privacy aspect. Because if you install that VPN service or...

Anything else that allows a commercial VPN to route other people's traffic out through your house on the open internet, you don't know what that's going to be. And a lot of it is probably going to be really freaking nasty.

Because the geofencing I mentioned, that's the biggest thing that gets most people into VPN type stuff is they want to be able to watch streams from different countries. Then the next most common thing is, you know, I want to be able to hide my piracy activities. And you might or might not mind that somebody's BitTorrent traffic is exiting a VPN at your house and now pieces of the latest blockbuster Marvel movie are being downloaded directly

to your house from when they disappear elsewhere and you never had access to it, but you might still get the strike from your ISP. But that is again, we haven't gone very far down the list of shady things people use VPNs for.

You don't know that they're not going to be looking up child porn from your house. So you want to think really hard and really long before you install some application that allows other people to proxy their traffic out from your IP with your name on the bill. Again, I can understand the appeal to some of this, especially as some of the video streaming services actively block VPNs now. Pardon?

partly to satisfy rights holders, not because they care. I can understand why you'd want to exit on a residential IP instead of an obviously this IP belongs to a VPN. But at the same time, if you're going to let people exit through your home internet connection, make sure you understand all the implications of that before you do it and then probably reconsider and don't do it anyway. The worst thing about it is somebody is making money off of that huge risk that you're taking. It ain't you. You ain't getting a penny of it.

You are just taking all the risk that somebody is going to be downloading child porn, uploading child porn, pirating software or, you know, Hollywood properties that, you know, the Gestapo are going to come after you for potentially, you know, doing direct online attacks against things, you know, using you as one of the notorious seven proxies that the 4chan B user is behind when he attacks somebody.

You don't know what all that's going to be. But again, the thing that you can be sure of is every last bite of that traffic is going to be something that somebody didn't want coming out of their house. They would prefer that it come out of yours. Yeah. If the VPN is free, you are the product. Well, yeah. And also you've got to look at the business model of something that is free. If that business model is we're going to show you some ads. Okay. That's not ideal, but what can you do?

If the business model is we're going to sell access to your IP to other people for pennies, then shit like this is going to happen. Do we really want to be using VPNs that have to have a bespoke app you can't see inside of instead of that use a standard VPN program that, you know, maybe I already have or that is open source and I can trust as opposed to, you know, this app's so sketchy you have to sideload it on your phone. Or in this case, I guess it was specifically being loaded on the VR headset.

So that's a little more complicated. I'm not entirely sure of the details of what can and can't be loaded on that headset, but I kind of feel like VPNs in general might not be allowed directly on the headset. So you might have to sideload any VPN no matter what, because that class of app is not something that you generally want on a VR headset. Because what would you use it for? Well, you'd use it for piracy. You'd use it for cheating in games. You know, what legitimately would you need a VPN for on a VR headset?

So you probably have to sideload no matter what, which would certainly explain why so many people would use exactly that service that was in the guide that gave the instructions on how to sideload it. For our own listeners, you know, if you want to cheat on your VR game and it requires a

either adding latency or let's just go ahead and say geolocating because if you just want to add latency, well, Alan will tell you all about using dummy net on FreeBSD. But if you want to geolocate it and you don't want to become a FreeBSD user, essentially what you're looking for now is you want some way to route all the traffic from only that one device.

through another machine that then uses a VPN to yada, yada, yada. You don't actually have to have the app on the device. You just have to make sure whatever the device is using as a gateway will then route that traffic through whatever VPN you elect to use.

In any case, whether it's your phone, a VR headset, whatever, if you're sideloading an app, you have to realize that it's not going through the same reputation checks and so on. Not that the app stores are great at this, but sometimes they do provide some protection or at least will take down things that are actively found to be malicious. If you're sideloading, it's now all on you to check and make sure the image you're loading is not a malicious application.

To Alan's point, although none of us are happy with how well any of the app stores are curating their applications, really, with that said, it is a huge mistake to think that those efforts, as underwhelming as we might find them, aren't worthwhile. Because you think about all the times that you've seen a dodgy app on the app store,

I mean, the alternative is it's like the difference between going between a normal social media platform, you know, something like Blue Sky or Mastodon or whatever, and, you know, going on 8chan. That's the difference. There may not be much moderation, but even a little makes an incredible difference over absolutely none. And you kind of have to have experienced what absolutely none looks like to realize the magnitude of it.

Proof-of-concept exploit released for OpenSSH arbitrary code execution vulnerability. This sounds quite scary. So this is the vulnerability that got fixed last year that we got everybody scared about, and they've now actually released the proof-of-concept to show how it works, now that hopefully everybody's already got this patch installed. Right, and so if everyone's patched it, why do we care? Because not everybody has patched it. And we get to find out how it actually worked.

which is, you know, the more technically interesting thing. Are you new to the internet, Joe? You think everybody patches their systems on a regular basis or at all? Yeah, surely everyone has unattended upgrades or the equivalent installed. Again, are you new to the internet? Have you not looked around? There's also a good point here in that this is such a convenient backdoor into a system where

I could easily see somebody reintroducing this vulnerability as a way to get back in if they got kicked out once they got a toehold some other way.

Tools that test really well-known vulnerabilities that are instant game over and you get a shell are a good thing to have around because they're a way to make absolutely certain that you aren't vulnerable to a nasty trick that's out there in the wild that other people know how to use. It's kind of funny you mentioned that, especially in that this vulnerability was found and fixed in 2006 and then accidentally reintroduced in the code base and then...

found and fixed. And so yeah, having this to a as part of the regression tests for open SSH to make sure that it's vulnerability doesn't help again. But also, like you say, oftentimes, rather than depending on the version string that spat out by your SSH server to say that it has this fix, using the proof of concept exploit to make sure that it fails when run against your SSH server.

especially when you're patching older versions and so on. And it's not necessarily going to say, you know, I'm version 9.9 now. It's going to still say I'm 8.5, which is in the vulnerable range, but you've patched it. Being able to confirm that by using the proof of concept is a really good way to be sure that the patch is applied and working. It's a good idea regardless. And generally you want to have, and by you now, I really kind of mean the entire industry as a whole.

But you kind of always want to have a library of simple testers that will absolutely directly check to see if well-known vulnerabilities are around, because like those are the kind of things that you build into libraries that tools like, you know, the rest in peace Nessus used to use.

network vulnerability scanners. Network vulnerability scanners usually will have a couple of different modes. And in one mode, they'll just look for version strings and say, well, this version string looks bad to me. But they'll also have like, you know, a hard test mode because one of the ways that you harden production systems is you remove the things that will output the version string to make it harder to figure out what version to make it hard to figure out which vulnerabilities work or don't work.

So how do you test those to find out are you vulnerable or not? Well, you try the exploit out and you see what happens.

And when you're building a general purpose security audit tool, like I said, you want a whole library of these things because you might actually want to try several thousand exploits out against a system that you're really trying to make sure is hardened versus all the known exploits. Another thing to Alan's point, being that this is the second time in a couple of decades that the same vulnerability has been introduced here.

That brings up the idea that in certain applications, you have kind of a strange attractor where like there's a way to do things in code that looks easy and looks safe and sane, but actually isn't and introduces a vulnerability. If you let enough time go by after that vulnerability has been fixed before, somebody may reintroduce it because the same idea looked good. It's something that naturally tends to appeal to people who haven't seen that particular issue and don't know better.

So when you have that kind of strange attractor where you very well might have the same vulnerability crop up again, again, you want to have a tool where you can actively test it. It's kind of like how, you know, you have fuzzing tools that will just go ahead and automatically try, well, what if we flip the first bit of a command or a string of data? What if we flip the last bit? Let's check all the corner cases.

And this is, again, exactly why, because these kind of corner cases are the attractors that devs who mess up tend to make and leave potential vulnerabilities. So you always want to be able to test them directly and easily. Okay, this episode is sponsored by Entroware. Go to entroware.com. Entroware sells computers with Ubuntu and Ubuntu Mate pre-installed.

They've got a range of desktops, laptops and servers and most parts are configurable so you can pick the CPU, RAM and storage that's right for you. If you can't find exactly what you want then do contact them and they'll work with you on a bespoke solution that's perfect for your needs.

They ship to the UK, Republic of Ireland, France, Germany, Italy and Spain. And if you do buy one of their machines, there's a little drop down at checkout and you can select late night Linux so they'll know that we sent you. So go to entroware.com for all your Linux computing needs.

An updated USB logo will now mark the fastest docking stations. Finally, they're going to just put the number of gigabits per second right on there. It's getting pretty ridiculous when, you know, USB 3.2 Gen 2. It's like, what do you got the .2 if that's not the generation? Like...

Why is it not just 3.3? Why is it 3.2 Gen 2? It's like, how am I supposed to find the cable or understand how fast this port is supposed to be? I could pretend to hate reusing a joke, but we're right back to USB 3.2.2-23. Yeah. And then they had their marketing style thing of a super speed USB 10 gigabits. It's like, that's almost getting there, but...

high speed versus super speed and so on. It's like, yeah, I'd understand that was in the spec originally, but like, no. And don't forget, like sometimes there's the lightning bolt and sometimes there's not, there's like three or four different ways to label any given variety of quote, fast unquote USB connection from the last couple of decades. And it's really hard to figure it out. And also the big thing that like has got me going, I don't know if I'm excited as Alan is, is

I'm not convinced that every USB port is actually going to be marked accurately, properly, or visibly because out of all those branding options we already have,

So commonly, either they're omitted, the wrong one is used. It's like embossed black plastic on black plastic. You know, just sometimes you say, oh, well, you know, you just look for the blue plastic on the inside of the port and that tells you it's one of the fast ones, except when the fastest port on the machine doesn't have the blue insert, which I've seen happen pretty frequently. I don't think I've ever seen the absolute slowest old school USB 2.0 with the blue.

But I won't swear that I haven't either. Well, the one that I didn't know existed until trying to shop for stuff is USB-C cables that only do USB 2.0 speeds. Yeah. But that's a thing. And it's like, God damn you for that being a thing? Like, what the actual hell? Why do we have the new connector if it doesn't do the new USB 3 speeds? Like, I'm not even talking needing the 80 gigabits.

Why can't it do 20 gigabits like every other USB-C port? How is it? Oh, it's just for charging. Well,

Well, they've also got some logos that show you the potential power delivery of cables. Yeah, again, something that'll be really nice to see. But at the same time, now you're trying to fit even more font. And like Jim said, if they're etched or embossed on the side of your laptop, you remember how thin laptops are right now, right? How are you going to read the font to see if that says 20 gigabits or 40 gigabits? And is that a 60 watts or 80 watts?

And now let me invoke the other magical curse word that we haven't so far, Thunderbolt or not. Yeah. I thought the idea of USB 4.0, which hopefully they're never going to call USB 4.0, is that there's not going to be a difference with Thunderbolt. It's just always have it or whatever. But yeah, it's a whole nother thing. And...

At least this maybe increases your chance of being able to find the right cable because looking for USB 3.2 Gen 2 on especially things like Amazon that have the most dog shit search functionality ever. It's like you're never going to find a cable. It's like you put a length in your search term and it'll show you every cable of every length that's nothing to do with the protocol or length that you asked for. Pro tip, if you're searching Amazon or other online vendors for USB cables, don't

don't choose a version of the USB protocol. Search with the throughput or the power handling capacity that you're looking for in the term. You will still find a bunch of false positives and bullshit. However, you'll also find the things that you're actually looking for. You know, this stuff is supposed to be for consumers and it is more complicated than buying like SAS cables for servers. Well,

When we're talking about something as far down the food chain as like a cable, you know, or the simplest chipset dumb devices, you have these issues where you've got, you know, manufacturers who they come across, you know, an entire warehouse full of some shit from 20 years ago. And they'd like to find a way to reuse it because they got access to that warehouse of crap essentially for free. So if they can manage to build a product and get it out the door with that, well, that's just, you know,

free stuff into the bottom line. And they won't be able to make it once that stockpile of ancient crap runs out. But until then, they're printing money. This happens a lot. And I suspect this is a lot of reason for all these weird dodgy cables. If you're starting from new parts, I do not really believe it costs enough more to make a 40 gigabit power handling cable than an old school, you know, USB 2 is all it can really handle. There's just not that much more going on there in the manufacturing pipeline. But,

But if you're managing to avoid some of the manufacturing pipeline by reusing old parts, well, there you go. So I kind of feel like we're looking at the results of an industry where vendors have been allowed to do exactly that. It's like Intel CPUs. You know, what features does it have? Who the hell knows? You know, any given product cycle, you know, they release like

40 variants of essentially the same freaking thing. And it's market segmentation. Well, Intel's case, it's market segmentation. And other vendors, sometimes it's market segmentation. And again, like I said, sometimes it's reusing ancient crap. You see it all the time in like Wi-Fi stuff. You know, you have these things come out and like you still see IoT devices that say like, you know, 802.11n Wi-Fi. Why do they do that? Not because somebody's manufacturing 802.11n chipsets in 2025.

because somebody has access to just skip upon, skip upon, skip of those things for free. Or, you know, they're just that much cheaper because they're trying to get rid of them because every day that goes by, they're worth less and less money.

But to Joe's point originally, like they added the power to the cables, but they added an extra one that has just the power and not the bandwidth. Now, the thing about that is all the power and a little bit of throughput is just effed up. There's no two ways around it. However, there are some specialty cables out there that are power delivery only.

only for when you don't want to risk a potential malware issue plugging your smart device into something that you don't know what data is coming through from it. So you can buy cables that are power delivery only, will not pass a single byte of data. Like the USB condom from the old days. Yep. And those, I think, those are perfectly valid. Those are absolutely worthwhile. But yeah, if you're coming across something that's like,

USB 2.0 throughput with like USB power delivery, 60 watt charging. That's some bullshit, man. 240 watt in this logo here. High speed USB, which means 2.0 with 240 watt power delivery. What's that delightful Britishism, Joe? A shower of bastards.

Let's do some free consulting then. But first, just a quick thank you to everyone who supports us with PayPal and Patreon. We really do appreciate that. If you want to join those people, you can go to 2.5admins.com slash support. And remember that for various amounts on Patreon, you can get an advert-free RSS feed of either just this show or all the shows in the Late Night Linux family. And if you want to send in your questions for Jim and Alan or your feedback, you can email show at 2.5admins.com.

Joe writes, not me, looking online I see it is possible to replace the motherboard in my TrueNAS system. I was just wondering if there are any things to watch out for or to be prepared for when changing out hardware in my NAS. It's TrueNAS scale running the latest release and I do have a backup to Backblaze that occurs weekly.

So I assume we're talking about a third-party system that you have installed TrueNAS Scale, the community version on, or Enterprise, whatever, doesn't matter. But the point being, just it's a box that you built or you bought from some place other than iX Systems and then installed TrueNAS yourself. Yeah, you can absolutely replace a motherboard. TrueNAS Scale is just a Linux distribution and Linux in general is fine with motherboard replacements.

The big things that you should really watch out for are different device names, where this is the most likely to bite you, tends to be in network interfaces. Very frequently, you'll go from one motherboard to the next, and you're using onboard network adapters, and it's maybe F0 on the old one and a string of 20 letters with a zero on the end of it on the new one. So you may need to update some of your configs to reflect the new network interface or what have you, but that's usually the worst of it.

And you'd be surprised how often you don't even need to do that much. You just pull the drive, put it in an entirely new box, and stuff just works. Yeah, and TrueNAS is a bit more of a special case in that

Because of the design of keeping the pool separate, you can lose the entire installation of TrueNAS and just do a fresh install of TrueNAS, plug in the existing hard drives, import the pool, and it will load the config, copy the config it has on the pool. And, you know, the entire OS is disposable even. So yeah, I wouldn't expect any problems just swapping the motherboard and not having to do anything. It'll just boot up and work. You might have to set the BIOS on the new motherboard to make sure it boots off the right drives. But other than that, it should just work.

Does TrueNAS use UUIDs then? It uses ZFS, even for the OS. So it boots off ZFS. And so ZFS just finds the drives by reading the labels on the hardware. So it uses its own GUIDs rather than UUIDs, but same thing. Right. Should be very simple then. Yeah. And I've definitely replaced the motherboard or taken hard drives out of one machine and put them in a completely different machine and had it just work. Yeah. When I upgraded my NAS, this was just an Ubuntu system.

I exported the pools, built a new system, installed Ubuntu again, and just imported the pools, got going straight away. That's the beauty of ZFS, right?

Yeah, and in particular, TrueNAS will store the configs for things like your Samba shares on the pool, not on the OS. So that if you just, when you import the pool, it can actually re-pick up all the Samba shares and everything. There are still going to be some cases where you may lose some configuration going from one set of hardware to another. For a simple TrueNAS installation, sure. If you've got something that's considerably more advanced and complex, in particular, if you've got multiple network interfaces installed,

that do different roles and you transplant that system into a new motherboard and it has differently named interfaces, there is absolutely no way for TrueNAS to know what configurations go on what interface. So you may have some minor reconfiguration to do there. I want to make sure everybody hears you may have some minor reconfiguration. It may not be literally you just turn it on and you're done. However, it will be simple reconfiguration. You can do it. It's not a huge deal.

Right, well, we'd better get out of here then. Remember, show at 2.5admins.com if you want to send in your questions or your feedback. You can find me at joe-rest.com slash mastodon. You can find me at mercenariesysadmin.com. And I'm at Alan Jude. We'll see you next week.