2.5 Admins 238: Hyperbranded Nonsense
28:23 Share

Two and a half admins, episode 238. I'm Joe. I'm Jim. And I'm Alan. And here we are again. Before we get started, your customary article plug, Alan, is ZFS Space Accounting Explained.

Yeah, so we had Jim come over and write another article for us talking about how space accounting in ZFS works. So where you'll see differences between things like DF and ZFS list and figuring out which snapshot is taking up all the space and why the size of each snapshot will change depending on how many blocks it shares with other snapshots. Also, why trying to figure out how much space you've got left using zpool list is always a bad idea. That's for sure.

Right, well, link in the show notes as usual. Let's do some news. Google apologizes for Chromecast outage in email to users. So it seems that in these second generation Chromecasts, they had a 10-year certificate hard-coded into them, and sure enough, it's been 10 years since they launched Chromecast.

people are still using them and they stopped working as a result. Anybody out there who has managed the same open VPN server for an entire decade, you know exactly the pain that Google is going through right now. Although Google is going through it at a rather larger scale and probably had considerably less excuse for letting that get away from them than you did. Yeah, I do know one person that hit that open VPN thing twice in their deployment lifetime.

Do you think that they just didn't expect them to be around for 10 years? I think it's more that, you know, they made the certificate, it was so long they didn't think about it, and they didn't even realize that they had left a 10-year ticking time bomb. But yes, I also think they didn't expect anybody would still be using a 10-year-old Chromecast, but if it still works. I just don't think any of the tech giants are thinking 10 years ahead anymore. It just doesn't seem to be where people's heads are.

It's not, I don't think, so much that they thought, oh, nobody will be using Chromecast in 10 years. It's that they just weren't thinking 10 years in the future. Yeah, at all. You're looking for your next quarterly bonus. Maybe you care about where you're at next year. Will you get offered that promotion or not? And after that, everything just goes kind of hazy and gray. Who cares? The particular problem here is that the Chromecast audio, which is 10 years old,

There's not really anything else on the market that isn't ridiculously expensive that does the same job. And so people put them into service over the last 10 years. They're doing a great job and they want to just keep using it. But for now, at least they can't.

And Google does say that they are working on a fix for this, but it's going to be a pretty tricky problem to solve, especially if people have factory reset them, which you can't blame people for doing. That's probably what I would have done instinctively. To try to get it to work again, yeah. Yeah, that was the big surprise to me is that Google said there was going to be a fix. I was firmly expecting Google to just say, yep, that's 10 years, pack sand. I mean, this is a company that has...

end-of-life dates for every Chromebook that comes out. You know, why would you expect a Chromecast to be different? Yeah, but that's the problem, that they hadn't anticipated this. They didn't have it in their calendar. They didn't publish anywhere that it was going to be discontinued. If they'd done that, they arguably could have got away with it. Yeah, sure, but it lasted for longer than any of those Chromebooks that did have an EOL lasted.

I mean, it was a whole decade. You got like an extra five to seven years out of your cheap little gadgets you bought from Google. What are you complaining about? It seems every one of these certificates is from March 9th. I guess, is it just all second gen? And, you know, when is this going to happen to the third gen Chromecasts? Because I don't think they were very far behind on the calendar. The other thing I found interesting is they said they're going to share updates and guidance via the Nest community page.

So suddenly the Google Chromecast is now in the division with the Nest thermostats and so on? Well, that's because Chromecast doesn't exist anymore.

Everything consumer hardware from Google is Nest now. So the smart speakers are no longer Google Home. They're Nest Audio now. It's all Nest, whatever. Everything except the phones, I guess. Well, and if you're going to go there, the tablets as well. There's a Pixel tablet and a Pixel watch. But as far as the consumer infrastructure devices, smart speakers, Wi-Fi, thermostats, whatever, all that stuff is branded as Nest now.

It's pretty depressing, though, that your takeaway from this is, well, you got longer out of it than I thought you would. Suck it up. It shouldn't be that way. Well, and also, surprise, Google's actually fixing it. I'm kind of on board with that, too. I agree with you that it should be that way. It's just very clearly not that way. And it is a surprise that Google said they would fix these things at all. Because again, we're

We're talking about 10-year-old inexpensive gadgets sold by Google of all companies. So I half expect to see a follow-up story about somebody getting fired for promising to fix these things. Well, just like we've even covered stories relatively recently about devices getting the, sorry, we're remote bricking it even though it was still working after like a year because, yeah, we're not going to have these devices anymore. So it's over. And they've

They broke it on purpose. This is Google. They broke it by accident after 10 years, and they're saying they might fix it, which is, yeah, I'm actually quite surprised. Microsoft is shutting down Skype in favor of Teams. As it has been doing for, what, 15, 20 years now? Yeah, I was surprised it was still going, to be honest. But I feel like we have to mark this passing because...

Skype was amazing 20 odd years ago, wasn't it? It was great before Microsoft got their hands on it, but it's like Microsoft has been trying to kill it since day one. The whole split between like original Skype and Skype for Business,

Just the never ending kerfuffle where Microsoft reps would want you to come attend some presentation they were giving only you had to use a Skype for business login. And if you didn't have one, you're out of luck because you couldn't create one and it would conflict with anything you had going on with legacy Skype and just, oh, it was awful.

Frankly, at this point, it's kind of nice just to see the poor thing get buried and not still get trotted around like a zombie from now until the end of days, you know? Well, the big thing that back in the beginning, the name Skype comes from the original idea of Skypeer, which was that it was peer-to-peer. And that meant that, you know, there was no service behind it. And it used the same magic the original creators had come up with for, I think it was Kazaa, the file sharing app, to basically...

Basically, take two people that are both behind that and be able to make a direct connection between them so that they can share video or files or whatever. So Skype was really big then. It was a service where eventually I remember paying a couple dollars a month to have the premium account or whatever, where you could have a video call with three people in it instead of just two. Eventually that became free and free.

Yeah, when Microsoft took over, they got rid of the idea of the super nodes or whatever they were called that would do the mediation to do the NAT busting and move that all to Microsoft servers so they could do wiretaps on Skype accounts and so on for the government. But the real point was, yeah, Skype was a really big deal. It was a great way to communicate and for free back in the day. And...

It was a really big thing for a long time to actually be able to have face-to-face communications with people. And, you know, they tried some stuff during the pandemic. They added these weird modes where you could have, like, take all the people on a call and make it look like they were in the same room and a bunch of really gimmicky shit.

But yeah, like when we started using the Microsoft suite of stuff at my company, we were using Skype for business at first. And yes, it created lots of confusion whether, you know, that it's not really Skype. It was originally what Microsoft called Link and they just rebranded it to Skype for business, even though it wasn't really that related to Skype. But we had even worse problems when they rolled out Teams originally. So this is probably 2019, 2020 when

Microsoft really started pushing Teams. They switched meeting invites that you created in Outlook from Skype for Business to Teams. But this meant any meeting invites we had created that were recurring, so we created before the switchover, were still in Skype for Business. And every meeting we created after was in Teams.

So you go through your day on your calendar and you'll have one meeting in Skype for business and one meeting for Teams. And it was just really annoying and we burned it all down and went to Zoom. Yeah, I was going to say, you're still paying for the Microsoft stuff and you could be using Teams, but you're also paying for Zoom. Yeah, because we hate Teams with a passion. I think the original Skype was frankly just too consumer friendly to survive a Microsoft acquisition. I remember when Microsoft bought it, I thought,

Well, that's clearly just, you know, one for the boneyard because the original Skype was one of those things that was so clearly designed just for random people to use as a tool without extracting a rent from them in any way that it's like I could see Microsoft coming up with something like this in-house and letting people have it. But I absolutely cannot see Microsoft...

acquiring this, like paying actual money for it with the goal of it continuing as it is. The fact that they put the price tag on it to begin with to acquire it says there's going to have to be some way either of figuring out how to seek rent through it or just kill it off in the long run. Otherwise, it just doesn't make any sense. Yeah. So when eBay originally bought it, I thought that was a little weird, but I could see what eBay was going to do with it to try to, you know, let you talk to sellers directly without

Having to have long distance charges and so on. It didn't really make sense. But then, yeah, Microsoft bought it. And I was like, oh, here goes the inshittification. I might be having a mini panic attack at the idea of having to video conference with anybody I was buying some random thing used off the Internet from. No, thank you. No, no, no. Do not want that to or trying to sell stuff. And a bunch of random is calling you with video for no reason.

But when Microsoft merged all the MSN stuff in with Skype and created all the confusion and suddenly your Skype account had to be replaced with a Microsoft account, but your username might change or it might not and all this other nonsense. Yeah, Microsoft's been seemingly trying to kill it for a very long time. Even the article, they admit, you know, Skype hasn't really shrunk in any dramatic way. It's just stopped growing. And, you know, we want to let the air out of that balloon. Yeah.

And so my Skype that was working yesterday today just says, this version of Skype is out of date. Please download and install the latest version, which I'm guessing they actually mean Teams. Well, it is actually going to survive until May 5th. So there might be another version. I don't know. I'm guessing that version is the version that will self-destruct on May 5th. It just makes me sad because it was so great in my formative internet years.

Yeah, it was a big part of that for me. And I'm still waiting on the email from Microsoft when my data extract will be ready because there's 25 years of message history that I guess I don't want to lose. I think we just we're going to have to get used to this idea of platform shift. It's a lot more rapid than we would like. It's funny because.

The invention of the printing press really, really slowed language, human language evolution to a glacial crawl because as more people can see what other people have written, it slows the change of language because you want to speak the way that you've read other people speak and some of that is multigenerational. So the speed at which language changes slows way down.

The same thing, obviously, with, you know, audio recordings, you know, TV and movies, whatever that that also slows down the evolution of language. But the Internet seems to have just absolutely exploded the evolution of social platforms in the opposite direction.

where it's gotten to the point that, you know, if there's something useful in your life right now, electronic, you'd better be ready to replace it because it's not going to last for very long. Things are just going to iterate, iterate, iterate. And after so many iterations iterations, whether your thing is still any good or not, it will no longer be supported and therefore you're

will not be useful, whether it would have or not. So if you're not prepared to keep moving and keep changing and, you know, keep looking for different things to put in the same tool bag to do the same job, you just wind up failing and you do so, I think, a lot quicker than has ever really happened before. Yeah. And, you know, that relates back to the previous story we talked about with the Chromecast and stuff. But

Yeah, I can definitely see where some of the language evolutions come from as well. But yeah, I think we have to get a lot less attached to everything because the platforms are churning and the services are shutting down. You know, if Google has taught us anything, it's like, if you learn to love something, we'll kill it. Well, that's why you should have your own website, your own presence. Don't rely on social media, for example. Sure, but how much longer is even like the current trend

very general loose framework idea of HTTP and HTML going to be useful as it is in the current sense without some kind of an interpreter.

If you look at a lot of the things from 20-year-old websites right now in a modern browser, they won't work because the platforms have changed. Flash is gone and so on. Java. Java and the browsers that it was built for at the time was like Internet Exploder and that's not a thing anymore. And we're seeing the increasing concentration where everything is either Chrome or some other thing. You know, the Firefox is holding on with...

you know, it lasts the skin of its teeth. It's either Chrome or something that technically isn't Chrome. The same people who made Chrome are the 90% of the source of money for those other people that made the thing that supposedly isn't Chrome. Yeah. That's our current browser market. Yeah, and then you've got Safari and WebKit. That's a fairly big segment of the mobile market. But even more to Jim's point, what we have is more and more and more push to have, you know, the official app for something that is mostly just the web browser wrapped up differently. Yeah.

But you don't interact with any of those services via their website. You do it with their app. And so it's really to your point that, you know, we're getting away from having an open standard web to an app store full of proprietary apps. You know, it reminds me of actually, it reminds me of, it's kind of like the software version of where the hardware was in like, you know, the early and mid 90s when we still had the huge hardware diaspora. You know, you had,

really more 80s even than 90s, but before the big rise of the IBM PC to kill off everything. You had Apple IIs, you had Macs, you had Commodore, you had Amiga, you had VIC, you had PET, you had Trash 80s, you had Trash 80 color computers, you had Ataris. I mean, just on – I haven't even begun to list all the different kinds of mutually incompatible hardware yet.

Now we have that same kind of just explosion of mutually incompatible, brightly colored, hyper branded nonsense in the software space.

Okay, this episode is sponsored by people who support us with PayPal and Patreon. Go to 2.5admins.com slash support for details of how you can support us too. Patreon supporters have the option to listen to episodes without ads like this. And it's not just this show. There's Late Night Linux for news, discoveries, audience input, and misanthropy. Linux Matters for upbeat family-friendly adventures. Linux After Dark for silly challenges and philosophical debates.

Linux Dev Time about developing with and for Linux, Hybrid Cloud Show for everything public and private cloud, and Ask the Hosts for off-topic questions from you. You can even get some episodes a bit early. We've got a lot going on, and it's only possible because of the people who support us. So if you like what we do and can afford it, it would be great if you could support us too at 2.5admins.com slash support. They curdle like milk.

Warner Brothers DVDs from 2006 to 2008 are rotting away in their cases. I mean, this is not a surprise to me in any way, shape or form. Yeah, we've talked about this many times when talking about people wanting to use DVDs or CDs or Blu-rays or whatever to archive data. And it's like, well, you know, it doesn't last that long and we never know exactly how long. And we...

are told that pressed DVDs are going to last longer than burned ones because the ink wasn't designed to be changed by a laser, but there was really a race to the bottom to make the DVDs as cheap as possible, so I'm sure there's some problem. And lo and behold, here we are, exactly in the place where like, yeah, I bought all my stuff on physical media because I didn't trust subscription services to not just yank and delete videos that I wanted. And it turns out if you're not

ripping your DVDs and storing them somewhere where you can protect them from bit rot, it is likely that they are literally rotting. It is just another form of information and there's no getting away from it. If you want information to stick around, you have to back it up. If you're not backing it up, it will go away. It's ephemeral. So yeah, what really makes this news, I think apart just from so many people in the mainstream having no idea that, you know, DVDs go away like this,

What really makes this news, you know, even to technical folks is that, yeah, this was considerably quicker than we would have expected pressed DVDs to degrade. I think Alan's comment about a race to the bottom for, you know, the cheapest possible production costs is really on point here. And I think it probably isn't a huge surprise that it was Warner Brothers in specific that we see this happening to because, you

There's nothing concrete here, but I don't know about you folks, but I don't really have a perception of Warner Brothers as exactly being like a premium studio, you know, in that time frame. It seemed like they were kind of struggling with a lot of, you know, failing intellectual property. And it doesn't shock me that they were cutting corners harder than anybody else when it came to burning discs.

They had the Matrix sequels around that time, didn't they? That was earlier. Well, no, the sequels would have been around that time because the original Matrix was 99. Yeah. So the sequels took a few years to come out between each one. So, yeah. But, I mean, Matrix 3, how –

great was that, really? Well, all of the Matrix movies, apart from the first one, are terrible, and the first one is a masterpiece. Don't at me. I pretty much agree with you on that. Well, I don't know about the first one being a masterpiece. The first one was certainly vastly better than anything that followed. I'll give you that. Revolutions came out in 2003. Right. So, yeah, maybe you're right then, Jim. Maybe it was like after that and

Yeah, they weren't exactly a premium studio. It feels like they've been living off of legacy for quite some time. Once really beloved intellectual property brands kind of seem to be Warner Brothers'

Claim to fame, really. I mean, what's – The Matrix is the newest thing either of you have come up with so far. Like, quick, name the hot Warner Brothers property. I mean, I mostly remember it on old, like, Batman cartoons, you know, from the time frame. Like, they were making the, you know, the Batman cartoon sequels in that time frame, I think –

Maybe they got their name on some of the Justice League Unlimited type stuff that was going out on Cartoon Network in that time frame. Well, I just Googled it now and Joker, that was a pretty big movie. 300, again, that was ages ago. Inception, again, ages ago.

There's nothing hugely new that's good that I'm seeing here. But anyway, if you have one of the affected discs, apparently Warner Brothers is offering replacements for some of them if they're still in print. But for a lot of the material where they don't, you know, they're not pressing new DVDs of that particular movie, you're just kind of shit out of luck. Off to the Pirate Bay for you. We didn't back it up. We didn't want to let you back it up. And now it's out of print and sucks to be you.

Let's do some free consulting then. But first, just a quick thank you to everyone who supports us with PayPal and Patreon. We really do appreciate that. And if you want to send in your questions for Jim and Alan or your feedback, you can email show at 2.5admins.com. Michael says, for work, I have to run various Ubuntu VMs on Beehive on a free BSD host running on ZFS. I've been using ZFS in the VMs.

Should I be layering ZFS on ZFS like this, or should I use ext4? This is a concept that we've covered before, the idea of whether it's okay to layer ZFS on top of ZFS and, you know, VM infrastructure. And the answer is generally you want to avoid it, but it's not because something will break. It's just because tuning it for performance can be really annoying. So it's usually much easier to use a simpler file system like ext4 or UFS2

And then, of course, the other part of it is when you've got one simple file system and ZFS, where do you put ZFS? And the answer is as close to the middle as humanly possible. So ZFS goes on the host and the simple file system goes in the VMs. I really kind of wanted to answer this one, even though we've covered essentially this exact question before, because...

When we talk about using FreeBSD and Beehive on the host, I feel like that makes it a lot more likely we're going to talk about a nested ZFS situation in the guests because somebody who wants to run a FreeBSD host is a lot more frequently going to be wanting to run a FreeBSD VM. And while it's kind of a pain in the butt, you know, even with something like Ubuntu, it's a little bit of a pain in the butt and going off the ranch to use ZFS as your base file system on FreeBSD. It's the default. You have to go out of your way if you want to set up a FreeBSD VM not using ZFS.

So my answer there would be if performance is a concern, I would advise going with a simple file system inside the VM and CFS on the host. However, that doesn't mean it can't be done or it's unsafe.

And there may be some situations where you really, really have an interest in the VM having direct access to its own ZFS storage, its own pool. Maybe you want to be doing replication in and out of the VM. Maybe it's really important that users on the VM be able to roll snapshots back without giving them access to anything on the host.

And in this case, layering it can make a lot of sense. Just again, you do need to be careful of performance concerns because when you've got like mismatched record size or a shift or, you know, a million other things down the stack, you can wind up with really weird like edge case pain points in your workflow that may require some serious skull sweat and expertise to tackle. Yeah. But the easy one, the kind of free win you can get if you are going to do ZFS on ZFS for

for example, because you want ZFS boot menu or boot environments inside the VM, is just pick one side to do the caching at. Either have primary cache equals metadata only on the dataset that hosts the VM on the host, or disable the arc inside the VM so that you don't end up caching the same block both inside and outside the VM using twice the RAM for no reason. That's kind of the free win you can get. And then as Jim said,

Depending on how you're setting it up, you really do want to try to get the A shift of the pool inside the VM to match the record size of the backing storage outside of the VM. But sometimes that's not always an option, especially if you're using zvols and it gets very complicated. But depending on your use case, it probably is worth the trade-off. If you have a use case where ZFS will help,

then yeah, it's totally fine to do ZFS inside the VM and outside. But if you don't really need ZFS in the VM, then go ahead and use the simpler file system like ext4 or UFS or whatever inside the VM. That's a great catch, Alan. Yeah, obviously you need to be worried about using tons of RAM to duplicate your cache inside and out. That's a bad idea.

The other thing about that, you know, Alan mentioned that you can work around that issue by setting ZFS set, you know, primary cache equals metadata to only cache the metadata on one side or the other. It's almost always going to be a good idea to cache the metadata, even if you're doing it in duplicate, go ahead and cache that on both sides, but not the actual data. So then that raises the question, okay, well, where do I cache the data? Do I do that on the host or do that inside the VM?

And typically, I would recommend doing it at the host level and then setting inside the VM is where you do primary cache equals metadata. The reason for that is because it's really nice to still have a hot arc when you reboot a VM. And if your data cache only lives inside the VM, it goes away when you bounce the VM.

If it's on the host layer, you're still booting directly from RAM when you reboot that sucker. So if you're at all concerned about the impact of reboots, not only on the time it takes to reboot the VM itself, but also on how long it takes for the services that VM is offering to come back up to speed, you want that caching on the host.

The one time that you would want to do it inside the guest and not on the host is when you are absolutely willing to deal with all the cache going away whenever you bounce the VM, but you need the absolute bare minimum of CPU cycles executed in order to manage the cache.

If that's what you need, you're going to get about a 5% to 10% performance bump out of having the cash inside the VM rather than out. But me personally, I can count the number of times I wanted a 5% boost over knowing that the cash would still be hot when I rebooted the VM.

I can count those instances on the fingers of one foot. Yeah, I think the only other case for that is if you're running all these VMs, but other people are using the VMs. If you do all the caching on the host, it will cache whatever is going to be best across all the VMs. So if one of the VMs is the busiest, it'll take more of the cache. And maybe that's what you want. That'll give you the best performance. But it might mean that department A is now using all the cache and actually making things slower for department B.

Whereas if you account for how much cache they're going to have inside the RAM of the VM and have all the caching happen inside the VM, you know that department A's VM can't use more than the RAM we give to their VM. And that means that department B's will have some RAM for their cache as well. But usually if you're dealing with noisy neighbors, you need something more complicated than this anyway. Well, also that's only really going to have an effect if essentially you don't have as much RAM as you should for the amount of data that you're handling to begin with.

This only makes an impact if it's causing your cash hit rate to artificially be much lower for one client than another because it's the fault of the other busier client. And that can really only happen if you don't have enough RAM for as much arc as you need to maintain a decent hit rate to begin with, in which case the real answer is you fix that. You don't penalize your users, but

Yeah, yeah, add more RAM. And if you can't add more RAM, then you start considering, you know, sharding out your workload and you have two hosts instead of one host and you move some of them off until you actually have enough RAM to effectively cache everybody. Because doing it the way that Alan was talking about, it doesn't really make things work better. It just blocks out that one very, very specific scenario, which is predicated on

on you not having as much RAM as you really ought to to begin with. Because remember, the ARC is not a simple FIFO cache. It maintains things in the cache based on how frequently they're accessed. So in order for one really busy neighbor to use up too much of the cache so that a less busy neighbor can't, you basically already have to be looking at an issue where you can't keep all of the frequently accessed things in cache to begin with, which again means you should be adding RAM.

Right. Well, we better get out of here then. Remember, show at 2.5admins.com if you want to send in your questions or your feedback. You can find me at joerest.com slash mastodon. You can find me at mercenariesysadmin.com. And I'm at Alan Jude. We'll see you next week.