2.5 Admins 233: 2.005 Admins
30:43 Share

Two and a half admins episode 233. I'm Joe. I'm Jim. And I'm Alan. And here we are again.

And before we get started, you've got a plug for us, Alan. Key considerations for benchmarking network storage performance. Yeah, so over on our website, we have a new article that kind of walks through the things you need to consider when you're benchmarking storage, but in this case, over the network. So why is my NAS not saturating the network and things of that nature? We cover NFS, SMB, and iSCSI, and the unique considerations you have for each to make sure you're getting the most out of your storage. Right, well, link in the show notes as usual.

What is some software or principles and techniques that you love but aren't that well-known? And obviously, ZFS and related stuff doesn't count. Yeah, I wouldn't say it's not very well-known, but maybe just underappreciated. I think the biggest one for me is the way subnetting works in networking. The fact that we have this subnet mask and we tend to use only the specific ones very often. But the fact that you can take an address, whether it's IPv4 or IPv6, and then the mask you know of it

And by doing a math operation called an AND, where you're just only the bits that are set in both the original and the mask, you can basically take two addresses in a mask and instantly be able to tell, are these two addresses in the same network or are they in a different network? Basically, do I have to go through a router to get from A to B? And the fact that your computer does this so many times a second, but mostly that it is so inexpensive. And the fact that then we build a routing table out of

doing this, but doing what's called the longest match or most specific match where we kind of like look for the one with the longest subnet mask because that's more specific. And so that means, hey, if I'm trying to go to address and it matches more bits than this other route, that's the route I want to take. Especially when I was teaching like subnetting and TCP IP, you would see a couple of people start to get how interesting the math for this was. And other people are just trying to, you know, memorize what 255.255.255.0 means.

Instead of understanding that it's just how many bits from the IP address are the network versus the host part. And the fact that you can, in a super, super inexpensive math operation, be able to tell this list of addresses, which ones are in my area and which ones are foreign.

just by doing this simple AND operation. Well, super inexpensive if you happen to be a binary computer, not so much if you're talking about your own organic meat computer, in which case it does wind up being a whole lot more efficient just to remember, oh yeah, 2525250, that's class C. There's 255 hosts in that subnet. Admittedly, when I was teaching this, I'm writing out the binary with a marker on a whiteboard and showing how it works.

And that also makes it much easier for people to understand where, like, slash 24 and how that relates to 255.255.255.0 and understanding what the difference is between slash 24 and slash 29 and slash 17 and so on.

But once you understand the very, very basic concepts, the fact that the whole Internet's based on this like tiny, tiny little bit of math that's not that hard to understand. I've just found really interesting just the way that that super simple thing could just scale to be this massive thing on the Internet. I think for my well-loved but not well-known principle or technique, I'm going to go with custom benchmarking.

Obviously, benchmarking in general is a topic near and dear to my heart. I've done a ton of it for a lot of different reasons, whether we're talking about Wi-Fi or storage or what have you. And one of the things that it took me a while to learn was the value of rather than just looking at a common, widely used benchmark and saying, do I have bigger number or lower number than that? That means my device good or my device bad.

actually sitting down saying, what does my workload look like? And can I just test using my actual workload? And if not, all right, how can I make a standardized, repeatable, artificial workload that very closely models my real one so that now I can rank things based on how they get through the actual day they're going to have to get through if I implement them in production.

This can make a really big difference. As an example of this, when I first started benchmarking Wi-Fi stuff for the Wirecutter,

I needed to figure out, okay, in real world terms, which of these devices are better than the others? Because everything on any box for any Wi-Fi devices, just, it's always a lie. You know, if there's some truth on there, it's almost accidental included, you know, in the part of making a larger marketing lie. Every single router is faster than every single other router is faster than every network interface you have or any workload you could have. It's just, it's all nonsense.

So when I went to evaluate this, I was like, OK, I can't really just like run a speed test and call it a day. I already know that routers that look good when you just run a speed test and call it a day frequently work like crap in the real world. So how can I capture that?

So then I start thinking, okay, well, what are we doing the most of when we use a router? And of course, the answer is we're browsing web pages. We're watching YouTube videos. We're sending emails. What does all this stuff look like? Well, the majority of that, again, is actually going to be browser based. These days, even the email stuff is usually browser based. So you're looking at a whole bunch of HTML and you're looking at a whole bunch of individual resource calls for every single page that you load, right?

In reality, you're making hundreds or frequently thousands of individual requests to various different backends. And until a certain number of all of those requests get fulfilled, your browser doesn't even put anything on the screen other than just a blank white page.

So this is the thing that you're waiting on. This is the thing that really pisses you off when your router sucks and you're staring at this white page and be like, why won't this web page load? And you're thinking about all the various workarounds, unplug the router and plug it back in or whatever.

unjoin the wifi network and join it again, you know, all these things to see if it will then work better. How do we trigger this? And so the answer became, all right, well, what happens if instead of just sending a bunch of packets across an interface and noting the latency of those packets and just focusing on the median latency, what if instead I say, all right, well, since I know I need to make a few hundred requests before this page loads, rather than looking at individual packets,

Let's look at what happens when we actually make several hundred individual requests. And the real answer is what happens is you emphasize the unusually bad return. When your 99th percentile latency is terrible, but your median latency looks great, you might think, well, this is a pretty good device. And if every request completes all by itself individually, it might be a good device for you.

But when everything that you want to do involves waiting for several hundred things to all complete and you've got really, really bad 99th percentile latency, that means you've got really bad application latency for every single thing you do. And it's terrible. But you wouldn't know that if you didn't

stop and think about what is my workload? What am I trying to do? And come up with an artificial workload that actually matches that, respectively. Yeah, I had to do something similar to that for one project we did where we needed to

create a workload that looked like people streaming Netflix. So doing basically random range requests from big files off of a server and making those align and deal with all the encryption for it. So it's like for this network, we need to make it look like there's a whole bunch of different people all trying to pull Netflix style things where the

The Netflix client will download a couple of minutes of the movie and then pause and then come back later and download a couple more minutes of the movie, right? It's not streaming it in quite the same way as like a Zoom call where it's going to be a constant data rate. Netflix is going to download some chunks and then wait a little bit and then download some more chunks and so on and be able to switch servers between that and so on. And so...

Being able to model that using Lua in the WRK benchmark tool meant that we could make our own benchmark that wasn't just download one big file as fast as you can or have a bunch of clients all trying to download the same file, but have a huge kind of library of files and clients pulling random pieces from them all over the place.

way that makes sense for each individual flow. But in the aggregate, what the server sees is like everybody's asking for different things all at once and nobody asked for the same thing twice and kind of like defeating caching and so on. And it was very interesting to see just how a good benchmark tool will let you kind of customize it to basically turn it into your own benchmark that matches your workload.

I thought this one was pretty interesting. So up until now, you have had the option of attaching a USB external drive to your Xbox and storing games on there. And you could have up to 16 terabytes on your USB attached drive.

No more. Now the news is that Microsoft has added a feature where if you attach a drive of some sort, a USB storage that is larger than 16 terabytes in size, you can partition that drive into multiple 16 terabyte partitions, each of which the Xbox will be willing to format and presumably just sort of transparently JBOD together in the background, although I don't really know the details on that one.

And I found this interesting for a lot of reasons. For one, it's neat to me to see that Microsoft, not even Microsoft, really this is more about like the console gaming community, is getting to the point where their IT demands are enough that they're

A segment of the community is basically starting to want like a combination Synology, NAS, and gaming console, like all in one. It seems like it's really kind of where this is headed. So this idea that you have this truly massive library of games, many of which are

are truly massive individually in size, but they're all accessible all the time. You may potentially even be using your external drive for backup of your internal. It's janky and primitive compared to what professional system administrators are used to, but it's a big step forward for the community.

That aspect is interesting. But the other interesting aspect is that Microsoft needs to ask you to partition your large storage device into 16 terabyte or smaller partitions, which it will then glom together on the inside. Now, the reason for that is that the Xbox uses kind of a hacked up version of the standard Windows file systems for the external drives.

Depending on which Xbox and which version and which year, these things have changed a little bit. Initially, the external drives were formatted with a variety of FAT32. Now they're formatted with a variety of NTFS. Most of the change revolves basically just screwing with the MBR in a way so that PCs will refuse to read the drive.

Microsoft doesn't want you to be able to easily unplug your external from the Xbox, plug it into your PC, monkey with the data using your PC, and then plug it back into the Xbox. So they monkeyed around with the MBR, which gives you a little bit of security by obscurity. It's easy to patch back and forth if you know how, but you have to know how.

So this was interesting because we're looking at this 16 terabyte file system limitation and your first thought is, okay, well, obviously we're going fat here. So let's laugh at Microsoft for still using fat in the year 2025, but it doesn't appear that they are anymore. You're not given the choice as to which file system you want to use when you format the drive the first time you plug it into your Xbox and you do have to format it from the Xbox when you first plug it in.

Remember, even if you theoretically had a compatible file system, the MBR is on there wrong. So you're not gonna be able to store games on it. The Xbox is not going to recognize it as belonging to it. So, if Microsoft is using NTFS, which can easily address more than 16 terabytes in a file system,

Why this limitation? And I think the answer is going to be that for whatever reason, they've hardcoded the cluster size to 16K because the file system limitation for NTFS with a cluster size of 16K is, in fact, 16 terabytes. So I guess that just brings me to wanting to laugh at Microsoft for hardcoding the cluster size and not coming up with a better solution for that when they wanted to expand the addressable spec.

space than just, oh, we'll just JBOT a bunch of other things together and it'll all work out in the end. Going back through the history, it looks like the Windows 7 or Windows Server 2008 R2 versions of NTFS also had the limit of 16 terabytes less than

64 kilobytes. So it's actually not 16 terabytes exactly. It's 16 terabytes minus one extra sector, 64 kilobytes. Well, it's also not 16 terabytes at all. It's 16 teppibytes, less whatever. Yes. But I do wonder, like, you know, knowing that Xbox just stood for DirectXbox, which is just let's turn a Windows gaming PC into a console, what version of Windows is actually running inside? And hopefully it's not the equivalent of Windows 7 still, but that seems to be the version of NTFS they're shipping with it.

I don't have much in the way of insider information on this one, but I would –

Imagine and everything that I've seen leads me to believe that it's probably better to think of the quote Windows unquote in an Xbox as a cousin of desktop Windows rather than like, you know, a particular version. I think they diverged at least 10 years ago and have continued to diverge in opposite directions. Yeah. And so they probably took whatever was the current NTFS driver at the time. And that's what's in the Xbox. And that's why it has this limit.

Okay, this episode is sponsored by SysCloud. Companies big and small rely a lot on SaaS applications to run their businesses. But what happens when critical SaaS data is lost due to human errors, accidental deletions, or ransomware attacks? That's where SysCloud comes in. It's a single pane of glass to back up all critical SaaS applications, such as Microsoft 365, Google Workspace, Salesforce, Slack, HubSpot, QuickBooks Online, to name a few.

syscloud also helps with ransomware recovery, identifies compliance gaps in your SaaS data, monitors for unusual data activity, and can automatically archive inactive SaaS data to save on storage costs. Plus, it's SOC 2 certified, so data remains secure in the cloud. Over 2,000 IT admins already trust syscloud to protect their SaaS data.

Head to syscloud.com for a 30-day free trial and for a limited time, use code 25admins to get 50% off your first purchase. That's syscloud.com.

Migrating away from BcacheFS, this is a post by Steinar H. Gunderson on a decidedly web 1.0 looking website, which I really appreciate. For those who aren't familiar, BcacheFS is a file system built onto the Bcache tiered storage application for lack of a better word.

Bcache was something that you could put in front of a file system so that you could divert very heavily used blocks to a more rapid hardware device than what you had the actual main file system on.

Bcache FS is an evolution of that that attempts to be the entire file system. And it does offer, I believe, tiered storage a la the original Bcache. And I think it's more important to just describe it as it's yet another attempt at a next generation file system along the lines of ZFS and ButterFS. Much like Butter, I think that there were a lot of people who had some really high hopes for Bcache FS, but...

Unfortunately, also like butter, the dev process hasn't really lived up to expectations. And after an awful lot of years, we're still left with something that's largely non-functional. There are also some pretty ugly personality conflicts between the founding developer of the file system and the package maintainers at various distributions.

Packaging is a lot more complex than it really needs to be. And in general, it seems like enthusiasm is really,

really waning for this and it's sad to see. But hang on, it got put into the kernel and it's been absolutely hunky-dory ever since. Yeah, it got put into the kernel and Linus hadn't stopped bitching since the day he did it. Yeah, we covered this a couple of months ago on the show where they were like, no, you can't just ship a newer version in order to integrate this regression fix because we're in the middle, like we're late in the release cycle. You can't just bring in that much diff.

And yeah, there's been serious threats about removing it from the kernel. But to Steiner's post, he's got a valid point that a file system developed by mostly a single person is putting a lot of trust in that one person, especially if they have strong personality conflicts with both the Linux kernel mailing list and the package maintainers of the operating systems that are trying to ship support for it.

They're attacking from both sides there, making it that much harder to actually use the file system. At risk of being called a zealot, how many hundred developer years have gone into ZFS? A couple at this point. I think when it left Sun, it was at 150 developer years. And that was...

I don't know, like 20 years ago. Yeah, and it's still actively being developed by slightly more than one person. Yes. And when we dig into the details of the current complaints that we're covering right now, a lot of it does center around not only the difficulties in packaging and distributing BcashFS, but the way that the founding developer, Kent Overstreet, is being reported to...

There are widespread reports that he is essentially asking his user community to attack Debian maintainers over Debian packaging. Now keep in mind, we're talking about user space tooling at this point. Debian pulls down a Linux kernel, so they get the kernel part of BcacheFS automatically when they pull down a vanilla Linux kernel. However, you need the user space tooling that actually interacts with the kernel in order to do anything with it. And that's where the packaging difficulty is coming in here.

There are a whole lot of packages, mostly built in Rust, and the dependencies of the BcacheFS tools and the tools themselves

They've been built from a wide array of versions, basically. And you get into a DLL hell situation where like, okay, if I'm not literally just grabbing the binaries that Upstream produced and just chucking those down the wire, which is not how package distribution works, you're supposed to be as a distribution compiling from source code yourself and integrating with the distro and have the full package

chain of custody, essentially, to get from source code to binary. That's what you have to do as a package maintainer, which means you need to be able to build this stuff and you need to be able to build it in ways that are compatible with all the other things that you're building on the operating system. This is not anything new. This is not anything really unreasonable. It can get difficult to deal with and kind of unwieldy, and it genuinely does cause conflicts between reasonable package maintainers and reasonable upstream developers pretty frequently.

But trying to weaponize your user base against maintainers is, that's a step beyond, man. I can't hang with that. Well, and apart from anything else, this post talks about technical problems with it. That's ultimately why he moved away from it. He tried it for a bit.

And it just didn't work for him. I think the wider story here is that it's part of kind of an unfolding debacle around BcashFS. And it makes me genuinely sad. I frequently get called things like the biggest ZFS stand on the planet. And, you know, fair point. But I would love to have options. I would love to be in a situation where I'm like ZFS is great. And this other thing is just as great. And I can use either one if I want to and feel good about it.

But right now, I don't really have a great replacement. And Butter wasn't it. And even apart from the packaging, it doesn't look like BcacheFS is going to be either because in addition to the packaging difficulties, there are wide reports of features that don't work and bug reports being ignored. And, you know, all the other features of ButterFS development that I've grown to love and expect over the years.

You said either or, but both would be great, right? You could have some systems on a different alternative and some on ZFS, and then you'd be protected against bugs in either. Sure, and if I had two great alternatives, that's absolutely what I'd be doing regardless because, I mean, nothing's ever exactly identical. So if you have two things that are awesome, they're still going to be slightly different, and there's almost certainly going to be situations where you want one over the other at one point. And yet, again, just...

I would love to see greater diversity in true next generation file systems. We don't have it right now. It feels a lot like time traveling back to the 90s and, you know, everything is Windows. And that sucked too. I don't like monocultures.

And on the one hand, it's kind of weird to talk about a file system monoculture because, like, everybody's got a file system. You know, there's EXT4, there's UFS2, there's ZFS, there's Butter, there's BcacheFS, there's MacOS, there's – it has APFS and, you know, yada, yada, yada on down the line. I haven't even scratched the surface, but none of them really offer one.

what next generation copy on write file systems do. If I want a truly workable alternative to ZFS, I don't have an option short of going and buying, you know, like a million dollar Hitachi SAN. Alan, you tell me, what option did I miss other than going to massively expensive proprietary SANs to get the full feature set that ZFS offers or a reasonable facsimile without it just catching on fire every now and then like, you know, Butter or BcacheFS is likely to do? There's none that I'm aware of.

Precisely. I want more. And apart from anything else, it's the license thing, right? It would be great if we had like a GPL or... No, GPL would be terrible. No, that'd be great. To your point again, Joe, that's one of the things I was talking about when I said, you know, there's no such thing as two things that are different but exactly equal. So that's one of those differentiating features that it could easily put you in a situation where it's like, okay, yeah,

I slightly prefer ZFS, so I like to use it here because of licensing difficulties. It doesn't work in this environment that's, you know, allergic to cuddle. So luckily, I have this wonderful MIT-licensed file system over here or GPL or, you know, whatever. I want alternatives. I want usable alternatives. I hoped Butter would be that. It wasn't. Then I hoped BcacheFS would be that. It ain't looking good.

Please? Somebody? Well, BotRef has worked perfectly fine on my jankiness that we talked about in Linux After Dark. So I don't know what you're on about. And now the show is 2.005 admins, everybody.

Okay, this episode is sponsored by Factor. Ready to optimize your nutrition this year? Factor has chef-made gourmet meals that make eating well easy. They're dietitian approved and ready to heat and eat in two minutes, so you can eat well and feel great no matter what life throws at you. Factor arrives fresh and fully prepared, perfect for any active, busy lifestyle. With 40 options across eight dietary preferences on the menu each week, it's easy to pick meals tailored to your goals.

Choose from preferences like Calorie Smart, Protein Plus or Keto. Factor can help you feel your best all day long with wholesome smoothies, breakfasts, grab-and-go snacks and more add-ons. Reach your goals this year with ingredients you can trust and convenience that can't be beaten.

Jim tried Factor and said the meals were quick and easy to prepare, and liked that there was plenty of variety. Eat smart with Factor. Support the show and get started at factormeals.com slash 25A50OFF and use code 25A50OFF to get 50% off your first box plus free shipping. That's code 25A50OFF at factormeals.com slash 25A50OFF to get 50% off plus free shipping on your first box.

Let's do some free consulting then. But first, just a quick thank you to everyone who supports us with PayPal and Patreon. We really do appreciate that. If you want to join those people, you can go to 2.5admins.com slash support. And remember that for various amounts on Patreon, you can get an advert-free RSS feed of either just this show or all the shows in the Late Night Linux family. And if you want to send any questions for Jim and Alan or your feedback, you can email show at 2.5admins.com. Michael writes...

Do remote help programs such as Chrome Remote Desktop or Remina expose the supporter to vulnerabilities on the supportee's machine? In other words, can malware jump through the connection? So that depends a bit. More likely, if something like that's going to happen, it would be a vulnerability in the application, like Chrome Remote Desktop or Remina or whatever you're using, because that's the only code that's executing on your machine.

Although you do have to be careful when you use any file transfer capabilities that are in those programs to copy a file over, then obviously if you're bringing a file from a suspect machine, then that file might not be safe. But in general, unless there's a vulnerability in the application or you're using some remote health application you're not familiar with, then I wouldn't be too worried about a vulnerability on the machine you're helping with jumping to your machine. I do think it's important to note that anytime...

a program accepts external input, there is a potential for compromise because there might be some bug or vulnerability you don't know about in that program that's not properly sanitizing that data or that allows somebody to do something with a string of commands the person who wrote that program didn't think about. And so now you have a potential for compromise via that application.

Remote desktop applications are no exception to that. And by definition, it's two programs talking to each other across the network. So is there a possibility for malware to spread? Absolutely. Always is. Two machines talk to one another. You could have malware. But like Alan was saying, that would be a vulnerability getting exploited within that application. Like it's a bug. It's a security bug. If the question is,

Can malware just see, oh, there's a remote desktop connection to this other computer. Can I just automatically silently infected over that? No, the only way malware would be able to use a remote desktop application that doesn't have any vulnerabilities in it to install itself on another machine would be to literally use it exactly like you do and remote control the desktop of the thing and open a browser and download the file and run it like, you know, all where you can see what's going on.

The big concern that you had with like basically the invisible kind of automatic infection, not a concern unless there's an issue in that specific remote control application. Yeah. So if you're remoting into someone's machine to clean up the malware, your machine is not going to catch the infection. Now watch somebody tomorrow release some malware that like listens for incoming connections, notes the IP address and immediately tries to chase it back.

That's mostly a joke, folks. Just to let you know, I mean, I talk about that and it's that's something you could do. You could absolutely log the IP address of incoming remote control connections and decide to go attack that IP address yourself, you know, as this infected system. But in the absence, again, of a compromise in that actual remote control application, that doesn't really get you much.

Because the other machine would have to be listening for that connection. You would have to have credentials that worked on that other machine. You know, there's a lot going on there. You really don't have to have a huge concern. I used RDP to look at somebody's computer that had malware. Do I have malware now? No, not from doing that. Not realistically.

Michael also asked, "Does something like FortiGate make sense for a small professional office, or is it more complexity than value?" Well, for a small office, I think FortiGate is more complex than you usually typically need. If you're getting one of their smaller models, they're not hugely expensive. You will probably pay about twice as much as you'd pay for, you know, like a Netgear Nighthawk or something like that. You won't get any higher performance.

you will get some additional features and you'll get quite a lot of additional complexity in the interface. It's going to be considerably more difficult if you aren't intimately familiar, not only with networking in general, but with FortiGate's specific interface, it's a lot harder to get things done.

It's not to say that you can't. You absolutely can. And one of the big potential value adds for going to FortiGate is you may be able to more easily find people with FortiGate experience because it is pretty commonly used in like small offices, especially I see it a lot in like small local government where they don't go Cisco. They'll frequently go FortiGate.

So yeah, it's a perfectly cromulent product. It's not really my favorite. If you want to use it, that's perfectly fine. I would not get upset and rip FortiGate out of any network of mine. But if you're asking, do I need this? You probably don't need it if you would prefer something else.

If you're looking for potential replacements for FortiGate that tick most or all of the same boxes, have the potential to make higher ups happy, you know, offer all the features like, you know, you can do multi-win, load balancing, failover, all that kind of stuff. I think my personal choice that would override FortiGate, the next thing I would jump to would be OpenSense installed on a generic x86 box. Or if you prefer, you can buy a hardware direct from the company that produces OpenSense and

They don't usually have cheap models in stock, though. Their stuff that's direct from them with it pre-installed is usually the big boys, you know, in the like $1,000 and up category. If you just want like a $100 to $300 router, get yourself a cheap x86 box like Joe just bought a nice little N100 box with two NICs on it. Custom, I think about $150. Get something like that. Install a community version of OpenSense on it. You'll have all

all the features that you had on the FortiGate. You will not be in any worse of a position security-wise, and that would be my preferred approach. Right, well, we better get out of here then. Remember, show at 2.5admins.com if you want to send any questions or feedback. You can find me at joerest.com slash mastodon. You can find me at mercenariesysadmin.com. And I'm at Alan Jude. We'll see you next week. ♪