Deep Dive
Shownotes Transcript
Two and a half admins, episode 239. I'm Joe. I'm Jim. And I'm Alan. And here we are again. And before we get started, your customary plug, Alan, understanding ZFS in the real world, mistakes made, lessons learned, and future plans. Yeah, so I was on Lawrence Systems' YouTube channel last week, and we talked for about almost an hour about ZFS. And so wide range of discussion covering all kinds of interesting topics. Right, well, link in the show notes as usual.
Alibaba launches server-grade RISC-V CPU design. Yeah, I think the really interesting thing about this is that as we see specifically the Chinese government pushing more and more into RISC-V adoption and bringing this technology further to where it needs to be to be a true competitor with the heavy hitters these days, your ARM and your x86,
I don't think there's much that's going to slow that train down. I think there's a lot of room in this world for permissively licensed, very open hardware. And that seems to be the effort that's getting us there. And I'm all for it, frankly. Yeah, and as it's matured, RISC-V seems to have mostly come up with a solution to the kind of problem they had about compatibility, where every different RISC-V core would have some combination of 100 different extensions.
And, you know, if you didn't pick the right one, it turns out, you know, it wasn't good at division versus one that was and so on. But it looks like specifically the one they're talking about here for Alibaba, which I'm guessing they're trying to do for their hyperscaling cloud, has, you know, the common extensions like VectorMath and Crypto, but a couple others and is basically compatible with what they've defined as the RISC-V RVA23 profile, which is kind of just a baseline, I'm guessing, from 2023 that
It means that there will be some way to say, you know, this CPU is actually somewhat like this CPU. Whereas before every RISC-V, unless you got into the details of the 27 letters after the model number, it would be difficult to tell what features one CPU had versus another and was difficult to compare them. But now that they're getting this idea of profiles and kind of any CPU for this use case is going to be able to handle these specific things, I think it's a lot more interesting.
This has got some government officials in the US frothing at the mouth, not really understanding how permissive licenses work.
and being all like, we can't have China having this CPU stuff. But this was just inevitable, wasn't it? It's permissively licensed. Anyone can do what they want with it. Yeah, exactly. I mean, there were never really any secrets to keep there. Now, you can get kind of tangled up in the terminology here because it's not always a copyright issue. Sometimes it's a patent issue. And
You can have an issue where completely open in terms of copyright code can nevertheless not be copied because doing so would violate a patent and vice versa. With that said here, though, I think you're largely correct. I think it's just basically –
It's probably a lot to expect an average U.S. government official to know anything about CPU hardware or intellectual property licensing and expect them to know both of those things and be reasonable. Well, I don't know what you're doing right now, but it's not living in the real world.
But the bottom line is there's not really anything they can do about this. Well, also, this is not really the part that's the secret IP. The ISA, the instruction set architecture, is important, but it's really not the part that a lot of companies' IP is based on, right?
In the end, Intel and AMD are using the same ISA. They have some extensions to it that are proprietary to each other, and they tend to eventually achieve parity with each other. But this is the parts that are common. All this RISC-V stuff does not include any of the IP for PCI Express and the chips that actually do that and all the other parts that go on a motherboard.
Although we know that most of the Chinese manufacturers already have licenses for a bunch of those pieces and so on. So, you know, the RISC-V part isn't everything you need to build a whole computer without needing anybody else's IP. And so there's plenty of IP that can still be not exported or shared or whatever. I mean, it all is already being exported, though. It kind of has to be because, quick, name a thing that has no Chinese parts in it and has a motherboard.
But yeah, as far as, you know, electrical property revolving around PCI Express or anything like that. Well, if you have to have it to manufacture a motherboard, China absolutely has it because that's where all our motherboards are coming from in the first place. I mean, I shouldn't say all. Sure, if you look really hard, I'm sure you can still find the occasional Japanese manufacturer motherboard. But the vast majority of everything electronic is being manufactured in China these days. So, yeah.
The idea that we can realistically keep intellectual property away from the people who are building that intellectual property into real world devices seems kind of pants on head stupid to me. Yeah, the South Korea as well. But yeah, the point still stands. Well, also, you have to differentiate the two Chinas in that particular case. But yes.
Well, we had an email from Richard that relates to this. He writes, recently you all spoke about changes to ARM and their new efforts to make their own CPUs and how this may make RISC-V more attractive. Joe made a comment about there now being a third architecture, but this ignores IBM's Power, PowerPC and OpenPower. Power is fully open source and others have been working on other non-IBM power CPUs such as Microwatt.
And then he talks about Raptor computing with their Power9 systems, and there's even a PowerPC laptop project. The elephant in the room, the devil in the details that we have to take a look at here is when Richard mentions raptorcs.com as an example of an outfit that's selling power-based systems right now, their idea of an entry-level desktop system runs you five grand.
Just a single quad-core power PC CPU runs you one grand, and it's backordered. And that's not a very impressive processor. And I don't want to be completely unfair here. I mean, you know, price is a thing that, to a large degree, tends to get determined by how many people are buying something. And obviously, the world is not rushing to Raptor's door right now to buy, you know, thousands upon thousands of these power-based systems.
It's one thing to make allowances for that in an up-and-coming technology that you see gaining a lot more market share and taking off, which is what appears to be happening with RISC-V right now. Whereas power, like I said, it just –
That is not something that I think many people are seeing in the future. That's something that most of us are looking to in the past. Yeah, Apple have been from power through x86 to ARM at this point. Well, that was a much older version of power, but yes. And I think really to Jim's point, the problem seems to be economies of scale, right? The power PC can't get enough demand to get the price down to be competitive.
Power9 in particular looked kind of interesting before the likes of Ampere and so on came out with ARM CPUs that were on performance parity with x86, where you could get like an 80-core server that would beat out the biggest Intel you could buy at the time. And now the powers with only, you know, four or eight cores per CPU and having to do multi-socket systems in order to get anywhere near, say, 64 cores, whereas with an ARM, you can get 64 cores in one chip, and
It just seems like, yeah, we leapfrogged power nine. And even if they come up with a power 10, again, if they can't get enough momentum behind it to get the economies of scale to get competitive prices, then it's going to be really difficult. Arm has the advantage of, in general, the quantities you're shipping are so high that even if it's a slightly different or higher end chip, your manufacturer is going to have a lot more of the setup for it than trying to do something completely different like a power nine.
And RISC-V has got a lot of the low end as well, like microcontroller stuff in appliances, hard drives, that sort of thing. Yeah, stuff below kind of a general purpose computer type CPU. Yeah, RISC-V is really even coming for the Arduino segment of the market.
One suspects that eventually we're going to start seeing – and this is not obviously the most positive spin, but eventually we're probably going to start seeing RISC-V pop up in a lot of the counterfeit electronics out there. Recently, we've seen these counterfeit Bluetooth chips using the – what is it? ESP-32 chips.
And that's the kind of thing that we'll probably see a lot of RISC-V stuff being used for in the future because it is more powerful, it's very cheap, and it can be built to exactly the specs that you want it to. So it's kind of the perfect use case for counterfeiting things. Yeah, but at the same time, I expect the real ones to adopt it as well. Like the genuine chips is like, why would we pay for a license on some existing thing if –
RISC-V has got to the point where people are manufacturing chips that will do the job at enough scale that they'll be cheaper or even cheaper than something like the ESP32. That's kind of one of my watermarks for like when you're in the good period of the manufacturing and consumer cycle. When you're at that sweet spot where like the only thing wrong with the counterfeits is that like they claim to be the more expensive thing. But that's the only difference is it's not actually the nameplate that you're trying to spend more on, but it functions exactly the same.
I love that part of the cycle. I very much would like to get back to that part, please. Okay. This episode is sponsored by ServerMania. Go to servermania.com slash 25A to get 15% off dedicated servers recurring for life.
ServerMania is a Canadian company with over a decade of experience building high-performance infrastructure hosting platforms for businesses globally. ServerMania has up to 20 Gbps network speeds in 8 locations worldwide for optimized global reach. They have flexible custom server configurations tailored to unique needs as well as a personal account manager offering free consultations.
With 24-7 live chat support and support ticket response times under 15 minutes, you can always keep your systems running smoothly. Alan's been a happy ServerMania customer for over seven years, so support the show and join him at a hosting provider that truly delivers. Get 15% off dedicated servers recurring for life at servermania.com slash 25A and use code 25ADMINS. That's servermania.com slash 25A and code 25ADMINS.
I saw a couple of related stories recently, both about surveillance tech. The first one, why Combinator deletes posts after startup's demo goes viral. Yeah, so this one was terrible, but expected.
So the startup basically makes AI that looks at the security cameras from your factory floor and says, hey, employee number 11 is being 16% less efficient than everybody else. You should go yell at them and terrible things like that. And, you know, they made a pitch video directed at
the people that would buy this thing and that it was seen by the people that would be monitored by this thing. And they were like, hey, fuck you. Shocked Pikachu face, it didn't go over well with the people who would be expected to live beneath this system. Just the lack of clue in the pitch deck was just incredible. I mean, these are clearly folks who have either they've never actually worked at
as workers in their lives, or they have absolutely no truck with the idea of like class loyalty, because it is just relentless in the whole like, you know, oh, hey, it looks like this guy's slacking a little bit. Oh, hey, it looks like this person, you know, hasn't hit this target in this amount of time. And it's not just a bloodless sort of a, hey, we analyze this worker's performance, current and past, and are offering you these statistics.
The pitch deck is very specifically a middle manager just absolutely crawling up some poor worker's butt and confronting them with video this and the system the other and just –
One would like to use the word Orwellian, except it's just not gross enough. I mean, we titled an episode Kafka-esque recently, and that almost seems a little bit more appropriate just in that you feel like there's got to be cockroaches in here somewhere. Yeah, like if this one is like, well, all workers are just fungible, replaceable cogs in the machine, right? So if this one's not as good, we should just get a spare out of storage and replace the light bulb that's not as bright as the others.
Well, I wasn't really surprised about this because I've heard so many horror stories about Amazon and other kind of warehouse jobs being just subject to terrible surveillance and performance targets and constant cameras on them and all the rest of it. But there's a piece that is on Ars Technica that was originally from Wired.com called The Surveillance Tech Waiting for Workers as They Return to the Office. And it's quite a long read, but it's worth it if you want to be horrified because it's
It's unbelievable some of the tech they've got now, at least in the US and increasingly in Europe as well. Things like sensors under the desk to see whether you're sitting there or not. And in meeting rooms, they've got like humidity sensors to see if they're being used. And it's just...
a dystopian nightmare. Yeah, although if you think you have to be back in the office to have that kind of stuff, you're being a bit naive. A lot of companies are like, oh, you have to use a company-issued laptop, not bring your own device, and it's because they're by policy installing surveillance tech to make sure that you're working from home enough.
Whether that's if the camera's on and actually looking at stuff, but just, you know, have you moved your mouse recently? If not, then we're going to assume you're not at your computer and report to your boss that, hey, you only spent four hours actually using your computer this week or this day. And so you shouldn't get paid for eight hours of work. To the point where you can buy on Amazon a dongle that impersonates a mouse moving random patterns and they advertise them as 20 different patterns and stuff like that. Yep.
I hate to say this, but this stuff isn't going to go away for as long as people tolerate it. There has to come a point at which you say, no, I won't do this. No, I will actively resist this crap. You want to put sensors under my desk and turn a camera on me. I mean, at some point you have to say, look,
This is oppressive. This is effectively warfare. I'm not on the same side you are. You're not on the same side I am. And if we can't negotiate something more livable, we just plain aren't on the same side and it's going to have to stay that way. But so many people are trapped in jobs effectively. Yeah.
I agree with that completely. I'm not trying to shame anybody and say, oh, well, you know, you specific person should have done this specific thing. There are a lot of people that are trapped in a grind. God knows I have been trapped in some horrible grinds myself from time to time. But everybody, absolutely everybody knows.
Every once in a while has some kind of an opportunity to do something. And if you take the lazy option, if you take the just get along option, if you just do whatever is easy, well, then you keep getting railroaded into whatever the folks who are doing the railroading want.
And it's pretty clear right now that the folks running the railroad aren't real interested in the passengers. Well, at risk of coming across as a complete lefty, I saw a panel at a recent conference from a tech union. And their whole pitch was, yes, if you're working in tech, you should be in a union. It's not just for blue collar workers. And they made the point that at least in the UK, it's illegal for the company you work for to even ask you if you're in a union.
And so my takeaway from that was join a union, basically, and collective power to stop this kind of shit that we're talking about here. Join a union, absolutely, where that's possible. I certainly recommend joining a union, but also you have to take some...
individual personal responsibility and accountability. And it's not as hard as a lot of you think it is, I promise. There have been a lot of jobs that I have not taken until I lined through entire paragraphs in red ink in the contract they offered me because the contract was not reasonable. I have yet ever to not get a job because I refuse to sign a contract with onerous terms in it until we fix those terms.
What I have found is that, you know, when I started out doing this kind of thing and I had my heart in my teeth every time I would say, no, this won't work in this contract. Oh, that's it. They're just going to let me go. And that never, ever happened once. What I did learn is I was wasting my time trying to come up with less onerous restatements of, you know, what they wanted in those contracts. You just whack through whatever you don't like with a red pen and leave absolutely nothing in its place.
It's not your job to propose something more reasonable. And in fact, they're not going to accept it if you do because that would require further legal review. Whereas frequently your manager can just say, oh yeah, whatever, with something just lined through as opposed to something else that you've inserted. So by all means, you need to read your contracts. You need to object to things that you don't like in them. You need to treat it as though you're
you do have part of the power at that bargaining table. And you really do, I promise. Yeah, they're always going to prefer to use their language. So as Jim said, don't waste your time coming up with a counterproposal. Just say, this doesn't work for me. And they'll either say, okay, or how about this instead? And in the vast majority of cases, in my experience, it's just a straight line through. Nothing else replaces it.
And when you start to realize how easy it is just to whack through entire offending paragraphs and clauses of terms that you don't like and how they don't even bother coming back with anything else when you do that, you really eventually should start asking yourself the question, why in God's name have I not been doing more of this? And like, how bad do I screw myself every time I just blindly accept whatever ream of nonsense some company throws at me?
And, you know, to some extent we all do it. We've we've all just clicked to get through the TOS, especially, you know, anything electronic where it's like you can't open the program until you say you agree. And, you know, there there are no provisions to like line out parts of the contract or whatever. OK, yeah, we're all guilty of not reading the whole thing and just clicking and saying whatever.
I will say this, though, even when you do that, what should be going in the back of your mind is this is nonsense and I do not agree to this in any way, shape or form. What I'm actually signifying with this click is fuck you, I don't care. Now, that won't hold up in a court of law, but it absolutely should be how you're thinking about your relationship with this company, which says you can't play our fun little game on your phone until you agree to a five page terms of service.
I feel like we're coming at this from our ivory tower of entrepreneurship, and at least in mine and Alan's case, affordable healthcare. There are a lot of people in the US who don't have much choice but to just accept this bullshit. And it's fine for you to say don't accept it, Jim, but...
I feel like there are a lot of people who have no choice. It goes back to your point about the unions. If everybody in the office agrees that they're not going to put up with a censor under the desk, that's a lot different than one person not doing it. Yeah. But if you aren't the one person who refuses to do it, why do you expect everybody else to have the courage that you didn't?
And, you know, your point is valid, Joe. You know, different people are in different situations, but I'll push back on that and say this is 2.5 admins. We're addressing folks who largely are they can either afford to have some kind of an IT hobby pretty seriously or, you know, they are IT professionals.
I don't think it's fair to just say, oh, well, you know, we've got a bunch of carpenters in the office who have no choice but to go and do whatever carpentry job, you know, the office threw at them or whatever. Like, that's not who we're talking to here. Man, carpenters make more than IT people in the UK. You'll notice I didn't say anything about, you know, the relative affluence. Just you maybe get more in the way of like complicated contracts and stuff if you're IT, and that's where you get the chance to like redline things more. But yeah.
Honestly, I mean, even if you are working as a carpenter, yeah, I mean, read through your terms of employment and, you know, if they're laying claim to your firstborn child, don't agree to that crap.
Okay, this episode is sponsored by people who support us with PayPal and Patreon. Go to 2.5admins.com slash support for details of how you can support us too. Patreon supporters have the option to listen to episodes without ads like this. And it's not just this show. There's Late Night Linux for news, discoveries, audience input, and misanthropy. Linux Matters for upbeat family-friendly adventures. Linux After Dark for silly challenges and philosophical debates.
Linux Dev Time about developing with and for Linux, Hybrid Cloud Show for everything public and private cloud, and Ask the Hosts for off-topic questions from you. You can even get some episodes a bit early. We've got a lot going on, and it's only possible because of the people who support us. So if you like what we do and can afford it, it would be great if you could support us too at 2.5admins.com slash support.
Let's do some free consulting then. But first, just a quick thank you to everyone who supports us with PayPal and Patreon. We really do appreciate that. And if you want to send in your questions for Jim and Alan or your feedback, you can email show at 2.5admins.com. Jono writes, I'm building a server running TrueNAS scale and looking into disk encryption. The concern being someone gets into the garage and swipes the machine off my rack.
Any best practices here? I suppose I need to SSH or log into the web interface every time the machine reboots to manually enter the key. If the machine reboots, should I have it alert me somehow that the drive needs decryption? Also, what about self-encrypting drives? Is that worth looking into? So I'll start with the self-encrypting drives. Those don't do what you think they do. So a self-encrypting drive generates a random key and encrypts all your data with it, and
and basically has a command you can send it that will generate a new random key and replace the old key with the new key, meaning all the data on the drive is now not readable because it was encrypted with a different key. That is now gone. So self-encrypting drives only really allow you to securely erase them. They don't actually protect your data in most of the other ways. Or some of them will have something where
Again, similar to the software-based hold disk encryption is you'll have to send a command feeding it the key in order to decrypt the drive every time it power cycles. And you would have kind of the same problem you have in here where you need to SSH in and log into the web interface and then feed it the key.
The biggest problem with all that is obviously you don't want your boot device to have this so you can still bring the machine up. And the fact that, yeah, you need some way to decrypt the data before you can access it. And that can get rather problematic. One other option that TrueNES Skill and ZFS have are the dataset encryption, which is not whole disk encryption. It's different.
but allows you to have different data sets have different encryption keys so that you can encrypt some of your data and not all of it. So that, you know, some of the data will be accessible without the decryption. So after a reboot, most of the data is usable right away. It doesn't have to wait for you to SSH in and enter the password or the key. But anything that's really important will make you wait and then go in and put in that key to do that.
I do have a suggestion for a fun workaround here that's kind of in the middle ground. Now, I'm not sure how much of a pain in the butt this is or isn't to implement on TrueNAS specifically.
But like on a vanilla machine, if you wanted to have something that addressed the concerns that you have, that somebody will swipe the machine and get access to your data, but you don't want to manually put in a key every time it boots, you can be a little sneaky. Hide a Raspberry Pi in the ceiling connected to your Wi-Fi network that will supply the key to a machine that shells in and grabs it, have your main machine boot,
boot up, shell into the Raspberry Pi, temporarily grab the key and unlock the drives, and then you're good to go. So what happens then is if somebody steals your server but didn't know about that Raspberry Pi, you've got stashed on the ceiling. Well, they made off with your server, but they can't get to your data when they get away because they left the key behind. So ZFS makes this even easier. You can set the key location for a ZFS dataset to be an HTTPS URL.
So when the machine boots, it'll reach out over HTTPS to the Raspberry Pi and the Raspberry Pi will then do whatever it wants to do to decide, yes, you're really that machine and I really should give you the key and then we'll give you a key to decrypt the drive. Then if somebody steals the machine, you just tell the Raspberry Pi to stop giving out that key or, you know, they're not on your land so they can't reach the Raspberry Pi. And so you do all that with even less automation.
The fun part here is since it's HTTPS, I kind of feel like we shouldn't have to spell this out, but we definitely need to spell this out. You need to make sure that that HTTPS URL is not publicly accessible or you will defeat the purpose. And that also means you're going to need to figure something out in terms of like private certificate infrastructure because you're not really going to be able to easily do let's encrypt certs with non-accessible systems.
I can see Alan wincing over in the other window over there. And, you know, yeah, you can absolutely set something up where one machine is publicly accessible and completes the Acme challenges to update the cert and passes it down. But I was thinking of the DNS based Acme validation so that it works for internal stuff. Fair enough. But the point is, you need to be certain that you set it up so the HTTPS part actually works while in your house, but will stop working if the machine leaves the house.
Yeah. And so generally you'd have more than just an HTTPS URL. You'd have to have it give a username or some other mechanism, basically make an API call, prove that it's the right machine and that it's still on the right network and it has all the other information.
But that's kind of still a bit left to roll your own solution, which is probably not what we normally advise. So if somebody has a strong enough use case, obviously, you know, your NAS and your home garage is probably not that. But if somebody has a commercial use case for actually getting this to work so that your servers have encrypted data sets and they automatically decrypt, but, you know, based on the TPM chip or an authorization server inside your network, we'd love to
design a real solution for this. And just one last note here, now that we've gone through the whole ball of yarn in terms of the ability to use the HTTPS URL as the key location, if setting up your own
Local DNS so that you can resolve a private IP address to a publicly accessible domain that you're using DNS-based authentication to get ACME certs for. If all of that sounds like more than you can really face right now, you also have the option of just setting up a script to shell into your Raspberry Pi, grab the key, and use that to...
unlock your locked data sets after you boot every time. That's going to be a whole lot easier to just, you know, kind of hack together in your home lab. Yeah, or you can do the thing where you just have to SSH in and run ZFS load key dash A
and then type in the passwords or whatever. Yeah, just don't set automatic reboots on it. Just do all your reboots manually and just be there to attend to it. No, no, no. We're trying to leave people with a setup where they don't have to manually enter the key and unlock things because there's locally present infrastructure that will let the unlocking happen essentially automatically. But it's a little bit of security via obscurity because you're banking that the attacker won't be bright enough to know they need to look for hidden infrastructure with a key to unlock the data automatically.
But in this case, that seems like a fairly good bet when all you're really trying to do is secure the server in your garage in the first place. Yeah, if somebody's targeting the server in your garage, they probably know a little bit more about your network than they should. Like that there's a NAS in your garage. Well, or that they just know that, you know, your garage was something they could get into with a crowbar and they want to look for shiny things inside. In which case, are they really going to be interested in your data? Yeah.
Probably not. Probably not, but, you know, why take the chance? The person who buys your stolen ads off eBay might be, though. Maybe, maybe. Right, well, we'd better get out of here then. Remember, show at 2.5admins.com if you want to send in your questions or your feedback. You can find me at joerest.com slash mastodon. You can find me at mercenariesisadmin.com. And I'm at Alan Jude. We'll see you next week.