KCP, with Marvin Beckers
31:45 Share

ABDEL SIGIWAR: Hi, and welcome to the Kubernetes Podcast from Google. I'm your host, Abdel Sigiwar. KAZELIN FIELDS: And I'm Kazelin Fields. ABDEL SIGIWAR: In this episode, we speak to Marvin Bekers. Marvin is a team lead at Kubernetes and contributor and maintainer of the CNCF Sandbox Projects KCP. KAZELIN FIELDS: But first, let's get to the news.

Docker announced their subscription prices are increasing. Starting November 15, Docker Pro is moving from $5 to $9 and Teams is moving from $9 to $15 per month. The price for Docker Enterprise remains the same. The Linux Foundation announced intent to form a Developer Relations Foundation or DRF for short.

The new foundation aims to address some of the challenges DevRel organizations face across the industry like lack of clarity, recognition, and the difficulty of measuring impact. The DRF will partner with industry leaders to address these challenges and already received support from multiple companies. NetApp Insights took place September 23rd to 25th in Vegas. NetApp CEO George Kurian emphasized the importance of data for AI workloads.

The company is integrating AI functions into its storage platform, enhancing interoperability to eliminate data silos, and optimizing storage for cold data. Partnerships with hyperscalers like AWS, Azure, and GCP, and NVIDIA are highlighted as key to delivering AI solutions across diverse workloads, emphasizing cost optimization, security, and sustainability in data storage. And that's the news. ♪

Hello, folks. I'm today with Marvin Beckers. Marvin is a team lead at Kubernetes and the contributor and maintainer of the CNCF Sandbox projects, KCP. Welcome to the show, Marvin. Thanks for having me. So before we started recording, you told me that KCP does not stand for Kubernetes Control Plane, which is the thing that I was, that's how it was described to me the first time I heard about it. So what is KCP? KCP is a control plane.

So it's not the like, it's not an official acronym, but I think the description is kind of spot on. So it's a cloud native control plane for Kubernetes like APIs. So you get the idea, even if it's not what KCP stands for. Got it. Okay. So the cloud native control plane for Kubernetes like APIs is

So then the next question is, what is actually KCP? Like, what does people use it for? KCP is basically the idea that the Kubernetes resource model, like all the engineering that went into the Kube API server, it has become more useful than just using it for container orchestration. So it's like a decade now of engineering and technology that went into it. And KCP is basically the idea that this is extremely useful for hosting any sort of API project.

And I think in the ecosystem, we see this already with a lot of projects that don't really use it for container orchestration. They use it as a control plane. If you think of projects like Crossplane, for example, that uses Kubernetes as a control plane and builds on top of the API, not necessarily on top of the container orchestration part. So the next question becomes, why would people use

KCP as a control plane instead of just using Kubernetes, right? KCP has several features built in that are, well, primed for this specific use case. So because KCP thinks about APIs and not primarily about container orchestration, KCP has ways to manage

APIs at scale. And KCP also has a multi-tenancy concept that you don't find in Kubernetes, which is called logical clusters. And logical clusters basically allow you to serve multiple Kubernetes API endpoints or something that looks like a Kubernetes API endpoint from the same KCP instance. So this way you can basically talk or you can basically get a lot of Kubernetes API server endpoints with very little cost. Interesting.

So you mentioned control plane as an example, and that was one of the, I mean, yeah, Kerem, I've seen Kerem before, the Kubernetes resource model,

And when you come into the context of anything that does something like cross-plane, then the question that naturally people ask is, do you actually need to run... I think you answered this question partially, but do you actually need to run the entire Kubernetes control plane with the API server and all that stuff just to run cross-plane? And it seems to me like KCP is the answer to this question. If you don't need the container orchestration part, if you don't need the container runtime part...

you could just run KCP, which gives you the control plane part, right? Yeah, that's exactly the idea behind KCP. It's basically kube without the container orchestration. Interesting. And then you have the multi-tenancy part, which basically...

I mean, I'm assuming here, correct me if I'm wrong, is that only for cloud providers trying to offer things on top of KCP or would that be useful also for like end users? I mean, we consider KCP kind of a foundational technology, sometimes call it a framework, but it's not like a programming framework. So that sometimes confuses us a bit.

But basically, it's an API server, it's a control plane. So what you use it for is kind of up to you to build it, what you want to build with it. So for example, if you build a ZaaS, like what you say, like a cloud provider or like a ZaaS provider, they could basically start with KCP with a strong multi-tenancy setup and go from there. We could also think of this in terms of being useful for things like platform engineering.

where basically you, well, you're not a cloud provider, but you're building an internal platform for developers or other people in your organization to use. So this is kind of like two of the, I think, several use cases that you could bring KCP into. Right. Interesting. Okay.

And I just forgot to say this as the beginning of the introduction. If the sound looks a little bit weird compared to how people are used to hear it, it's because we're actually recording at a conference. So we're here at Container Days Hamburg. It's my first time at this conference. It's pretty cool. We can probably talk about it at some point, maybe, if you want. But let's just stay on the KCP topic. So I went to the documentation. I did some reading. I read some terminology.

And it's a little bit confusing, but it's something you would expect from, as you say, like a building framework, like something that is meant for people to build stuff on top of. So let's go through these basic concepts. What is a logical cluster? Let's start there. Yeah. So a logical cluster, it's basically a way to partition like an API server. I'm not going to say Kube API server because it's not exactly the Kube API server, but it's

something built on top of that. It's a way to partition basically the storage layer, so etcd in most cases, and your access to the API server into multiple ones. So basically each logical cluster that you talk to, it has its own space in etcd, in the key value store, and it has its own endpoints

that you can talk to, and it acts like a single, standalone Kube API server. Even if behind the scene it's only one instance of KCP? Exactly. You would be able to expose as many endpoints using logical clusters, because there is one endpoint per logical cluster, right? Yep, exactly. And then how does the multi-tenancy at etcd done? Do you add a value to this? How is that done? There's a prefix, basically, that is prepended

oh and that basically includes okay which logical cluster is this data related to and then the api server kcp it's well takes that information and returns different data depending on well what logic cluster you're talking to got it got it and then does that mean that every logical cluster implements the full permission model of kubernetes

Each local cluster has an adjusted authentication and authorization layer of Kubernetes that we all know. But because each local cluster has its own objects, that means that in each local cluster you will have different RBUG objects.

being applied to your request. So the cluster roles are, well, they, I mean, they're cluster scoped in like normal Kubernetes terminology, right? In this case, like we could call them like workspace scoped. So cluster roles are different from works, sorry,

from logical cluster to logical cluster and that basically gives you different permissions depending on what logical cluster you're talking to. Got it, got it, interesting. And so then you also have a concept of physical cluster. So what is a physical cluster then in this case? Yeah, so this is something that I think we want to talk a little bit about because KCP at some point had like a bigger scope. It was basically looking more at solving the multi-cluster scheduling use case

Since the project is part of the CNCF Sandbox, we have decided to scope down a little and basically focus on the control plane API parts.

In KCP, physical clusters, and I guess to some degree they still do, it's Kubernetes clusters that you can run workloads on. That is basically the idea, because KCP is only a control plane, and because we took away the container orchestration part of it, if you want to execute software in the Kubernetes model, you still need something powering that.

So in this old component of KCP that at the moment is well frozen, that is called transparent multi-cluster, physical clusters where basically your workload clusters, where you were scheduling your multi-cluster deployments or something that were then spread around multiple clusters.

That is no longer the case. But of course, you still need to think about physical clusters because KCP is only, again, like a control plane. So that means if you want to run software against it and you need to have your own controllers, your own operators maybe, then of course you still need Kubernetes clusters that power that. So in a sense, physical clusters are not...

formal term anymore, but you still need to think about them. Got it. And so we would, I mean, I'm talking about the previous scope, right? You would still run KCP, create logical clusters, which will give you this like separated control planes, let's say, or endpoints. And then you would

make a physical cluster connect to a logical cluster, right? Well, make a physical set of clusters and by physical means virtual machines or whatever. They will dial up to the endpoint which is part of a logical cluster. This is basically one of the patterns you can implement. So because in KCP like all we think about is APIs. But if you have, if you define an API where you would say, okay,

I want something to happen and most APIs are doing something. Then you can basically implement a syncing component or controller that takes the information from the central control plane that you're running with KCP

and distributes that to physical clusters. And that can also be higher level resources, right? We don't necessarily need to create deployments or something in KCP. We can say, for example, let's say you want to implement providing databases. So then in KCP, you would create like a database API. You would create that and then it would be synced down to physical clusters where like the actual database instances then would be started.

Got it. Does that mean you could, I mean, this is like a hypothetical example. You tell me if I'm hitting on the right nail here. Would that mean that you could potentially use KCP if you are a cloud provider that runs managed Kubernetes? So you could use KCP as the control plane to create user clusters? I think that's definitely completely in scope for it, yeah.

Maybe using something like the cluster API, for example. You would run KCP, run the cluster API on it, and then when a user goes, like, give me a cluster, then that becomes a YAML file that you submit to KCP, which then provisions a Kubernetes cluster with all its components. Yeah, so usually implementations, they need to be KCP-aware, so you can't run, like, stock cluster API against it. But from the principle idea, yes, exactly. That is how you could use KCP as a central control plane and let it handle...

and sync it down to the workload clusters. Yeah, the reason I was asking this is because it's a pattern that is very common, actually, in all the providers that does some form of managed Kubernetes. Even like on-prem Kubernetes can do the same idea. You have a control plane, which then provisions other control planes and worker nodes, right? And so how are logical clusters designed

than virtual clusters? So I think that depends a little bit on your definition of what a virtual cluster is. So usually, I think this refers to a concept also known as hosted control planes, where you take a subset or maybe a full Kubernetes control plane and put it in pods basically as just another workload on another Kubernetes cluster. But of course, that means maybe they even share a data store, but you need to spin up a new API server. You need to maybe spin

spin up maybe new etcd for each of them, for each virtual cluster. And this is more efficient than, of course, provisioning three nodes and make them the VM three node kind of control plane pattern. But still, logical clusters, because they only exist in the data store and the API server does the separation, you're not spinning up any new processes and you're not spinning up any new data stores. So even hosted control planes, they can come up pretty fast.

Yeah, because they're pods. Yeah, but a logic cluster in KCP takes like a second or so to be there and ready. Yeah, so that's actually a very, very important distinguishing because in typical hosted control plane virtual clusters kind of implementation, the control plane is just pod, right? That runs the entire control plane stack, which has the API server, the controllers, and everything, right?

But in the case of KCP, it's essentially... I've seen it in the documentation. It's like a YAML file that you create, but it doesn't really spin up anything. It just separates its CD storage layer and then implements some sort of authorization authentication on top of that, essentially. Yeah, exactly. It just takes basically what you give as a name, as a partition key, puts that on the storage layer, and then it's basically done. Provisions like, I don't know, five, six resources...

And yeah, then you're done. Oh, awesome. Because you talked about workspace. What is workspace? Yeah, so workspace is kind of a higher level resource for managing logical clusters. So a workspace basically models...

a Kubernetes API that you are provisioning the way that you want to. So in a workspace, when you create a workspace, you get a logical cluster. And basically workspaces are the user-facing resources because they are logical clusters that don't have a workspace representation. They're like the system clusters, basically something that the administrator would interact with, but not a normal user. So users primarily create workspaces and then get...

So they get a Kubernetes API they can talk to. Yeah, just a way of organizing resources, essentially. Cool.

And then there is this other concept, and I'm still going through the concepts from documentation. API exports and API bindings. What are these? In Kubernetes, when you create a CAD, you do basically two things at once. So first of all, you define a schema for the API resource, but you also register it with the API immediately. So you have the schema definition and it gets immediately registered. But these are kind of two steps. And KCP separates them. So in the API exports,

a service provider, like someone who wants to provide an API, they would define a schema of the API, but this schema would not become available as an API resource. So it's just the schema, basically. It's just, okay, here is an API that I am offering. Got it. And the API binding is then the consumer side where the consumer says, okay, I want to bind this API because I want to have it available within my workspace as a resource that I can create objects from.

Got it. So you basically separated the way Kubernetes does CRDs into two separate steps. You would create the schema first, and then to make it available, you have to create the API binding in the workspace to be able to consume that API. Exactly. And that's why you can also convert the CRD very easily to a resource schema in KCP. Is that done for this implementation? Is it for security reasons or for efficiency reasons? Why was it...

implementing this way. It's more so that API consumers can mix and match which APIs they want to have available. Because in Kubernetes, when you create a CAD on a cluster, well, that is the API resource that you'll have available, right? Yes. And usually it can only be provided by a single provider, like a controller, an operator that you start using

And in KCP, you can do this mix and matching and you can also consume APIs from different providers. You can say like the same API is provided by two, let's say, teams in your organization. And you want to choose which of the teams should be providing that. And then you select one of the API exports, basically. Got it. I mean, it's hinting toward platform engineering in a way, right? That you could...

It's kind of like having a catalog of APIs available through the exports, and then you can bind your workspace to an API to be able to use that specific API, essentially. Yeah, exactly. Got it. Cool. Awesome. So we talked briefly about stuff that KCP can do, like examples and stuff. But I want to go back to the concept of multitenancy. Like, why is multitenancy important?

I mean, I don't know if it's easier or more complicated, but how does KCP solve multi-tenancy that cannot be solved with traditional Kubernetes? So we talked about this earlier, right? With virtual clusters, you could spin up a Kubernetes API for everyone. Each team that wants to basically work on their own, want to be self-sufficient in a way, and want to do self-services. But that is quite costly. And because KCP workspaces are basically, well, they're very cheap, they're easy to create,

and because you have these interactions between workspaces. I can create an API export in one workspace. As a service provider, I own the workspace. And service consumers in their workspaces, which they own, which they have full permissions to,

they can create the API bindings and consume APIs across these boundaries. So teams are basically using different Kubernetes API endpoints, but they still share the system from which they can consume APIs from the service catalog that you mentioned. Yeah, so it sounds like it gives you the option to reuse APIs, while if you would do something like virtual cluster, then you don't have that option. If there is a CRD, you need to install it basically in your own control plane, right?

Cool. And also the other thing that is now coming to mind is like it makes, I mean, multi tenancy is not only about separating the worker parts or like the data plan if you want, it's also about separating the control plane, which is, as you said, it's cheap in KCP, because you're literally just creating a YAML file, right? They're not spinning up anything.

So how does KCP itself relate to the Kubernetes upstream control plane? There is like a cap here that you mentioned, 4080. Can you talk about that? Yeah, so KCP is built on mostly Kubernetes code. So we're using large parts of that. And during development of KCP or like in the last few years, it became clear that

In Kubernetes, there is a Kube API server, and this has all the container orchestration parts built in. This cap is basically more or less about code organization, where there is a step before the Kube API server, which is like the generic control plane. And basically, the Kube API server is now built on this generic control plane, so it takes this generic one and adds the container orchestration bits to it. Oh, got it. Okay.

So if you want to use the Kubernetes Resource Model and what Kube API Server makes Kube API Server, you can take this and build your own API component. So we've been trying to work with Upstream to clear out the code and make sure that it gets this generic control plane available. But we think that if you have this generic use case and KCP is maybe too much for you, then hopefully this will now enable

community members to build their own control planes based on the Kubernetes resource model. And when you mentioned that this cap is all about code organization, does that mean that up to before this proposal was implemented or is being implemented,

Does that mean that in Kubernetes itself, the control plane plus the scheduler and all the other components, they were tightly coupled together? Not the scheduler, but within the Kube API server, previously, basically the container orchestration bits, like the core APIs and all of that, were all kind of built in. Not the best one to speak about this, but basically if you take the earlier code out of it that just initializes the Kubernetes resource model and all that,

But without registering the container orchestration bits, you have now the separation in the Kube API server. You've got this prototype, but this generic control plane, and then the Kube API server built on top of that. Which then has the container orchestration built in. Okay, so it sounds like separating Kubernetes itself from Kubernetes to make it possible to spin up the generic control plane if you just need that, or run standard Kube API server plus container orchestration plus whatever, right?

Cool. That's more like code organization. All right. So we talked about this briefly. And I mentioned that when you were talking about this concept of like consuming APIs across workspaces, it sounded to me like platform engineering, in a way, like always got its platform engineering. How do you see like KCP use today in the context of platform engineering? Like, do you have actual use cases that you have seen before?

I think we see these use cases out there every day. Like with all these projects, like we talked about them earlier, like a lot of platform engineering is at the moment is based on extending the Kubernetes API with useful APIs. And I think this is basically, well, exactly the use case that KCP fits in and centralizing this because usually I don't think you can, you know, have one central Kubernetes API. Like that's not what it was meant for, right? Mm-hmm.

but you can build like one central KCP control plane and which then everyone can go to.

have their, well, the bits of platform engineering in there. So say, okay, I want to create databases from it. I want to create infrastructure as a service from it. And basically from there it gets, well, scheduled where it needs to be or where it needs to go. But yeah, I think what we're basically seeing is platform engineering has already taken all the benefits of the Kubernetes resource model. Yes. And probably is...

well more suited with something like kcp because it's meant for this kind of use case got it yeah like one of the ways i could see kcp being used in the context of platform engineering is if you need to run for example a central cross plane just an example you would just run it on kcp you don't need kubernetes for that right you could just

KCP, cross-plane, and then that's it. And then people can do whatever they want. I think that would be the idea, yeah. So the controllers usually need, well, they need to be KCP-aware. They would need to run like a cross-plane, like a version of cross-plane that is KCP-aware. But in principle, yes, that would be the idea. Yeah, I mean, I was talking to somebody from Kubernetes, and they were mentioning, without mentioning a name, a company, right? Let's say a company, a nameless company that does...

hardware, right? And they were thinking about using KCP inside their hardware for

Can you talk about that? Do you have an idea about that? I think in general it's fair to say that we've been talking, like the maintainers have been talking about KCP for quite a while. I joined the maintainers last year and have also been to conferences and talking about it. And I think what we're seeing is more interest because more and more organizations are realizing, hey, the central control plan could actually help us consolidate what

what we're running here. And so I think there's a need for that and a lot of companies and organizations are looking at it. And yeah, we think that that hopefully will result in like official adoption at some point. Got it. Got it. Interesting. Okay.

Cool. So the last question would be, how can people contribute to KCP? It's a sandbox project, right? Yeah, exactly. So where can people find you? Are you looking for contributors? Obviously, we're looking for contributors. There is a website, kcp.io, which is the project website from which you can find the documentation. You can find also our bi-weekly community meeting, which is happening on Thursdays right now.

Everyone is welcome. We really like to see new faces and see them join, introduce themselves and maybe talk about how KCP can be useful to them. We have a Slack channel on the Kubernetes Slack. So that's kcp-dev. So we also love people joining there and talking to us. I think that's the main part at the moment. Our code is obviously on GitHub.

And so we're very much looking for contributors, contributors to the code, contributors to documentation, because obviously I think we found out that this is like a fairly complex topic to talk about. So the more documentation and the more we help people get into it, the better. Awesome. And you're going to be at, I mean, the project is going to be probably at KubeCon North America. Yeah, there's going to be one of the CNCF projects there.

I think they're called. In the project pavilion, but yeah. Yeah, exactly. And there will be some maintainers from KCP there. Awesome. So then people can go find the maintainer and talk to them. Thank you for being on the show, Marvin. Thanks so much for having me. Thank you very much, Abdel, for that interview. We both are familiar with another employee of Kubernetes,

Mario Faland, whose last name I probably butchered right there, but who I work with closely in open source. And I know that you've interacted with it, events at least, and things before. So I was excited to hear about this topic, which is also something that I think Mario works on. So tell me a bit about it. Yeah, I should start by saying thank you to Mario, because Mario provides actually the hardware. So we recorded the episode at the Container Days Hamburg.

For the background, Hamburg is a city in Germany and that's where Kubernetes is originally from. That's where the company was created originally. And so the conference is run by Kubernetes and we were chatting with Mario and he said, yeah, I'll just bring all the recording equipment and you can just use them. So he provided all the equipment. I didn't have to bring anything with me. And he also provided the guests. Thank you, Mario. Thank you, Mario. So yeah, that was pretty cool.

First of all, if you hear any background noise in the audio itself, if you have heard any background noise in the audio up to now, it's basically because we recorded in open space in a conference. So that's the background there. So it's just, you know, it's not our typical recording setup.

But no, we had a super interesting conversation. So I heard KCP before. I was under the impression it means Kubernetes control plane, but it doesn't. Ah. Yes. It clearly doesn't. And they are emphasizing it doesn't mean Kubernetes control plane. It is KCP. That's it. Okay. It doesn't particularly stand for anything, huh? No, it doesn't stand for anything. It might have started as...

KCP at some point, but they don't intend to mean because it is a universal control plane. It's not only a Kubernetes control plane, essentially. So it's a universal control plane for other types of platforms as well as Kubernetes? Yes.

It's universal in the sense that you can use it as a control plane to run crossplane, for example. And then what you would use to control with it or with crossplane is not necessarily Kubernetes. You would, you know, use crossplane to spin up all sorts of cloud resources. So it's universal in that sense that it's not necessarily just for Kubernetes. You can use it for anything else, technically. Okay.

Remind me what Crossplane does. Crossplane is this open source tool that allows you to use Kubernetes resource model, Kerem, to declare cloud resources and then it will spin them up for you. So you could define a YAML file that defines, for example, a Cloud SQL database or a PubSub topic or something like

So instead of using infrastructure as code like Terraform or Pulumi, you... Use KRM. Yeah, exactly. Exactly. All right. So it's creating a control plane that uses a Kubernetes-style API to manage objects? Correct. Yeah, that's a very good way of putting it, yes. Okay. And then if you write your own operators and CRDs, then you can use it because that's just standard Kubernetes stuff, right? Right. Okay.

So this is a sandbox level project with the CNCF. It just became... How long has it been a sandbox project? I don't remember exactly, but very recently. Okay, yeah. That's what I thought. Apparently they are on their way to not graduation. What's the thing? Incubating, yeah. So they are on their way to incubating. Yeah. So at this sandbox stage, usually a sandbox project within the CNCF is kind of just getting its legs under it from a maturity standpoint. They're...

Establishing, of course, what the project is, working on building the resources so that it's easy for more folks to join in and contribute, working on getting more users. So once you reach the incubating stage, you're going to have to work on building the resources.

You are usually starting to branch out in terms of recruiting more contributors, hopefully from other companies than the original founding company of the project, since most projects do come from some kind of company. And then you are starting to have a few users that you can refer to.

So did he talk at all about the journey that they've been through so far? I mean, the project started from within Kubernetes because they have their own platform that they basically they provide a platform to be able to spin up Kubernetes on-prem or on other cloud providers.

But there was reasons why they didn't want to mention some names. I mean, they didn't want to mention the modern day recording. I know some big companies that are actually using KCP internally. Usually with sandbox projects, it's kind of like that. Like a few companies are kind of testing the waters on it. They don't want to call it out until they're confident in it.

Correct. But from what I understood, they are pretty confident in the maturity of the products itself or the projects and the maturity of the community. So they are... Of the technology, right? Or the technology as well. So they're very confident that they are on the right path toward incubating at least. Cool. I look forward to seeing more about it.

As you can tell, I haven't listened to the interview yet, so I'm going to have to go and do that after this. But thank you so much, Abdel, for conducting that interview. I'm excited to hear more about this sandbox project in the CNCF. Thank you.

That brings us to the end of another episode. If you enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on social media at KubernetesPod or reach us by email at kubernetespodcast at google.com. You can also check out the website at kubernetespodcast.com where you'll find transcripts, show notes, and links to subscribe. Please consider rating us in your podcast player so we can help more people find and enjoy the show. Thanks for listening, and we'll see you next time.

so