Public safety Kubernetes
01:07:20 Share

One.

This is Ship It with Justin Garrison and Autumn Nash. If you like this show, you will love The Change Log. It's software news on Mondays, deep technical interviews on Wednesdays, and on Fridays, an awesome talk show for your weekend enjoyment. Find it by searching for The Change Log wherever you get your podcasts. Ship It is brought to you by Fly.io. Launch your app in five minutes or less. Learn how at Fly.io. ♪

Well, friends, I'm here with a good friend of mine and maybe a good friend of yours as well. Adam Jacob, co-founder and CEO of System Initiative. They're defining the future of DevOps automation, collaborative replacement for infrastructure as code. Now, Adam, I know sometimes for people to try something new, they need just an invitation.

Give them an invitation. Yeah, look, if today or in the last, maybe let's call it week, what you had to do was log into GitHub or into one of the cloud platforms and review a plan before you applied it and think about cloud state.

and how you are synchronizing your infrastructure as code, the thing you need to come do is use System Initiative and see how much better your life can be just by using System Initiative to do that same work. Like how much better it is to see that happen in like an architecture diagram that happens in real time that you can play with your friends. Like it is crazy how much more of your time you're going to get back and how much more fun it is to do this work again. Like actually having the people that you want to have review your work

come and join you to do the work is sick. It's so much better to actually do that that way with each other. And you should try.

because it's great. Okay, so they have a generous free tier for you to play with. So if you're hearing this for the first time and you're thinking, I got to try this out. All you got to do is go to systeminit.com and try out their free tier, kick the tires, see what you think. Is it the future? Yes or no. Again, systeminit.com.

Hello and welcome to Ship It, the podcast all about everything after Git Push. I'm your host Justin Garrison and with me as always is Autumn Nash. How's it going, Autumn? It's been a week, y'all. It is. Like, I want to say good, but like, I don't even know if I can lie at this point. Yeah, you're here. We showed up, like...

It's November 7th right now as we're recording and we are here. We are recording this. I think it comes out at the end of November. We are here and we are sober. Be proud of us. What's a sweet? Thank God me and Justin don't really drink because it would be wild.

That third voice you're hearing is Mark Borstein. He is the CTO at Tremolo Security. Welcome to the show, Mark. Yeah, thanks for having me. We were talking a little bit on LinkedIn about what you're doing with running Kubernetes in the public sector. So can you ask us specifically what infrastructure or software you're responsible for? Sure. So here in the DC area, it's kind of a unique place. We've got a lot of different governments, a lot of layers of government.

And so what we do, at least here, is we run an identity system for first responders. So go back 20 years, 9-11, you know, airplane gets flown into the Pentagon and you've got units responding from Arlington County, where the Pentagon is, City of Alexandria, D.C., Maryland, all these different jurisdictions. And they can't talk to each other.

I'm not even talking about digital communication. Their radios don't work with each other. That's wild. Yeah, it's wild to even think about. And so one of the many things that came out of that was a report that basically said here, especially in the D.C. area, because of how unique we are here in the national capital region, we need to have a way where everybody can talk to each other.

Because it's not just the different jurisdictions that need to talk to each other, but the jurisdictions work with the feds, work with military police because you've got a large military presence here, work with state governments. So there's all these different layers of interaction from a public safety standpoint.

And so they built a, there's this big giant grant that came out of DHS, Department of Homeland Security, specifically out of FEMA. So they're the people who, whenever there's a natural disaster, they're the ones who respond to it. And so this big giant grant for all sorts of public safety stuff. One of the things that came out of that was a fiber network that could survive another 9-11 type attack if the commercial internet went down, so that there was still a communications backbone.

And so they built it. Nice giant pipe. Worked beautifully. And nobody wanted to go on it because they didn't want another username and password. That's crazy. So they had this big giant network nobody wanted to use. So they all use Google single sign-on, right? Not even. No, that would have been awesome. That would have been amazing. No, this is – so we first got involved with this in 2011, right?

When we first got involved, it was actually kind of funny because I had just left my full-time job to work on Tremolo. And I was at a pitch contest, a mass challenge.

So I was up in Massachusetts with my partner and I was working for a consulting company at the time that dropped this thing in my lap. And I was like, you've given this away. There's no way I can implement this and be profitable. And, you know, I got back and started talking to my customer and I was like, you know, Tremolo could do this. And they're like, OK, let's pitch it to them. And, you know, the sun, the moon, the stars aligned.

And they're like, okay, let's do it. It worked beautifully. But it was Active Directory. And with Active Directory, local government typically is very, very Microsoft-focused. So, you know, everything from their on-prem all the way up to the cloud is almost entirely Microsoft, at least in this region. Different regions are going to be different. But at least my experience has been most localities are very, very heavily Microsoft-focused.

And so they had Active Directory. And if you've ever worked with Active Directory, you know, the only way to get two Active Directories to talk to each other is to just take all the firewalls and throw them out the window. So they didn't want them like hooked up directly into this thing. They didn't want them talking to each other. You know, Active Directory, I mean, that's the keys to your entire kingdom.

So you don't want to just like, yeah, we're just going to merge these things and make them all work together. That's a little dangerous. So we came in and said, all right, we can link these things together in a secure way. At the time, we were doing everything via LDAP virtual direct.

So if you've ever worked with an LDAP directory, you take a proxy that knows LDAP on the front. Think of like an HTTP proxy, but for LDAP. So it talks LDAP on the front and then it talks whatever you need on the back and creates this big virtual tree. So we created this environment where it was this big virtual tree of all these different infrastructures.

And then we were working with at the time, I think it was SharePoint 2011. And so we had to integrate that in, which was a whole kind of fun. We got all working beautifully. And so they're like, yes, this is great. We want to move forward. And then I learned about government procurement. So it took us years.

Almost two years to get the procurement through. And we finally got it up and running. For them just to be able to say, yes, we want to pay for this and you can go ahead. Yeah. And it was just going through and figuring out like, you know, who are we going to buy it through? Because if you've ever sold something to like a large company, you've had to go through like a reseller, like a CDW is a big software shop that you might have to go through. Insight's another big one. Cinex.com.

broken off of insight but you know if anybody who's listening knows who any of these names are I'm sorry and good people you've already got a lot of trigger words on this episode with proxies and so like you know this was our first customer and we were just like

I'm sitting there texting my partner. I'm like, do we have a bank account? Like, where do we put the money? Like, what do we do with this? And so it's just, it took about two years to kind of go through that minutia of all the different layers of procurement. And when we went live, we went live with five different jurisdictions all over LDAP.

And it was an LDAP virtual directory. And what was really interesting was at the time, they were saying, well, the grant that was paying for this very specific grant called UASI, U-A-S-I. And the goal of the UASI grant was all of this money that was being spent to build out this infrastructure was supposed to be temporary money. Like it wasn't meant to fund it indefinitely. It was meant to fund it to the point of maintenance and

And then at maintenance, the localities take it over. And so the assumption was, was that we were going to build it and then hand it off to someone. And so they wanted everything to be as Microsoft focused as possible. Now at the time, this is before we were an open source company or had any open source projects or anything. So at the time we had a virtual appliance, right? You know, containers before there were containers and it was built on CentOS 6.

And, you know, we would drop it in. They were like, well, can you give us a physical appliance? And I was like, well, you know, I'm just going to buy something off of Dell.com, spray paint it red, put my logo on it. You're going to get better prices than I am. So just, you know, buy it and let us deploy it. So we did that. And we didn't have an interface at the time. It was all just APIs. And so we built an interface for them based on .NET, and it was the ugliest interface ever.

I wish I still had a screenshot of it, but imagine the most bugliest HTML marquee. It was bad. But it worked, and people really liked it. And we got off the ground, and we went through...

three data centers before we finally were able to go live. They just kept moving us from data. Like they wouldn't, you know, initially they didn't have hardware for us. So we had this hand-me-down hardware.

And they're like, so, you know, you're going to have to put a hypervisor on there. So VMware, they're like, no, no, we're not giving you anything like that. No, you get to go with Hyper-V. I'm like, okay, I've never done that before. And like, okay. And I'm like, I'm sitting there in the data center deploying it. And then they moved us to another data center. And then finally, when we went live, they ended up putting us in a third data center.

And so we finally go live. And ironically, so I was telling you about this big network, right? Our first app that we went live with didn't run on that network. It was hosted on the commercial internet. So, you know, we were like, okay, you know, whatever. Customer's a customer. We don't care. It's an app. And so we went live with it in July, 2013. And it was one of these things where like the people who started using it were like, what do you mean I don't need another password?

It didn't compute with users because the SSO wasn't a big thing at this point. You know, this is when we were competing with like, at the time it was NetEgrity and then bought by CA, Oracle Access Manager, you know, ForgeRock, you know, a lot of these legacy identity names.

And so SSL was still not something that was super common. And in this case, your app that you put on the network, people are coming to use it. You have this LDAP proxy and that proxy is going to each group's own Active Directory service. So it's like the user just exists and they just need the full username at whatever domain with their password and they don't need to sign up again. So you're just proxying all those back to whichever LDAP. You're like, you are a proxy, you're a router at that point. You're like, oh yeah, you go over here, we'll figure out if that

is the right password with the hash and then we'll send it back and say yes or no.

Exactly. And so, and no hash though. No, we got the password. We just handed it back off to AD. Of course. This is 2013. Yeah. Okay. Thank you for the benefit of the doubt, but no. So, well, I didn't like it, you know, it was painful. I mean, our biggest pain point, it was networking and it was certificates, right? Because certificates expire. Nobody tells us anything, you know?

And we don't know until the user starts complaining, right? And at the time it would take like a month to get a new certificate signed, right? It didn't take a month to get, no, getting the cert was actually pretty easy because I mean, it's AD, so it's all self-signed certs. Sure. I mean like a publicly trusted. Yeah. Okay. Yeah. So, but what would happen is people would start complaining that AD is broken. And so I came from a background of having to deal with broken certs all the time.

And so we had built into the technology the ability to just suck the cert right in. But I still had to go in and do it. And at the time, the monitoring that was in place was capable of doing something that would be like an LDAP synthetic transaction that could tell us it was broken. But the team I was working with had no idea how to do that. And every time I tried to explain it to them, they'd be like,

So this is an HTTPS.

No. And like trying and the way the networking was set, it never worked. And they would always say, yeah, the monitoring says everything's OK. And I'm like, but I can tell you customers are complaining they can't log in. And so we were in this place where it just like it was bad. We're in the bad place. And as technology got better, we did a really fun job of making getting it into our systems.

Like, I mean, this is a system that's been in production now for over a decade and it's not been static at all. Like it's just been growing and growing and changing with what's been available. And I've been very lucky because as a customer, when I say we should try this new technology, you know, and you'd think they'd be risk averse because we're talking about public safety systems. They'd be like, no, we trust you. You're telling us we should try this. Let's try it. So I'm very lucky to have a very forward thinking customer.

And so we slowly migrated. We went from an all Windows infrastructure besides our Linux VMs to then our portal moved from .NET and that fugly interface to a nice bootstrap-based Java interface.

And then, you know, we eventually got rid of AD and we moved up to Azure and we went from being just like these virtual appliances that we installed in ISOs to running on, I call them faux containers where I had Ansible playbooks that would deploy everything. But we would like to play it all from scratch, right? Like I could point it at an empty Ubuntu server and the whole thing would be running.

And as that was happening, we got rid of the virtual directory. So as ADFS, Active Directory Federation Services, became more common among the jurisdictions, we were getting rid of the virtual directory components, which really made things nice. And then when we moved to Azure, we did something really cool where we took over all the monitoring and we stopped playing this like, okay, well, there's an ops team and a dev team. And like, no, we're just going to own everything.

This is our problem now. And so we were able to build out a Prometheus infrastructure that I now knew when there was a problem before my customers did. And we were running those synthetic transactions and we had a dashboard and we had alerts and it was tracking things. And like that was the first time. So that was about 2019. And that was like the first time where like, I felt like, wow, I really know what this infrastructure is. And it was...

it was all automated. It was amazing. It was beautiful. I feel like that type of observability is what people should be shooting for. At the end of the day, tech is a business, right? You want your customers to be happy and you want to figure out what they're experiencing before they do. The fact that people are just like sleep on the fact that

it would make it so much easier to iterate and make your product better. Just boggles my entire mind. Well, I mean, the big part there too is like the org chart matters of who's getting the alert, right? Because if it's the ops team getting alert, then they have to go wake up the dev team. That's not fast enough. And that's not actually helping. You're just putting someone else in the middle. And I know devs don't typically like to carry pagers and they're always been resistant to that, but also it makes the product better when they do. Are there really dev orgs where they don't

carry pagers? Oh yeah. Oh yeah. How do I get into those orgs? Where have those jobs been my whole life? Any listeners right now, if you are a dev on an application and you don't carry pagers or at your organization, the devs don't carry pagers. Yeah. Like this is, it is a big thing. I need to know. Like,

Even in 2024, almost 2025, that is absolutely still a thing. I also think it depends on how big the feedback loop is. There's some applications that even going through the feedback loop of ops and then dev, it would still be better than waiting till you get a ticket from your customer or until people just decide to move off of your stuff. You know what I mean? Like even in the worst like...

case scenario, it's better than the way that so many very high quality teams in fang land are taking their product feedback from customers. Sometimes I'm just like, what do y'all do? If I can Google, if you are using a technology and I can Google other people asking you about my problems that you haven't addressed on multiple forums,

What are you doing? I mean, to this day, the single best alert I ever set up in my life was before I went to bed, the night Disney Plus launched. And I put a if this and that alert on Twitter to say hashtag Disney Plus is down. And that woke me up before my alerts came. But that's what I'm trying to say, though. Like, go

get an RSS feed. Like you can look at like the, you can take the data from Stack Overflow. Like there's so many ways to understand where your customers are struggling. And half the time we work on things that aren't even that impactful to our customers because you're not getting the right feedback loop. Like getting that proper feedback loop will like, you know what to prioritize. You know what I mean? Like, well, you know where to put your dev hours, which is the most expensive thing you have. So like, it just baffles my mind that like,

There are things that you can do to automate it that make it so easy, you know, and there's so many tools out there and free ways to collect that data. And people are just like, I don't want to be better at my job. Well, and that was one of the things that was so difficult about this. So bringing back to public safety, all of the public safety people that we work with, if it's something that is like really life or death,

They have manual fallbacks and they're trained that if the technology doesn't work, don't just fall back to the manual. Yeah. Because people die.

So our feedback, like you would think we would know immediately that like, oh, we can't get into the thing. Like, well, no, they just switched over to the manual and then back to the CIO, you know, at the weekly meeting a week later. So we don't actually find out there was a problem sometimes until like a week after it happened or two weeks after it happened. Yeah. They worked around the issue. They needed to get something done. And they're like, I can't, I don't have time for this. Someone is in danger. Right. Yeah. So it became imperative for us to be like, not just to,

From a convenience standpoint, but sometimes we didn't even know if there was a problem. If we weren't running synthetic transactions to make sure what was happening, we wouldn't even know that there was a problem.

Was this application in, was this in the hot path for like 911 calls or is this dispatch afterwards? Okay. That's just such an interesting use case. Cause sometimes it's like, I love technology and sometimes we get excited about solving problems that are some random problems that I don't know if anyone really asked for it, but that is such a cool use case because you're really impacting, making like things better for people, but it's still like deeply technical.

Like there's only so many original ideas still left in the world. And the fact that you solved such a cool, like important problem is really neat. Thank you. I really appreciate that. So the thing with public safety is that they very siloed industry. I mean, even more than a lot of enterprise tech that you're used to, it's not used to working with other systems. It's not designed to work with other systems.

I mean, they built a whole internet for it. It's a private fiber network that no one else is allowed to use so that we can. Well, and even then, you know, you're dealing with an industry that's just very used to one being very slow for updates and slow for changes. I'm not going to shame the big giant company that does this. It's a company that if I told you who they were and showed you where they are, you'd be like, oh, that's them.

And like, they were so like isolated that when we first integrate the SSO system with them, they're just like, well, can't you just sync everybody's passwords into 180 and let us run that? And it's just like, no, we're talking to 20 different jurisdictions across three states and the District of Columbia. You want to tell their CIOs to do that before you close the deal on the software? And like, yeah, no, we don't want that.

And so, you know, very isolated. And then because we deal with the feds, federal standards for security, like I always like to say in the battle between compliance and security, compliance wins every time. Like it always wins.

Security never wins in that battle. Shouldn't they almost go hand in hand, though? It's so weird to me that they are always treated as two separate issues. I know logically we know that, but they're so separated, which you would think that a lot of compliance would also be like,

I don't know, mandating or influencing because of security. And it's so odd that they're such separate subjects, which make it like, you know what I mean? Like you're just sitting there like, what's the point if you're not going to talk to each other? Well, perfect example. So CJIS, the criminal justice information system, this is the IT security standards that the FBI puts out. So if you're going to work with their systems, you have to comply with this set of features.

So here in the DC region, you get arrested, your picture goes into a database, and we provide the security for that database. Well, one of the things that CJIS says is that you can't have access out to public networks from the workstation that you're working on to do this because they don't want you exfiltrating these photos to somebody, right? That makes a lot of sense. Except remember how I said every jurisdiction uses Microsoft?

Well, starting in 2020, that became Azure and Azure AD and Entra. So instead of us handing you off to the ADFS running inside of the corporate network,

We started handing everybody off to Entra and Azure AD. So Azure AD is now rebranded Entra to anybody who's listening and hasn't caught the new documentation. I was not up to date on that. Thank you. Yeah, that's only like six months old. So most clouds, and I will shame Microsoft on this one, most clouds have some kind of direct connect, right? I forget what AWS calls it.

So you don't go out to the public Internet, right? Large enterprises don't go out to the public Internet to get to their EC2 instances, right? There's a direct connect with Equinix or something else, right, to do that. It's either a direct fiber or even like a virtual appliance VPN, something like that. Right. Microsoft has this for Azure, but not for Entra. You have to go to the public Internet in order to authenticate to Entra.

There is no way to bypass that, which is bonkers to me on multiple levels. But that's the way that they set it up. So now I'm explaining to these cops, hey, you need to open up all these firewall ports to go talk to. And of course, Microsoft, it's not just one URL. You've got a whitelist, like 30 different URLs.

And you can't do it by IP either because they're all dynamically generated IPs. You know, the range is 0.0.0.0.255.255. Here's the Microsoft block of IPs. Yeah, exactly. I think Microsoft still has the slash 8, right? Exactly. But that's the, you know, in that particular instance, I was like, look, you can either hand everybody their own username and password.

or you've got to get an exception to this rule. That was the one instance where security won out. But Autumn, to your point about security versus compliance, why is this a thing? Because a lot of the compliance rules aren't written with our integration ideas of today in mind. They're written on a very narrowly focused. But when you look at it from a broader picture, well, that's not going to work, right? And so that's been a big part of our fight

has been when we work with the feds especially, that we had one federal agency, they'll go unnamed, that they were originally going to integrate us as an identity provider to make it easier for the localities to access their system. So we have federal users that access locality systems.

Because we already have all the localities integrated, this one group wanted to go the other way. And they're like, are you guys level four certified? No, not at all. The cost to do that is through the roof. And they're like, yeah, no, we're not going to do it. We're just going to hand out usernames and passwords.

It's like, but there's no MFA there. Yeah, but we don't want to, like, your compliance is going to be too much of a pain for us to deal with. So we would just rather do username and passwords. And we're just like, really? Like, this is what you want? And they're like, yeah. Okay. So it happens all the time and it's painful to watch. ♪

Well, I'm here with a good friend of mine, David Hsu, over at Retool. Big fan of Retool. Big fan of internal tools built with Retool. But David, not everybody knows how to use Retool. What can't you build with Retool? Yeah, so Retool is really good for building any sort of CRUD application where you care mostly about security, authentication, authorization, sort of internal facing things, basically.

If you're looking to build, let's say, like Google Maps, for example, probably you should not be using Retool for that. You know, go write some custom JavaScript or React in order to go build a really complex application like Google Maps. But if you're looking to build a CRUD app, Retool is great for that. And what we see especially is that backend engineers who have less experience in the front end or less interest in the front end really gravitate towards Retool.

Because for a backend engineer, sometimes all you want to do is you want to get a formal type of database, test that it's working. You want it to go test an API, for example. And spinning up a quick app in Retool is so much faster. They're trying to go learn React, learn Redux, learn state management, learn all these different parts of the frontend stack. And it's kind of so complicated. And so I think that backend engineers in particular really gravitate towards Retool for building fast code apps.

Okay, friends, the best way to build internal software is by going to retool.com. You can seamlessly connect databases, build with elegant components, and add your own code on top. You can accelerate mundane tasks by not learning React, not learning Redux, and freeing up the time you need to work on the things that matter most. Once again, retool.com. Start for free or book a demo. Retool.com.

Going back to the infrastructure you're setting up, because you stopped selling most of it in 2019 or whatever. You had all this automated and you were starting to go into this open source. Previously, you were setting up a lot of stuff there. It's all AD integration. It's adding more applications. It's giving people the identity on this network for first responders. What happened after that? What changed or what are you doing since then? I don't want to say COVID was good in any way, shape or form.

It really accelerated the understanding of the need for identity infrastructure because everybody switched to work from home. So I'll give you an example. So our primary customer for the caretakers of the system is Fairfax County, Virginia. So about 1.1 million people. We're larger than eight states. And we tend to take the lead on all of the regional initiatives around here.

And so in a county of 1.1 million people, you know, they've got a very sizable county government. And I was one of like 10 people who had a VPN.

Of the, I think, 3,000 people in IT, I was one of 10 that had VPN. They weren't handing out laptops in 2020. They were still handing out desktops. And all of that identity still is based on network access. Like you're right, even the VPN is like, oh, you have one of our IP addresses? We trust you. You're good. Yeah. And so, you know, COVID happens, work from home happens, and all of a sudden, identity matters.

And, you know, they're getting rid of ADFS. They're moving up to Entra. Like they're building out that identity infrastructure. And so that gave us the ability to be a lot more flexible because we could ditch our legacy virtual directory technology that we were working with. So that happened. The other thing that happened was Kubernetes. We had tried to do Docker containers in 2015. And it was one of the most beautiful failures I have ever seen.

Like it was such a cluster that like we're like, OK, you know what? We're not touching this Kubernetes thing until we can use a managed service. I've been doing Kubernetes since 2015. So like, yeah, I feel comfortable deploying Kubernetes. Right. And I'm like, no, I'm not dealing with this until we got managed service to go.

And so we finally moved up to Azure. You know, it was one of these things that like, we're going to move, we're going to move some point, we're going to move. Oh, the contract's up. You need to move us into Azure in the next 30 days. Go. The data center contract? The data center. Yeah. Okay. The data center contract with the county had expired. So we kind of did a lift and shift where we got everything into Azure, where it was still running on VMs. And then at that point, when we were on Azure, it was like, okay, well, we can Kubernetes now.

So we got started with Kubernetes and we did our first rev on Kubernetes and moving to Kubernetes was so much harder than we thought it was, which no one has ever said before. This is still Windows and .NET? No, no. This is all entirely Java at this point. There's no .NET anymore. When did you switch from .NET to Java? So our backend was always Java. The only part that we had that was a frontend was a .NET frontend.

So that switched over to Java in about 2017, 2018. And then we actually got rid of a separate portal application. The original architecture was the portal application was a J2EE web app that communicated to OpenUnison APIs using mutual TLS for security.

And then in 2018, we got rid of that, switched over to an AngularJS frontend, where your frontend was all in the frontend, and it communicated directly to the APIs using tokens. Then we proxied those APIs back. So there was a DMZ in there. And that really simplified our architecture because now we didn't have to worry about this portal thing anymore.

And then we moved over to Kubernetes in 2021. We went live with that. That was great because we had automated the heck out of everything. Like even to the point where we had a system in place that if Ubuntu patched a library that was on a source container, that would trigger the rebuild, get deployed into devs.

We had the automation and dev watching. Now, dev didn't run 24/7 for cost reasons, but I would know that there was something that changed. I could just fire up dev, do a quick eyeball scan, see that everything was working, push into product. It was great. We had automated our monitoring, and then once we had that going for a while, we then decided to move into GitOps.

and to move all of our manifests into Git. Can you tell us more about how you automated it? Yeah, so it was almost everything was Azure DevOps. So, you know, Azure DevOps is Azure's like GitHub actions, basically, right? And so we were able to put in place

a combination. I mean, it was a bit of a Rube Goldberg machine, but it worked beautifully where because we're the vendor, right? Because we're a software vendor. We're not just a professional services. We're not, you know, we're, we were the vendor that took on ownership of this rather than, you know, this being our primary product. So we were able to integrate it into our vendor systems in such a way that when we knew OpenUnison got patched,

That would then trigger a webhook that would patch a private Git repo, which would then trigger Azure DevOps to run and to rerun the build, push the container into an Azure registry. And then before we went to GitOps, it would patch the dev Kubernetes instance, the dev AKS instance.

And that would then get the ball rolling. Then when we were happy with everything, we just had a simple script that would run that would take the version that was running inside of dev and patch all the manifests and prod to move everything over. And so it was all very clean, very easy.

Is the backend of Azure DevOps like an open source tool? It's not just like Jenkins or Spinnaker or something like that, right? Like it's a custom thing or? I don't know. Okay. I wasn't sure. I don't know. I've never used it. So I was kind of curious there. Our stuff is all YAML bash, you know, amalgamation. But yeah, I don't know if that's a good question. I don't know if it's, if it's a open source backend. I don't think it is.

I think it's something custom, but I never actually looked into it. So I was kind of curious. The second question there is the move here from we have this push-based pipeline to a pull-based controller, right? GitOps controller versus what were the... You could still store manifest in Git in Azure DevOps. You could still do a lot of the same things, but the push versus pull...

Why did you feel like you had to move to that? What was the leading thing? This is going to be a bunch of work. It's going to change how we do things. What were the benefits you projected that you were going to get out of it? So there was actually a really specific reason that triggered. It was one of these things that we always wanted to do as long as we had time and budget. But what the actual thing that triggered it was, we wanted to have a secondary instance. So when everything started, we realized,

We were very heavily focused on this fiber network, right? Stuff that was running on the fiber network, stuff that was accessible via the, and we had a couple of apps that were hosted on the internet, but for the most part, everything was on this fiber network.

Well, times change. I mean, it's been 20 years and most of the apps have moved to the commercial Internet at the same time. We're at the point now that like if America turned off our Internet, like the world would survive without us. Right. From an Internet standpoint, like we're no longer. I think that's true. I don't think I'm making that up. I think that happened like in the last decade.

Even if US East 1 goes down, half the world stops. I don't know. Well, that's a little bit different. But just from the bytes going across the wire, I think at one point, and I want to say it was like sometime around 2010, there was a threshold cross where there was enough cable infrastructure and backbone around the world that if somebody severed the US from the physical internet,

the global internet would continue. And that's kind of true here in the US too, right? The backbone of the internet has just gotten more and more and more, especially here in DC, where you can't throw a rock without hitting three data centers. So people became less reliant on this idea of a backbone network. Commercial services became more and more important to government services. And so more and more of our applications started running on the internet.

including some of our critical public safety applications for SaaS solutions on the internet. Also, the identities we were working off of were all coming from Azure. Like they weren't coming from some AD, they weren't coming from some ADFS that was running, you know, or some ping federate server that was running on it. They're all coming from Azure. So,

there was no tie anymore to the NCRnet except for the NCRnet only application. So somebody said, what if this network goes down? Now what? Like, does that mean all of these critical services go away because this network goes down?

And not to belittle the engineers who built an amazing network with multiple redundancies and all this other stuff. But that's a failure mode. So they said, we need to have another instance of this. I was like, it's all self-contained. It's not rocket science. It's just a bunch of web apps at the end of the day. But I'm not maintaining configuration manifests between two different clusters. That's a losing game. I'm never going to win that game.

So that was the main driver to say, you know what, we're going to move from everything stored as manifests inside the API server to GitOps. So that way we can have these two instances of this environment in two different regions that could survive an outage of the NCRN.

But like GitOps isn't a requirement, like because you can have two DevOps in two different regions with a Git repo that pushes both places, right? Like this isn't a... You could, but this is the way we wanted to do it. Like we wanted to be able to say, here's our manifest, because if we needed to rebuild this system quickly, we wanted to be able to say, here are all the manifests, it's ready to go without necessarily like saying, okay, we're going to have a, you know, we're not using like a terraform or...

a Pulumi to build it out, right? We're relying on Kubernetes APIs. The manifests themselves are all fairly static. Like these are not, the infrastructure to handle this is really not all that complicated. It's a web app. It has a database.

You know, it has a message bus. There are a couple of different apps that are integrated with it. But, you know, you've got a front end app. You've got a back end app. You've got a few different APIs. You know, we're relying on a combination of Azure's secret management and, you know, the database and whatnot. But like it's not a stateless app, but it's a most like the critical parts are fairly stateless. So GitOps was a good route for us to be able to go that way.

Does Azure have the, I don't, I literally do not know this. Do they have like a GovCloud sort of thing like AWS does? Is that where you're running this or is this on the like standard public? It's all commercial. Yeah, it's all commercial. As a group building something that is...

very sensitive, information sensitive and critical to a lot of people's livelihoods, I guess. I mean, just in general, like this is emergency sort of services. How have all of the security vulnerabilities of Azure impacted you over the, I mean, you've been there for quite a while. We've been there since 2020. So we've been there for about four years. Right. Which like is, they haven't gotten less, uh,

over time. And I'm kind of curious, like, how does that impact what you're doing? Because if you're not in a special cloud environment, like you can, you can protect all you want about your applications. But if the underlying, like there was literally just like, you can read another account's database, right? Like in Cosmic or whatever it was, it was just like, oh, this is just like, we found this thing and people could read across accounts without any security guidelines whatsoever. Like, how does that impact you of what you're doing and what you're thinking?

We haven't really been impacted by any of the Azure-specific security. Like the Cosmo one didn't impact us. The log4j was a big one for us because we're Java-based. That was a weird day because I've got other contracts inside the county that use us. And, like, I got up and I started looking at Twitter. And I was like, huh, log4j, oh, great. Well, let's see what's going on. I'm like, oh, sh**.

Wife, please go drop the kids off. I'm not going to be, I got to focus on this now.

So that was a day. But like Azure's issues from a security standpoint, we've mostly been insulated just because we haven't used the services that have gotten beaten up that badly. I mean, you're running Java and Kubernetes. This is a pretty good service area. I'm not accepting any hate for Java and Kubernetes. It works beautifully. Thank you. Tell him. Tell him. Yeah. Java, Kubernetes.

Okay? They love each other. See? I like Mark. Yeah, we're cool. He understands me. And we as a company, Tremolo, we've been doing the whole S-bomb thing since way before it was popular. Our releases going back at least seven or eight years...

Our goal is always to have goose eggs on the scanners when we cut a release. You know, it doesn't stay that way more than 30 seconds, but at least we try, right? You know, Bart, you know, with the cake, at least you tried to toss it in the trash. But, you know, so we're constantly staying updated from that standpoint, protecting the front door.

No, the pain that we tend to have when it comes to Azure is they change stuff all the goddamn time and they're constantly deprecating things. Like a template that I use to deploy an AKS cluster within a year is completely unusable. And it just they just have this tendency to like make these breaking changes.

And it's not to like Kubernetes itself that's a problem. It's like, oh, well, we're changing the way we're doing log aggregation to this thing. And it's like, I just want to run a Kubernetes cluster.

Can I just run a Kubernetes cluster? Is that okay? And, you know, they're just, they're constantly making those small types of changes that just make automation difficult. It makes life infuriating. You know, I'm doing a talk next week in Salt Lake City and I need a Kubernetes cluster and I've got all these free Azure credits. I went to launch it and I couldn't access my Ingress controller. I couldn't access Ingress Nginx.

It wasn't a network policy. It was just, you know, KubeNet. So there was no policies to be had. It wasn't a networking issue from me to the IP address, the public IP. The health probes looked like they were working. I could not access it. I was like, what's going on here?

So I have found that Azure, when it works well, it works really well. I feel like they took the people from AWS that failed at interface design and built their interface for Azure Cloud. You're saying that the failed AWS designers are Azure is like, whoo.

I know. And I don't like tossing shade, but like, so I'll give you an example. We use Azure Secret Manager. And on the AWS side, I'm a big fan of ParamStore. And ParamStore, it's got kind of a clunky interface, but as long as you know how to do the path-based process, you can narrow it down at worst fine, right?

you can't do that in Azure. There's no way. So you've, you're just hitting next, next, next. If you've got to go through the UI and do something. That's a spicy take, but I can't wait to see how this is going to turn out because it's not going to be good. Like if that's how you feel now, give it six months. Cause the other one is not going to be better. Yeah. So Azure has definitely had a lot of challenges, um,

One thing I will give Azure big props on. So we're using their MySQL database and they needed us to migrate from one offering to their new offer. And, you know, we don't have a large database, but like it's kind of critical data.

And so, you know, I'm trying to figure out how to do this. I'm like, no, we want to get on a call with you. And I'm like, oh, God, they're going to have like 500 people on a call, none of which actually know what they're talking about. And they're all sales reps. And I got on the call and they had really smart people on there who knew what they were doing. And they said, yep, we understand what you're worried about. This is the right way to do it. And it worked without a hitch.

And I will say I was not expecting that based on past experience. And they knocked that, like that migration, it did exactly what they said it was going to do. It did a one-time migration plus live data. So we were able to get the whole thing up and running. And then we were able to pick a date to do the cutover and everything worked beautifully. And like, I was like, yeah,

Y'all had props. I'm not going to lie. I had zero confidence that this was going to go off without a hitch. And the only reason we had any downtime is that I was an idiot. You were like, zero confidence. You were like, I had no hope. But like, y'all pulled it out though. Y'all did a great job. And the only reason why we had downtime was my own stupidity. I made a mistake. Otherwise, we would have had zero downtime. It was beautiful. Thank you.

so

Okay, friends, I'm with a good friend of mine, Avtar Swithin from Timescale. They're positioning Postgres for everything from IoT, sensors, AI, dev tools, crypto, and finance apps. So Avtar, help me understand why Timescale feels Postgres is most well positioned to be the database for AI applications.

It's the most popular database according to the Stack Overflow Developer Survey. And Postgres, one of the distinguishing characteristics is that it's extensible. And so you can extend it for use cases beyond just relational and transactional data for use cases like time series and analytics. That's kind of where Timescale, the company, started, as well as now more recently, vector search and vector storage, which are super impactful for applications like RAC.

recommendation systems, and even AI agents, which we're seeing more and more of those things today. Yeah, Postgres is super powerful. It's well-loved by developers. I feel like more devs, because they know it, it can enable more developers to become...

AI developers, AI engineers, and build AI apps. From our side, we think Postgres is really the no-brainer choice. You don't have to manage a different database. You don't have to deal with data synchronization and data isolation because you have like three different systems and three different sources of truth. And one area where we've done work in

is around the performance and scalability. So we've built an extension called PG Vector Scale that enhances the performance and scalability of Postgres so that you can use it with confidence for large-scale AI applications like RAG and agents and such. And then also another area is, coming back to something that you said, enabling more and more developers to make the jump into building AI applications and become AI engineers using the expertise that they already have. And so that's where we built the

PGAI extension that brings LLMs to Postgres to enable things like LLM reasoning on your Postgres data, as well as embedding creation. And for all those reasons, I think when you're building an AI application, you don't have to use something new. You can just use Postgres. Well, friends, learn how Timescale is making Postgres powerful. Over 3 million Timescale databases power IoT, sensors, AI, dev tools, crypto and finance applications, and they do it

all on Postgres. Timescale uses Postgres for everything, and now you can too. Learn more at timescale.com. Again, timescale.com.

How is the applications you're running? Is this more than just identity now, right? Is this still focused on identity for those districts, for first responders, or is this beyond that? So we got a couple of other apps in there that we manage. So the main one is an open unison instance that does the identity side plus the portals.

So that's the main application. We have a Mattermost. It's a kind of a Slack clone. We rolled that out. We call it regional chat. I love it. I think it works great. But the thing is, we're competing with teams. So it's kind of hard to, which is weird to me, but it's kind of hard to be like, no, don't use teams. Use this thing instead. Because even though everybody loves teams,

It's there and it works 99% of the time. So that never really got the traction we hoped it would. We run a Wikijs instance that we use just for random stuff that we need to be able to host documentation on. So those are the big ones. We have an email server, which is actually kind of fun. But Office 365 is right there. What's the...

No. You're running your email server in 2024. Everybody that can't see Justin's face. Before he said that shade, the smile on his face was so big. Like I just like for context, before he even got it all the way out, he was like mid-laugh.

And like his, like the smile was like half of his beats. - I'm trying to understand. Like you got, like, you're like, oh, Teams is over there. But also like you have Office 365. Like there's some email servers. - No, you're more correct than you think. - He wasn't trying to understand. He led you into the shade. He was like, come work with me.

Oh, no, I went willingly. I knew that was coming. I wanted to see if y'all called me on it. What's the email server? What are you running? You're on DoveCut? What are you running? Wait, wait. Mark was like, I went willingly. I was so down. Like, I can't with y'all. Wait. So, okay. There's a story here. So SharePoint is now SharePoint Online.

So one of our largest customers for this service is a regional SharePoint system that we have provided SSO for for five or six years. It was an on-prem system, and we had to do some fun stuff to make that work. Well, they got sick of managing a SharePoint system. They wanted to move it online. Yeah, no, this is like day one. Don't run your own SharePoint server. Right. And this is the second iteration. So they want to move to SharePoint online.

When you do SSO, so Azure has this concept of B2B, right? Company to company doing some kind of SSO.

and B2C, company to customer. Well, we're an SSO system, right? We're an identity aggregation system. So in theory, what we would want, excuse me, you go to SharePoint online and they bounce you over to us to authenticate and then we route you over to the right thing. That's a problem on two layers. The first layer is if you're a tenant inside of Azure,

They just go straight to you, right? So I put in my fairfaxcounty.gov email address. They just bypass our system. IAMS is what it's called. The National Capital Region IAMS. Identity and Access Management. So that was the first issue, which wasn't that big of a deal because we still had stuff to do with provisioning. The second issue is the U.S. federal government. So I mentioned that we're running in a commercial tenant. So is the SharePoint.

Feds do not run in commercial tenants, and you cannot invite a guest into your tenant if you're commercial. That is government. So we had two issues that we had to deal with. One, we don't just maintain a broker with other identity systems. We also maintain what are called external accounts.

So maybe you're a contractor, maybe you're working for an NGO, a non-governmental organization, but that works with governments, right? That you aren't part of our SSO party. So we maintain a credential for you. And so we have those credentials as well. Now we integrate with the federal government, so we can authenticate any US federal user. So the question was, how do we get Azure to recognize us as an identity provider?

The other thing that we ran into that's a technical issue is you can say to Microsoft, hey, for tremolosecurity.com, go SAML to them.

The problem is that we have to prove that we own tremolosecurity.com because they don't want somebody proxy attacking. I get that, right? But they also have no mechanism to work around that. They have very specific exceptions, like Okta as an example. You can get around this. So what we ended up having to do was for U.S. federal users and these external users that don't have jurisdiction accounts,

We had to create an email domain alias that would be recognized. So we run an email service. We had to completely re-architect everything to make sure it was isolated. But we run an email forwarding service where you have to have an email, right? Otherwise it gets rejected. So you have to know the email of the person that you're sending it to. It has to exist.

It'll forward it to them. So now when we register the invitation with Microsoft, it's at our domain instead of their home domain. So they're still using their home credentials, but they're using our email forwarding service. So we don't maintain like email boxes. We have to forward it.

But that was a whole other bit of fun because we were originally doing it with like just straight up SMTP. And it worked fine when we first went live. And then like three weeks later, people stopped getting emails. And we found out that like basically if an email comes from any of the three major public clouds, it gets blocked by everybody. So like if it comes out of Azure, it comes out of Google, if it comes out of AWS, doesn't matter, it's getting blocked.

And so we ended up going to AWS and saying, okay, well, we'll forward everything to SES.

And that solved the problem because now we were able to rewrite it with all the right headers and everything. And then I went up to SES. Is this actually like a VM running an email server? Are you forwarding this to the, yeah, because you need that trust, right? Like email has this huge like network of what IP address am I receiving this on? Like, are you spam or not? Because it just gets abused, right? It's just an open thing that's on the internets.

And so someone signs up and they say, I'm a contractor. I don't have this account. I need to sign up, but I'm gonna sign up under your domain. I'm still going to get the email forwarded to me through Amazon's service that goes back to my actual proper domain. But then when I sign in, I'm signing that under your subdomain because that is what is, it gets trusted here of like, yes, I own that subdomain. Even if the email comes back to me. Yeah.

It's a bit of a room goldberg. It works really well. Once we got the whole like, okay, we can't send the emails ourselves. We have to go through an SES-like service to send it out. And once we got that and all the headers and everything figured out, it works beautifully. But no, it's a container that runs in cluster. We got everything locked down with network policies. So, you know, and that particular container has no service account.

So it's got no, like, if that thing gets compromised, you can't get out of it. Like, you can go out to the internet, but you can't hit the API server. You can't hit anything from inside those containers. And there's no identity. So even if you did try to, you know, even if you had access to the API server, there's nothing to abuse. There's no keys. But I mean, they have the emails that flow through. Yeah, the emails flow through. Yeah.

Which is like the thing that people would probably want from that. The only thing that it is, is it's notifications. It's like, hey, you've been invited to this tenant. Somebody updated this document. It's not like a general purpose. But I mean, you're collecting email addresses of where, not as a SES would know where it's going. Well, an email address on its own doesn't really count as PII.

Because it's not associated with anything. You have to have, I think it's three pieces. Yeah, and I think going all the way back full circle to this, like...

Email is not a good identity. We've used it as identity for so long. No, it's a terrible identity. And that is one of the problems that like, yeah, this thing has to exist somewhere because that is like the still method of getting information to someone in some form, right? We don't have another system that does that, but also like it should not be your identity at all. No, never. I always talk about to everybody that will listen, do not use email address. It's an anti-pattern.

Names change all the time for a bazillion different reasons, right? Domains change and all that. But not even that. People change their names.

It's not that uncommon of a thing. It's a pretty common thing in a lot of different contexts. Dude, nobody talks about the fact of when you get divorced and then you change your name. You can't change your email, so you're just stuck with it forever. Yeah, you're stuck with that. Yes, it's so annoying. I'm like, I guess I could redirect it, but it would be such a pain in the butt. And you have to have so many logins. It's such a bad idea.

So, you know, there's name changes for that. There's transition. There's a bazil. People just say, I don't want this name anymore. Right? Like names are a terrible way. Human names are a terrible way for computers to identify a child. I didn't even pick my name. Right? Like my parents decided it was a guy. Exactly. Exactly. There are a lot of people who say, I don't want the name that my parents gave me. And who the hell am I to tell them no?

Yeah. It's a terrible identifier. So you're in Azure. You got GitOps. What's next for the system? What are the new improvements? What are you looking at for the next five years? Oh, five years. I mean, we don't have to talk about all five years, but in general, what's the thing that you're going towards? We're in a pretty steady state right now. We're always adding new applications. The big thing that I really want to do, so OpenUnison...

is got two ways that you can configure it. The legacy way, which is XML embedded in the container and the new way, and it's not really that new, we've done it for a few years now, is CRDs running inside of the cluster. Custom resource definitions for anyone. Yeah, custom resource definitions, exactly. So today...

I've got different variations of OpenUnison for different roles inside the system. I've got an identity provider. I've got a provisioning system. Those are the only two roles we have anymore. We used to have a virtual directory. We used to have a proxy.

And so, you know, those are their own code bases that if I want to make a change, I've got to run a build, generate a new container, push that container out, test it out, and then move it over. And that's a long cycle. Like that, you know, could take a couple minutes to go through that long cycle. In which case, now I start looking at Twitter or Blue Sky and my mental health goes to the sh**.

So what I would ideally love to do is convert that to the way that most people run OpenUnison today, where there's this one base container and everything gets pulled in dynamically as a CRD. It's hot, right? Like it's not just awesome, but it's live, right? I make a change and OpenUnison is itself an operator. So it sees the change and picks it right up.

And so then I could also move all those manifests into the same repo that stores all of the operations manifest. So I can have one giant repo for everything. So that would be where I'd like to go in the future. Right now we're in the process of changing our interface, which I'm really happy about because we have had the same bootstrap-based interface for like...

10 years almost. And we're switching to, if anybody who's listening uses, it's like bootstrap V2. Yeah, no, we're switching to a nice, beautiful material based UI. We rewrote, we rewrote the open unison UI to use react with material. If I ever have to do front end, like react and material is my favorite. It's the only one that doesn't give me like full hives. Like, yeah,

It is the best. It was an adjustment for me because I'm not a front-end developer by trade, but I like it. It's just fast. I never realized how slow and clunky our old interface was until I started using the React-based one. I was like, oh my God, this is so quick. It also is very user-friendly. It makes more sense to my brain than a lot of the other front-end ways of doing things. Yeah. Yeah.

And so, yeah, so that's going to be the next major project is finishing up the interface revamp. And then like, those are the big ones. Like we're, we're already pretty bleeding edge on a lot of things where we've already externalized our secrets management.

Yeah, I'd like to switch. Right now we're using CSI secret provider. I want to switch to external secrets operator. That'll make it a lot smoother because with CSI, you still have to have some, even if you're working off of secret objects, you still need something to trigger the mount of the secret in order to sync in the secret. I mean, if it's a file, right? Yeah. Yeah.

And so like external secret operator removes that step. So we're probably going to move to that at some point. So that'll be a big deal. And then whatever, you know, new infrastructure people come up with next,

If it turns out to be better than Kubernetes, we've been around now for a decade. We've gotten to the point where people just expect us to work and are totally underestimating how hard it is to run infrastructure. Like, well, this just works. Can't we outsource this to someone? And our customer's like, no, are you insane? You're like, it works because I'm good. Yeah. This isn't easy.

So yeah, let's see what the infrastructure world comes out with us next. Mark, thank you so much for coming on the show. I have one last question for you. Are you using Flux or Argo?

Argo. Argo. Okay. I'm a GUI guy. I'm Enterprise. I love my GUIs. Yeah. I mean, in the GitOps war or whatever it is out there, there's a lot of opinions on either side. And I'm just kind of curious. I think I see more and more people going towards Argo now. Yeah. And I'm actually talking at ArgoCon next week about multi-cluster identity.

By the time this comes out, it'll be two weeks in the past. Two weeks. So I've already talked about ours. Awesome. Well, thank you so much, Mark, for coming on the show. Thank you everyone for listening this week and we will talk to you again soon. Thanks for having me.

Thanks for listening to Ship It with Justin Garrison and Autumn Nash. If you haven't checked out our ChangeLog newsletter, do yourself a favor and head to changelog.com slash news. There you'll find 29 reasons, yes, 29 reasons why you should subscribe. I'll tell you reason number 17, you might actually start looking forward to Mondays.

Sounds like somebody's got a case of the Mondays. 28 more reasons are waiting for you at changelog.com slash news. Thanks again to our partners at Fly.io. Over 3 million apps have launched on Fly, and you can too in five minutes or less. Learn more at Fly.io.

And thanks, of course, to our beat-freaking residents. We couldn't bump the best beats in the biz without Breakmaster Cylinder. That's all for now, but come back next week when we continue discussing everything that happens after Get Pushed.