DOP 291: The Future of Software Development in an AI-Driven World
51:42 Share

The saying I've heard is, your job won't be taken by artificial intelligence, but your job might be taken by someone who knows how to use artificial intelligence better than you do. That's the key. This is DevOps Paradox, episode number 291, the future of software development in an AI-driven world. Welcome to DevOps Paradox. This is a podcast about random stuff in which we, Darren and Victor, pretend we know what we're talking about.

Most of the time, we mask our ignorance by putting the word DevOps everywhere we can and mix it with random buzzwords like Kubernetes, serverless, CICD, team productivity, islands of happiness, and other fancy expressions that make us sound like we know what we're doing. Occasionally, we invite guests who do know something, but we do not do that often since they might make us look incompetent.

The truth is out there, and there is no way we are going to find it. Yes, it's Darren reading this text and feeling embarrassed that Victor made me do it. Here are your hosts, Darren Pope and Victor Farsen. Victor, here we are, coming up to the end of 2024. I can't believe it's the end of 2024 already. We both still have jobs, but some people are starting to lose jobs. And some people are starting to get jobs.

That introduction, if you made it, like, I'm not sure how long, let's say 100 or over 100 years ago when industrial revolution started, would be equally valid, right? That's very true. But their fear wasn't the robots from 2025 or whenever Terminator shows up. I've lost track of when Skynet is supposed to take over. On today's show, we have Derek Ferguson on. Derek, how are you doing? I'm doing well, thanks. How are you? Good. So we're going to be talking about...

AI could kill software jobs. But again, as Victor alluded to, maybe AI means it's now a new hiring spree, just not for prompt engineers. Is that even close? I went back and I looked at the historical precedence for when new technologies and new ways of doing things have come into just even software engineering, like the move from assembler to C, the move from C to Java, so on and so forth. And

Every single time productivity increases, I think from similar to C, it was like a 300% productivity increase. And the demand spiked after that, because it seems like when you get the ability to do a lot more software with the same amount of resources, people find uses for software that were never conceived of before. If you think of like

where software was when assembly and pure binary was the only way that you could code it. Those were just all military applications, right? You had huge metal in order to run them. Nobody ever thought of anything with consumer software, right? See, that suddenly becomes possible. I think I tend to be an optimist. I mean, I'm a universal optimist on pretty much all things, but this is one of them. When we look at the productivity gains that we see coming out with things like generative AI coding, uh,

Those are on the order of 25 to 50%. It's nowhere near the leap that's been made with previous jumps. And I'm not saying that it won't get there because, you know, as is often said, this is the worst AI that we will ever use, but yeah.

In general, what we've seen every single new generation of software technology that's come out, it enables use cases that were never dreamt of before. And then there's more demand and somebody needs to maintain all of that sort of stuff. And so I remain sort of an optimist about all this sort of stuff. I feel that the major flaw in doom and gloom

type of predictions, in this case, related to AI, is something you hinted earlier, is that they all assume that the output will be the same. And that leads to inevitable conclusion that, okay, so we need less people to produce the same output. While my understanding, and I guess what you're saying as well, is that, no, we can change the output. And that's the key. That's what somehow people miss.

I will be able to do this alone. I don't need my team anymore. As if this stays the same. People are going to be doing different things. I often tell the university graduates who work for me, my senior project, and I hate to date myself with this, but

was literally a webpage when I graduated university. And that was, you know, the norm back then, you know, we were, we were happy that we managed to get the scrolling marquee over the top. And it was even before the year of the dancing baby. If you remember that now that wouldn't even be a full homework assignment, right? Because people aren't handcrafting HTML anymore. I think it very likely that a decade from now, it will be a,

half day's programming task for a professional programmer, create an app that does X, Y, Z start to finish. But it doesn't mean that there will be less software because the demands of your consumer always increase.

Proportionately, right? Whatever your company is doing, your competitors will have access to the same tools. And so they'll be, yeah, it's hard to predict, but maybe we'll be entering the era where we create custom applications for each of our customers because we just have that much capacity for software development and software maintenance.

That's the real killer. When we think doom and gloom, we often think scenarios where you produce software and then it's perfect and it never needs any modifications. But we all know, of course, that's not the real world. As much as generative AI can enable us to create tons of software to make those modifications and to make them in a smart way, you still need people who understand the underpinnings behind the controls, right?

Let's play out the scenario, real world scenario. Since you threw in a couple of phrases there, real world is one of them. I will buy that at some point in the future, we'll be able to describe an application and expect it by the end of the day that it's out in production. From zero to production in a day. I buy that. What I don't buy, well, okay, but assume that I buy it just to keep you happy, Victor. Okay.

My problem with it is that what happens on that second day when the changes start coming in, I can't expect the AI or anything else to rewrite that application to match what the customer is requiring. Or can I? You could get to the point where AI will be able to do some modifications, but I think you're still, I mean, it's like autopilot on an airplane, right?

How long have we had autopilots on airplanes? And yet we still, we're only just now starting to talk about the possibility of having completely pilotless airplanes. And most folks I know still aren't super comfortable with that, right? You might get to the point where you have an AI that can suggest, here's what I think you want to do in this modification.

I think it's going to be a long, long time before we're completely comfortable with having AIs do those modifications without the supervision of folks who really know what's going on under the hood. Greenfield development, inherently a lot safer than something where you've already got data in the database and it's going to make modifications to the code that are going to change that data in some way, shape or form. It only takes one, you know,

drop table command going in the wrong place or some misunderstanding that, you know what? All right. When you said to make that modification, I assume that you meant make that modification and start from scratch, right? You didn't need any of that pesky data, right? So I think there are going to be humans in the loop for a long, long time to come. So I think your concern is, although I might be a little bit more optimistic about the power of software to make suggestions for modifications, and I've seen it in many cases where

I don't think we are particularly close to near where AI can do stuff like that unsupervised. I don't think folks are going to feel comfortable with that for a long time. I know I won't. So what you're saying is I'll be retired. By the way, when you said your senior project was creating an HTML page, I graduated from college when there was no HTML. So we won't even go there. Did you use the blink tag? If you use blink, then you had to have passed. Yeah, I did.

You'll be retired. I'll be retired. Victor will be retired. Will our kids be retired when this happens? Are you talking about like the whole 10 year olds? Like, so in 50 years, if we were to play this out, right, let's, let's just go along. Right. In 50 years, like to me, you're set up right now, the Greenfield thing, I see it as spring boot or any of the other sort of bootstrap things on steroids.

We can define it a little bit more. We get something there a lot faster. I'm using Spring Boot because I'm an old Java guy, but there's many others. I see that being true. And within five years, your vision seems to be semi-practical, if not completely practical. But then it's that day two of the modifications. Does that happen in 50 years? I'll be dead and gone by then. So that'll be easy. The 50-year horizon is such an interesting one.

I'm not an expert in it by any means, but aren't most of the projections for quantum computing fairly revolutionary at the 50-year time horizon? And once you get that kind of computing power in place, all sorts of different scenarios change, right? I mean, once you have the ability to solve a preponderance of problems with brute force and to crunch the data that would be required for...

sort of general purpose artificial intelligence. That's a tricky one. I think anybody who engages in trying to figure out what things are going to be like 50 years from now, I remain optimistic even about that. I mean, if you think about that sort of superhuman computing power, presumably has the ability to solve superhuman problems, right? Climate change, disease, all that sort of stuff. Yeah, but will they be able to power it?

Didn't I just hear that a lot of the firms that are trying to do this production of the next generation of LLMs and the generation after that, they're starting to turn to like their own private nuclear power sources, their own generators and such. Yeah. So, you know, okay, we're going out way too far here.

Let's try to bring it back to today. Even though I disagree with Victor on this, I think the bootstrapping process can be better than where we're at today with a lot of things. Victor, would you agree with that? Maybe not the full app, but the bootstrap would be at least further down the line. Oh, yeah. I think that the whole history of software industry, and I think that that includes AI, is about easy tasks becoming or tasks becoming easier, right?

Some tasks are becoming so easy that you don't need a fully qualified, very experienced developer to do them, right? And as a result of those types of changes...

What we as developers need to do becomes more complex, right? It makes no sense. I would never... I don't think that any company would hire pure HTML developer, right? Kind of, we don't need those anymore because it's so easy. It wasn't necessarily easy 20 years ago or 30 or whenever, but now it is. So...

What was previously considered a job for a developer now is considered, at best, a junior position, right? At best. Most likely not even that. And that forces developers to think about solving more complex issues, knowing that simple things are easy. And I think that that's the same for a long, long time now. And from that perspective, and I'm generalizing now, I see AI just...

being one more of those pushes towards, hey, you don't need to focus. You can focus on complicated things because easy things are handled. Don't worry about it. With the note that the definition of what is easy is changing, just to be clear. I sometimes think of the model. So my wife is a nurse and so was her mom.

When her mom started as a nurse, you had people who were in the hospital for a week to have their tonsils out and all that sort of stuff, right? Hospitals were places where you had some folks who were having minor surgery and were in for a long time, some folks who had stuff that was serious.

It ain't that way anymore, right? You know, everybody who can possibly walk is in and out of the hospital in 24 hours. And as a result, the job of nurses and doctors, for that matter, they see a lot more and spend more of their time on the really sick patients rather than giving the same amount of care to the folks who aren't.

And the whole thing's become a lot more efficient because for various and sundry reasons, we figured out how to treat more people at home and all that sort of stuff. But the day-to-day of the average medical professional has gone from being a little bit more varied, dealing with some folks who aren't quite so sick and some folks who are very sick, to a steady procession of folks who are...

If you're in a hospital setting, very sick because that's just the definition. The more cynical side of my nature wonders if 30 years from now, is that what software development is? Are we dealing with just the, I can either say, are we dealing just with the really heinous, difficult problems or more optimistically? And I think the way you sort of phrased it, are we just dealing with all of the really interesting problems? I like to think that we're just going to be dealing with the interesting problems. The problem though, I think with dealing with the interesting problems is

Is will our managers care if they're interesting or not? Because as soon as they see, maybe not the managers, but further up the food chain, AI can save us this much money. So let's put AI on all these cool things and we'll let the humans deal with the, the rote and mundane stuff that the AI still can't figure out. I think it's the competitive factor. I always come back to because it's never being done in isolation, right? Our bosses, competitors have access to the same tools. And so whatever the,

they produce or whatever they do, whatever cuts they make or whatever will be duplicated on our side. And like, let's, let's take the pessimistic case where they, rather than building more, it's using the same number of, or using fewer people to do the same amount of work. What history shows is it always builds back because after you get those efficiencies and you're like, okay, fine. Yeah. It's, you know, we're going to do whatever we need to do to get those efficiencies. The next thing that happens is, um,

customers want more. Some competitor comes up with a new service offering, and then all of that company's competitors need to have the same service offering, and it always winds up building back. So I think even in the case where in the short term, the industry went the way of taking the cost savings rather than the productivity boosts to build more stuff. That always winds up being a short-term play because you're

Humans, being the creatures we are, thankfully continue to have new ideas all the time. And people need to come in and build those new ideas. As those ideas come in, you build back and you wind up having to bring in more folks. There will be companies. There always are companies that will...

reject the change, right? Reject those productivity boosts and say, we've been doing like this forever and that works for us and so on and so forth. There are always companies like that. It's just that they will disappear eventually, right? Even before AI, that was happening all the time, right? So I don't think that companies will all adopt it and all become more productive, right?

It's just some will, some won't, and they will not exist. The saying I've heard is your job won't be taken by artificial intelligence, but your job might be taken by someone who knows how to use artificial intelligence better than you do. That's the key. So going back to AI helping us and the tools no longer, or actually the tools are a competitive advantage or not a competitive advantage, depending on which way you look at it. It almost sounds like in the future,

If you want to actually be what we would consider, quote unquote, a true programmer, you're going to be working for a vendor because those are the people that are solving the big problems. And then the consumers of the vendors are just going to be bootstrapping just the standard, simple business applications. And then nothing else pretty much happens except for the mainframe stuff that's still running in the background that's paying all the bills. Yeah, but look at it today. It's not only vendors, right? Netflix is solving real problems.

Netflix is not a vendor, at least not a software vendor, right? And they deal with complicated things. But those are outliers. But see, they're a technology company. No, they're not outliers. They're those that survive. They're survivors, right. But to me, Netflix is not a streaming company like Blockbuster was trying to be a streaming company. Netflix is a technology company that happens to masquerade as a streaming company.

No, Netflix is a streaming company that understood that they need to become a technology company. Well, they're a DVD company that decided they needed to be a technology company. I still disagree with you. So Derek, you sort of started chiming in there. Where do you think that falls in play? I honestly, in the case of Netflix, would tend to think of them as a entertainment company, understands technology.

technology is what they need in order to move forward and be competitive. I think there are a lot of companies that say we're a technology company first and foremost, but this just happens to be our vertical. And this, I do tend to be a little bit cynical. I think at the end of the day, you look at the power structures within those companies and who is really calling the shots. Is it sales and commercial or is it the technologists?

And the preponderance of those companies, I think you see that it's the sales, or perhaps in Netflix's case, maybe it's the creatives, the folks who are actually creating the content who are in charge. And so I tend to think, and on the topic of, is it just going to be ISVs that are solving the big problems and everyone else just uses their package software? I think not specifically for the reason that

Because in the preponderance of organizations, it's the business, the folks who are actually creating the stuff for sale that are leading the show.

They very rarely are content with using stuff out of the box, right? Every business is convinced that their use case is just a little bit different. Yeah, that works for company ABC, you know, one of our competitors, but we do things just a little bit different. We need that change.

As long as that power dynamic is in place, I think there is always going to be the need for folks at different organizations making those customizations and maintaining those customizations. Even if it's just overseeing an AI that makes and maintains those customizations, I don't see everything going to a centralized model because I don't see businesses ever just saying, yeah, you know what, we'll do things exactly the same way as comes out of the box with package software. Let's talk about the power dynamic for a second.

Again, right now we're in 2024. Most of the people that are in charge at these larger ISVs are of my vintage, late 50s, early 60s. As we start aging out and the next generation kicks in, is that going to still be true? I think of my daughter. She's in her 30s. The way that we approach business is very, very different. So I'm wondering as my generation ages out and the next generation comes in,

Are those same drivers going to be in place? I mean, I could say that that's going to be true probably for financial services, anything that has direct money, but other things, I think all bets are off. When your generation moves, we will finally be able to get rid of mainframes and DB2s. You think that? Here, I'm going to put the stake in the ground right now. Mainframes will continue to exist well into the...

I'll be a hundred years old in 66. There will still be mainframes running. IBM will still be making money hand over fist in mainframes in 2066. Still making money hand over fist on mainframes. Yeah. So based on a statement like that, it sounds like it's not just that you think they will be around because the cost of removing them is so much.

You think that they will actually continue to be actively grown and to provide value? And that's the reason that they'll still be around? I'm interested in that perspective because I thought you were going to say, look, this stuff's been around for so long. It has its fingers in so much stuff. It'll just never be worth organizations' time and trouble to extricate themselves from it. But it almost sounds like – is that – I go beyond that point. I mean, that's baseline.

Because you're not going to be able to get rid of, in my opinion, get rid of mainframes in financial services. Okay, 2066 might be too far, but it's going to be hard. Well, that might be true. Will you be able to get rid of financial services that use them? Not that they get rid of mainframes, but you get rid of them altogether.

FinTechs. I haven't thought about getting rid of financial services. That would be... No, no, not financial services. Financial services that exist for such a long time that they're still using and cannot stop using Mayframe. So you're saying basically retail banking goes away and potentially parts of commercial banking go away and are reimagined. Be replaced by other commercial banking. Whatever it is, right. Yes. I think that's going to be the only way you're going to get rid of them.

The big US banks, the big European banks, they're all mainframe. The big shipping companies, mainframe. How do we... I mean, this is turning into let's kill the mainframe, but do we really need to kill the mainframe? I don't think we do. It's still the right tool for a lot of things. As long as its interface continues to evolve so that you can get the talent that's needed in order to continue running it, right? That would be the real challenge. Like,

20 years from now, will it really be possible to hire folks who are knowledgeable, experienced, and wanting to do something like COBOL as a programming language, for example? Or in order to continue keeping those mainframes up and active, will it be necessary for them to offer all the latest programming interfaces that folks want to use? React and Python and all that sort of stuff. And I think...

I mean, I'm not as close to mainframes as I was in my previous career, but I think they've done a pretty good job of offering some implementation of modern stuff in a mainframe setting, right? I think so. But that's been true all along. We're going back to see this. This goes back to the regeneration story, Dr. Who reference that once something goes in,

people see that, oh, let's pile on that until it becomes, you know, because people want more and more of that. So once we realize, by the way, Kubernetes is nothing but a mainframe there, I had to get that into what's left for people to do. I mean, I think to your answer, once we're paying COBOL programmers three to $400 an hour, which inflation adjusted in 20 years will actually still only be like $10 an hour. There's always going to be a job for people to do the scut work that other people don't want to do. It's going to be hard to find the people

but it's not going to be impossible. Back on the topic of AI though, I wonder, isn't translation, be it human language or software language, isn't that one of the core capabilities of AI that one would expect to see built out? I mean, it's already one of its best strengths. I would tend to think, if you think about

AI building greenfield software and the complexities of dealing with a human being telling it what it wants and all of the ambiguities that go along with that. And you hold that in one hand and then on the other hand, you say, well, what about using AI to take software that already exists and translate it from one setting to another? I actually think that that's probably a lot more within the realm of near-term things that AI can do

100% and get right than dealing with human beings and taking sort of ambiguous requirements for greenfield development. So I would almost think if you have a system that's running on mainframe, it's not difficult to imagine an LLM having sufficient knowledge of mainframe systems and sufficient knowledge of modern technologies that you could potentially go in and say,

And on more of a near term basis, Hey, I've got all this AS 400 stuff. It's all in COBOL. Can you give me some clean Java and react that replaces it all? And that's the sort of stuff, particularly something like an, an, an agent based AI. That's something I could actually really envision getting done. I think that's probably true because COBOL has been around longer than Java, believe it or not. Some people may think that's crazy, but,

But if we were able to get a good model, there should be enough corpus for COBOL to where we don't need humans to do it anymore. But you're saying translate it from COBOL to Java plus React or whatever you want to do. It doesn't really matter. But is that really what we want AI doing for us? I mean, I think the answer is yes, but also the answer is no, because there's going to be nuance in about two lines of what it writes that could either

make the company a ton of money, or sink it. That's the optimistic case for developers. It must be, at least in theory, easier than with human languages, right? Because there is no ambiguity. Code does what it does, period. There is no... I mean, we might have difficulty understanding what the code does, but code is exact. It does what it does. There is no doubt over there. We can just not understand what it does.

Following on what you said before, even if the biggest problem now is not necessarily that we cannot rewrite those things into something better, more modern. That's not the problem. The problem is that we cannot ensure that what we just built is doing the same thing as what we have, right? If I rewrite my mainframe in Go...

That's less of an easier thing to do than confirming that it's doing what it's supposed to be doing. Even if you don't take AIs to rewrite stuff, just go over there and tell me whether this code is functionally the same as that code. I would take that even. It doesn't need to do all the work. I would think on mainframe systems, the existence of full suites of unit tests...

might be less likely than, say, Java, for example. So maybe you could see moving a bunch of Java code to Python, because presumably whoever's been creating that Java code has hopefully been doing unit tests, and that can be something that the... And I think some of... That's actually how I think some of the agent-based AIs that focus in translation work are.

nowadays, right? You know, they see the unit test screen, they make the changes they think are appropriate, they rerun the unit test, then they zero in on the tests that have failed to figure out what they need to regenerate in code. But it was interesting what you said earlier about how an AI translation is going to miss nuances as it goes from COBOL to Java, for example. As I was looking back at the era of the move from

simpler language into C, of course that was a lot of what was said at the time too, right? It's like, well, you know, when I'm down at the level of individual CPU instructions, I can futz with them and I can squeeze out that last bit of performance. Whereas, you know, this C compiler is just, it's always going to give me these blocks of code that may or may not be optimized for the situation. But over time with CPUs,

compute capacity growing exponentially over time, that became less and less of a concern, right? Because folks were like, well, we have so much more processing power than we had just a few years ago. Who cares if the code's not quite as efficient under C as it had been previously?

I think that's probably true here also, right? You'll have AIs that are creating software that might not be optimized, but then either A, that becomes the work of the developers to go in and take a look at 10 different suggestions from the AI and pick the one that's the most efficient, as well as meeting the business need, and then B, going in after the fact and doing those little tweaks and modifications. Just like the first C developers, we're always going into...

the debugger and looking at what the underlying code was that had been produced and doing tweaks that very few people do nowadays. Right. What would be your suggestion to people like me that are fine with AI, but don't see a whole lot of value in it today? What would you say to me? It's like, suck it up. It's too late for you. Go ahead and retire. What's the answer?

Too early. Like I said, the other thing I hear frequently said is this is the worst AI that you'll ever use. Just last night. So I do these quarterly staff updates and it's getting to be about time to do the Q4 staff update. So I went back and looked at the Q3 staff update. It was funny because one of the slides had an AI generated graphic in it, which was state of the art three months ago.

I thought, you know, let me go back and actually try to just generate that same graphic. And it wasn't anything fancy. It was one of these 15, apparently they're called 15 square things. It's the kid's puzzle where you've got 15 numbers on a grid. One of them's missing. You have to slide them around to get them in the right order. So I'd asked AI to do that three months ago. And I said, put software developer job titles on these tiles.

And as AI was frequently doing at that point in time, it couldn't really put words properly onto the tiles. It all came up messed up and everything. So I tried it again last night and it was photorealistic and it was properly labeled. Same system, but the model had improved just that much in three months. So

I guess I would say give it some, you know, I wouldn't even say you need to give it a lot more time. Things are growing at a pretty exponential pace. I would also just say in my experience with AI-based code generation in particular, you just got to remember we're dealing for the first time on a widespread basis with non-deterministic algorithms, which means that

You're going to have to, in most cases, ask it a few times and provide feedback in order to get what it is that you're looking for. But once you get over that hurdle and you realize that this is a model where it expects to interact with you and it expects more guidance, you're going to get more productivity, even if you have to ask the AI to give you a block of code five times before you get what you want. Once you get what you want,

To take that block code and copy it and paste it, you're still, according to industry estimates, anywhere from 25% to 50% up on productivity. For myself, I just find that super compelling. That's happening right now with people as well. If you ask for a block of code from a developer, you're not going to get the block of code you wanted. That's why we have code reviews, right?

That's why we go back and forth on that block of code until it's just right or acceptable at least. So it's not as if what you just said about, hey, you need to ask a couple of times and change what you're asking. That's happening without AI as well. So true. We just now get blocks of code faster. I guess it's the cycles. So what I'm hearing now is...

The engineering managers, the bosses, can actually do all the coding and they actually don't need any human programmers. So does that mean the engineering managers now become developers and the whole chain gets pushed down? No, because I think that if it's a 50% productivity boost, the demand boost is going to be 60%. If it's a 25% productivity boost, the productivity demand is going to be a 35% increase in productivity demands, right?

But isn't that dangerous? Now you're pushing the humans beyond what the machine can help them get done. That's always the case to a certain extent, right? Our desires always slightly exceed our grasp. That's what it is that drives us to build new tools, right? Ever since the dawn of humanity, people have wanted a little bit more than what they could easily get. So one has to figure out

how to work smart rather than working hard. And that's where tools excel in many ways as a species. And I know obviously other primates have used primitive tools, but we are really that species that develops and specializes in using tools to extend our capabilities. I don't see AI as any different fundamentally in that case. We'll go back to the 15 square thing real quick. So three months ago you ran the prompt and you got a result, right?

Now you ran the same prompt and got a different, better result. Were you self-hosting that AI model or were you using a service? That was a service. So where I'm heading with this is will big companies be able to self-host these models and see that kind of turnaround in three months?

It's basically the long story of if you're self-hosting something, it normally takes longer to update it versus if you're using a service. That's my thesis on it anyway. That's a very broad brush. But with AI, it seems like it's a lot messier than even just old things. Self-hosting is super tricky when you have that level of resource demand. And is the concern there what data escape or helping train other people's models or...

Something different? It's not my concern. Okay, so the concern would probably come out of privacy. But then to me, it's just going to be, okay, from the service, the model is being updated all the time or being at least massaged to where it appears it's being updated all the time. Whereas internally, if I have the model, yes, I've got everything inside my four walls, but the bosses are expecting what they're seeing on the outside world, the mid-journey level type churn.

anything from open AI. But internally it's like, Oh yeah, we grabbed the latest foundation model from IBM because we're still basically an IBM shop and it's a foundation model. And we're just trying to figure out how to rag it or something to, to make it useful for us. And the bosses, the high bosses are seeing, okay, all this stuff's happening outside, but inside nothing's changing. We're still doing things exactly the same way internally. How are people going to have to change how they work in order to, um,

train their AI overlord that's going to be taking over their job in the next 12 months. Or maybe that's the thing is they're not going to train it. So they still have a job in 12 months. The whole idea of testing, I think is real interesting in that context because QA for as long as I've been in the industry. And I think as long as any of us have been in the industry has been a fundamentally deterministic task, right? Yeah.

You put in facts X, Y, and Z, and you can reasonably expect to get back answers A, B, and C. All of our test scripts up to this point have been written with that expectation. I'm going to put in these inputs, and here's the exact outputs I'll get back. You put LLMs into the equation, things get better.

a lot more interesting if you're a test engineer, I think, nowadays, because you've got to figure out ways to interpret results. And what I've seen done thus far is you wind up getting LLMs actually as a part of the testing equation also, because you essentially need to have an LLM to interpret the responses from another LLM. If you're using something that's hosted and being constantly updated,

the need for that becomes even more intense because you're having a technology that's evolving so rapidly, the world is sort of shifting under your feet. Today, even if you have something like RAG in front of it, maybe you've got some sort of search through your content or something and you're used to getting back. When they ask the following question, you should get back an answer, which means approximately the following thing. If you've got a model that's constantly under update and it's serviced,

Maybe that answer fundamentally changes. That's where I think having that acumen and answer to your earlier question about how do things have to change, the need for automated testing has never been greater.

I think one skill set in the technology arena I would really, really, really be focused on is the testing skill set and making sure that you're evolving from manual testing into next generation automated testing because it's like a whole new world where when for the first time the answer is that someone can get out of software are going to vary a little bit on every level.

And yet there's many valid ways to say the same thing. That's where I think a lot of investment is going to need to be made is revisiting test scripts. What does that next generation automated testing look like? It just feels like we're still throwing spaghetti against the wall. I think it's testing that has to, I had, was it set cannon against cannon or fight fire with fire, depending upon how you like the, the metaphor, but,

It's going to have to incorporate artificial intelligence just like the systems it's testing. So you mentioned RAG earlier. You'd like to think, or I guess I should start, the way that things have always worked is if you go to a system and you put in a question and it has to be a highly prior to LLM, so it would have to be a very specific query, maybe with dropdowns or whatever, to get a specific answer.

That's the traditional way that testing has worked. And it's, you know, we use stuff like Selenium and Quick Test Pro, all that sort of stuff. Just it's used to saying, all right, you put these values here, you should get these values out here. But I think now I see a lot more with test engineers and

Because the answer is going to be something that comes back in a natural language, and that phraseology changes every single invocation, even if the same question is asked. The test scripts need to incorporate an LLM on the back end with a very specific prompt, which is, I want you to look at this

sentence or paragraph that has been fed into you. And I want you to tell me yes, if it boils down to the following fact, I want you to tell me no, if it doesn't so that you've actually got an LLM on the test side, which is capable of dealing with the variability in the way that the response might come back and might be phrased. We think,

It's going to be 30 degrees warmer next month. Next month, we think it'll be 100 degrees. All the different ways that you can express the same thought using different words has to be interpreted by a test script on the back end, which is using LLMs to say,

You might see this phrased a hundred different ways, but at the end of the day, the only appropriate response is one that tells you that its prediction for the temperature next month is going to be a hundred degrees. Give me a pass. If the response you get back equates to that, give me a fail. If it doesn't, that's where I think QA needs to go. I think, I think QA is going to become a much more technology focused profession over the next five years. I think,

manual testing in the era of AI. I don't know that it's going to cut it necessarily. I really honestly hoped that that same response would have come long before we started even talking about AIs. Honestly, I still don't understand how manual testing is done at large scale these days. I understand, okay, mom and pop shop, you created a website and

You try it out, all that stuff. I understand how you manually test it. Even I wouldn't waste my time writing automated tests in any form or way. But something like a Google-sized project, right? Forget about AI. I don't understand how people are still expected to be tested manually.

Now, when we're talking about millions of users, random parts of those users, nobody can predict what they do. We're talking about thousands of developers. We're talking about releases going to production multiple times a day. How are we even still talking about manual testing is beyond my comprehension. I tend to agree. And yet, I think in many cases, we still are. And it's interesting because it gets into the whole...

conversation about do we steer towards a model where requirements documented to the degree that you don't require an SME to do the testing or are they a high level requirement

and you have development staff and testers who know the industry that they're in well enough that they can perform those interpretations on top of the requirements and just get it right. And I've seen it done both ways. Probably the worst combinations are where you try to have the inexact requirements and you pair them with the technology staff who aren't particularly close to the business.

and can't make those sorts of interpretations, which is a, that's a topic in and of itself. But to your point, a lot of the same issues exist whether you have AI or not. What I think AI does is it exacerbates bad situations, and it probably accelerates good situations. If you have those fundamental good practices in place,

You're going to be much better positioned for AI than if you've languished a little bit and not developed the right policies and procedures and ways of doing things. I want to throw in the one thing about the documentation. This takes us back to business requirements documents. Yes, those things used to be a thing. I think there's still a thing in some areas. I could see that a fully flushed out business requirement document is what feeds that test agent document.

And that is the rule. I mean, that's what we're always supposed to do manually testing is go against the business requirements documents. Have you covered all the cases there? What could get more interesting is that the test agent now could actually, hey, did you think about these things? Go back to banking. We're wanting to do transfers. Well, we can't do a negative transfer, right? And I don't see a case for that. Do you want me to write a case for you for that? Those kinds of things could automatically show up for us. I feel that you just described

during testing as it should be in a way that I can easily write code that validates whether your requirements were implemented. What I cannot easily write code that validates is whether it makes sense, which is my interpretation of what you just said, Darin, right?

That's where manual testing, from my perspective, really jumps in. I'm not going to validate that if I put those two numbers as input, I get some of those two numbers as output. Any developer can write code that validates that. Whether summing numbers instead of multiplying them makes sense, that's what I can do. I sort of interpreted what you were saying as...

Using AI to look at the set of tests that exist against a requirement and suggest edge cases that might've gotten missed, right? Correct. Which I think is a super interesting application for it. Good. But you both proved my point. I set out a piece of text. I set up prompt and I got a response from the Victor model and I got a response from the Derek model and they were different.

But now we have to resolve and figure out which one is correct. Or are they both correct, depending on the situation? And the answer is yes. Welcome to the world of programming and AI. So all of Derek's contact information is going to be down in the episode description. Derek, thanks for being with us today. Thank you very much. We hope this episode was helpful to you.

If you want to discuss it or ask a question, please reach out to us. Our contact information and a link to the Slack workspace are at devopsparadox.com slash contact. If you subscribe through Apple Podcasts, be sure to leave us a review there. That helps other people discover this podcast. Go sign up right now at devopsparadox.com to receive an email whenever we drop the latest episode. Thank you for listening to DevOps Paradox.

♪♪♪