#486: CSnakes: Embed Python code in .NET
01:02:45 Share

If you are a doctor, developer or working a place that has some of those folks wouldn't be great to fully leverage the entirety of pipi with its almost six hundred thousand packages inside your dot code. But how would you do this? Previous efforts have let you write python syntax, but using the full libraries, especially the sea and rest based ones, and has been out of reach until the snakes.

This project by Anthony shaw and iron power unlock some pretty serious integration between the two languages, and we have them both on the show today to tell us all about IT. This is talk about on me, absolute four hundred and eighty six, recorded monday, november fourth, two thousand and twenty four. Living to mi kennon python to me, life important organ.

And this segment was made with python. Welcome to you talk python omy a weekly pocket on python. This is your host Michael Kennedy. Follow me on mid on my at m knee and follow the podcast using at talk python both accounts over at Foster on dot org and keep up with the show and listened to over nine years of epsom at talk paths dot of m if you want to be part of our life episodes, you can find the live streams over on youtube, subscribe to our youtube channel over at talk athon other film, slash youtube and get notified about upcoming shows.

This epsom is sponsored by posit connect from the makers of shiny public share and deploy all of your data projects that you're creating, your impact on streaming dash, shiny boat, fast, A P, I, flash quarter reports, dashboards and A P S. Posit connect supports all of them. Try posit connect for free by going to talk by thon that a firm flash posit B O S IT.

And this episode is brought. You buy a blue host. Do you need a website? Best get blue host.

Their A I build your world press site in minutes and there are built in tools optimized your growth. Don't wait. Visit hackathon on dot F M slash blue host. They gets started, everyone. Before we jump into the conversation, I want to talk about infrastructure.

Actually, there are a couple of interesting things going on over here at talk python, and I write them up on our new talk python blog, which I will a link to both of these articles in the shower note. First, we moved our web hosting over to a brand new hosting sitter in the U. S.

Run by a german company called header. And it's super interesting. I wrote this all up and I think people who are considering me where should we run our apps if they're unhappy where they are, it's worth thinking about.

I just rewrote all of talk python, but f ms code, about ten thousand lines of path on to move to an a web framework. Which one was IT? Well, that was part of the journey.

And so I did a big, long right up on migrating from permit to court, which is a all the other frameworks I consider doing that, how I went to the migration and how it's working. So if either these two sound interesting to you, check out the links to the articles in the showing. Te, as always, thank you so much for listening.

Let's jump in the conversation. Hello, Denny. Hello, everyone. Welcome to talk back.

And me I .

think great to have you here and always, always good to have you back on the show the most popular guess and see where IT land where this puts you in the the list. But you're you're right near the top of the guest page out of all I love having on the show. Thank you. And iron welcome is great to have you here. So even though we've been on a bunch times, Anthony.

maybe a real quick influence just here. And I do hope on to open source projects. And I read a book called to sea python internals, which is how most people know me these days, not me.

Yes, we're going to talk about more sepi than internals, maybe putting python inside of dot net. I don't know. It's something is going into the internals of the something else. We'll figure out which way that goes. That is a very project Better about yourself.

Well, i'm definitely old one out here. I am someone who is pretty much never written path in my life. I merely I I now know a bit more about IT having work from this project with Anthony, but i'm a doctor developed by by trade being doing that longer than.

I admit.

on a recorded audio. At ten years, yes, we ll go with at least ten year.

I'm set a good ba bo us.

a good one. I also make soft down. I an team, Jason dan's doing advocate around or match and yeah oh wise.

I also write java script, so do everything I can to avoid having to to get into the python space in my time. And here you are and here I am. Yes.

amazing. yeah. So it's it's a different world, right? Coming from dot net that's guaranteed static typing lot of Simon Collins lot the types. And then I can says and White .

space python loves its White space. But then again, like I have a very small spot for shop. So i'm not like I kind of get the wide spacing aspect of the python language bit the same time I do miss my Kelly brackets.

Yeah the trick is if you use a good editor, you don't really notice IT. If you don't, it's pretty ad. So use a good other. I also to IT and that he would be an up to what's .

new in your world. Been up to this project for a few months .

now is now I was for .

a few months. Yeah, i'm try to get ready for. We've got a massive conference, one of our two big conference, the year coming up in a couple .

of years time.

working hard on demos to try and get for partner ference.

The is so big thousand?

Yes, exactly.

I went to a prepare mica. I think he was thirty thousand people. Podcast rose.

So microsoft had me out as one of ten podcast or something. I thought, what is this? Hardly ever heard of this. I've heard about conference and .

at the moment, but I have been working on this project for a few months, and we kind of kicked IT off in my idea, kind of floated around in may at the build conference, which is Marks of developer conference of this, talking to a lot of different developers and people in microsoft were kind of kicking around the idea of, if we looked at this again, so like running pacing code from dont like how would you approach that? I kind of had some ideas, put some ideas together and then started working with on a really early prototype just to see like if IT was even possible. And then we kind of got something working character in the been pretty much started again and then redesigned. And then yeah, we have been of fiddly .

with that ever sense from A M person. But as someone in the dont ecosystem looking at the the breath of library that exist for python, there's a lot of things that I would have loved to views doing some of the A I stuff that we've been doing main project recently. This there's just not as many packages or options in the dont space.

So it's it's always been that like, uh, okay well, if I want to earn this, the role is in recommend using his evidence I was was always kind of like I we bridge that gap like that was kind of one of the the early thinking of just that's that's a clear problems. But I like a lot of people are learning that. But then going into industry, maybe python isn't the language that the the organization that working with the using. So and we still have that a fundamental .

knowledge explosion and possibility that is, if you can say in, by the way, there's almost six hundred thousand other libraries, including the biggest community of A I based resources that are now, as far as you are concerned of, to do you .

sun in, imagine you are six hundred thousand python packages. You had that to is like four hundred thousand packages on new, yet million possibilities.

Yeah, it's amazing. And just for people have seen primarily, primarily by the audience, new get is dotted its equivalent of our pipi OK. You this idea body said, hey, how would you go about doing?

And then you were suck. Yeah, I was such of designers mean, they set up with looking at how you might integrate python into dot net for a while. Like, if you were to approach, how would you do? And I kind of them through all the projects that try this in the past, or there are different ways of doing IT and posing coons of each and kind of the posing coons of a really low level integration on a high level one. So like a high level one would be stick a fast A P, I on top of your python APP and just call IT via H D. P.

rest. Somehow auto generates some rest and points and then make those do their thing.

Yeah, you could use that like p, open A P, S, back on top of the in auto generate client and that that's like that's like the high level option. And then like the lowest level option is like, you know, you get into the bike code level, which i've explored years ago with pigeon and yeah, that's a very deep, dark tunnel.

It's full of terrible education and corners and sharp edges.

And and one of the chAllenges there is that the like patterns implementation is changing so rapidly right now, like, but just in past and three thirteen and looking at three fourteen now. And like all the changes after now, like this is continuously changing. So IT needs to be a happy medium. And in terms of how we 嗯 blend the two together.

right there has to be some level of disconnecting ess a little bit, right? So that that python can do is have its changes without .

you constantly chasing in, right? Yes, because we would do the integration at the sea level using pattern C A P, I. There's a way of consuming python through that like an embedded mode. Not many projects to do this, but is definitely possible. So you can spin up a present interpreter or inside another process, which in all cases.

that process yeah and I think blender does that. I'm pretty sure that there is a bunch in the film and movie industry that do that for their automation pipelines. I don't think it's that great.

What I talk to some of the people from the movie and video space, and they have most must work on python too, because when they wrote the implementation to integrate the thing, integrate sea python, it's done. And like kind of said, pardon moved on in that way, a lot of work. And then we got just leave on python to and that's that's not great.

Yeah exactly. So we looked at this and said, we said we wanted support the new version of python. We want to support the of practically old version, which I think settled around three point eight, was like that as old as I was willing to go.

I don't think we may be present four to three point nine because there was some specific features we need on the dont side. Again, there's a lot of change is happening and may be can share IT more. But like you know, that is going through just as many changes. So we ve got like to moving constantly in platforms, and we want to put make an integration between the two of them. So let's think about like how can we do this in the way we does, he people, and we can make IT, you know, we can test that and we can make .

a stable as well. Yeah, one thing maybe you could speak to as part of this, people might know dot net as this windows programing library that use visual studio with. right? But it's it's changed a lot since then. There's da core, there's source that runs on linux. There's there's a lot of different things that have been changed about IT.

right? Yes, it's a little of conception we ve got is that the people look quite dot net with windows. And like historical, that's that's true.

Like that that's what I got started developing on windows and i'll be once and still the windows developing what my primary O S. Has been over. I flat with macos and linux over the years, but windows is just where i'm more comfortable.

But there has been a bit over a decade now since the the essentially relate ched, the net was originally called don t call or so. So now we kind of have these two streams. And currently you ve got done ET framework, which is the one that is what we traditionally thought of the is on IT IT chips kind with windows and stuff like that.

So it's it's very much designed around that longer tail support life cycle and and what not IT IT is very stable but also getting minimalist investment just because IT has that kind of need to be stable and consistent year in, year out, then we have done to core or now we just kind of refer to what is DNA. Naming things is hard, let's less be honest. But so we we have done IT and every year and and your release comes out and this is like my major framework, like a major revision to the wrong time.

So you done at nine is actually coming out next week, at least the time for calling its next week. And this is considered like A A short time support life cycle. You are familiar with support life cycles of a lot of run time out there now. So I know populist this quite a lot now part in standing to adopted a bit more that can if you have this yearly rolling life cycle and and you have long term supports and shorter support, the long term is every even number. So done at eight being the most recent long term support ah done at six was the previous one, which is actually going as support next week really is where people are and developing our you couldn't just target python three, thirteen and and don't at nine as the only support of versions because be honest, most most packages are that somewhere most projects are gonna be. So we have to make some some .

calls that you have have a little bulk in there even for me, some my web apps and stuff. I the way a few weeks upgrade to three thirteen because some dependency, dependency has some of that had been removed, had been different for removed, and I didn't work. And well, try that again in a few weeks until IT works and ever caught up.

It's definitely a chAllenge. This portion, a op python omy, is brought you by posit the makers of shiny, formally, our studio, especially shining for python. Let me ask you a question.

Are you building awesome things? Of course you are. You are.

Develop a day a scientist. That's what we do. And you should check out posit connect.

Basic connect is a way for you to publish, share and deploy all the data products that you're building using python. People ask me the same question all the time, Michael. I have some cool data science project or note of that I built.

How do I share with my users? Stakeholder teammate, I need to learn fast. A P I, A flaskful. Maybe you will react is now those are cool technologies and i'm sure you benefit from them, but maybe stay focused on the data project. Let posit connect handle that side of things.

With posit connect, you can rapidly and securely deploy the things you build in python streaming, dash, shiny poke, fast P, I, flash, quarto ports, dash wards and A P, S, position net supports. All of them in posit connect comes with all the belden vessels to satisfy IT and other enterprise requirements. Make the played the easiest step in your workflow with posit connect for limited time.

You can try posit connect for free for three months by going to talk by them at a film. Slash posit. That's talk by thunder F M slash P O S I T.

The link is in your player shows. Think to the team at posit for supporting talk by done anything. When we open this, you mentioned there had been other attempts, other ways.

The one that comes to mind for me is iron python. Iron python is A C, L, R implementation of python and is a pretty big difference right there than what you're trained. So maybe contrast IT with some of the things people done previously.

Yeah every three projects I thinks that have been done previously, that kind of IT is set in the same space, but they have different goals. So I mean, the main goal for our project is that there are lots of delta um that have existing applications. They are typically quite and like a city enterprise based a lot of IT.

So they are like quite big, big applications. They got teams working on them and there's a technology in party on they want to use that. Technology is often package or library or an example, especially like data science now and a lot of the ma I libraries that are fatting around a lot of the idea for there's in the demand from developers was from you.

We we want to use these libraries are we want to use this particular package in python and like how do we import them into our dinner application. So like that kind of the angle that wave up at this from which is like how can we make the best of both words? Like you've got a you like a fast mature enterprise application in the net.

And then is like a piece of technology is in python that you want to use. I python is an implementation of python written in and that was done quite while ago, I think. And yeah like old versions of donate old versions of python.

And it's kind of similar to Jason as well. Like I think I think put in the same bucket is Jason, which is I wait as well, you like. And at that time, a lot people were making different implementations of python, the language in other works.

And so yes, you can write python syntax and run IT in the C, L. R. But you couldn't do something like important on pie.

And that's the difference. The most packages in the ecosystem these days are designed and tested pacy python. And this is all as at at least they're tested for pie pie. But all the other implementation are pretty much, I know like like pandas, emp ire, like a learn, all the designer tools.

especially the see in a up type of stuff yeah.

And I think of written about this in the past how you know patterns. But one of the solutions to python s before once problem is that you implement extension modules and see which is a great solution, or you cythia, which is the same kind of thing, which is a great solution. That just means that the ecosystem is more, more tight to see python as the interpreter or choice. So what I didn't want to do is, right, a home new interpreter like that goes back to the compatibility problem and the maintainability issue. Bottle time we published IT would be out of date and we just be opening yourself for a whole later work.

It's not even about is not even matter whether you can do IT and you can keep up with the language as we talk and there's a million library you can use. But if you did this, he wouldn't be able to.

Yeah, i'm in is one pigeon we've covered in the past episode, that is A P, P, Y, T, L, P, Y, J, I.

O N, J, I, O N.

This is a git for sea python that happens to use the 点 net core jet engine， because that was one of the most plugged jets at the time. There are other jets available. IT could have been written in as IT could have been made in L, V, M jet.

I didn't know who actually work on I python so like he knows the space for too well, broken and worked on pigeon originally and then to remote IT a few years later. That's a different problem. That doesn't help you integrate dinant python together.

IT just happens to use components from dot net to to implement a jet. That project is effectively dead. Some of the benefits of IT have been rolled into sea in versions three ten and three eleven.

The things that made IT faster, most of those were so to pick up anyway. And in the third one is python don't net, which is probably the closest to the project that we've started. This one has been around for a while ah.

This one been around for is a similar thing, but they they mostly focus on the integration in the other way around. So like how do you call dont library and things from python? You can call python.

Oh, right. I don't net your way to that.

And this project has been around for a while. So the API is pretty stable. We originally looked at this one actually is a potential solution that found a number of things that they kind of tie themselves to in the design, which you're gonna go away and on nine.

So IT wasn't really an option, an option for us. And also like a lot of way, the C A P H is was different to how we wanted to do IT. So that's kind of the .

alternatives. And so you ended up writing room because just none of these really match, right? The iron python was just never gonna keep up, and I IT didn't have this package support for was the other way around.

And yeah, so well, I guess so I could think to think about, well, why did you think you could do that? Like a lot of people, a lot of smart people put a lot of time into this. Why did you think this is gonna? This is going to work well. Do you have a fresh take on?

And what are you thinking? I really wanted to use some of the modern benefits of boys python and do not to do this to design IT. And one of the biggest changes in such as we've been implementing this, we're learning and eighty percent of the chAllenges of related typing and aren't smiling because he agreeing with me, like integrating two different ecosystem together like this, is about calling functions, methods, whatever.

And but calling things is about calling them with arguments ready and passing data back was in force. Muscling information and getting things back was in force between different types like that. If you integrated, I don't know, some function that you are in python into dot net, you're gonna calling IT with something like you're gona be sending at some data and expecting some data back. Now in python, there are types. IT is dynamically typed, but the types do exist .

even if you don't put IT in your. In your function signature, there are still instance strings that up being installed over .

the place behind the scenes. Yeah, exactly the the net, like you, you could call a function in from the net in python, but you'd have to write a load of code. That said, like I think this parameter is probably this types to convert that to this. And then when he comes back, then it's probably this one to try that one first.

And so we just said right now, typing type pilotage and more popular in python, what we wanted to do was basically, you give IT some python code when you write like a function, which is the wonder you want to call and you and 猜 的 function， you say IT takes, you know, has these arguments, and they are these types, and IT returns back this type. And then what we did is row a code generator. Could a source generator that would, in in your project that would look at the the feel that you wanted to embed IT, would see what functions are defined, what their signatures are and what types they take.

And then I would generate a function that can call that for you with an equivalence. And then IT handles the the Marshall, like the transfer of the types back wasn't forwards between dt, python and like that. Most of the project is really about that partition.

okay? So it's leveraging the C A P, I python kind of talk directly to the runtime .

you can write function in python, you know that takes string a list on a couple and returns back a dictionary like is a very common to do yeah and you would give IT type annotations in python, and you dropped that python file into your net project and you run, build again, and you would see a type in the net with the same name with the function, that is the one you just defined, and with arguments which are dot net equivalent to that. And you just call the function, and IT just runs important.

How complicated can this data exchange me? Can I come up from my own types and exchange them? Or do they have to be six or seven core types? I think the data exchange is actually one of the really tRicky bits, right? Here's a list of a million items. Won't you copy them all over to a dott type N N Y over on the run thing and then copy back to a dott list more im in IT or something crazy like that?

Rain, yeah. If you jump to the documentation in the reference page, there is there's a list of the ones that we support at the moment, then there's a list of the core ones that we support and then anything beyond that. So we do generic for examples that if you say it's a list of strings, we will give you back a read only list of string.

And in dont if he is a dictionary and you type the key in the value, we will give you back. I do not key value. Now in python, often IT could be in any or IT could be a union or IT could be something complicated. So you've got the option to say IT could be a dynamically defined type IT that's still supported and then a run time you can look at the type to see what IT is and then converted .

into something you need. I see a lot of read only on the darnell side. And this is the thing .

receiving the data from python picked up on I I decision when we do in the dot net service area for this of that, that any time we're receiving data back or and when we started sending data, this should be represented as we only just to indicate that if you're calling function has been exposed from python from dot knt and it's returning a collection like a dictionary list or not, that they should be returned is read only just to make IT very clear to you as the consumer that you don't own this data like you earn the one that there is nothing inside of your application technically that created that data that you've then got responsibility for. You can then transfer that into some kind of meal data structure if you do need to mute, but then you then you're making a clear code choice to transform that from the original source of the daughter into something that is then immutable with inside of your new .

run time space OK. So had an read list of IT, you could create a list of IT, and then you'd have to pass that to the constructor.

Something and I would do A I item could be for doing like, like creating an a right from an existing collection of of daughter. And then that becomes more 迷。 无聊的 啥时候 like a list， like a the read only list, for example, that then plug strait into links so that you can do the the the query in tax or the expression methods in text to to perform Operations on the those collections of data inside a dont space, once it's being hydrated back out of out of python。

So I guess you get a lot of these read only a neural type of things. You could do the link on the l yeah, I wish we had this thon. I know we have something kind like a thing, but it's the really want to make sure .

that we were tackling with this and and dancing point before of the that you've got two very different types system is that you're working with and structure the way that you would write coding part and he is going to look different to the way that you write in down that. So we didn't want to to feel like the cogen that we do for you is that we've just made this is just donate casing of python code.

We wanted to feel like as I do not develop and i'm working against the do not in a face, so when you work with the types and the the the the modules that you expose out to and we do to, we then generate you a class behind the scenes. That is then we then generated in a face for you, and then that's registered into the depends on action tiner. So you then can just resolve that elsewhere with inside your application.

So now as I don't not developer, i'm important I am getting an interface injected into a class that i'm using elsewhere with inside of my code base. I haven't had to to go OK. Well, no, I am calling some kind of like an import or i'm doing some kind of resolve.

Also think that might be unfamiliar to me is and not developer. This is what i'm doing is just IT just feels like i'm running I shouted, say so dona, i'm running sea shop and I we've optimize this sea shop. I said, have soft shop, the f shop up and kind of works in f shop. And if you if we do not develop A V I, we optimize this to to the kind of development experience you would expect when you're riding a shop code. And we just so happen to have you you haven't run this, some of the c shop code we have just automatically generated for you and injected IT into a so when .

we're talking about exchange in this data, this is sea python in the same process. Is that right in the ony? yeah.

How's that yon C A P. I. Normally if you call so c python is the interpreter that you would use to run your pin code.

IT is implemented in sea, mostly the standard library modules of written some of the marine python, some of the many say you can write modules for python in c and the way that you do things like create numbers and dictionary and less is used. This A P, I, is to see API. And part of that, C, A, P, I is the ability to spin up a python interpreter inside another process.

And that's embedding mb dde python. There's different A P S for doing that. There's a lot of different ways of approaching IT. But really what we do is that when you spin up, don't you see next you inject a python interpreter into the environment.

It's not like a sep process, like it's not IT hasn't got like come on windows running in the background and embedded into the process is like loading. The library is running inside the process. And then we have full control over everything in sea python.

So we can then import modules. We can convert things at a types back was and forwards. So we're kind of doing a lot of the work that often gets done either in the same extension module or inside the interpreter.

So if we want to create a pattern number, for example, like an interview from a dont integer, we do that the sea levels. So we can do that like a really high performance A P. I. If we want to look at the size of dictionaries, of the types and stuff at that, we do that the sea levels, so against its super high performance.

And really importantly, while the typing stuff is really important to the other major chAllenge for this is, you know, do not freely uses threats and thread poles and and carrotty and stuff like that pison can you threading? But there is a luck. So you have to be really, really careful.

And this is something I don't didn't see like a novel solution from the other projects that are looking in this kind of space about how they would tackle IT or IT was kind of based around like older styles of python. And things have change so much recently. So we kind of want you to make sure that you could run this in the multiplicative environment, whether that's like A A U I, for example, like a detox interface or is A A web APP.

So like one of our demos is like is a doctor web APP like A P I um that each request can run a part in function if he wants to. And they were running in separate threads, in a thread pool, in dot net. And you shouldn't have to worry about the girl like you didn't have to know too much in terms of what is and how IT works. So we've done quite a lot of work to make sure of that. That can happen.

This portion, a talk path on me, is brought you by blue host. Got ideas, but no idea how to build website. Get blue host with their A I design tool, you can quickly generate a high quality fast loading website instantly. Once you need all the look, just hit inner and your side goes live, really that simple.

And that doesn't matter whether you're a hobby est entrepreneur or just starting your side, husband blue host, has you covered with built in marketing and e commerce tools to help you grow and scale your website for the long hall. Since you're listen to my show, you probably no path on, but sometimes it's Better to focus on what you're creating rather than a custom build website and add another month till you launch your idea. When you look great, the blue host cloud, you get a hundred percent of time and twenty four seven support to ensure your site stays online through heavy traffic.

Blue host really makes building your dream website easier than ever. So what's stopping you? You've already got the vision.

Make IT real visit talk python df m slash blue host right now and get started today. And thank you to book host for supporting the show. Python has A C, can wait A C share, pass A C in away.

They're similar, but not the same. yeah. Can I have an aging function in python that is called by a shirt?

No, no, yes, just a test out.

Run easy. I don't run the completion sort of thing and then keep going .

yeah I would love to do this. But there there is no A P I for on the python side to like manage event loops and in bed an event loop into another process or to write a custom of event loop. It's a what kind of bill into the standard library was put into another module. There's been a couple of proposals in draft peps, but another have been accepted. And if they were, then they might be in three fourteen anyway, which we can't that would compatible support yeah .

I do not quite there time wise yet. Do you find that? I think I will. Event lives and paths are hard to juggle, right? Like you have to create one if IT doesn't exist, but if he does exist and IT doesn't want you to create anyone. But there's not a great see if that exists. And if you don't have to rect access to IT how you get that, it's just a IT always feels to me a little bit of touch to fully take controller .

of that and work changes between Price versions like you gets. It's gotten easier recently like this is like a dot run function now and I have to like, get the event opens, create one instead of that. Now you can just run the current is still tRicky.

And the idea of basic writing a custom event loop and then giving a routine objects and like polling them and leaving them from another run time. I just couldn't figure out the API i'd love to do IT. If anyone knows, then please ping me.

Well, U, V, look, does IT yeah somehow, but maybe they just swap out .

some pieces of IT.

not all of IT. yeah. I know you set the event look from so I see in the steps, install python.

Put your projects in the sea shark file. Use invitations on your python functions, right? That's a really important stuff he talked about.

Install the sea snakes runtime package. So maybe from this point on, let's talk a little bit about some of the steps. So this sea snakes runtime, this is a new get package that you put into your that project. Is that what i'm read new?

yes. So we essentially have two binaries, leadership. One which is going to do all the code generation for you. So I from a dont perspective, you'd be IT wouldn't be uncommon to have multiple different projects with inside of your solution, which is then like one might just be a bunch of services, which are they going to then talk to a database or what of the case may be.

So you might have that as just a like that would be way you would have the code in that you want to do so where the python files are getting the sea shopping rop generated for you. So you have one project with that and then we have a second, second binary, which is then the run time actually like make that work when your application is running. Now we do ship this is a simple consolidated package just for these of views. But that's kind of why we we differentiate some of in terms of a coach in verses runtime because the different concepts and one is a compilation that versus obviously run .

time yeah OK with .

that run time package and the the documentation talk a little bit more about this is just a few steps you go to do to make sure that the sushi compiler is capable finding the byand files. So we actually do stuff with inside of the c shop compiled pipeline to generate code that is then add into IT. So and this also means that we can vary early on, find out if you're going to have a compatibility issues.

So if something is is type to the type that we don't understand or the the python court is syntactically obscures, what i've learned is that there is a waste. You can write valid python code that is very obs, and he should be some that ah some some things that you can do. But please, I don't. And yes, so like you actually file a completion step rather fell run time persisted. And I think like again from not all the perspective like here, we use to that static typing, that compilation step, that that belief the compiler has done a whole bunch of checks us to give this code that should just rather than just like that more typing system that hit out of path and or like in my experience, like the joy of things.

definitely expected if you're doing c share plus plus if you compiles, at least it's clicked together in A O way. Where is where by thon were or pressure? We are no Fisher, but or pretty sure.

So in python typing, you can lie, I could say a taste, the list of energy and really, what IT took was strings, which had numbers of character. Who knows? right? What happens if I lie to the sea? Sharp code jen thing. So I just crashes.

Don't do IT yeah, specially you just just crash winkles. Because IT IT has to make the assumption that what you've told IT is correct. The air in the net, if i've got a couple of vents and I give IT across and is like I actually want .

a strings going to IT.

IT will rise yeah. Because to get for example, if IT returns a list of strings but you actually return a list of inter, you in order to give you back a list of strings in sea sharp, we first check that is a list if IT isn't or if it's not a sequence type like IT could be a and if you could have inherited from list a major own customer less or you could make something which is a sequence, like if it's not that, then we will raise an exception.

IT doesn't just crash was joking and we got a lot testing to make. There's a whole like a test environment is around like lying coated and type hints and stuff which are completely invalid. And in for the strings, you know, we need to look at the python string and then copyin.

Most of uses U, T, F A. And don't know, most of uses U, T, F. sixteen. You know, we need to convert from one to the other. And if it's and if it's an interview, not string in the first place, and that's never gna work. So we would raise an exception there and say.

you taught .

us that was a string.

I we've tried to like the exceptions we raise, I donate exceptions, which are relevant to the concept of the problem that we've cover. The same time, IT actually will give. I imagine if something failed in the python as well, like you type annotated, that IT was like a bunch of strings.

And we and I know a bunch of east and we gave you a bunch of strings. Well, okay, the python court is going to fail there. So where we then surface out the python exceptions as well as like embedded with insider of the dot net exception. So you can you can actually get down to like the lines of code with inside of python that failed.

which I think is kind of service actually includes he didn't .

IT raise from IT. With that, will sharp as an inner exception in dont as well. So we did a lot work on the exceptions, most of because we had to like debug the stuff we were developing IT and and that .

someone else .

was going to have. You see you can see this.

that ice. You get like all the frame variables and stuff for that. So things you would probably want from, like baby, you can see on the dot net side. So if you're running the debug a in T, T and IT crashes in python like IT all raises an exception or something. You see the python exception with all the details, and then you see like inner exceptions and stuff like that as well as well as the locals in the global and any other frame information.

Yeah, I was pretty impressed when I saw that, that how integrate, how much I felt from both directions, kind of natural and not just bolted on. Another step is for code, for source generation. This feels like a visual studio, twenty whatever year.

The last release of visual CEO was where was the number twenty two? okay. So the reason I bring this up is a lot of people here a visual studio, especially in the python space, I think in VS code.

by sometimes open in VS code, I see you have likely to install the shop extension to get a Better developer experience on that. But what we have to do is because because the the source generation is actually running as they step with inside a shop compile pipeline, we needed to know how to find these files and we can to tell with the idea of just do like just do a directory scan.

everything that ends in that P Y. Will go and after .

that's not particularly efficient. And also like if you've just got modules that are using other modules that you want, how do you craft the surface area in python that you actually want to expose rather than just expose everything? So we are inside of the C.

S. Prod file. So this is the the father is the instructions on how to compile a more complex and application. We can then and add notes to that export structure to say, like here is a python file is and that for the sea shop compiled to pick up so that when the source genre hits IT IT hits now our steps IT goes OK.

I'm going to find all these additional files that you've said I need to be aware of and then I can read them in. And it's actually got some optimization for how like internally will read that file in rather than then having to do file I O or anything like that. So it's just it's more physically aware of them and that IT will make those available so that they can then gender out the the curry really you like I said that I was having to try and guess what pothon files you want to have. They have to be in a special holder or they had to be named a certain way or we just directory can everything and we'd dot you like.

yeah you wanna really careful about which ones you're trying to do this to because they require type notation and you don't want to just expose all the guts of some bit of code you want. May be talking about the surface area right? As that that people you have these protected private internal, protected internal.

So Anthony is the last step to instantiated the python environment. This is feels almost like, uh, an import statement. Then you use a named space style. Later you wanted .

talk about that, like to embed python into the net process. You need to create python environment. And as a few things we have to know, like where's python installed? What version is that? Do you have a virtual environment? Is IT just like the global environment? Do you have like packages? Is like additional things that we need to know that you would in and you would have to set those elsewhere.

Like if there was a viral environment, you would have activated, you know, if you were running python, then you would have set which version that wasn't where IT is installed. So you've in the net, you haven't done those steps. So we need you to tell us, like, which version python do you want do you to create, and where is IT? And we created a hope and code to help you find that. And A D V team, I have been solving a similar problem with, like spitting up python and running python. So I us to see like how they tackled IT and we might try and look at bringing U V N as one of the mechanisms for finding Price and for you because it's such a complex yes.

I wanted to talk you about is you've got to a have hythloday, so i've got ta get the dependencies .

and we're not going to store time for you.

Yeah but with U V V V give IT a piton version IT confection and just run IT in its cash and maybe you could even that as a resource compiled time, there's interesting possibilities there .

on viral environment support uses either conor pip as the package manager depending on what you want. And you can say his my requirements file and IT will garden and pip install them as well like I can do those things for you because we kind of be stemming that yeah, you've developing this locally and you're running a locally, but it's gonna deployed somewhere else.

So you don't you don't want have to like log into the production service and like create viral environment on the common line stuff like that, like with hopefully fifteen and twenty years past people do like. So you should be have to programmatically say this project needs a virtual environment, and IT needs these packages and stored and needs this version of python. And then you can either use the version of python that comes with your unit contained up, for example, or you can pull in one from somewhere on that.

You can actually download IT from new ga as like you can actually add IT as a dependency in. And that's a really simple solution for windows development for mac waz. If you've install IT from python to org, like using the official or installed you you just tell IT that you just say use like the mo s python and give IT the version number.

And then for linux, s is a lot more complicated because there's no standards. So yeah, we've kind of support every which way of finding IT on on linux. Once she've kind of put the python environment into the process, they should only be one of them.

Python itself does not want you to ever spin up more than one part in environment in the same like Operating process, you will have all sorts of problems if you if you do that, so is basically put in as a singleton. And then you import a module by creating instance of IT. And then on that module, you can then call methods which other functions in the module.

So like if you had a called demo D P P Y, then in your environment you'd see like a factory for like effectively like import demo and get back in of IT. And in on that instance, you to have methods, which other functions in the example you've got in a screen, this is function called hello world, the texas arguments in a, in a file called demo or P Y. So you just say environment demo, like make a demo instance.

And then on the instance you can call dot hello world with the argument and they give you back to the results. So there's no really in a few lines of code like four, five, you can spend up environment in A A function. So we really try to minimia place code by and then following the patterns at best for both platforms.

So on the places you could go with this, this cool. For example, if you, if you got a single in, which is great for envision, but with a service interpreters, you get A D create these in parallel. One thing that stands out is maybe a little not expected in this is like you have module equals enva demo, right, where you're crying this. But in python that this thing is called demo, would you be a pattern to maybe named the variable .

to same as the mode name that's probably from were not dogs, but yeah, what we're probably lacking from because this is i've been dogs and just mark down code is that so when so we will get back our ipad environment from our dependent injection in china and then things like the the demo method is exposed, they would then return an I I demo. I think IT is or and not is then the representation of the that demode O P Y five is important to yeah yeah. So then then always methods then export on that a moderation. If I talking about this is help looking at IT from python, like, okay, the module is that python file and that's what I get back when I say, like, I I I want an instance of the in the facing to that file.

Yeah, he does. Honestly, IT doesn't matter how I feel about this as a python person, IT feels about IT matters. How does that people feel about this when they look because that this is a sharp good, that's their code. Their world is just so happens that see snakes make make really nice and possible to talk to the python roth. yeah.

So one thing I wanted maybe could spend A A couple minutes talking about this, as I saw an example where you you showed integrating open tim ely and asp on net and post gress and all sorts of stuff together, not just a you know hello world sort of example, which shows some really cool stuff, but maybe doesn't convince people that they can do cool stuff. With that you won. Just maybe you talk us through some of the moving parts that they give people of a more realistic example.

yeah. So we have another demo, which is a web up medicine in the project. This is kind of what we thought was a more realistic integration example. So you've got a web out britain in unh. Well, the front of in just the the A P I is written dot net. You're using posters, grosses the database and you've got like a feature that you want to add to your APP where you want to use some pothon light, recent, some path code. And we were looking at whether everyone loves a, where the M O just .

real enough come, it's just real .

enough the way the demo is like the onic dont demo of when you do find new API projects, like here is a weather up. So I from a di, very much .

at home, creating the one .

that people would expect. Now on the down that side, we also said, right, and let's be realistic about what people would want to do, because like modern applications, you would write the A P I in a pit on that, and you, you know have A P I N points that you would and route can figure those, and you would have a probably relational database like post graal server or or something as the the storage environment.

And then now we support open telemetry ary in platform quarter, by which I can probably explain in in a second. So open telemetry is is a news standard for i'm like tracing and instrumentation and observably generally of of application britain in any language, I mean the net you can have. So for this APP, for example, like you know, somebody makes a request to the web APP and you see the request come in and all timing and IT doesn't log records in the or of anything crashes or like it's all associated with the same. Now if we're going to spin up and run a python function and in python, you do import logging, you know logging logging info or logging or era, like where does that go? Because you just looked something in python does disappear .

into the air is i'd like because into the .

the terminal output of the container is not easy to manage. And we're coming at this the angle like people write big enterprise apps in the net. So like they would expect that they do logging and it's all on the same system.

So this demo basically says, okay, one of the A P I M points in on dont IT runs a function in python, which spins up IT, uses pandas and as a data frame to look at the weather or IT calculate some averages and shifting means and stuff like that to give you back like a prediction for what the weather going to be like this week and we just use see our towards the data because it's easy because it's I was gonna in and then it's not actually was that IT doesn't no. So everyone's aam python. And in the python code, we use open telemetry ary to show the trace of IT talking to the database, and we we use python logging on that log information.

On the trace information all gets put into the same trace. So from dot net, from start to finish, you see the request coming. On the front end, you see the dot net code executing, you see the python de, and then if pison crashes or the logger entries going, they're all in the same environment.

So in like an open to elementary dashboard, you see the whole trace across both do not empathy or in the same environment. And you can just see, is logging, as you would Normally and does not aspire, comes with a like a free open telemetry dash water. You can run locally like a doctor container. Really nice to save you of spinning up yager or something.

Yeah, this is really cool because you've got an a speed on that API. But then when it's called IT uses one of these code gensec snakes, python and integrations. But a lot of stuff just flow us through, like the post gress connections string flow us through the open kilometres context, through the logging go off talks, the python, trace back.

And your example is just your special connections and stuff. But psycho P, G, two. But you could use pretty much any R, M, or anything, right? There's no broken .

strength there. Yes, you are trick pandas and psychopath is is anyone who's deployed that to production in contain is or in different environments now is is hard because like is that which binary do use? And like importing IT hard and you know you need lip q and like is is a bit of a night night.

So IT is demonstrate that the like this is this is a real thing you can do. And IT shows something which is practical in python and something which is practical and dont net, which is like a proper APP with tracing and logging. And all the you know all this kind teaches and on the pi inside, we're using pandas, which you know and we using no pie, which is the same extension.

We're using psychopath, which is like depends on a home bunch of like that party libraries as well and springing IT altogether. And IT just looks and feel seem as like you see the demo working and you would never know that the the weather prediction comes from python or even that is running in this way. Like IT just looks like he is implemented like is part of the same P M.

function. And one of the libraries and off IT goes, but are all connected together like this. I like either we're about at the time, but I either we take this, that was a web example. What about something like W, P, F, or mari, or any other that top APP .

possible? Yes, so because so now because this that the the modern U I firework, you give a really.

really quick definition to listeners. You probability of that as .

part of a way it's kind of the next evolution of of zamora, which people with of the way that you do cross platform you eyes i'm definite not the expert on male, but the idea is that you can have something that has been more portable from um a application experience where you want to that target like a mobile application like an android on I O S or run as a as a this top application windows like that.

So just like a level possibility struction between U I concepts. But fundamentally that's just running donut behind the scenes. So there is no reason why we couldn't also drop in A A bit of see snakes in there and have that year work with some kind of back in service are you could use this and and this is something that anything I ve talked about, you trying to get to them of being like a local model running with you.

You pull a model of of hugger face or something that just runs up locally to do some kind of A I influence or whatever the case may be for the scenario you want to tackle. But that way you don't have to be calling out to an extension service. Now while sure running like a local model is doable, we don't IT.

IT is just way easier to do that with patent. And all the examples you would find, I am going to be written in path first and for most, so trying to then go OK. Well, so this is do well in athons.

So what's the sea shop equivalent? How do I find the equivalent? Don't IT while I read on you here and and so on.

So far, no, like with snakes, you could exposed that model, create some surface areas in just python ap eyes. That is then going to be control by a dott application. So he has so you can definitely do not until entity, you can do IT on mobile.

I think that's probably stretching in a bit far just because of the way that we have to bug trap python inside of the a, and you probably be limited from a mobile O S run time to be able to doing that kind of boot shopping. But at the very least like a test stop application or if you want to, to tackle with like across platform U I framework like available ia. So avona is a cross platform dotted U.

I framework that means you could build for linux and windows and macos at the same time. You could then run, see snakes under the hood of that to then have that they still experience. That is leverage python for a way, a python, he's going to be Better suit to attacked a problem, don't have the a different time and set of problems is Better suit for but just inside this application way, you would have no idea as the end user you like. That's what's actually happening.

Very compelling, very compelling. Our infinity. Let's close out with one final thing. I think this is import, and we haven't really touched on IT yet.

You guys started the conversation by saying this really report for data science, right? Da science use cases. And I made a joke about copying a million items to add one to the list. But what about sharing data like num pie across different on the dot net side and the right side?

So on, we have tackled that specifically so that the buffer protocol is a pet from a few years ago. It's kind of an A P I for the internals of something like an dara in numba. No point. And the buffer protocol wish that we will support that. So if your function returns a buffer, which is a type, then we can access the internals of that buffer directly without having to convert the data. What this means in reality is that if your function returns a empire like an array, and the internal values in the end array are a specific type, like a dad tupid integram, you can access them as thirty two in internet without having to a copy data.

That's also you right right inside of the the india age. Just grab .

its actual memory and just say, got IT, let's go. And then which is you right? The data as well. So you can do us and you can actually just modify the values inside the empire directly from the net with I hadn't to copy back and forwards. And we support pretty much every, like a low level types that none of supports.

And that means you don't have to do a dot to list on the empire, right? Because like that is really expensive and also don't networks in a similar low level numeric types that none pie does know. I will work with thirty to be int and and thirty two or sixty four bit float, for example.

So like if you're storing a two materials in thirty two bit floats in no pie and then you wants to send that to to donate to, I don't know, like pick out some special values or like dim calculations in in donut, then you can just shock that back as a buffer. And there's information in the dark about how to do this. And then you can just access the values directly at a memory. So it's superfast. And if you even want to like phila bus with values from internet, then you can do that using the write table spam.

Pretty awesome. So you that really is leaning into the data science benefits here as well .

as in nothing ever goes wrong if you're sharing memory across one times for different programing languages.

I see others could .

go a very fine and that first make sure we supported really only span yeah right.

okay. No, I think it's great. I think it's great. And the U I. Stuff with avalon, ia is something i'm in a look at that just really called to play with.

So right, final thought, you guys, people out there going, oh my gosh. H, I cannot believe we can finally do this with our doctor. Open log in some other things, should they is IT ready, somebody where to take this build? The thing is that a good idea?

Should they wait, dot, give you to the anything will possible changes at the number of the A P, I. Based on feedback, we try to keep stable as possible right now. So we're making changes that compatible. And yeah, we just want to see more use cases and people get more feedback. So yeah, please pull IT, grab the saplings in them up, do something, tell us what you made and and give us any feedback.

congrats. That looks really, really nice. Def definitely looks promising, and I think IT solves a lot of problems that the other attempts had.

So this has been another explode of talk python to me. Thank you to our sponsors. Be sure to check out the offering. IT really helps support the show.

This episode is sponsored by posit connect from the makers of shiny share and deploy all of your data projects that you're creating using by thon streamer dash shiny fast, fast P I flat quarter reports, dashboards and A P S. Posit connect supports all of them. Try posit connect for free by going to talk by thon that a firm flash posit P O S I T and this episode brought you buy blue host.

Do you need a website? Best get blue host, their A I build your world press site in minutes. And there are built in tools.

Optimize your growth. Don't wait, visit hot python dot FM slash balu host that gets started on level up your python. We have one of the largest catalogues of python video courses over our talk python.

Our content ranges from true beginners to deeply advanced topics like memory and asic. And best of all, there's not a subscription insight check IT out for yourself at training that tuck by thon out of them. Be sure to subscribe to the show, open your favorite podcast APP and search for python.

We should be right at the top. You can also find the itunes feed that slash itunes, the google play feed that slash play, and the direct R S feet that slash is is on top by thon out of him. Word live streaming most of our recordings.

And these days, if you want to be part of the show and have your comments featured on the air, be sure to subscribe to our youtube channel at talk by and a film slash youtube. This is your host, Michael knee. Thanks so much for listening. I really appreciate IT. Now get out there and write some python code.