On this page On this page
Episode 10 – zerocopy with Joshua Liebow-Feeser.
A conversation with Joshua Liebow-Feeser, the developer behind netstack3 and the creator of zerocopy, a crate that makes zero-cost memory manipulation effortless. Originally built for Fuchsia, zerocopy is now used by over 400 crates on crates.io, with nearly 300 million downloads.
We explore the origins of Fuchsia, the creation and purpose of zerocopy, how it works, and why you might want to use it. And of course, we get to know Joshua and his journey so far.
If you like this podcast you might also like our modular network framework in Rust: https://ramaproxy.org
00:00 00:39 01:34 11:41 17:21 23:03 42:57 53:10 55:11 59:23 01:04:20 01:11:13 01:16:50 01:32:13 01:42:42
Music for this episode was composed by Dj Mailbox. Listen to his music at https://on.soundcloud.com/4MRyPSNj8FZoVGpytj
Elizabeth (Plabayo)
0:14 | 🔗
This is netstack.fm, your weekly podcast about networking, Rust, and everything in between. You are listening to episode 10, recorded on the 14th of October, 2025, where Glen has a conversation with Joshua, a software engineer at Google, where he drives adoption of zero copy and similar design patterns forward. Hello, welcome to netstack.fm. This week I will have a conversation with Joshua. He's one of the original developers behind netstack 3 and he's also the creator of ZeroCopy, which is a crate that makes zero cost memory manipulation effortless. It was originally built for Fushia, but ZeroCopy is now used by over 400 crates on crates.io with nearly 300 million downloads. So welcome, Joshua Yeah, thanks so much for having me. I'm really looking forward to this. Yes, I'm very happy with this conversation because we talked in episode 8 with your colleague Bruno a bit about netstack3 and he mentioned a bit about Zero Copy. He also mentioned a lot about netstack3 but he was less familiar with the origins of it. So before we dive into that, could you tell me a bit more about your own origins, Joshua? Sure, So, yeah, I got into the software world in a way that lot of people do, I think, which is I was originally interested in middle school, so I was maybe 12, 13, in just making my computer do things that it did not want to do. And I ended up sort of through requirement, Necessity is the mother of invention. you know, having to like Google, how do I do this thing? And then, you know, you'd find a blog post that says, you type this into the command line and I wouldn't really understand what I was doing, but, you know, and then it would break and whatever. And so I sort of got into computers that way, just sort of wanting to make my computers do, you know, like I got into, into let's, say the IT side of things before I got into software, like into programming. and then I took a, software engineering class, like programming class. in my senior year of high school and just fell in love with it. I mean, for all of the reasons that all of us who are listening to this podcast, I'm sure can empathize with. So I ended up majoring in computer science in college and then did a follow on masters. And throughout that experience, I was really interested in both like the math side of things. I was never like amazing at math, but I was pretty decent at it. I like the sort of computer sciency, the discrete math and proofs and models of computation and Turing machines and stuff like that. But I also really, really enjoyed the system side. And obviously that's sort of where I ended up. And as we'll talk about in this episode, there's sort of a through line there of figuring out how you can make systems feel like math. I think that's sort of a theme in my career is really liking the nitty gritty details. but being bothered by how ugly they are and sort of feeling like, wouldn't it be cool if the networking code that you write could feel as clean and pretty as your CS 101 binary tree implementation or something like that. So that was my history. I focused earlier in my career on security for the most part. So my first few jobs. I was at Cloudflare working on their security team for a bit. And then I moved to Google working on the Fuchsia operating system. And I was on the security team there for a while. And actually, NetStack 3, which as you said, we'll talk about in a second, was just a side project for a while while I was on the security team. And then I worked on NetStack 3 full-time for about four years. And then now I'm back on the security team and focusing more on. the programming language security side of things. you know, have, there's an old world of, you know, being interested in cryptography and, you know, more hardcore sorts of stuff. I mean, I had a, you know, a stint at Cloudflare where I was even like mostly working with the lawyers on like, you know, corporate policy, you know, how many days does a message get to exist in our chat system before it's automatically... removed as part of our data retention policy. So I had a previous life interested more in the sort of more real world, let's say, side of security, but those days are maybe behind me. We'll see. But that's not what I work on right now. Yeah, anyway, there's always a lot of connection with networking and systems and the real world. So, I mean, it's very cool. Do you feel that because not everybody was a computer science or a lot of people never even studied computer science, but nevertheless developed systems. Do you think it might help them if they know certain topics of mathematics, like, for example, if they would be better at discrete mathematics, would it help them? Yeah, that's a really good question. I have seen this both ways. I tend to be by like bias, let's say, like my personal intuition, I tend to push more for the theoretical side of things. It's not that I ever actually, know, that your average systems engineer needs to do theory on a day-to-day basis, right? I very rarely, even in my extremely weird niche job that we'll talk about later, I don't do proofs that often. I still, I do them occasionally. The vast majority of software engineers and even of systems engineers are never going to be doing like abstract math. It's not, the point is not that it's actually a job skill in the way that like learning calculus is a job skill for an engineer or something like that, right? For like a mechanical engineer. Rather it's about training yourself to think precisely. You know, thinking precisely is not something humans do naturally. It's not something that we're born to be very good at. And so you really have to train that intuition. And I happen to think that formal math training is one of the fastest ways of practicing precise thinking. So the recommendation that I usually give to people who are early in their CS careers, they're doing a boot camp or they're in college or something like that. is at least take a discrete math class, learn to do proofs, and take it really seriously. It's not that you're going to be using induction or set theory or something like that in your actual job. You probably won't. But the ability to think precisely is one of the characteristics that separates the great systems engineers from the good systems engineers. ⁓ basically don't know anyone who is in charge of large scale systems or really, really challenging, really advanced systems who doesn't have that ability, whether they learned it formally or not. I know a lot of people who are extremely good at software engineering who do not have undergraduate degrees, but all of them have the ability to think precisely and clearly. And all of them had to learn that at some point because it's not something that comes naturally. So however you get that, Practice thinking clearly and precisely. Okay, that's a very precise answer, thank you very much. so Rust helps us a lot because it has a very rich type system. It also has a borrowchecker and all kind of nice tools, but it doesn't help with all kind of logic bugs which like your system logic. Of course it couldn't. then next to that, sometimes you need to do unsafe actions. Later we will talk about zero copy, which claims it does the unsafe for you so that we don't have to do it. And we will also talk a bit about how you can apply this methodology to your own projects or your own libraries. Now as part of proving that you are safely doing unsafe, you make use of formal verification tools like Kani. does it help there to know the mathematics or is it like you could also once you have a kind of like mindset and thinking that's enough because it's really just about like how much mathematics is actually involved in these kind of systems because I only know about coq like c o q I don't know much about Kani like how much related and how does the mathematics help there Yeah, the answer in terms of something like Kani is that it probably helps to have a little bit of sense of the math, but it is not necessary. Now, the people who are building Kani, that's a super hardcore theory heavy. I mean, they're basically building on top of a theorem prover. I don't remember which theorem prover they're building on top of, but they're They're building on top of like a SAT solver or an SMT solver or something like that for, so apologies for the word soup there. But basically they're building on top of tools that are very math heavy. And so if you're developing the internals of Kani, you probably need to have a pretty solid grasp of the theory there. To build on top of it, to use Kani, I have not found it to be super necessary. But again, You don't need to know the theory, you just need to be able to think precisely. So an example of this is, ⁓ for the most part, and the Kani developers will yell at me for ⁓ oversimplifying, but a very rough overview of Kani is, or an example is that you give it a program like you would write a test, right? So if you think about, I have a function and I want to confirm that it is correct, I'm going to write a test and I'll provide a few different example inputs and I'll make sure that I get the output that I wanted. Kani basically allows you to do that for every possible input. instead of testing using some list of a few inputs, or maybe you fuzz, where you randomly generate inputs, what Kani does is it actually basically constructs a mathematical proof for you that says for all possible inputs, imagine that the abstract space of every possible input this function get, can I, prove to myself that the output will be the thing that I expect? Where what that really means is like it has a list of, know, that your code will never panic and it will never exhibit undefined behavior. So it has a few specific things that it means when it says correct. But basically that's the idea. It's thinking very precisely. So again, you don't need to actually necessarily understand the theory. but it's useful to be able to understand and sort of work with in your head a statement like, for all possible inputs, the program behaves correctly, right? It's a very sort of discrete math proof sort of statement. So tools like that are, they're super helpful. You don't necessarily need to understand the rigor underneath them, but it does help to be able to think precisely in order to know what it is that they are giving you, like the power that they're giving you. Yeah, I understand. Thank you very much. Now, we talked a bit about your background and I would like to shift our attention a bit to the start of Netstack 3. You discussed or you said before that it got started as a side project. Is this one of those things that you're allowed to work on some personal interests within Google? Is that how it got started? Yeah, so basically the history here is I joined the Fuchsia team in April of 2018. And very soon after joining, it was a very sort of like researchy project, right? So at that point in its history, there were a lot of people who were thinking very deep philosophical thoughts about how to design an operating system. The existing networking stack at the time was this thing called NetStack 2. I won't go too much into the history here because Bruno is in your previous episode, I think it's episode eight. Bruno already went into a lot of the background here for NETstack 2, but basically it was a networking stack that had been not forked is the wrong term, but like borrowed from another team at Google. It's not forked because they were still, you know, they were contributing upstream patches and all that. It was written in Go. And there was a desire to move to a new networking stack that was (A) written in a different language. So there was a lot of rust in the fuchsia operating system at the time. And so the thinking was, if we do this in rust, then it'll be better from a performance standpoint. It will be more ⁓ predictable, right? Garbage collected languages tend to be unpredictable in terms of their performance. They tend to like introduce latency spikes at different points. ⁓ It'll be better on memory usage because you can more tightly control when you allocate on the heap as opposed to the stack and how eagerly resources are freed. So all of those things were in the air. People were thinking about this, but no one was taking the idea seriously or no one was actively working on it. Basically, this is so long as I have the platform of a podcast, I'll use it as a bit of soapbox. This was a classic example of, I'm going to say that the phrase is like ask for forgiveness, not permission. That's a big glib. It wasn't like I went off and did something that people didn't want me to do. But I was this junior engineer, right? was two maybe, yeah, I guess two years into my career at this point. It was sort of absurd for me to claim I'm going to go build this networking stack. But I did it. I just went to one of the people on the networking team and I said, hey, I want to do a networking stack from scratch in Rust. And this guy, Tim, Tim is still on the team. Tim's a great guy. Tim, to his credit, was like, sure, kid, go ahead. Like, you know, like, I support you. I think this is going to be awesome. Let's see what happens. And just to give a sense of the hubris, when Tim first asked me, he said, OK, like, how long do you think it's going to take us to get to like TCP? My previous networking experience at this point, I had worked like, you know, on networking software in general, but I had not worked in the networking stack itself except for an academic context. So I had implemented the TCP IP. I had implemented a networking stack as part of a course in college, and then I actually TA'd that course, so I implemented it again. So I had done it before, but in an academic context. And those of you who are professional software engineers will know those are very, different things. And so Tim said, when do you think you're going to get TCP? And I said, ah, three months from now maybe. I think it ended up taking us like four years to even start working on TCP. So it was definitely the sort of thing where a lot of it was just me being like just sort of naive, right? Being like, like this will be easy. Well, you know, I've done a net stack twice before, right? Like how hard could it be? But again, to Tim's credit, he was willing to sort of, you know, take a chance and, you know, as we now know, net stack three ended up being very successful. And so, you it was definitely the right call, I think, to invest in it. ⁓ So, yeah, to get back to your specific question about the history, it was a side project for me for about a year. So I was working on it, you know, at first it was literally a 20 % project, which is like a formal Google thing where you can, you as you said, that you can work on a project of your own interest. And then... It switched to being sort more of like a 50 % project where my boss on the security team was letting me work a little bit more on it. We ended up having external contributors during that year. So that was how Bruno came on as one of the people who was working on the project within its first, I don't know, maybe six months, nine months, something like that. We had a few other people who were interested in it. I was very lucky that ⁓ a netstack is a very sort of like... you know, sexy project to work on, right? And so we had a lot of interest from other people at the company who were really excited to donate their time to it. But that was sort of the history. It was just sort of a scrappy, like, know, us and our free time sort of thing for about the first year. And then within a year, it became very clear that it was not going to be possible for me to give it the amount of attention that it deserved as just sort of this like part-time thing. So I actually switched in early 2019. I switched to working on the networking team. So I actually moved bosses and all that jazz and then worked on it for, I guess it would have been another three years. So think I worked on it in total for about four years. And yeah, so I was working on it for that time. And then when I moved back to the security team, which is where I am now, and back to working on Zero Copy, which we'll talk about later, Bruno took over leadership from me at that point. And Bruno has been the tech lead of that project since then. And so, you know, all of the cool shit that they've done over the past few years is, you know, credit to him and the current team. And they've done some very, very, very impressive things, which I'm sure we'll talk about. So he deserves a huge amount of credit for shepherding that project to where it is now. And we've just had a really, really good, really talented set of engineers who have worked on it. But anyway, yeah, so that was sort of the history. As Bruno mentioned, it's now at the point where were mostly transitioned, Fuchsia is mostly transitioned from netstack2 to netstack3 I don't know that, again, I'm not on the team anymore, so I don't know the exact details. I don't keep like, you know, super, super good tabs on it. But I think they're a good chunk of the way through that transition is my rough understanding. Yeah, that's also what I got more or less. Now, in your own words, would you be able to tell me like why they believed in it so much that they allowed this project to be worked on more and more because, okay, first it was just a little side project and Tim believed in you so far, I get it. But at some point they do need to see something that they think like, okay, okay, this is definitely even though it's like a kid doing it, like clearly this is something like Would you be able to know why that was that they certainly believed in it so much? So that's a really good question. The honest answer is I don't know. I can speak to the virtues of the project, and I'm happy to talk about that. What was their actual calculus? Impossible to know what's in somebody else's head. You should have Tim on the podcast. He would be a good guest. I think, I don't know, do you feel like this is maybe a good opportunity to talk about some of the project we were talking beforehand about? you know, some of the interesting things we did. I think that's maybe a way of answering that question without actually speculating about what was in Tim's head at the time. Maybe that is a thing we should transition to. Okay. Yeah, exactly. Again, you should have Tim on. He's cool guy. So, yeah, basically, I mentioned before that a theme in my career has been really enjoying system software, but being Yeah, yeah, that's fair enough. We definitely don't want to start speculating. bothered by how messy it is. And I think, again, this dovetails really nicely with the thing we were talking about before about thinking precisely. One of the things that I have always really valued, and this sort of comes from the security world, I think, is that in the security world, especially in math and when you're doing cryptography, for example, right, the very sort of like mathy side of security, you are taught to never, never be confident that something is the case unless you can completely prove formally, rigorously that it is actually the case. Right? So, you know, the history of cryptography is littered with, I think the Bruce Schneier quote is, any person is smart enough to construct a system that is clever enough that they are not smart enough to figure out how to break. So it's very tempting to feel like you have designed something that is perfectly secure that is not actually secure. And it contains, you know, vulnerability that you didn't think of. And so you were constantly trained to be adversarial towards your own designs and to really understand what it feels like, the sort of emotional, psychological feeling of being certain and to distinguish from this feels basically true to no, I am I am 100 % confident that this is true. And so that cryptographic experience, like learning cryptography and taking cryptography classes and working through exercises and things like that, really gave me this sort of appreciation for the difference between I understand how this works and I really understand how this works. I think that was one of the really useful parts of my education. for the work that I do now was being able to distinguish between I understand how this works and I really understand how this works. And so, as I said, the theme in my career has been trying to marry these two things, right? Enjoying systems programming, but wanting to get back that cryptographic type of feeling of, I really understand how this system works. And systems programming in general has historically encouraged you to live more in the former domain. It's just often not possible to so completely understand the details of the system that you have that level of certainty about how it behaves. And so maybe there are some super hardcore kernel hacker, blah, blah, people who really, really understand the system. But in general, there's a lot of sort of like, look, you understand it as well as you can, and then you make progress. There's this sort of pragmatic thing. And so when I started working on netstack3 I wanted to have that. feeling, right? I wanted to be able to say that I actually understood how the system worked. And the thing that I love about Rust is that it basically turns what normally is a systems programming type of problem, where you just kind of have to like vaguely understand what's happening and it's going to give you access to lots of low level system details and just sort of, you know, in a C or C++ style, trust that you're holding it correctly. it basically in the type system encourages you to think more precisely and say, no, I need you to tell me exactly the behavior of this program, right? If the lifetimes don't match up, I'm not going to compile. If the ownership isn't perfect, I'm not going to compile, right? It's forcing you to think very precisely about the behavior of your program and encode that behavior in a way that is legible to the compiler. And so that is a lot of philosophy. But there's a question then of how you actually turn that into practice. And so in netstack 3, what that looked like was basically saying every time there is some form of correctness, some sort of property that we might want the netstack to have, we're going to try to figure out how to actually represent that as a first class abstraction with its own types and its own representation. And then we sort of push that very far. One of the things that's really nice about working in the networking world, and I think I've been in some sense spoiled by this, is it is completely unambiguous what a correct program looks like. If you were building a video game, there is no sense in which you have built the correct or incorrect video game, right? Maybe you've built a better video game or a worse video game or something like that. There's no definition of correctness. In the networking world, the definition of correctness is, did you implement the RFC correct? And so artists talk all the time about how requirements, restrictions can actually be very freeing because they sort of free you from the paralysis of choice. The same is true in the software world. Having a very precise specification of what you're supposed to be building is very freeing because it lets you innovate not on what you're building, but how you're building it. And that was sort of one of the things that we experienced in NETSTACK 3 was... Having such a precise constraint about what we were supposed to build, let us focus all of our creative energies on the software engineering aspect of things instead. And that was really sort of how I conceived of NetStack 3. thought of it as this, like the main challenge was the software engineering, right? How do you, if you've taken a networking class, you may have seen the five layer model or the seven layer OSI model or whatever. and you have this very clean abstraction boundary, The ethernet sits there and doesn't know anything about how IP works and IP sits above ethernet and doesn't know anything about how ethernet works. And TCP sits above IP and doesn't know how IP works. doesn't even know that ethernet exists. That is a complete lie. That is a beautiful, nice, simple picture to provide a mental model that we can teach to students. But if you've worked in the real networking world, you know that we violate those those abstraction boundaries all the time. So there are instances in which TCP actually needs to reach down not just into IP but actually through IP into the boundary layer between IP and Ethernet in order to muck with the ⁓ neighbor table entries in IPv6, if that's a term that the listeners are familiar with. If you're not familiar with it, doesn't matter. The high level point is just that the clean modular picture that you were given of a network stack is a complete lie. These things are complected together to high heaven. And so your natural inclination, if you're going to read all of these RFCs and then sit down and say, how do I actually implement this? You're basically given what is sort of like the RFC equivalent of spaghetti code. And so your natural inclination is to write spaghetti code. the natural most obvious way of implementing a networking stack is just to have every piece of the stack call directly into every other piece of the stack and all be completed together. And there's no way for you to pull apart an individual component in order to reason about its behavior in isolation. And so, as I said, the benefit of working on a net stack is that the requirements are specified for you and so that you can aim your creativity at the software implementation. And what we did in that stack three was we aimed our creativity at the question, how do you recover good software engineering practices and principles out of a world that is just trying so hard to make you implement an ugly spaghetti code mess? How do you basically push back against that and say, no, we're going to do it right. We're going to do it in a way that is testable and modular and easy to reason about and so on and so forth. And so that was the main challenge. And Rust's type system was so, so, so helpful in achieving that goal. Because basically there are so many instances in which one part of the stack needs to reason about what another part of the stack is doing. And if you were writing in a language and you were not, know, a language that either didn't give you these type, the rich type system of Rust, or even if you're writing in Rust, but just not making use of the rich type system, you would be really tempted to just go ahead and do that, right? TCP is going to call directly into NDP. That's just going to happen. And if you want to reason about the behavior of that interaction, you just kind of have to understand how the whole system works. What Rust lets you do instead is say, okay, TCP is going to call into NDP. That is unavoidable. But we can very carefully shape the API in such a way that what it means for that interaction to be correct is completely expressed in the type system. And so if we hold that interaction incorrectly, it will not compile. So yes, there is complexity. These things are calling into one another. That is unavoidable. But all of that complexity is made explicit in the type system. And what that lets me do is anytime I want to change things or experiment with different ways of expressing that complexity, I am sure that if I get it wrong, my program won't compile. And that is so much more pleasant of a developer experience. And it makes building complicated APIs and complicated interactions actually feasible because I can just iterate quickly. I can trust that if I made a mistake, I'll catch it immediately as opposed to the experience that I think a lot of us have had in large code bases where you say, boy, I could do that. but I would have no way of convincing myself that it was correct. And so I'm just not going to do it. There is an engineering virtue in simplicity. There is an engineering virtue in holding yourself back and saying, I am going to sort of exercise humility and not do the crazy optimization because I can't trust that it's correct. What Rust says is, no, go ahead, try it. As long as you encoded it in the type system, I'll tell you whether you got it correct or not. And so you don't, some sense Rust is like a language of hubris, right? It's a language that says, ahead, try your crazy idea. I'll stop you if you got it wrong. And so that was really what we did in Sstack 3 was to say, let's actually go ahead and try all the crazy optimizations, but we'll be very careful to encode correctness in the type system so that we know when we got it wrong. And that is the origin of a lot of our abstractions. It's the origin of zero copy, which we'll talk about later, is that sort of attitude of saying, let's push this as hard as we can. But at every single step, make sure we're doing good software engineering and never getting out over our skis. Hmm, that's very cool. And I totally agree. I mean, my original background is C++ and compared to that developing Rust is a world of difference. Now we are also the developers of Rama. It's like an open source network framework and it's also there where we also really think in these layers and where we like, and it's nice because like you said, when you implement these RFCs, which is a very pleasant experience because you can really, as you say yourself, think about how you implement it. Then it's very nice. And the way we implement this is that we can very abstractly, okay, let's say we implement the SOX 5 RFCs or we implement the HTTP or SOC. Well, there are many HTTP RFCs, but like we can really think about it and implement them in a way that's very abstract. like, let's say SOX 5 needs to work with some kind of byte stream. really matter which byte stream. You don't need to worry whether it's on TCP or not. It also means the way you stack it because are you familiar with the towers system and the tower service stage? have heard of it, I know a very small amount about it, so basically no. Yeah, so to introduce you a bit to it, it's basically a very simplistic trait. which is inspired by original framework by Twitter in back in the days, which called finagle if I'm correct. And it works around one single service trait, which takes an input and needs to produce an output, which can either be like, okay, or like an error. And that's basically it. And that's how you would like in a framework like Rama, like stack your layers because you would basically start with like something which maybe like a TCP server and it needs something that can serve TCP streams. So that could be like some kind of TLS acceptor or it could not be, it could be like a SOX 5 and you kind of like just stack these things on top of each other because they just wrap each other or they stack on top of each other like a turtle, however you want to think about it. And it's very nice and that's one example where I like it a lot and where we do make use very well of a type system already. where I feel that it's much harder sometimes, it's like where you have to work with dynamic states. And I have a feeling that in netstack3 you have a lot of clever tricks to work around this. So examples of this or things like, I don't know, let's say the TCP accepted something and it injects socket information into kind of extensions of your input because we, like I said, the service trait accepts inputs. which could be a request, but could also be like a TCP stream with some extensions, extension or basically just you can inject anything into it. And so in order to access those, you always have to kind of like check is it there or not. And sometimes you might assume, this is always there because let's say I can always assume that it's socket information, but still you have to do an unwrap there or you could just say it's fallible. And then another example of things where you kind of have to Yeah, I don't know, like those are things like all those dynamic states, I find that a lot harder to make very good use of a type system because you don't have static verification of like, yeah, this is always there in the flow. And at the same time, it's also sometimes, example, in our case, we have these stacks and then you can have like a very giant mountain of genetics, which like stack in the stack, like it wraps, it wraps. And so at some point you might just choose to box or something because yeah, it's just easier. But then again, also you erase your type. yeah, like, because I know from like a talk with Bruno that you had some clever tricks. according to Bruno, where example you might change behavior based on whether it's IPv4 or IPv6. And for example, we also deal with IPv4 and IPv6, but in our case, we just use match statements, right? Like we check, okay, it's an IP address. We know that that's the type system says that the type system also allows us to match in an exhaustive way. we do have that and we could even use there, but I wonder what other, do you ever have to work in that stack with dynamic state or do you always get away with like somehow, I don't know, not having to inject such states? Yeah, so I think in general, the philosophy that we tried to push towards was the only things that are going to be dynamic are the things that are actually dynamic. And what I mean by that is like, you know, the protocol defines them to be dynamic, right? So for example, a UDP packet is allowed to have a missing source port if memory serves. I'm pretty sure that's true. There's one of the fields that's allowed to be optional. I think it's the source work. And people ⁓ listening can go confirm that I'm right about that. But there's one of the fields that's allowed to be optional. And so that's an instance of inherent dynamicism. There's just no choice. Your netstack has to be able to handle the case that there's a missing source work. And so in those cases, what we do is we try to encode that in the type system. So we'll say, you know, I think if memory serves the UDP packet parser that we wrote, if you want to extract the source port, the source port is represented as a non-zero U16, and the getter for that field returns an option of a non-zero U16. The caller is forced to deal with that. What do I do if there's no source port here? By contrast, if there are things that are guaranteed to be but, excuse me, guaranteed to be present, and it's just a limitation of the type system, then we try as hard as we can to figure out how to introduce new APIs that actually guarantee the presence of that thing. So I'm trying to think of a good example of this, but. Basically, there are instances where you know that the RFC requires a particular thing. And so in, let's say, like parsing a packet, you might say, if this field is missing, then that's a parse error. So we're never going to parse and produce a parsed representation of this packet where the field is missing. And so that becomes an internal invariant. And then your APIs, to access the fields of the packet can be infallible. are lots of like weird optional, yeah, go ahead. reason why I brought it up is because you were talking about when you're parsing and that's within a single layer and I feel there it's a lot more easier and also in the case of something missing, once you develop long enough in Rust you would quite quickly think about making an option for that, so far so clear. What for me is much harder is like, and I'm sure you have that same problem set in netstack, is where you cross boundaries. because between, let's say, a TCP socket, which you accepted, and your application layers, let's say, HTTP, there are a lot of different layers, and sometimes could be more, sometimes could be less. For example, in like a very extreme case, you could end up with eight layers in between. Right? Because you're dealing with SOX 5, you're dealing with TLS, you're dealing with all kinds of stuff. Maybe also like HA proxy. mean, you can, you know, you can think of a lot of them. And so there lies the problem is because if I would just deal with, okay, my service is accepting a TCP stream. So that's one layer. Then I can guarantee very easily like, yeah, I can, I can get from that my client address or my server address or whatever, or my peer or whatever you call it. But once I go like, okay, now I'm having a service, which is accepting like, let's say an HTTP request. I'm already very, very far in the app structure And at that point, it's a lot harder to still, even though like conceptually, you know for certain that this comes from like a TCP packet. I mean, it somehow has a connection to that, right? Like, okay, somewhere is something a task, ⁓ parsing this and whatever, and dealing with an actual TSP socket. But I find it much harder at that point to, to, without making a huge burden on, on, restrictions on the system to at that point and force like, I always can guarantee that you can get from this, like a socket information, unless I would put that kind of information directly into an HTTP request, but that doesn't seem to really fit there either. So I think that ⁓ a big answer that we have in the netstack3 code base is just that we use types a lot. Like a lot a lot. And so I do think this is one of the limitations. So a point of philosophy here is that the types in general, the concept of using types, compare that, contrast that with like Python or assembly or something like that where you don't have any typing. The concept of introducing a type system tends to be really good at reasoning about properties that are local. As in, you have, what's a good example of this? If you have a function that takes an integer and returns a bool, and if I have an integer, then I can call that function and I get a bool back. It's a lot less good in general at talking about global program properties, right? It's a lot less good at expressing the idea that, you know, the, what's good example of this? you know, an instance we ran into in Sstack 3 is that you have interfaces and then you have sockets that can be bound to those interfaces. Those live in very different parts of the code base. but they're actually related to one another. So in theory, if the interface goes up or down, then, so let's say you have an interface, you have a TCP socket bound to that interface. If the interface goes down, the TCP socket, the correct behavior might be to close the TCP socket. That's a sort of configuration thing. But let's say, just for the sake of argument, that you want to close the TCP socket because the interface went down. How do you express that in the type system? How do you express the idea that this is a TCP socket and that it has an interface, and that the interface that it is on is definitely up because the TCP socket exists. That's actually really hard. And I will say, this is one of the things that netstack 3 ⁓ didn't have a good answer for the entire time that I was there. After I left, this was one of the things that they tackled. I know that Bruno, I think, mentioned in his episode that they ended up with a fancy arc-based system. in order to track resources like this where you need to have multiple references throughout the stack to, for example, a particular interface. I don't know the exact answer of how they deal with this, but it's actually genuinely hard. And I think the Rust type system lets you encode information that is non-local more powerfully than other type systems do, but it is not perfect. And if you wanted to... Like there's, right, the Rust type system is turing complete. In theory, you can do anything you want in the Rust type system. That doesn't mean that it's going to be ergonomic and it doesn't mean that it's good idea. And so in theory, you could imagine structuring a system where somehow you encode the information in the type system of like the liveness of every interface. That's probably going to make your type signatures on every single part of the stack like awful. And so Partially the answer to your question is we didn't have a great answer for this. It was like an open area of research. Again, I think Rust makes it easier to encode non-local information or at least to encode more complex information. And I can talk in a second about an example of that. But yeah, I think it's a really hard problem. And I don't think we had a great answer for it. I think that being able to encode the local information alone is such a win compared to C/C++ that it's still big difference, but that does not mean that the ⁓ effort is done. We still have a long ways to go in figuring out how to develop large system software in Rust in a way that is completely infallible, in a way that if it compiles then it works. If it compiles then it works is true in Rust way more than it is in other languages, but it's not a hundred. Yeah, yeah, for sure. And anyway, in the end, you're forced to deal with the fact that it's not there or there and you can fail it. And in the end, I learned to live with it. Like in the beginning, I tried to do everything with static states and it was, it became after three years of development, such a disaster that recently we switched away from that. And at some locations it makes sense and you can do it. And other layers, most of the layers, you actually just learn to deal with some things are just dynamic. And given the fallible nature of networking, that's fine as long as you do use, like you said, locally, the Rust type system very well to deal with those things and not just like panic or something or assume like you do deal with the possibility, even if you know most likely it's not because you want to make sure that depending how you stack it, works. So, I mean, I learned to live with it, but yeah, sometimes I do wish like a lot more could be done, but yeah, it's a trade off and okay. So we talked a lot about systems already. We talked a bit about designs. We talked about how Rust helped you. We talked about your backgrounds. Now, what is not clear to me and which also allows us to segway into zero copy is that at what point... Zero Copy became a thing, how it was conceptualized and at what point did you start to work on it. Yeah, so zero copy was like a classic necessity as the mother of invention project. We actually started it probably within a month of starting netstack3 So netstack3 dates to maybe May of 2018, if memory serves and zero copy, think I wrote the first lines of code in like June of 2018. So super, super early on. Basically the problem that we were trying to solve was We want to do, so I should clarify here, zero copy as a phrase is the name of this crate. Zero dash copy, which unfortunately is pronounced identically, it refers to a style of parsing. So if you receive a message from a network or you've parsed, you've read in a file format off of the disk or something like that, and you want to parse it, for performance reasons, you would like to not have to allocate more memory elsewhere. You would like to be able to reuse the existing buffer that you have ⁓ and just sort of operate on it in place. That's what zero-copy parsing refers to, is the idea that you're not making copies from the buffer into another location. So we wanted to do that style of parsing, and in C/C++ this would be fairly straightforward, right? You would define a struct whose memory layout happens to be the same as the memory layout of the packet format that you're trying to parse. And then you would take your pointer, right? You would have like a car star or something like that. And you would do a cast into your struct type, right? So let's say you're trying to parse a UDP packet. You might say, okay, I'm going to cast my car star pointer into like a UDP header star or something like that, right? And now I have a pointer that is typed, right? It's a struct pointer. And I can just go do field access, right? So I can take my pointer and I can do, you know, pointer arrow source port or something like that. So that was roughly what we wanted to do in NetStack 3. of course, doing that in Rust is unsafe. And, you know, if you've worked with this stuff, you'll know. And if you haven't worked with it, then you can just take my word for it. It's extremely, extremely subtle. There are many, many, many edge cases that you have to consider. in order to determine whether this is sound at all. And I'm not talking about edge cases at runtime that your code has to handle. I mean, edge cases in order to figure out whether it's even correct to make this program at all. So edge cases in the type system, basically. Yeah, no, I mean, I'm very familiar with it because, mean, originally I come from the game industry and we did a lot of that as well. And it's also probably why in the RFCs, a lot of them, I mean, I feel the layout is often made with the exact structure in mind and some RFCs even explicitly mention the structure. And it does work, except of course, where you deal with flags. mean, if you... Maybe my memory is a bit vague here, but as far as I know, once you deal with flags you do have to at least do some kind of bitwise operation to turn it into some kind of boolean, because there is no way, even in C as far as I know, that you can just turn it directly into a struct, or am I wrong? Yeah, in C I'm not exactly sure. I don't know off the top of my head. Something tells me that there are subfields, yada yada, but I'm speeding out of turn there. In Rust, the answer is no. We don't have any types in Rust that are less than one byte in size, except for types that are zero bytes in size, but that's separate thing. There are crates out there, like the bit flags crate, or bit fields or something like that. that will handle this for you, but it's a little bit higher level than ⁓ we needed because we need to be able to precisely control the memory layout. I may be speaking out of turn there, maybe that is something that they support, but for whatever reason, it wasn't something we ended up reaching for. Luckily, it's not that big of a deal to do the bit manipulation stuff. It's a bit of a pain. write some manual code. Luckily, it's not a memory safety thing. The thing that really mattered for us was the memory safety aspect. The big things that you have to deal with here are you have to make sure you've done bound checks, right? So if I give you eight bytes and you try to cast a pointer to eight bytes, or a slice reference, right? A U8 slice reference that's eight bytes long, and then you convert that into a reference to some struct type that is longer than eight bytes, then that's undefined behavior, right? Because it'll allow you to read off the end into some other garbage memory. The other thing you have to deal with is alignment, right? A lot of types in Rust have some form of alignment requirement where they have to only exist at even byte addresses or byte addresses that are a multiple of four or multiple of eight or something like that. In general, if you have a slice of U8, you don't know where in memory that lives, right? It might live at any arbitrary byte offset. And so even if you've already checked the size, you also have to check the alignment. And so for these reasons, basically what we realized was doing all of this over and over over again, every single time we write a packet parser was going to be just a source of bugs. And it was also just going to be really annoying to deal with. It was like, even if we got it right, it would take a lot of effort every single time that we wanted to get it right. And so what the zero copy library gives you is a type level representation of properties that your type. satisfies. So it's a bunch of traits, basically, and if your type implements one of those traits, that tells us something about the memory layout of your type. So the base traits here we have FromBytes which basically says any bit pattern is valid for this particular type. So any pair of two U8s is going to be a valid instance of a U16, for example. Any bit pattern is okay for U16. This is not true for all types, right? So a boolean, right, a bool in Rust can only be 0 or 1. If you try to interpret the byte 2 as a bool in Rust, that's undefined behavior in your program that's going to be arbitrarily wrong. So basically that's the idea, is we have these traits in zero copy that encode the idea of... whether your types have certain properties. And then we have a bunch of functions that will do these conversions for you, but uses those trait bounds to force you, the user, to prove that this is actually going to be an okay conversion to do. So for example, if you want to convert a reference to a slice of U8s into a reference to your type, you're going to have to prove that your type is from bytes, right? That any possible bit pattern is acceptable for your type. and you're going to have to prove that it is unaligned or that has no alignment requirement. If you do those things, then we will provide to you an API that will at runtime check the size. We'll make sure that you've actually given us the correct number of bytes. And then if the size check passes, then we'll do that reference conversion internally and we'll give you back a reference to your type. So that's the basic premise. It's this very low level little like Swiss Army knife type of thing, right? It's very low level utilities, but the point is that the API is 100 % safe and you can build up much larger abstractions on top of that. And so all of the network, all of the packet parsing in netstack 3 is built on top of zero copy, right? So it's all you give me a buffer, I'm going to convert in place the type of the buffer into a type that represents the memory layout. of the particular packet format that I'm trying to parse. And then I'm just going to go on my way using normal Rust operations. So that's sort of what zero copy does. It has grown a ton over time. Like it was very small in the beginning. Now it's a much, much larger crate. It's the full-time job of two different engineers. So it's grown a lot since then, but that's its origin is that basic operation of just wanting to do zero copy parsing. Yeah, I mean, it's awesome because I feel I was like I was reading through the source code of zero copy earlier this week as I knew we would or actually last week because we're just Tuesday. yeah, sometimes the week boundaries are unclear to me. I have three small kids, so they flow so quickly. So the last couple of days I was reading a bit through it. And yeah, it seems like then you have the zero copy derived crate and there I mean. It seems that you are doing all the hard work that otherwise you would have to do manually because you're putting the correct annotations, you're putting the correct operations and checks, and you also make use of the type system to verify statically what you can verify statically. mean... Is that like a fair summary? That it is just instead of doing it like all manually, like now you can just use like your procedural macros and the type system that you get the system of type that you build around it so that you don't have to do all that work yourself each time. Yeah, that's exactly right. Basically, there are two steps here. The second step, I'll say the second one first. The second step is if you have proven that your type satisfies these properties and that that proof is carried by the type system in the form of one of these trait implementations, then we'll provide for you all of these runtime functions that will do various operations that would normally be dangerous. then that raises the question, how do you actually prove that your type satisfies these properties? And that's where the derives come in. Most of these properties are recursive. So basically a struct can be from bytes if all of its fields are from bytes. And so what that means is that it bottoms out at the core primitive types. So we basically just hand wrote in the library. implementations of these traits for all of the built-in types. So, you know, U8, U16, U32, I64, et cetera, et We have implemented the appropriate traits for all of those built-in primitives. And then the custom derive will allow you to automatically derive an implementation of these traits for your type. And that derive will either fail during the processing of the derive itself, or it will emit code that will fail to compile if your type doesn't actually satisfy these requirements. So for example, if you have a struct that has a boolean in it and you run the drive, then the emitted code from the drive will say, sure, your type can be from bytes if boolean is from bytes. But hey, bool is not from bytes, so it fails. And then in addition to that, there are extra constraints that we have where in the derive itself, we can check it. for other conditions and sometimes we fail during that pass as well. But yeah, your high level description is exactly right. The derive lets us determine whether your type satisfies certain properties. And if it does, then we turn those properties into trait implementations and then zero copy itself, the runtime code consumes those traits as a bound in order to prove that we can do certain operations soundly at run. Yeah, very cool. I in a way, like I feel, let's for example, within the framework Rama for now I'm not using zerocopy myself. I feel I could definitely make use of it because for now I'm doing a lot of that work myself. At the same time, I do wonder because like, okay, it's nice to do zero copy and that's possible because you're borrowing from the memory of the slice. or you are just basically viewing it through another lens. Well, I suppose that sometimes it just also just means that you are copying, you are copying, right? It's just on the stack. No, I mean, in the end, it's not like you are literally just, I suppose you just mean you don't make any allocation on the heap, but you are copying it to another, like on your stack. No, am I wrong there? Like if you, for example, would interpret a slice byte as some kind of, I don't know, TCP header or any kind of other protocol. In the end, you are copying it, but like on the stack, right? Yeah, so basically the thing that is getting copied here is the pointer, right, or is the reference. So if you have a TCP packet that you read in as a slice of U8 in a buffer, that is represented as a U8 slice, right? So ampersand, open square bracket, U8, close square bracket. when you do a zero copy conversion, you're not actually moving the bytes that are the referent of that reference. You are leaving those where they are, but the reference itself is going to be a stack variable, right? So that the reference, as in the ampersand thing, is going to live on your stack. What zero copy is letting you do is just change your mind about the type of that reference. So it's going from a slice of U8 reference to a UDP header. reference or a TCP segment header reference or something like that. The bytes themselves that are live somewhere else, maybe they live in somebody else's stack frame, maybe they live on the heap, those stay where they are and they don't get copied around. But the reference to those bytes, that is going to get moved and copied and passed around and all that jazz. And it's just a matter of changing their type systems mind about what type lives on the heap or in somebody else's stack frame or wherever it is that you're pointing to. Does that mean that you have an opaque struct and to get those properties you need to call methods and it's only then that you copy to the stack or how I just want to like figure out like yeah, yeah, yeah. Right, exactly. Yeah, so let's say, I mean, it doesn't have to be opaque necessarily, but the idea would be, let's say that I want to operate on the destination port of a UDP packet. What I'm gonna do is I'm gonna take my slice of bytes, I'm gonna convert it into a reference to a UDP packet header. Let's say that's the name of the type. That type is going to have fields, right? So it'll be a struct that has a source MAC dest MAC length and checksum are the four UDP header fields. When you actually access one of those fields, let's say you do packet.dest MAC and you pull that out, that is in the compiled code that Rust generates, the Rust compiler generates, is going to be copying those two bytes from somewhere in memory into your stack, maybe into a register or maybe your stack, whatever, and then you can operate on it like normal. But basically the idea is it should generate the same machine code as if somebody had passed to your function a reference to a UDP packet header. That's sort of the idea, right? Let's imagine what would have happened if we had been operating on a reference to a UDP packet header this entire time. Ideally, that's gonna be exactly the same machine code that gets generated as if you're using zero copy. ⁓ There is a slight asterisk there, which is that we're always going to have to do a length check at runtime, so it's not going to be identical, identical, but it'll be pretty close. That also probably means that let's say you have a function and you use that address 10 times, I suppose you want to just call that method once and then just always use the final version, right? Because you wouldn't want to do that work 10 times, I suppose, in a single function. Yeah, yeah, yeah. Right, like you wouldn't want to do the length check 10 times. And so yeah, usually you will just do this conversion once and then you'll operate on the typed reference once you have it. You'll just operate on that from then on. Okay, cool. And then a lot of these protocols, also... I mean, it's very nice if everything is like in a fixed location and for the, for the, like the metadata ports and the, and the flags. And so that's usually the case, but then usually you also have like optional parts or, or length, like, like let's say string values or something like that. It could be like a domain name. And then usually they encode like the length in the first two bytes. then maybe the, mean, or they go even fancy to maybe they even do something like where if it was a small value is like one byte if it's a big value, it's three bytes length and then you know the other length amount of bytes is your string value. Does zero copy deal with that kind of stuff too? Yeah, so this has been a huge headache for us as the answer. Basically, zero copy today is capable of supporting any byte pattern that can be represented as a native Rust type. So if you can build a struct or an enum or a union whose memory layout is the same as the memory layout that you're trying to parse, then you're golden and zero copy can help. But there are certain things, there are certain memory layouts that just don't correspond to any native Rust type. So the format that you were just describing, which is called TLV or type length value, is basically a variable length number of objects. And the way that you parse them is you have to of chomp them one at a time, where you look at the header and that tells you how many bytes to eat. And then you eat that number and then you parse to the end. and then you look at the next thing's header and that tells you how many bytes to eat and you iterate that process over and over again. There is no native Rust type whose layout is equivalent to that thing. And so zero copy can't really help you. It could maybe help you parse an individual one of those things, right? So it could maybe help you parse like a single instance of the TLV format, a single record in the TLV sequence of records, but it can't help you parse the whole thing. And so what we did in netstack 3 is we have a crate called ⁓ packet. So generally speaking, there are ⁓ three sort of utility crates here that we talk about a lot. There is net types, which contains a bunch of our low-level networking primitives. This is like IP addresses and MAC addresses and stuff like that. There is packet, which is low-level utilities for packet parsing. So it builds on top of zero copy and gives you a framework for doing packet parsing, but it doesn't actually implement any specific protocol. And then we have packet formats, which is all the packet formats. So TCP, UDP, IPv4, IPv6, MLG, sorry, MLD, IGMP. The list goes on. It's an absolute alphabet soup of different packet formats. So in the packet crate itself, which is this just like framework, we have a set of helpers for parsing repeated records. It turns out that in the networking world, there are many different formats for repeated records. TLV type length value is just one of the instances of that. So TLV style parsing tends to show up in IPv4. It shows up in TCP. Both of those have like optional variable length options fields. IPv6 uses this like crazy thing called extension headers, which can be nested. So you can have multiple different extension headers that follow after the main IPv6 header. And then inside each extension header, you can have variable length options. It's crazy. The amount of code that we had to write to parse IPv6 packet headers is insane. It's like thousands of lines of code because of how complex the format is. So zero copy can only help so far with so much with these things. As I said, it'll help you do like a single struct, a single union, something like that. But it's not going to like, you can't just like magically generate code that does all of that. That'll do the entirety of IPv4 options or IPv6 extension headers. We would love to move in that direction at some point, but it's an open research problem of how you actually represent that. What we do in the packet crate is we have utilities that will at least help you structure your code when you are parsing iterated records. So generally speaking, what this looks like is you give us a slice of bytes, a slice of U8, and then you also have to give us some amount of code that tells us how to parse an individual record. And then out of that, we will give you back an iterator over all of the records. So for really bespoke things like IPv6 extension headers, it's fairly low level. We'll just say, like, listen, if you can parse an individual IPv6 extension header, then we'll let you parse like multiple of them one after another. We'll sort of like... structure will scaffold all of the code around that. For some more common types like TLV, we actually have a native TLV parser in the packet crate, but that is not in zero copy. For those of you who are interested in using these things, zero copy is on crates.io. You can use it right now. Net types, packet, and packet formats are not published on crates.io. It would be super cool to do that at some point, but we're not sure when we will or what it would entail to do that. you are more than welcome to just copy paste the code out of the fuchsia repository. It's all there. It's open source. You can read the license and determine whether the license is acceptable for you. But it's not like on crates.io So it's not super nice and easy to use. But yeah, so all of the record stuff that I was talking about is in packet. So it's in the fuchsia code base. You can go copy paste it, but there's no like crates.io thing you can just go take a dependency on. Yeah, cool, no, I'm definitely going to least study it. It's always interesting to read other people's quotes. It's one of my things I like to do next to reading RFCs. I used to read the newspaper more, but nowadays I'm reading mostly RFCs. Yeah, yeah, you learn a lot from it and nice to read. Anyway, so that's one thing. Okay, cool. So sadly not in zero copy, but at least you solve it in a different way. RFCs are lot less depressing. And it's an open research problem, so the future is bright, who knows. Okay, another issue I have, or not really issue, like problem space I maybe create for myself is this, which deals with the function coloring problem. And for example, Rama framework is heavily async. And sometimes it's not clear, okay, when you want to make async and when not. And so for clarity, a lot of stuff we do is async just because... depending on how you implement a trait, you might anyway want to have some kind of async behavior because I don't know, maybe you need to validate something, but you need to check something like in IO or some kind of database or whatever. having it async first helps a lot there, but it does mean that you have, that you cannot really as easily make use of something like zero copy. would think because like I couldn't just like reference to a slice and then Move it over like you requiring send+sync trait bounds whatever like I couldn't just move it over because it wouldn't work I mean, but I think within netstack3. I don't think you have those issues because it Yeah, actually Do you have those issues there? I suppose maybe maybe you're arcing the bytes there or something and that's how you get away with it I don't know Yeah, so I'll say a few things. First of all, I'm not intimately familiar with these issues. I have not done much async programming myself. So I may be speaking out of turn. My general feeling about powerful frameworks for systems code is I try to avoid them as much as I can unless I really need them. My view on async is that the Rust async solution The whole async system is an engineering marvel that you should try to avoid if you can. I work with Tyler Mandry, who is one of the core designers of the async system. He has done amazing work. The Rust folks have done amazing work on designing async. It is a beauty. But it's really complicated. It is necessarily solving an extremely complicated problem. And it makes it difficult for you to reason about... the behavior of your code. And so you should use it if you need it, right? If you're writing some highly parallel web server and it makes natural sense to use async, like go ahead and use it obviously. We tried to avoid it in that stack three as much as possible because of how tight the control is that we need over the execution model of the code, right? I need to know not just like when the code runs, what does it do? I also need to know when does the code run? And what does the control flow look like? And async makes it difficult to reason about what the control flow is going to be. so Bruno talked about this in the previous episode, which listeners should definitely go listen to. a good, you know, it'll provide a lot of extra context for what we're talking about here. But I'll repeat briefly what he said, which is that 90-ish percent of the code in NSAC 3 by lines of code. is in a crate called the core or a part of the code base called the core. was a single crate at some point. Maybe they split it up since then. But the idea is that philosophically, there's this region of the code base that is completely platform agnostic. So if you go and you read that source code, you won't see anything specific to Fuchsia. And in that code, that's where all of the actual protocol implementations live. That's TCP, IPv4, IPv6, yada yada yada yada. The whole protocol soup lives in there. Inside of that part of the code base, there is no async. It's all just straight line functions. So you call in and the code executes to completion and then it returns. In the course of executing, it might call back out into the external system. have basically there's a dependency injection mechanism that we have so that code that lives outside in what we call the bindings that basically is the thing that binds the core to the specific system. So the bindings is where the fuchsia specific code lives. We have a mechanism for the bindings to call into the core and then a dependency injection mechanism that allows the core to call back out into the bindings. This is what lets you do something like send a UDP packet. You can call from the bindings into the core and say, I would like to send this packet. And then once the core bottoms out at like ethernet and needs to send a message to, you know, an internet driver. it then calls back into the bindings and says, hey, I need to talk to the driver. Here's the ethernet frame that I want to send. So we can call into the core and the core can call back out into the bindings. But that whole system is entirely synchronous. There's no async involved. It's just straight line code. If I wanted to do that call, the UDP example that I provided, the bindings would call into the core and it would just straight line go all the way through. At some point, the core would call back out into the bindings and say, hey, I want to send this ethernet frame to the driver. That call would happen, right? Either in line, the bindings would call, would actually do the communication with the driver, or maybe it would just queue that work for later. And then it would return and the function call would return all the way through the core and back out into the bindings where it was originally initiated. The bindings does have async. For various reasons, it's important to have async there because there's a lot more asynchronous stuff going on. The communication mechanism that Fuchsia uses to communicate between processes is very naturally asynchronous. And so it makes a lot more sense for there to be async code there. So there's async code that is sort of like initiating synchronous calls into the core is basically the way to think about it. But inside of the core, there is no async. And again, just to reiterate, the reason is that we wanted to be able to have a very clear mental model of how control is flowing through the core so that we can reason about performance and like resource liveness, right? When are certain resources live or dead? The possibility for, you know, like deadlocks and stuff like that. And generally speaking, I have found that it is a useful, you know, sort of rule of thumb. to avoid async until you actually need it. Once you need it, Rust async is super cool. And I don't want this to come off as like me denigrating the folks who did async. Async is is absolute magic. It's so, so, so cool, but it is a very sharp tool and you should only use it once you know that you actually need Okay, that's fair enough. But then I do wonder, as you said, the driver or whatever, like the bindings call the core and the core can call back into the bindings. Like how do they then make sure that, for example, that stuff doesn't block everything else? Because then there you deal with, think, is because they give it like an entire thread to do what it wants or how that works. My, okay, so the most accurate answer is I don't know. Most of the stuff was written after I left. My rough understanding from having talked to Bruno since he took over the project was, is that basically if they have a queuing mechanism where they will queue packets or frames to be sent to the driver at some point in the future. Now, naturally a thing you might imagine, and I don't know whether they do this, this is me just speculating. But naturally, a thing you can imagine is you could do a fast path. You can say, if there is room in the buffer that I share with the driver at the point at which the core calls into the bindings and says, please send this frame, maybe they opportunistically, as a fast path, go ahead and write into the buffer. I don't actually know whether they do that. You can imagine them doing that. Certainly, they're not going to block. You're definitely not going to block on that operation. You're going to queue it for later and then immediately return up through the core. Just to give a sense of numbers here, NetStack 3 aims to be a general purpose networking stack. You want it to be able to use on any system. And Fuchsia is a general purpose operating system. You want to be able to use it on any system. And so you want to be able to have this stuff be as performant as possible so that someone in the future, let's say there's some like, I don't know, This is complete speculation. I'm making this up. This is not like real, you know, this is not a real world example. But like, let's say that a decade from now, someone's like, you know what, I want to build like a router, right? I want to like use Fuchsia to build a router. Well, if you're going to do that, if you're going to use Fuchsia to build a router, then you would need to be able to turn around a packet in, I think the numbers are something like a microsecond, right? Like a modern, you know, take your random, you don't know, net gear router that you have in your home networking setup or whatever. you have about a microsecond, right? A millionth of a second to receive a packet, parse it, decide what you're gonna do with it, decide where you're gonna forward it, turn around, serialize the new version of that packet, forward it out. You have about a millionth of a second budget to do that entire thing, right? That entire operation. So the point is that if NetStack 3 has the goal of just being like a general purpose networking stack that people can use for general purpose networking stuff, then you need to be able to like hit those performance goals, right? And so if you have a millionth of a second to turn around a packet in the most extreme case, then you definitely cannot be blocking in the core. And so the idea is that, yeah, you want all of the extra scaffolding around that sort of stuff to happen in the bindings, where you can have more opinions about what sorts of performance trade-offs are acceptable, but you definitely do not want to be blocking the core. And you might say, well, maybe I have this thing like blocks in the core, then that's fine. Like it blocks trying to send to the driver, but that's fine because a different thread could come in and do work at the same time. True, but what if the core has like, you know, locked a mutex in the interim, right? And so in fact, you're blocking like other resources at the same time. So generally speaking, the core tries to be as just like get in and do your work and get out as possible. ⁓ This, by the way, for those of you who are interested in like networking optimization stuff, is generally speaking the direction that high performance networking has gone over the past decade. If you read design docs from the Linux kernel, example, there's a really great text file. I don't know, ⁓ Glen, maybe we can put it in the show notes or something. I'll find the link to it. There's a really great design doc that someone wrote that just describes all of the high performance stuff that the Linux net stack does. And a lot of it basically entails, how can we be as dumb as possible? That's basically the idea, right? Like stop trying to synchronize between different cores and different threads. Stop trying to be fancy about stuff. Just do the dumbest possible, no work at all. Get in, do your work, get out as possible. There are various things that this looks like where you can like pin different sockets to different cores and you can like do, you like. hash-based bucketing so you know a priori which core you're going to send the packet to. There's all sorts of fancy stuff. But all of it boils down to get in, do your work, get out as quickly as possible. And so for that reason, async is a bit antithetical to that because it just robs you of the ability to make your own decisions about what that execution flow is going to look like. Again, this is a very niche. The netstack use case is very niche, especially from a performance standpoint. So there are lots of other cases where async is a very appropriate tool for the job. As I said before, if you're doing a web server and you're just going to have a bunch of people coming in and your web requests are very IO bound, someone makes a request and then you need to ⁓ issue a read to the database and that's going to block for a while and yada, yada, yada, async is the obvious choice there. But if you're doing low level system stuff, it is probably a good idea to think very deeply about the control flow inside of your application, in which case async is something you should think carefully about. I'm not saying don't use it, but just make sure you understand how it's going to affect your control. Yeah, for sure. mean, first of all, it's very exciting to talk about this. Like it always makes me feel very warm. But yeah, it's a... Yeah, Rust is a good... Rust is a warm and fuzzy language, that's why I Yeah, but even the networking world, mean, it's kind of like for me until like eight years ago or seven years ago, networking was always an abstract concept. Like I was using it when I had to and I knew. how to open sockets and I knew how to manipulate them. knew how to write them. I knew a lot about like all even writing our own protocols for the games because that's what they did. I don't know why, but everybody was inventing their protocols. So I have lot of experience with that, but it's only once I really got into, I don't know, security and data extraction and the web world and all those kinds of stuff and then proxies and yeah, again, security in another kind of capacity that I really got into networking world. Yeah, but still, I talk to someone like with your experience working in those kind of systems, I feel like there's still a whole world to explore where I definitely didn't scratch the surface yet. it never ends. mean, that was, you like, despite the fact that, you know, I was able to spend nearly half a decade working on this stuff. In my time there, I never got to touch a lot of the higher performance stuff. never actually, you know, the current synchronization model is basically just mutex based for NetsuDuctory. I was always really excited about some hypothetical future in which we could go like super hardcore and do the core pinning stuff and like, map sockets to different threads and whatever. We actually, wrote up a design doc for that once and basically just, there were so many correctness edge cases that it just, we ended up deciding that it was too complicated to justify it in the short term. ⁓ But man, you can go so deep. never, at least while I was there, we never did any of the ⁓ hardware offload stuff, ⁓ having the hardware do the checksumming for you or like coalesce TCP segments for you. I mean, man, yeah, it's easy to understand how you could make an entire career in this industry and never actually, never get to the bottom. It's very cool. I mean, yeah, I mean, that's why I live. love to learn. I soak it like a sponge. Now I also, we already mentioned it a bit before and it's also written in your README, which by the way, I mean, in one aspect, I love the README of Zero Copy. Like it's very beautiful at the same time and maybe just me, but I feel as an outsider, if I wouldn't have read the actual code, I would find it very difficult to know still how to apply it just for the fact that It doesn't really contain any examples or examples I could easily find like for example one might expect, yeah, this is like a struct and that's coming from this kind of slice and that's how it's and then you see all these concepts coming together. Well, now I feel it's like a laundry list of like concepts and then like It's a lot of work on the reader I find to then put it together. Maybe that's part of it because you need to read and understand that. doing that work, kind of like it's maybe it's like a mathematical proof where you have to go through the proof yourself and only then you understand it. You couldn't just read it. Is it like that or is it just because no time? You were being very kind. No, mean, the answer there is twofold, right? mean, a large part of it is just like, zero copies is two engineers and we don't have time is the real answer. I would love to invest more time in documentation. And certainly, hey, let this be a call. Those of you who are still with us an hour and a half into this podcast or however much this gets edited down to, I'm seeing an hour and a half on my screen here. Those of you who are with us this far in the podcast, we would love open source contributions. Zero copy, love, love, love it when people come and contribute. Always looking for open source contributors, always looking for people to use it in new contexts and give us feedback, all that jazz. Yeah, the documentation thing is largely just a lack of time. There is somewhat of a challenge here. It's not entirely just a lack of time. There's a little bit of this, is just that it is a new way of thinking about how to do systems programming that a lot of people aren't really used to. And so we oftentimes find that people come in and they are... It's not just, do I do X? But fundamentally, I don't even understand what it would mean to solve my problem. And so it is zero copy really does have this sort of like... We will give you the tools, but we can't just document what tools we're giving you, because oftentimes people don't know what tools they need. Zero Copy really is a very powerful Swiss Army knife. It's very powerful. It'll let you do really cool stuff. But it's not a high-level framework. It's not a high-level network packet parsing framework or something like that. We have aspirations to build such a thing on top of Zero Copy at some point in the future. But right now, our time has been entirely taken up just building the Swiss Army knife, basically. right, not even using it to build higher level stuff. ⁓ Well, mean, definitely reach out to me if you get to that. Maybe ⁓ it could be like a partnership, I wouldn't mind. Hahaha! yeah, mean, yeah, certainly if there are people out there who are interested in helping to build a high level framework, that is certainly something we are interested in. As I said, we are IO bound, so to speak. We are not limited by a lack of ideas. We are limited by lack of time. So yeah, anyone who's interested in, if you have some crazy idea for a feature you could add or high level framework we could build, please, please, please, please come talk to us. I would love to talk to you. There's a Discord link on our... ⁓ docs.rs homepage, so that's a good way of getting in touch. Yeah, it is, generally speaking, I think people are just not quite familiar with what tool they want to be reaching for. And so that's one of the big challenges is how much do we document what it is that we provide and how much do we try to be prescriptive about if this is your problem, here's what you should reach for. If this is your problem, here's what you should reach for for different problems. And part of that is it's actually kind of difficult to enumerate the entire space of possible things that people could use zero copy for because it's extremely diverse. At a high level, it's basically just any time you care about like the exact shape of what's happening in your memory, zero copy might help you. And so that's a pretty wide, you know, that's a pretty wide problem definition. I remember once I was giving a talk internally about about Zero Copy and someone came up to me and said, hey, I want to add an example to your list of example use cases. This person was doing video game hacking. So they basically had a video game and they wanted to hack it in order to get extra statistics about their own gameplay. And so they were reaching into the memory of this video game and pulling stuff out. And they were using Zero Copy to make sure that they were parsing the memory of the video game correctly. not something that had ever occurred to me that was a possible use case, but it sort of makes sense. So that's a big problem for us is like, we want to be careful that we, it's not super feasible for us to just list all the possible use cases. Cause it's, it's, you know, it's sort of be saying, it would kind of be like saying, you know, what are all the use cases for a for loop? It's like, I don't know, man, like it's a for loop, like you use it for a bunch of stuff. So that's sort of the challenge, but, also that's, as I said at the beginning, that's not an excuse. We should have more documentation, but it's, you know, there's never enough. Yeah. And then there is like project safe transmute. And you say it's kind of like about moving zero copy into the Rust compiler. Would it ever be the goal to just purely like, like, like, kind of like this not destroyed with like archive zero copy. And then it's just the compiler. Yeah, so... Yeah, so I suspect that we are... Well, let me give some background. The short answer to your question is probably there's not a history, there's not a future history in which zero copy and crates like it, like bytemuck is another big one here, there's not a future in which they go away entirely, but there is hopefully a future in which they get much smaller. So basically zero copy and crates like it, for the most part, solve the problem of... having a pile of bytes and wanting to convert it into a particular type, or having a particular type and wanting to look at its bytes. So it's a bidirectional transformation between bytes and some other type. It turns out that you might want to do more powerful things than that. So you might actually want to go from a particular type, like some particular type T, and convert it into some other type U. And that is something that zero copy is not good at reasoning about, because it's no longer a piecewise operation. So zero copy today will let you go from T to U, but only if you can intermediate by going to just pile of bytes. So if you can go from T to a pile of bytes, and then from a pile of bytes to U, then zero copy will let you compose those two operations and just do them together as just T to U. But there are a lot of pairs of types where that's not valid. ⁓ So for example, if you have like an enum that has three variants, and you want to go from bool to that enum, that should be allowed, right? A bool can be either 0 or 1, and then maybe I have some enum that is also one byte long that can be 0 or 1 or 2. In principle, I should be able to go from a bool to this enum type. The problem is, zero copy, the only way that it knows how to reason about that operation is to go from bool to a pile of bytes. Once you have a pile of bytes, now you just have a u8. And from zero copy's perspective, if you try to go from a u8 to this enum, you're basically telling it, hey, can I take any u8 at all and convert it into this enum? And the answer is no. What if it's the value four or five or 27 or 38 or whatever? So, just saying I can go from T to a of bytes and a pile of bytes to U limits you in the types, in the number of types that you can actually support, the number of conversions that you can support. So that's the reason that zero copy and crates like it are limited. So the safe transmute project is a project that attempts to support that more powerful class of transmutations in the compiler itself. Basically the way that we do this is we have a representation inside of the compiler of the layout of a type. So there's a way that the compiler can reason about what is the set of every possible bit pattern that I might actually see for this particular type. And then it has a way of asking basically a subset question, it's doing a subset query. If I have a T and I'm trying to transmute it into a U, is the set of possible bit patterns in the T a subset of the set of possible bit patterns in U, right? Is it guaranteed that no matter what T I have, it's going to also be a valid instance of U if I just convert the bytes from a T to a U? So the compiler, we're building a way in the compiler to do that transformation, right? To ask queries of that form. It turns out that is actually like a very interesting CS theory problem. We've had to develop some novel algorithms and there's been some really interesting engineering there. So it's a long-term effort. But the goal is that eventually the compiler will be able to answer questions of this form for you. Once it has done that, then you can rip out all of the analysis part of zero copy, right? So the custom derives can basically just go away and be replaced by this built-in compiler analysis. Now the high level stuff in zero copy still stays there, right? So you basically what the compiler does is it gives you this trait implementation. that you could use to say, is this a valid transmutation? But then you actually still have to write the code that does the transmutation. So all of the stuff in zero copy that says, if this is a valid transmutation, then here's a bunch of extra stuff you can build on top. That stuff is gonna stick around probably, right? That stuff could in principle be moved into the compiler or at least into the standard library, but there's a large set of features there and we're probably not gonna move all of it. Certainly not anytime soon. So more likely the short-term world is going to be, we add this analysis to the compiler and then we remove or strip down zero copy derive, but the zero copy create itself stays largely the same and maybe gets to grow new fancy features that are built on top of this. But that's sort of the future that we expect. Okay, yeah that makes sense and still very exciting. So is this like for now a very theoretical working group or is this something like no this is like getting very concrete and soon we will learn these kind of things. Yeah, so it's concrete in the sense that we have an existing experimental implementation. So you can go today to the Nightly Rust and find in the mem module, there is a trait called, I think it's currently called transmute from. We have renamed it at various points. I can go there right now as we were talking and answer that question. but basically there's an existing nightly feature that provides this query. So it's in the compiler, but it's unstable is the short answer. There's good amount of, okay, here we go. looked up, yes, it's mem-transmute-from is the current name. So it's unstable, but there is a fairly fully featured implementation in the compiler. But there are a lot of edge cases. So currently we don't handle like non-zero types, for example. That should be a relatively easy fix. Again, anyone who is interested in open source contributions would love to have you. There's a lot of like pretty easy low-hanging fruit for people who are already familiar with contributing to the compiler for adding extra features. Then there are optimizations that we would like to do. So currently, if you represent an array of a gigabyte of memory or something like that, you would hope that the representation would basically just be like, you know, the representation of a U8 followed by a number that says there are a billion of these. That is not what we do. Currently, if you give us an array of a billion U8s, we are going to instantiate in the compiler a billion representations of U8, which is very stupid. So there's like low-hanging optimization fruit like that that we just need to like, you know, do the obvious optimizations. Eventually, the goal, of course, is to actually put this up and try to stabilize it and all that jazz. We're not quite there yet, but we're hoping to get there. It is something you can experiment with today. If you are using Nightly Rust, you can try to play around with this with your types. As I said, you may run into edge cases and stuff, but that's exactly what we want to see. We'd love people to experiment with it and give us bug reports. So yeah, the long-term goal is to put this up formally for stabilization. But there's a good amount of engineering work between here and there to get Okay, very cool. And then I feel we covered a lot of ground. I also feel we could talk a lot more, but I do think we should wrap up soon. Is there something that we didn't like discuss yet or maybe we touched on it, but you might think you want to go a bit deeper into it before we leave? Sure. Yeah, so I think the like, would say that I had a few notes written down of just like stuff I wanted to touch on at some point. And I'm just, these are gonna sort of somewhat disconnected from one another, but it's just sort of the grab bag at the end. One thing is if you are interested in systems programming and you are like wanting to go deeper on stuff, I happen to think that writing a net stack is one of the most interesting and challenging systems hobby projects that you can do. you would be surprised by how tractable and approachable it is to implement IPv4 and TCP for yourself. Buy a networking textbook that has an explanation of those things. Read the RFCs. They are not the world's most readable documents, but they're not extremely arcane. Try to implement it for yourself. It is... It's a wonderful challenge in how to build large scale software. I think it was one of the most educational experiences that I had during college was being tasked with implementing this, even in an academic context. So if you're interested in systems projects, highly, highly recommend it. It's a lot of fun. The other thing is I get asked a lot is there a sort of high level philosophy or attitude behind the work that I've done, like in netstack 3 and zerocopy and safe-transmuting and stuff like that. Like, how can I do, how can I like take a similar sort of engineering approach to my own work? The answer that I usually give, and this is definitely something that I like recommend for a lot of people, is Rust in particular, but this is applicable in theory to lots of other languages, gives you the ability to be extremely ambitious. about how nice your software is. And so what I would say is whatever you're doing, abstraction design, if you're writing a library, let yourself be motivated by the functions and the features you need to provide, obviously, but also let yourself dream big about how nice it could be to use your API, right? Let yourself sort of engage in daydreaming about, man, wouldn't it be nice if I had a system that let me do this? It was just this easy, right? That it was just. just worked out of the box or it was just like this straightforward to do this particular operation. Let yourself have that dream. And then the challenge to you as an abstraction designer is to connect the two things, right? To say, is there some way for me to provide the functionality that I need to provide, but in an API that is actually this nice? Rob Pike has talked a lot about this. If you want to go watch some of his talks that he's given at various conferences. One of my favorite talks that he's given is, simplicity is complicated is the name of the talk. He has this great, one of my favorite quotes, simplicity is the art of hiding complexity. Basically the idea is your job as an abstraction designer is to understand the thing that you are implementing so well that it's possible for you to provide a simple API. The simpler the API, the more deeply you have to understand the problem in order to be able to produce that API. And so, That's really my advice is let yourself be ambitious about how simple your APIs can be and about how easy and pleasant to use they can be and understand that in order to satisfy that ambition, you are going to have to just sit there and think deep thoughts about the problem for a long time and like mull it over in your head and really, really get to the point of deeply understanding how you could possibly shape an API. ⁓ that is as nice as you want it to be, basically. So that's sort of the like, you know, I encourage people to be as ambitious as possible in that front. ⁓ And if you get there, then that's how you end up with something like zero copy that you're sort of scratching your own itch. But then eventually you end up solving a problem that other people also have and you know, to use a bit of a sort of overused phrase from the product design world to delight your users or whatever, right? I think that... That phrase gets a lot of poo-pooing from more hardcore systems engineers types. But I do think there's something there. The idea that you use a product that you really like and you just have this sort of emotional feeling of release, right? You're just like, ⁓ it's so nice. I don't have to think about this anymore. That's the feeling you want your users to have. You want them to just sort of have that like, ⁓ this is the thing I've been wanting to use my whole life, but now it's here and now I can just relax. ⁓ Let yourself be ambitious about that. I think that's... That's all I got. Yeah, only that. It's a wonderful advice. yeah, and I feel especially in a language like Rust is definitely a place where you can be ambitious because for example, if we think about the framework grammar that I was mostly writing alone, but now I do get more and more contributors is like I would never build that if not for Rust, especially not on my own because... I have, I like, okay. So one thing I often say to people is like, rust, they often focus on like performance, et cetera. But what I really like about rust is I, and thanks to the type system and many more things is how I can also refactor things and change things. And I'm not scared to change it because it will, like, it will really just work out. I've done several large refactors and it just worked out like no scary surprises. If I think back to my long history in C++. despite how I like the language and how I like to work in it, I would never have pulled that off. Like I would have introduced so many bugs so many segfaults, so many all kind of stuff. Like it just wouldn't have worked. And so yeah, I like your advice a lot because it's something you can actually do in a language like Rust, for sure. Yeah, no, exactly. I mean, this is something that I have seen. It's a very powerful philosophy because I say that specifically in the sense that I have seen other people take that philosophy and run with it. So we had this guy, Alex, who worked on netstack3 for a while, and he did one of the coolest projects I've ever seen where he basically figured out a way to encode ⁓ lock ordering in the type system. So basically the idea is like you have a bunch of mutexes and what is the order in which you acquire those mutexes, right? He figured out a way to put that in the type system and prevent the netstack3 code from ever having deadlocks. Basically, if you attempt to acquire locks in an order that could result in a deadlock, it will cause a compilation failure. So cool. That was just an example of basically someone on the team who had seen this philosophy of trying to be really aggressive about how nice your APIs can be. And he was like, yeah, I bet we can do this. then he just, again, it was that same, what's the goal? The goal is I want this to be a magical experience. I want to be delighted by how easy it is to use. And he just sat there and he banged his head against it for a while. And then he came up with this really clever trick that now means that netstack 3 is never going to have a deadlock. And if you want, by the way, for the sort of like more business minded people in the room or in the audience, if you are interested yourself or you're interested in arguing to leadership at your company that this is something that's worthwhile in doing, NetStack 3 has this crazy statistic that I think really encapsulates just how this is not just a thing that engineers can do because we like correctness for its own sake and it makes us happy or whatever. It's not just like a nerd snipe thing. it actually really translates into business value. In the first year and half of NetStack 3 being dog fooded, as in used internally on the team, but as a real networking stack by people on real devices, sitting in their homes, using to do development, whatever, normally the goal of rolling out something for dog fooding is that you expect to find a bunch of bugs that you had not found during testing. You'd expect that to be in the order of hundreds or something. over the course of a year and half, you're going to see this long stream of bugs, you're going to burn them down, yada yada yada. In the year and half that NetStack 3 was in dogfooding, and again, this is after I left, I'll sort of refer to the third person here, the team, they encountered four bugs in the field. Four. Which is just a stupidly low number, an absurdly low number. And if you think about the cost from a business standpoint, of especially a networking stack, right, mean, you know, if you have a device in the field, and its networking stack has a catastrophic bug, that device is effectively bricked because it can't connect to the internet to take a software update. That's how important a net stack is. So getting it right the first time is crucial. And you think about the amount of developer time that is saved by not having to do debugging, you think about the normal life cycle of a product. If you as a software engineer, especially if you've worked on a C or C++ code base, how much of your time do you spend fixing old bugs as opposed to writing new code? Imagine what it would be like if most of your time was just spent writing new code. That's the value proposition from a business standpoint of moving slowly and carefully in a language like Rust, you know, to use the phrase that the Marines have, slow is smooth and smooth is fast, right? If you measure on the order of lines of code written, maybe you're moving more slowly, but that's not actually the right measurement, right? The right measurement is like features shipped stably to production. And my thesis, which is not based on actual numbers because people haven't studied this yet, but my experience intuitively is that if you measure over a longer time horizon, Rust is not just a safer, more precise, more careful language, it's also a much faster language because you don't waste time in development fixing problems that you introduced earlier in development. So if you're trying to sell this to leadership at your company, give them that quote, four bugs in a year and a half of dog fooding. with a small team that had not been working on this that long. Rust is a really special thing and it translates into dollars and cents. It's not just nerd sniping for nerd sniping's sake. yeah I couldn't agree more and I will definitely use that quote thanks for that so with this with this being said I want to thank you very much for your time and I wish you a very good life ahead of you Yeah, of course. Thanks Glen, this has been an absolute pleasure. Anyone else who's thinking of doing this, take Glen up on his invitation. It's been a great time. Appreciate it. Elizabeth (Plabayo)
1:42:42 | 🔗
Netstack.fm is brought to you by Plabayo building secure, open, and resilient infrastructure with Rust protocols, and purpose. This show is also made possible by Rama, the open source networking framework. Plabayo offers service contracts and welcome sponsorships to keep building and supporting its ecosystem. The theme music of this podcast was composed by DJ Mailbox. If you enjoyed this episode, don't forget to subscribe on your favorite podcast platform and leave a five-star review. It really helps others discover the show. Thanks for tuning in. We'll see you next time for the next handshake.