On this page On this page
Episode 9 – gRPC with Lucio Franco.
A deep dive into the world of asynchronous networking in Rust with Lucio Franco, maintainer of Tonic, Tower, Tokio, and Hyper. We explore the origins and design of gRPC and its Rust implementation, Tonic—how it came to be, what problems it solves, and why it matters. Along the way, Lucio shares insights into open source collaboration, Google’s work on gRPC for Rust, and what the future might hold for the broader async Rust ecosystem.
If you like this podcast you might also like our modular network framework in Rust: https://ramaproxy.org
00:00 00:45 06:50 14:45 29:48 36:33 43:38 54:12 59:15 01:07:03 01:09:48 01:14:19 01:16:38 01:23:16 01:27:08
Music for this episode was composed by Dj Mailbox. Listen to his music at https://on.soundcloud.com/4MRyPSNj8FZoVGpytj
Elizabeth (Plabayo)
0:17 | 🔗
This is netstack.fm, your weekly podcast about networking, Rust and everything in between. You are listening to episode nine, recorded on the 6th of October, 2025, where Glen has a conversation about gRPC and Tonic with Lucio Franco, senior software engineer at Polars and maintainer of the Tokio Tower and hyper ecosystems. Welcome for another week at netstack.fm Today with me is my guest Lucio. We will talk a bit about gRPC and Tonic. It lives within the Hyper ecosystem and for reference we talked in episode 2 with Sean, the maintainer of Hyper. It also makes use of the Async Runtime Tokio although I think you can also do it without. Anyway we will talk about it. For reference we talked about Tokio in episode 5 with Carl. So welcome Lucio. Yeah, so today we will talk a bit about gRPC and about Tonic, but before we get into those technicalities, I think I would like to get to know you a bit better. So could you tell me a bit more about your origin story? Yeah, sure. I'm a US based software developer. I've been writing Rust. I've been writing Rust since just after the 1.0 days, but I've been also writing Rust professionally since around early 2019. And since then, I've basically have been working nonstop with the Tokio stack, async rust, the open source community and all those things outside of the open source work that I do. have a huge background in distributed systems and networking. Hence why Tonic is kind of my creation or let's say the project that I decided to shepherd all the way through in the Tokio kind of group of projects. And yeah, and I've been working in ever since. It's such a beautiful language and I'm very blessed to, you know, have the opportunity to work with so many smart people in this community. Yes, and I noticed that you also used to work at Amazon, which is also where Sean used to work and Carl still works there. Is that also where you met them or that's just a coincidence? No, I actually, I did not meet them there. We've known each other, have talked many times before. We all started working at AWS. I actually met Carl back in, I wanna say 2018 when... Tokio was kind of a budding project. think it just 0.1 and it wasn't, know, this is before anything async weight was even an image at existing at all. so I met them way back then, actually back when people still used, ⁓ the tool gitter I don't know if you remember that, but the, like chat free chat service that you could attach to your GitHub repo. and so I, I just, I met Carl through that just cause I was answering questions about Tokio as super interested. And, we started chatting. and then I met Sean later on. when we all, they had actually joined Amazon before I did and I ended up joining after them and we ended up working on separate teams but still working pretty closely together, obviously. Yeah, I mean, it's pretty remarkable how many influential people of the Rust community have actually worked at AWS Even though they might not know each other, it's remarkable nonetheless. Yeah, mean, I think initially AWS had a really good approach to wanting to, you know, take in talent for Rust. ⁓ Clearly, Rust was a big target for them. Obviously, like... when you compare something like the JVM and Java versus Rust, you get a lot of benefits from rewriting software. And Amazon had realized that. And in fact, when I initially joined AWS, I was not on the same team as I mentioned earlier with Carl and Sean, but instead I actually had joined the Lambda team to help them do a massive rewrite from Java into Rust. And so they were investing pretty heavily. And I think in a lot of good ways they were obviously, I think the The world has changed a little bit since the initial hiring of all the Rust people and priorities have changed towards investment, which I think has created a lot more people to leave those positions where I think the community could have benefited a lot from it. But for the good two years that we had there, I think before things started to ⁓ kind of go downhill a little bit, there was a lot of really good investment, a lot of really good learning, at least on my end and with others. We were learning a lot about how people were writing distributed systems. internally at Amazon and figuring out how we could turn Rust into a tool that they could use. So that was a really cool experience. Okay, and so in a bit I will go back to my question around that a bit, but before that, like right now you work at Polars and it strikes to me like a company which, okay, yes, it's invested in Rust, but it doesn't seem to be really related to all the wonderful projects that you maintain, or am I wrong? Yeah, actually, I'm no longer working at Polar's right now. And you're right, they... they're investing into their own technology with the data frame stuff, but they're not investing that much into open source. And I would say in general, there's a big lack right now of companies willing to invest into open source in general. Something that I have noticed, you my last half at Amazon was working full-time in open source. And since then, a lot of companies are not really wanting to invest into the technologies that they're using. And this could be, you know, a moniker of two things, right? One is maybe Tokio. Tonic, Hyper are actually quite stable. and quite mature, which is kind of a cool thing to say when you're looking back at when these things were all started. And so I think people feel less need to invest and to get the stuff that they need out of it when they already can get what they need out of it. ⁓ overall, the investment I've seen recently, besides the companies contributing to sponsorships or open collective, to be pretty minimal compared to, I think, what we were seeing in late 2018. in the early 2020s. Yeah, okay, that makes totally sense and is there anything you can say about the current job that you're working at? I'm currently not working at a job right now. I've been taking some time off focusing on some open source stuff, which I'm sure will cover some of those details and also just taking a general break. Yeah, okay, cool. That's very nice. When you can do that, it helps a lot. Now, GRPC and Tonic, like Tonic, was it born while you were at Amazon? I think it was before, No, actually, so the story is actually kind of funny for Tonic. ⁓ So I graduated university in ⁓ mid 2018. I had left to go to New York City for a job, work at a small startup. The startup was really focused on initially Elixir, ⁓ which is a cool language, and I had some experience. with Elixir when I actually created a startup in college. Obviously it doesn't exist anymore, ⁓ created a startup in college and I had chosen to use Elixir and so they ended up hiring me because I had this niche technology background for a new grad. And then they started writing some rust for their new tool called Vector, which is now a Datadog product. It's essentially like a log shipper. Again, none of these things have anything to do with gRPC or any of that stuff. While I do think... Vector had gRPC support and that kind of piqued some of my interest. I was already interested in Rust and networking and You know, I had this genius idea that you know async await shipping I'm thinking this is like September of 2019 and I am now in May of 2019 thinking, you know, what, can I get my foothold in the community a little bit? ⁓ and I wanted, was eager to create a project. I just didn't know what the right investment was. and I was talking with Carl and, you know, I, I'd spent some time on tower GRPC and I mentioned earlier, I have a huge interest in distributed systems. So I realized that there was an interesting gap in the ecosystem. know, async I/O was coming, there was going to be a plethora of libraries coming hyper already kind of, you know, we were making plans for async I/O already. I was already contributing to Tokio and stuff and Hyper and we were making plans for one for for async await and I realized that there was a pretty big gap. You we were investing the majority of our energy into getting Tokio ready into getting Hyper ready and you know, kind of getting things ready for async await not 1.0 yet, but you know, think it was Tokio 0.3. I'm trying to remember whatever the minor patch release before 1.0 was, you know, we were working pretty heavily on that and I realized there was going to be a gap in open source products for people to use was, know, I'm, I'm async await has just landed. You know, I want to use hyper. I want to build something on Tokio, but actually I want to build something a little bit more complicated. And, know, the cool thing with async await was that it brought the ease of writing these state machines, right? Compared to before where we're using futures 0.1, where you had to kind of write state machines by hand. They were very error prone, very hard to do. We're just now with async await. It was quite easy to compose these things together, right? And you can think of the idea is I can focus more on building my business logic than I need to focus on. wiring things together. And so I had this idea that, you know, what if I could deliver Tonic as a async await first, like, you know, the library that's created with the baseline of async await. No, there's no old future style. Everything is designed to work with async await, you know, and take advantage of the new borrowing rules that we can, we can have. And so I set out to kind of improve tower GRPC and turn it into a new project. tower-grpc is like an archive project now in the tower repo [organisation], but it had a lot of the initial fundamentals for creating a gRPC library on top of Hyper. And so I ended up taking this and spending a lot of time on thinking about how to make good ergonomics and how to make async await feel very natural and things like code gen that automatically generated you into an async trait that you could just start implementing right away without needing to worry about how do I set up futures? How do I do all of this? It kind of just naturally worked. with tons of good examples and so I started writing that on my own and the kind of joke that I say which is not really a joke honestly but funny thought was, you know, I wanted to get a job writing distribute systems in Rust. And I told myself, you know, what's the best way to do that, but to create the library that everyone wants to use. And ironically enough, I released Tonic in September and around November, I had a friend reach out to me who was currently working at AWS and was like, Hey, like I, looked around, you know, he was pretty involved in the Rust community there. And he's like, Hey, I heard Lambda ⁓ was looking into using Tonic and their POC was using it. And I was like, oh, that's great. Can you put me in touch with their manager? So I ended up getting in touch with the manager. And that's actually how I joined the Lambda team because they were using that technology. And I was able to kind of realize that goal, which is kind of funny. Didn't really expect it to kind of play out that way, but it did. And then I continued. I had about 50 % time when I actually joined Lambda to continue working on Tonic and to grow to where it is today. Well, that's indeed a very fascinating story and I think there's a lesson for all there when like a new, like every, I don't know, 10 or 20 years, there's like a new language which take over a major scene and I have a feeling that if you are on that timing and you join that community and you are lucky enough that the language and its ecosystem indeed grows into something because not all languages flourish, then I imagine there is a lot of room there to make impact, which is exactly what you did. Yeah, and I have another funny story that's pretty funny too. Like, I only discovered rust when I was in university and, you know... we were taught Java and C++ and C and stuff. And actually I picked up Rust before even done my first systems class. I'd only really done Java and maybe a little bit of Python, but I think all the intro classes were in Java. know, all of the program that I'd done had just been in Java. And, you know, obviously I'd done some HTML and some JavaScript for some website stuff, but pretty minor things. And, you know, I got this interest in, interest into databases and networking and all this stuff, you know, through the classes, what we were learning. ⁓ I had sat down on my own my second year and I was like, hey, I want to start writing some stuff. The first question is what tool am going to pick? What language am I going to write in? I knew I didn't want to do Java because I tried to do some networking in Java and knew that Java just didn't feel right. You're in university, there's no other requirements but what feels right or wrong to you, right? So it's quite a liberating time. I was like, okay, let me go try Go because at the time Go was really blowing up, especially in the distributed system space. And I remember getting to the instructions for Go and you know, I had previously had done an internship the previous summer in Node.js. So I was pretty used to the like NPM ecosystem, the way that you would set up your projects. Like I had a code folder on my computer that had all of my Git repositories and Go on the other hand, wanted to create like a mono repo and a Go folder and a Go path. And I was still very new at, you know, Linux and bash and all that stuff. And the whole process just felt very counterintuitive to mentally how I wanted to think about it. So I basically just gave up on it. And so I went to the next program in the language, was Rust. And I remember opening the webpage back when, the initial 1.0 webpage, which was like very bare bones to what we have today. And I remember reading the like points on it, you know, like performance correctness. And I forget the last one, but it was like choose three. And I loved that. And then I saw Cargo and I looked at how Cargo worked and it was just like NPM. And I was like, okay, wow, this is cool. And that was the only thought I had. I was like, this is cool and this seems good. And so I picked it from there and I started riding Rust basically as much as I could. You forgot about Go completely, which is pretty lucky. I didn't really expect to hop on the train like that, but those initial feelings really were strong. And those are things that drew me in. And then I have to be thankful for that kind of intuition because that's brought me to where I am today. Yeah, we're also very grateful because you brought many fruits with you and I think I did a lot of things right and I don't think you're the only one that probably was pulled in by that initially. how easy it is to make a project, to pull in dependencies. Obviously they learned a lot from the Ruby community there and some other communities, but yeah, I mean, personally I come from C++ originally and C and those kind of languages and yeah, for me it was like amazing at how they tackle these problems and they solved all the issues I had with the other languages. So that's like 11 years ago and I don't think I really ever left. So with that said, you mentioned, okay, tower and then tower, what is it called? Tower GRPC is now archived. It moved into Tonic. So before we dive a bit deeper into the protocol GRPC and also like how you implemented them Tonic, I would like to talk a bit about tower because you also maintain it, I think, or at least sometimes help out there. But tower is exactly, like when we talk about tower, It is still using manual features mostly because there is no other way, because you have to use an associated type where you need to define the type which doesn't allow implement trait yet or anything like that. So once you deal there with futures, you anyway have to either like box a future or implement one manually. And like I tried in the past to make something like async tower that experimented worked very well for me. Then that morphed into what I took from tower and morphed into my own network framework called Rama where we still use something like that where we basically have like an implement traits in the meanwhile like of course tower is massively popular and a of people use it which might like also mean that there is less like you have less flexibility to easily make a breaking change because like you're gonna maybe like break the entire ecosystem in two because moving that to what is now supported in rusts is very different than how it was when Tower was invented. So I wonder like what is the future for Tower and how quickly do you think that it will evolve if it will still evolve at all? That's a really good question. There's many layers to this puzzle. There's both the technical side and then the manpower side and then also the societal side to it. Now, we can kind go back a little bit. Tower and in Tower GRPC and to a certain extent Tokio all exist mostly because... I won't say Tokio doesn't exist because of reasons. Tokio was, or something like Tokio was always going to exist. It was always in the plans of the Rust community and Rust language people for a long time. But the way that we have Tokio in its current iteration, Tower and Hyper, really comes from the Linkerd days at Buoyant where Carl and Sean and Eliza were working originally. And they had a need to write a... proxy that could run as a sidecar on Kubernetes clusters. So it needed to be small, fast, low memory overhead and all of those things that Rust is really great at and needed those features. And so a lot of these things tower and tower GRPC specifically, and I believe to certain extent, Hyper really exists and have flourished to their level because of the investment that Buoyant originally put into this project by hiring the developers to work on it. So. Obviously, Linkerd existed, which is there, the proxy, or Linkerd2, I believe, is a proxy that... existed was starting to be written before async await had even landed. this is like, know, Futures 1.0 or sorry, correction, 0.1 had been out, but Futures 0.3 had not been out yet, which is the one that brought async await And so a lot of the decisions for what they needed were based on those, what was available to them on stable. so Tower kind of grew out of their own internal need for something like the service trait. And actually you can even layer backwards and there's a project called Finagle, which is a Scala project that Twitter created. And it's essentially a RPC framework that can be used to build proxies ⁓ and other sorts of thick clients and servers. ⁓ it provides everything, know, that tower kind of, not everything a tower has, it provides a little bit more than what tower has, but the kind of inspiration for tower kind of comes from Finagle. This is because if I remember correctly, The initial Lincolte proxy used Scala and was implementing using finagle. And so when they started writing the next one in Rust, they kind of took a lot of ideas from there. So the service trait is a finagle idea. And I believe there's a finagle white paper out there called service as a function, which kind of goes over the initial core concepts of the service trait. Now I don't know how much everyone knows about Scala, but Scala is quite an intense language. It's well known for having a crazy type system that you can do a lot of things with. There's many ways to solve the same problem, which is both Scala's winning condition and also its losing condition, I would say. And a good example is Finagle. There's so much more that Finagle can do than what Tower can do. And we're purely limited by our abstraction tools that we have in Rust. Good and bad. Obviously, I Scala has a lot of other problems that come with this sort of implementation of craziness in the type system. But for when it comes to porting something like finagle into Rust, this creates a lot of problems. Anyways, tower kind of spawned from this and a lot of the investment into the initial tower layers were done by the buoyant folks. And I think with the goal that they want to use start using tower and I remember quickly they do use tower or they have some sort of adapter for the service and they essentially use tower like stuff within linkerd2 proxy. I will say it's been a couple of years since I've looked inside that code base. But what I remember from the snapshot that I have taken for myself, they were still kind of writing some adapters and essentially writing tower service base code. Now. When you think of Tokio as a whole, there's a lot of layers to the project. Obviously Tokio's runtime is kind of the low level side of things. You can kind of go a layer up and you have hyper. Hyper is pretty low level when you compare everything else in the ecosystem. And then you kind of have tower, which is kind of on top of hyper. And then you kind of have tonic on top of tower. And the further you go up the stack, the harder it is to create stable APIs because obviously for Tokio, well, there was a lot of work to 1.0 Tokio's APIs. We had a pretty good idea of what we wanted to do and what a stable API looks like. And that purely just comes down to the fact that we had a lot of experience. with the low level stuff and the lower the level you go, the kind of easier it is to build a stable API because you need to kind of make less decisions for users. And this is also why I believe, know, hyper is a little bit easier to 1.0 than say something like tower because again, it's a little bit lower level. can kind of make decisions for users, but tower introduces a sort of abstraction over something that is quite opinionated. There's a lot of approaches to solving these problems. You know, we can think about a token bucket. There's many different to achieve a token bucket or a load balancer. There's many different algorithms to load balance over your code and there's a lot of different opinions and needs and so this kind of opens up a can of worms that makes it quite hard to develop the project and move it forward. There's a lot of competing decisions. So this kind of should give a picture of why Tower is where it is at today. Really the problem with Tower is that you need someone to kind of chauffeur in the right direction and to manage these different competing opinions and needs to create something that's stable and that everyone can come to a consensus on. And when we look at that with the sort of manpower that we have in the Tokio ecosystem, what people are interested in, It's kind of hard to invest into tower without some sort of third party interest from some organization. and I compare this to tonic tonic is much easier to invest our time into because there was a real need for users and there wasn't as real a need for tower. You know, people were using it, obviously Tonic uses it, but we can kind of get away with more things. It's pretty easy to take a tower layer and write a custom version for your own needs rather than, you know, writing a custom Tonic is a little bit more challenging. You know, thinking of all the layers with code gen and all that stuff. So I think the priorities for tower has kind of been pushed to the side because of that. You know, no one was really that interested in kind of. championing it a little bit and we've tried to search over the years to find some that was willing to kind of champion it but it was you know it's a pretty hard project to champion you know it's it's also hard when you know we have a lot of ideas but we don't have the energy or time to invest into it or we don't see it as a priority to kind of put focus into that side of the whole ecosystem so unfortunately tower has kind of gotten kicked to the side a little bit You know, there was an opportunity back in 2023 when I was working at AWS, had gotten agreement from higher ups to invest about a year's worth of funding of my own time to work on tower and to move into the future, which meant, you know, working on Async Tower, as you mentioned, and all that stuff and trying to champion those decisions and writing RFCs. Unfortunately, my whole team had gotten eliminated by the time that we got an agreement on that funding. So, you know, the big layoffs of 2023 happened. and I no longer have the funding to work on it. But really that goes to show the amount of motivation and funding really needed to move Tower forward was something like a full-time engineer for a year. That's what we had kind of predicted to get it to a 1.0 state, which is really unfortunate that never happened and I'm quite sad about it, but the problem space for making Tower, the future is pretty hard. this also means like, we'd also figured out that moving tower into the future away from old style futures but into new style futures was also going to require the investment of the rust language and the rust team. mean, Async Await still kind of is an MVP state. There are problems with it. I'll be the first to admit there are some pretty big glaring issues with Async Await. in its current iteration. And the big problems are around how you abstract async code, right? Writing async code that's imperative and pretty straightforward is not too hard right now. Obviously there's a couple of problems with cancellation and all those things, but things that you can work around pretty easily. But trying to write generic abstracted async code is still pretty hard in 2025. And I don't fully see a path forward right now. Obviously I have not been following a lot of it, but originally when I was working at AWS, I was involved in the working groups and all that stuff. So I was trying to push forward things in that area. But again, these things are hard problems to solve. We're kind of running into the corners of what the Rust type system can really do. And because of that, you know, we don't want to have to 1.0 tower and then 1.0 it again. We want to make the next iteration of the tower trait be the final version that we want. And to do that, is a lot of work, but also requires work from the Rust compiler, which is something that we can't totally do. know, like the investment it takes to get something over the line in the Rust compiler is a lot. It takes a lot of energy and a lot of time to do that. And for us with, all the other work that we need to do for Tokio, it doesn't really make sense. mean, we're focused on customers that want to ship things to production and get things out there. can't be sitting there in Lala Land thinking about what the future can look like sometimes, right? And we have to kind of focus on what we need to do right now. I would argue that this sort of mindset is what has gotten Tokio pretty far. mean, the reason that Tokio 1.0 exists and people are using it pretty heavily is because we decided to ignore a lot of the external noise and just focus on what we thought people needed. And unfortunately, I think the Rust language is not in a position to bring Tower into the future where we would be happy with it. As you mentioned, we're still missing some pretty vital features to be able to, you know... write an abstraction that allows us to say, there's some later named return type that is some future. you know, just doing that in Rust is not super easy right now, especially with lifetimes and all of that. And so. That's really why Tower has not moved forward. And this is why I kind of say it's kind of threefold. There's the technical problem. We don't really have a good solution yet. We've had some ideas, but beyond adjusting it for future 0.3, the next steps are not really available to us right now. It's also a people problem. We have limited time for open source work and it's quite hard to put that investment forward. And then also a societal problem. If we make a breaking change to Tower, That's going to have a huge cascading effect, right? Like I would say doing intermediary breaking change with another plan breaking change in the future would do much more harm to the ecosystem than forcing people to write old school futures with tower, right? So there's kind of these multiple layered problems why tower hasn't really evolved. you know, obviously tower is kind of one of my big loves in the ecosystem. And I really wish we could have pushed it further. And I, I put my full force into doing that when I was at AWS and unfortunately things didn't work out as we'd hoped. And the reality is what it is. This is kind of the nature of open source. Nothing is going to be fully complete. Nothing is going to be fully perfect unless you have full-time people working on it that are being paid and it's their full-time job. Yeah, I understand. And anyway, I think how far it got, it's already amazing. like, I mean, I like the system a lot. I know it came. from before in the Scala days. I didn't know it back then but I traced the history some years ago and it's very interesting and I have a feeling I might need to have a chat with William Morgan one day about the creation of Linkerd. I'm sure there's a lot of things to unpack there as well. Now in Rama we did adopt Tower, like I went via this Async Tower, Fork first and then eventually I learned a lot from that and then we integrated directly into Rama I now also have like compatibility layer so I can also make use of tower layers if I want to and we use it not only on HTTP but also on TCP layer on any kind of layer and like stack them like that and what I like about it as well is not just how how you can get away with a pretty simple abstraction but also how it makes it very explicit how your entire stack is while many frameworks they work in that you have some kind of I don't know magical runtime somewhere and you somehow plug in layers when you work with tower instead you have to explicitly say this is the order of layers and we can clearly see them here where it does kind of get rough is indeed where you say yourself when you have a lot of generics the the error starts to get very complicated unless you box it somehow so there's a lot of stuff there also if you combine it with like work stealing mode in Tokio you also get into things where you make your life a lot easier if you just force send-sync traits etc because it just makes everything a lot easier but I get that it doesn't work for everybody. That being said, I mean I would be happy to one day help steer the next version of Tower and like move that again back to Tower itself and see how that works out but that's of course just like I mean yeah there's a lot of work involved there so as you say yourself to to make that work with all opinions involved That's like another story, but anyway, if I ever can be of service to help steer that, then you can always reach out to us and I will be like gladly do that. ⁓ Now that we discussed tower a bit, I want to get a bit into GRPC because just like... Just like Tower was made in a different time, feel that GRPC was also made in a time that since has evolved a lot, because as far as I know, GRPC runs on top of HTTP2, but then nowadays you also have HTTP3, which I mean, don't think people run GRPC on top of HTTP3, right? It's just HTTP2. There are some implementations. I think the gRPC Java project has one. Okay and so it is a bit like it's the first time in this podcast we talk about gRPC so can you maybe give a quick summary of what gRPC exactly is? Sure. So GRPC as the kind of stands for is, I wouldn't say it's a Google product, but it's a Google invention. RPC frameworks are like a request response remote procedure call. Essentially, the idea is that as a user, I want to execute some code on a remote machine. So I want to call some sort of function that looks like a normal local function to my code and have that code actually execute on a remote machine. So the whole idea behind RPCs is to kind of add an additional layer of abstraction on top of your normal request response networking protocol, like HTTP. to provide a bit more functionality. And the big thing here is there's two main angles that's kind of attacking. One is code gen. So generating code for you that kind of gives you the stubs and everything you need and you can just fill in the bodies. And then also providing a cross-language interface, right? Because networking can be accessed. TCP is a language agnostic API. And so, you know, I can talk from Rust to Java, Python to Java, whatever I want to do. And so that was the goal was to support users that want to have a nice abstracted interface to talk between two different machines that may be running offering two different languages. And as you can see, this is a pretty useful sort of technology out there. Lots of databases have multiple clients in different languages. And so they need this sort of ability to have something that can talk across different types of languages. gRPC itself is not like a novel concept. We've had these sorts of RPC frameworks around for a while. I believe SOAP existed before this. And there's a couple other examples before my time that definitely existed around this sort of like space. But gRPC is kind of Google's open-sourced version of their own internal RPC framework. know, most of the large companies like Facebook, Google, Amazon, all have of invented their own RPC technologies. Facebook has Thrift, Amazon has this thing called Coral, which now they're kind of creating a thing called Smithy, and Google has one called Stubby. So gRPC is essentially open source version of Stubby. Now... GRPC itself is actually more of an umbrella than it is a specific technology. It really combines a couple of technologies together to provide the kind of complete unit that is GRPC. ⁓ And there's a spec actually in the GRPC project. You can go find it. It's called http2 protocol, I believe. And it's basically a markdown file that explains how to combine these different kind of technologies together. So the main two technologies that it combines together is HTTP2 as you mentioned and Protobuf. Protobuf being another Google project, which I'm sure follows the same sort of path that gRPC did. You know, there's some internal creation of this format and you know, it's been open sourced. We're currently at Protobuf version three nowadays. And Protobuf is essentially like a message format. how to do that? I have a message that has a set sort of fields. How do I encode these messages and decode these messages? And there's a couple of additional features there that make it nice are dealing with breaking changes and forwards and backwards compatibility, which you can go and find all of these reasonings on the Protobuf website. And there's a ton of debate online over these sort of formats, but gRPC chose to use Protobuf. But, you know, a little known fact is that you can actually use gRPC without Protobuf. So Protobuf is just kind of the main core messaging protocol, but it provides an additional feature and that it provides a sort of declarative syntax, you know, where you define the message, you can also define an RPC and the RPCs then defines what we call stubs. Stubs would be some sort of like function definition. You can think of this analogous as a trait function where you define the inputs and the outputs of the function but don't actually implement the body. all this is read by the protobuf compiler output to some additional protobuf message that is essentially a descriptor of the what you want. the messages and their inputs and outputs or sorry the messages and their types of fields and then you have also like you can think of an array of stubs that you might want to generate from and it has the inputs what messages are inputs what messages are outputs is it streaming or not streaming all of that and from this we can generate the code generating language. So that's basically the core of what gRPC does it allows you you have this code gen. And then the second part is, and again, this code gen is something that you can import into your own code and then start implementing. So for example, in Rust, it's a trait. It's also a struct that implements the client. And then there's a transport or what I like to call the transport layer. I believe the correct terminology is channel and servers. It's been one of my culture shocks working with Google, which we'll talk about later about gRPC, but they name things completely differently than I have. So I call it the transport. And what I mean by the transport is essentially how do I take your encoded bytes and write those on the wire and then read some decoded bytes, whether it's the server or the client, which in gRPC we call the client a channel. And so there's the channel and server portion of the code, essentially knows how to write http2 frames and manage all the other networking stuff. So it deals with load balancing, retrying, all of that fun stuff that isn't necessarily a part of the main code gen code that you kind of interact with your domain stuff. So these are kind of the two portions and you combine this all together and you now have a gRPC client and server that you can implement in multiple different languages that can talk to each other. ⁓ I will say that the majority of the complexity in gRPC kind of comes in the transport code and more specifically the channels where it gets quite complicated. And this is definitely an area where, for example, Tonic is not as in depth as other gRPC implementations have. So I'd like to conclude really, gRPC is just a tool to be able to communicate over the network, providing code generation for you so that you can quickly get started and providing tools to do more advanced things like load balancing, retries, load shedding, whatever it might be. Okay, thank you very much, that's pretty clear. That said, I've used gRPC on a couple of projects in the past and I like it a lot, but at the same time it's quite a high bar to get into or build a project around in the sense that if I wanna like build an HTTP project, it works with pretty much any tool. And if I want codegen I could still opt in for something like OpenAPI. which used to be called Swaggers and which a plenty of companies use as well. And at the same time I can make use of all kind of tooling like Wireshark and Curl to easily call to my servers or inspect data or do quick tests while with gRPC you always have this codegen layer you need to work on it. then... I suppose there must be a good reason why you want that besides codegen I imagine it's because people need performance or they have heavy stream APIs or something like that. Actually, I actually think it's the opposite. I will say I'm a little biased. I've been pretty deep in the gRPC space. So to me, it feels pretty much simpler than using HTTP by itself. I think ⁓ the complexity comes from it being something new. And I think ⁓ as humans, it can be hard to try to remove prior knowledge and ways of doing things kind of like in the same vein that... coming from Java to C to Rust is a very challenging experience because you kind of need to reframe your entire thinking. Now, I don't think gRPC is as extreme as that, but I actually think gRPC is simpler. You define some messages and you define some, know, I want to do action A and I want to receive output B and that's all you have to do. And then you have code that just can automatically do that. You kind of to think a little bit less about the lower level details. Now, I have my own gripes with open API. I've always found that open API is quite confusing and quite open-ended and the sort of possibilities that you can do with HTTP can be endless. And on top of that, there's no enforcement of that. follow the spec correctly. So yes, maybe I write some maximum API that can generate, sorry, actually I can write an accident API that generates some open API spec, but inside my bodies, I can do things wrong pretty easily and pretty, it's pretty easy to foot guard myself or to write some very specific things. And we see this quite often. you know when people tend to write custom protocols they at first it seems much simpler and easier but What ends up happening is you end up writing some custom code that has some custom behavior and that custom behavior then is confusing for other implementers, right? Think about somebody who doesn't have the option to see the code, the server side code and wants to run a client and they're getting some sort of weird behavior that's not in the documentation. And you know, and I'll step backwards a little bit, like any spec, even the GRPC spec or the HTTP2 spec, right, has weird things that they don't explain, right? And they're pretty in depth for something that is that wordy and that tries to cover all the basics. is the fact that it can even have certain gaps in its implementation details that you will only notice when you get. in deep into the implementation means that it's hard anyways when you even attempt to do a pretty comprehensive description of what you're trying to implement, it's hard to cover all your bases. I kind of my big gripe with Open API is that it tries to be, tries to act like it's a pretty strict typing requirement, but actually it still leaves a lot of surface area for interpretation, which can make it hard to implement things. And, you know, these things can be things like can result in like catastrophic production bugs that you didn't realize was a thing that could happen and it wasn't really fully thought through. Whereas GRPC tries to provide a more closed world experience, right? There's less that you can do, there's less area for customization. It kind of forces you down a very specific path and because of that I think it kind of pushes you in the right direction. Now that doesn't mean that GRPC is perfect, there are definitely problems, but you know... Anything you can accomplish with HTTP 1 and your sort of open API thing, you can accomplish with Protobuf. The only really big problem with Protobuf when it comes to these sorts of things that think holds people back is that it's implemented on top of HTTP 2. And even though browsers definitely support HTTP 2 and HTTP 3, they do not give you APIs that allow you to interact with HTTP 2 in such a way that would be compatible with gRPC. ⁓ What I mean by this is that in general browsers just give you like a fetch API, which is kind of designed around how HTTP one works, but it does not give you low level access to streams and frames and actual data frames on HTTP two, which is what gRPC requires. And that is supports things like streaming and more specifically client side streaming. But on the other hand, there is another spec called gRPC web, which many people do not know about. We have this implemented in tonic. There's a tonic web crate. Essentially, this is a simplification of the protocol that can work on browsers. Essentially, it's eliminating the client-side streaming and a couple other kind of like status handling things. And you can throw a proxy on top of your regular gRPC code and that can talk gRPC web. So you can still use gRPC from the web, but you don't get things like client-side streaming that you would get with WebSockets. And again, it's implemented over HTTP one, not over WebSockets. But in general, ignoring that sort of part of the problem, if I am some client code that's talking to some server code and I'm not in a browser, I really don't see a reason why you wouldn't want to use gRPC. It just makes life easier. And, uh, there's a little learning curve, obviously, but I think the ability to kind of write your ideas out in a protobuf spec or a protobuf, a proto file, and then run a command to have that code generated. And I immediately just have a client that speaks the correct language and everything I know is going to work is a, is a pretty good feeling. Um, you mentioned the tooling as being a problem and yes, I will admit, you know, it's a little bit harder to, to kind of work with gRPC. It's a, it's a binary format. Um, unlike a plain text format you would have. with HTTP one and or JSON, but I actually have had pretty good success with Wireshark. There's a GRPC plugin, I believe. Not only can you see, it's been a long time since I've used it, but there's a way that you can kind of set filters and it will detect frames over TCP and it will parse them and say, this is an HTTP two frame. I'm gonna parse it and show you the HTTP two information. You can also do that for GRPC and you can see the actual individual GRPC messages with their protobuf definitions and everything. upload the protobuf definitions to Wireshark it's possible to work with it and then there's tools like GRP curl and some other sort of tools that are kind of like your curl your postman that you would normally use in the HTTP REST ecosystem that you can use with GRPC. Again it's not the same level obviously but I've used it in the past and I've had to use it to debug things that were you know there's a weird trying to get Tomik to work with GCP was quite challenging because there's some interesting issues going on and I found some bugs using Wireshark and you noticing the differences between how Go was sending frames and how Rust was sending frames. So it is possible. Yeah, for sure. And I appreciate your in-depth explanation there. Thank you for that. But then as an outsider, when I, for example, need to inspect an API of a customer or even a third party, like it's pretty hard to even engineer this data without knowing the schemas of the protobuf. And so there are ways, but it's like a very loose process and kind of like a guessing game, like how to interpret the data. Yeah, I mean, that's definitely a problem. I would normally expect if you're inspecting the data like that, you would have access to the protobufs, the definitions themselves, and that would definitely make life a lot easier. But yeah, you're right. mean, it's not gonna be the same, whereas I can just curl an API and see the JSON, right? Like that's completely different. But again, there are some pretty big problems with JSON that Protobuf tries to solve. there's, you know, as with everything in software, there's trade-offs. And I think that gRPC definitely fits better into the ecosystem of people that are writing things like databases or microservices where you kind of owned both ends of the spectrum. I mean, that's really where this, you know, this was all created, right? It was created for Google to need to talk between their services. So, or other companies need to do these sorts of things where they need to talk between internal services where you have access to all the Protobufs and you might even have access to the team members that might be able to help you out, right? So absolutely, that's definitely a gap. And I don't think gRPC is targeted to that type of customer, though I do think, you know, it can be useful in those situations, or at least the ideas around the strictness about API definitions and all that stuff can be helpful. Yeah, for sure. For the record, I do really like gRPC. Like I've worked with in the past and I want to work again with some... I have a future projects where I want to use it again. But yeah, I do find it a bit like... Even though I like gRPC, I still find HTTP a bit easier. But that said, I want to have a bit of a... dive a bit deeper into how it works. Because like I said, this is first time we cover it. So given it makes use of HTTP2... as the low level protocol layer, let's First of all, why the choice to do that is for example, like building it on top of TCP and then how exactly does it use HTTP2 to instead of transporting regular data, transport protobuf or gRPC data instead. Yeah, now the original reason why it decided to use http2 I don't have a lot of knowledge on their decision making and I actually don't know when the spec came out for the http2 implementation. So I can't say how new http2 was. I don't remember that much about that. But reasons why it uses http2 over TCP. Well, Http2 is an improvement over TCP in many ways, but actually it's quite similar to TCP. TCP is designed to provide ordered streaming with window and flow control, right? These are huge concepts. This allows connections not to get overloaded. know, when the receiving end can't receive fast enough, you don't kind of overflow the downstream pipe. know, flow control, again, is another thing to help with congestion and all that stuff. So think of like the ordered streaming and all that stuff. Now, One of the goals of gRPC was not just to support unary request response, but to actually support streaming, bi-directional streaming. There's a huge need in gRPC for this, and it's one of the core tenants. What bi-directional streaming really means is, as a client, I can keep some sort of connection, some sort of stream open to the server. I can send messages when they become available, and then the server can also stream messages back to me when they become available. This idea here is, The idea is to keep open some sort of live connection that can send data back and forth efficiently. Now, if we were to do this on top of TCP, which is totally possible, you would need to have one file descriptor available for every stream that you want. And maybe you have 50,000 streams open and you don't want to spend that much memory consumption and all the NIC and all that stuff on a TCP stream. So you actually might want to multiplex your data over that stream. So this is where HTTP2. kind of comes into play. Http2 essentially allows you to have many little TCP streams on a single TCP stream. Essentially multiplexing these abstract streams on one connection that might be more efficient. And for data servers that say, you know, something like Lambda where, you know, you open a stream to the worker that's executing the function and you don't know when it's going to return, but you want to make sure that you're still available for the return as quick as possible. You know, don't want to have to redo handshake. You don't have to deal with TLS and all that. You want to be ready, but you might have many sandboxes ready at the same time. Maybe you want to have the ability to multiplex multiple streams over a single connection because they're not really all sending data at the same time. So that's kind of why http2 kind of fits naturally into this sort of tool because it allows it kind of adds an additional layer on state machine on top of TCP to do things a little more efficiently. ⁓ So this is why it was originally chosen to do that. http2 really maps really well to gRPC because http2 by itself is designed to have a single TCP connection and open many abstract streams. And obviously gRPC wanting to support bi-directional streaming, that mapped very well, but also unary request response maps very well. It's just a single stream that opens up, sends one message. The sender side closes and the reader side stays open until it gets one message and then it closes, right? So a unary request response is again just a stream that sends one message and waits for one message response. And this sort of theme is throughout the code base, know, when we implement the unary RPC calls versus the streaming ones, they're just like a subset of the streaming one. And so that really made the choice for http2 to make a lot of sense. Now, I know probably a future question you're going to ask is why not HTTP 3? And HTTP 3 does support a lot of this stuff, but actually I still think it turns out that HTTP 2 is still the king of data centers. When you have a 10 gig connection between two servers, HTTP 2 does a much better job of saturating that connection. The problems with HTTP 2 really falls short on sort of unreliable connections like cellular connections, and this is where HTTP 3 really does shine. But... gRPC really was designed to be used in intradata center. And therefore, Http2 was an excellent choice for it. And it just provides more features over Http2. That's really what it is. You know, it provides a lot of the core functionality that we would want. And I mentioned the windowing and all the flow control. Http2 has all this stuff on top of it as well, right? So each individual stream has flow control so that way you don't have one stream hogging the entire connection. So it kind of provides this abstracted kind of... simulated I would call like a simulated TCP on top of TCP that kind of gives you all these really cool features. Yeah, indeed. Thank you again for this in-depth conversation and also the fact that you already predict my next question, so that saves me some time. Now, of course, in HTTP2, we have these streams, we have the concept of these different kind of frames. The data frame is obviously very important because that's where you have your data. You also have those flow control and those setting frames. But then you also have the header frames, which is normally used for like your HTTP headers. Is there a lot of use within gRPC for those frames? Yep, so again, like we can step one step backwards here. So we have these streams on HTTP2 and these streams send frames, as you mentioned. These frames are obviously sent forwards and backwards between the client and server. more specifically for the streams themselves, as you mentioned, there's the header frame and the data frame. And you're right, the data frame contains the actual message that we're sending. And actually the protocol is quite simple and run in gRPC. It's a length limited codec. So you have some sort of header in the data frame that says, Hey, here's how long the message actually is. And then you just tell us it knows what protobufs decode and it tries to decode that set slice of the bytes. So that section is actually quite simple. It doesn't actually pass that much metadata for the request. It just passes messages, data, the actual message back. and forth. The header frame is used very heavily in gRPC and it's additionally one of the main components why HTTP 2 was selected. JuraPC also contains this thing called metadata, which is essentially HTTP headers. They're just slightly called differently. And there's a couple small things that are different in them. For example, you can pass binary data, not just text data, but the binary data must be encoded in base64. And so Tonic, for example, provides a lot of machinery around, know, type machinery to do this sort of stuff. But at end the day, it gets converted into a header data that gets sent in the header frame. There's additionally a couple other gRPC specific headers. More specifically, there's a status header, which contains a number that it represents a gRPC status, zero being okay, and then one through like at least 14. I know there's more than that, but. One through 14 provide additional headers, sort of question statuses like unknown, deadline exceeded, you know, all the things that the GRPC client might want to respond to. For example, resource exhaustion might inform the retry mechanism not to retry again because it's exhausted and retrying would just exacerbate the problem. So the headers are used pretty heavily for that. There's some additional cool things that you can do that you can pass trailing metadata. So let's say that you send a stream of messages and then your system fails. You want to notify that you fail instead of completing successfully. There's a header frame that will be sent at the end. And this header string will also contain, know, http2 flags saying, hey, my data stream is done. You can close the stream and all sort of mechanisms within http2 So gRPC kind of really layers very heavily on top of the sort of http2 header mechanism and all of that stuff. And it's used very heavily. Okay, very cool. didn't know that. So that makes totally sense. So now that we discussed GRPC quite a lot and I think we cover most of the foundation there. I wonder like you have Tonic and so now I want to start talking a bit about that. And so let's say in Rama, we currently have our own HTTP stack and so we also have HTTP2. And so we already have that part. Is there a way for me, for example, to use like, let's say Tonic or like adopt it, but with without having to rely on the fact that it normally runs on top of hyper. Yeah, they're trying to, I believe there's, again, there's these two sections of the tonic code that I mentioned earlier. There's kind of like the transport layer and then the code gen layer. And I'm including the code gen layer includes kind of more of the core technology. So a lot of the code that knows how to encode and decode based off of codec traits that obviously Prost implements, Prost being the Rust protobuf library. And most of that code, and I'm trying to remember perfectly, but if I remember correctly, almost all of that core code, gen code. So the code that can read your protobuf definitions, can output some sort of domain specific code based off of those definitions and know how to encode and decode into some sort of abstraction and submit messages on the abstraction. None of that requires Tokio. because the entire abstraction is built off of the tower trait itself. And so as long as you can implement your transport layer that plugs into that tower trait, you can have any sort of transport that you want, right? It's not very strict. Mostly the abstraction is around the HTTP crate and the HTTP body crate, those being the core abstraction layer. So if you could implement those two traits and types into your abstraction, then you can plug it in. It was specifically designed to separate those two different layers. I will say that one of the goals of the project was not to make it necessarily easy for people to use other sort of runtimes. What I mean is I didn't explicitly design stuff to be malicious, but I did not put a terrible amount of effort into making it really easy to use other runtimes. It was not worth the investment on my end. So the answer is it is absolutely possible to plug in your own custom HTTP2 implementation or whatever you want. You know, it could be in memory. But it's not definitely not easy, I would say. I understand. And let's say there would be a commercial company willing to invest the time into, for example, refactoring some of the tonic code so that the protocol and encoding code would be a bit more separate and easy to swap into another stack. Is that something that you as a maintainer would be open to or is this like, no, this code is pretty mature by now and there is not really any good incentive to change that. So yes and no, it's a complicated question. Obviously this is not something that we have not thought about. Obviously this has been a reoccurring theme in the Rust ecosystem for a long time. I believe that the abstraction that we have right now is definitely good enough to achieve anything that you would want. The reason I hesitate is that I vaguely remember throwing in a Tokio spawn somewhere, but I believe that's still just in the transport code. that would be like the big blocker, know, it tries to call some Tokio thing, but I don't believe it really relies on Tokio in that core code and you can feature flag it out. That said, I don't expect the abstraction to change in any direction except for one. And I'm happy to talk about this later, but I've been working with the Google GRPC team for the last year and a half or something like that, a little over a year and half on them trying to integrate their own stack into Tonic. And so we are working together to figure these sorts of problems out. I believe the abstraction code is going to move in in a direction that will be even more abstract than what it is now, such that as long as you implement the traits and stuff, you can plug it in and it should be pretty easy to do. So the answer really is that I believe it should already be possible, but I expect the abstraction layer between the kind of code gen and the protocol code to change the next year to support what kind of Google needs. And the needs for Google are going to probably be very similar to what anyone else might need. So the answer is we will support this already. But it's likely that most of the decision making will be kind of coming from the Google end rather than coming from, know, they're the main driver right now, as you say, the big third party that wants to drive something forward in the project. Okay, very interesting. Thank you for that. And then so you also mentioned a bit about when we were talking about gRPC and a tiny bit about Tonic already is that that Tonic has some features missing that other maybe more established gRPC implementations just as the one of Google itself in Golang for example, or maybe in Java do have. Can you elaborate a bit more on that? Yeah. So again, Tonic is an interesting project because it was not created from Google's kind of mind share. It was always written kind of to be different than the other gRPC implementations. was never like I, when I wrote a lot of the code, I never really looked at the Google's other gRPC implementations. I looked at it vaguely and looked at some details, but I never Use their approaches as pure inspiration beyond for example naming the client channel but I kind of designed it just as if I was writing a Rust project, kind of just thinking about how would I design a gRPC implementation that is pure Rust and is purely Rust in its core? And so Tonic makes a bunch of choices that are different than what Google would make. And you can kind of think of this in the vein too that Google's a big company. The code they write is very different than the majority of what people that are using Tonic write. And what I mean by this is the scale. Google has to handle things very differently. sorts of load shedding and load balancing to support their needs but this is something that the majority of users don't need. And so Tonic was kind of aiming at this other subset of users, not Google, but the other users that, know, when you look at Rust back in 2019, the majority of users running Rust were hobbyists or people that were doing Greenfield projects that didn't care about this sort of scale. So really Tonic was targeting them. And because of this Tonic misses out on a bunch of features. And I will, I will call it one major feature being XDS. XDS is Google's load balancing protocol. It's not just a load balancing algorithm, but it It's a load balancing client, sorry, load balancing kind of implementation that has algorithms, many algorithms, but also can speak to a server to inform it how to act. very complicated to implement. It needs its own gRPC implementation to even do that sort of stuff. this is something that Tonic from the outset did not want to implement. We took Towers load balancing algorithm and said, hey guys, if you need load balancing, here's a tool. If you want more than this, good luck, please go implement that because we're not implementing this right now. It's not a priority. There's a lot of other priorities. The core code needs to be matured and stabilized. So. We really kind of skipped out on some of those features. Some of the other decisions being, you know, we don't really think about arenas. We don't try to optimize things to a certain level that Google does. But it's quite funny, you know. There was some code I wrote in Tonic originally that I thought I was going to have to optimize away. You know, there was a couple of boxes in there that I had to make my life a little bit easier. And at the time there was a huge zealotry within the Rust community about allocating and all of this stuff. But I always laughed away because the box was hidden in some code gen code that no one would ever look. And to this day, no one has ever brought up an issue with it. you know. Tonic kind of just tries to take that quick approach to get there and to be stable at the core and to be good. It doesn't try to extend itself to be and have the full feature set that the other, know, gRPC implementations have. Okay, and then despite all this and despite that some things are missing, why is it then that Google anyway wants to connect their stack to Tonic instead of, for example, let's say writing their own stack? Right, so I think it's some pretty good hindsight from Google. They have a history of kind of coming into the open source communities or coming into language communities and providing their own implementation of something. and this creating fractures and problems and animosity. And there's kind of a big sentiment with Google that they kind of come in and leave things hanging. And, you know, there's a lot of negativity around some of the things that they do. And I partially think it's some of it's pretty rightfully so. mean, it's kind of corporations are not necessarily compatible with your general open source language communities. You think about Rust where it was in 2018, the general group of people that were working on on Rust were not big company workers, they were people starting startups and doing greenfield things, hobbyists, and it's not necessarily very compatible with the way that Google wants to write software. So I originally got approached by the tech lead of the GRPC Go team with interest in trying to see if instead of them coming out with their own GRPC implementation, because obviously Rust has been growing within Google, there's desire to have a Google GRPC Rust implementation like they do for the other languages. And they reached out and said, hey, like we would like to avoid the ecosystem fracture. Can we work together to turn Tonic into GRPC Rust? Now for me, was a bit shocked at seeing this, but I thought it was a great idea. mean... For me personally, Tonic has gotten pretty far and it's a lot of what I can give to Tonic is done. You know, I don't have much interest in implementing XDS for example. It's a lot of work and I don't have a need for it so I don't have that much interest. So when they reached out, I was pretty happy to say, hey, yeah, let's work together. Let's talk and figure this out. Obviously Rust is very specific and the community has some very specific needs. So, and being in the community for a long time, I felt like a pretty good... kind of shepherd of what they thought and what they needed and was able to communicate that with Google. And so for the last year and a half, we've been working together and to try to figure out how we can turn Tonic into something that Google needs, but also something that the community can still use and adore. And a lot of this is the simplicity that Tonic provides and Google obviously wants to add a lot of complexity. So we're still in the process of figuring these things out. ⁓ have implementation has already started, but we're trying to figure out how we can satisfy both towards both sides of the spectrum. And I'll give a, it's pretty evident, you know, why this is a hard problem. And I'll say Google internally uses a lot of C++ and they want their Rust and C++ to be interoperable, which is a very valid thing. You know, when you're migrating code or upgrading code, you want to be able to do things piecewise. You know, it's, you're not going to be able to go and rewrite the entire C++ Google ecosystem in Rust overnight. So being able to share and have interop is really important. And we've seen this. Many, many companies do this. Amazon is focused a lot on JNI support for Rust and Facebook has done a lot of work on C++ and Rust integration. So this is not like a new concept, but this means that we have two different needs in the ecosystem. We have Google's needs and we have Tonic's needs. So a lot of the work that I'm putting myself energy and putting myself into is how can we support both of these needs? And, you know, what we touched on earlier, this, the kind of transport abstraction is a great place to do this. So this is what we're kind of focusing on right now is how can we give existing tonic users a much more solid implementation of the channel without them having to change their code, while also supporting Google's needs for wanting to have new code gen that is different, that supports their needs like arenas and XDS and all these additional more complicated things like C++ interop. So these are things that we're working on and we've been discussing them for a while. And there is progress. Obviously the big blocker of progress has always been getting consensus and then also funding for implementation. But I think we're making some good progress there and I'm hoping that I'll be able to like announce some good progress or some actionable things that people can can know about in the future but at the moment a lot of focus has just been trying to keep tonic as stable as possible you know that's maybe why people have seen that there's not much new features being implemented in tonic and all the all the investment has been in supporting security fixes and bug fixes while also kind of working on the new side of things what this means for tonic users is that you know your current tonic code should kind continue to work, Tonic will eventually be 1.0'd and there'll be some moving around of crates and stuff. But the idea is that your current domain Tonic code that you have implemented throughout your codebase should remain the same and that in the future you'll be able to kind of drop in a new channel that will be way better than the current implementation. And what I mean by this is Google's learned a lot about how to implement gRPC transports and in general, HTTP2 based transports. And so we're hoping to take a lot of those lessons learned, a lot of those battle scars. implement them fresh in Rust, which I think is a really exciting thing. It's just a lot of work, so that's the nice part is that they're willing to invest that time into it. Very nice, that sounds very exciting. And so you mentioned existing Tonic users, but also we mentioned at the start of conversation that like a lot of the world is still doing on HTTP, most maybe developers only know HTTP, don't know GRPC yet. So let's say someone is hearing this podcast and they think this tonic thing, this gRPC thing, this sounds really exciting. I would like to start using it. What Lucio is saying makes a lot of sense. What would you recommend them to start learning how to use tonic and maybe also in the wider scope, how to use gRPC for their project using tonic. So yeah, I think there's two main great resources. Obviously, if you Google gRPC, there's a website that has some explainers. I will admit, some of the explainers are hard to kind of grasp sometimes because they kind of are tackling some heavy problems that are common in this sort of space. But it should give you a pretty good overview of what gRPC is, what protobuf is, and I think learning those things shouldn't take too much time, but it's pretty useful. Tonic itself, we've strived really hard to make it easy to get started. So we have both a Hello World and a Route Guide ⁓ example or Route Guide Guide. Both of them are guides that kind of step you through how to create a Tonic project. The difference between Hello World and Route Guide is Hello World just shows you how to do a simple request response, whereas Route Guide kind of shows you some examples of how to use the streaming stuff. So if you're new to all this, try the Hello World one. In addition, in of the examples folder in the tonic repository there's a ton of examples there and there's a hello world example that you can just copy the code you know if you combine the the guide to how to set up the the project and as well as the hello world echo example sorry the hello world echo example those combined should be enough to get you going the amount of code is pretty small So it should be pretty easy to kind of grasp and follow. And the whole idea behind having these really easy examples was just to get you started. I think once you get that code running, it feels pretty intuitive to how it works. So I would definitely start there. The tonic repository should have the links. on the main readme. And we try to do a pretty good job of trying to explain those things. we also have a, ⁓ within the Tokio Discord, which there's a link also on the tonic repository, there's a tonic channel and ⁓ there are many people that are willing to help people out if you have any questions. I see people constantly asking questions and we have a couple of really, really great people within the Tokio Discord that do a great service of helping answer those sorts of questions. Anything from beginner questions to more advanced things, there will be someone to help you. the community in general at Rust, if you haven't experienced it already, it's super friendly and super welcoming. So we try to do the same thing in the Tokio Discord. So definitely check out those resources. Thank you very much. I will also link them in the footnotes. Now, when people use Hyper, there's also stuff like Hyperutils, when people use things like Axum for HTTP, there are also like other crates which are built around Axum or which are used in coordination together with it as well as many tower crates. Let's say someone uses Tonic for their project or they like other crates that are built on top of Tonic or or used together with Tonic that you would recommend people to check out as well. Yeah, I I haven't done a survey of more third party tonic crates, but I'm sure if you go to like any sort of crate searching website and you type tonic dash something, there'll be a plethora of crates that you can choose from. Within the tonic kind of ecosystem itself that we provide, there's a couple crates. I already mentioned earlier, there's tonic web. Tonic web kind of... you because we use tower and hyper supports both HTTP two and HTTP one, I think we have the best gRPC web implementation around. and that is we have both a server side kind of tower layer that you could put in front of your gRPC server that will intercept, HTTP one gRPC requests and convert them into something that Tana can read. And then it will on the out on the like the, the, the sending end will also then know how to encode into the gRPC web protocol. Also, is also, I don't know of any other GRPC implementation that has it, but we have the GRPC web client, which can be used to talk over HTTP one for GRPC. ⁓ The reason that I don't think any other project has this is because mostly gRPC web is used to talk from browsers. like from your JavaScript implementation to maybe a Rust implementation and most users put a sort of proxy in front, whether it's the tonic web proxy within their actual server code or actually as an external proxy, say like a Kubernetes proxy that the request hits before it arrives to your actual server code. The proxy will receive an http one and return and kind of for the requests in HTTP2. So, Tonic Web is a great tool if you find yourself limited by only being able to use HTTP1, which is the reason it wasn't put in. I had that problem. So that's a great one to check out. There's a couple other crates. there's... a couple well-known kind of server client implementations, mostly around health. There's kind of a well-known health Google, know, protobuf and RPC spec for how to expose health of a GRPC server. And so we have an implementation that basically gives you a little handle that you can shove into your code to update the health of the service. ⁓ there's also a reflection code, which, which is a, another server implementation that allows clients like GRP curl to understand. what that server exposes. So this kind of goes back to that problem that you mentioned earlier where it's kind of hard to understand what my gRPC endpoint has. Well, Tonic Reflection, you can expose that as part of another service on your server and it will take in the file descriptor set that the... Protobuf compiler read and was used to generate all the code gen and it can use this to kind of return what your server speaks. So you can kind of think of this as your open API server, right? And that's what it's used for. So it's a well-known thing and there's a ton of tools that work with it. So that's another one. And then we also provide... a well-known type. So, GRPC, or circuit correction, protobuf provides a lot of well-known types. You can think of something like timestamp or empty or, I'm trying to think, there's a couple other ones that are kind of, you know, sort of like primitives. So we provide a crate as well that you can use with PROS to generate these to kind of use these well-known ones and more specifically Tonic has its own well-known ones because there's kind of some well-defined status protobuf specs and like within that because I don't know if you know this but you know a status in protobuf on gRPC can be a gRPC status code it also includes a message but also includes a details which is a binary message or whatever binary you want to throw into it normally it's used to encode a more complex status type that then you can put in there and you know can things have like validation lot error types and all that stuff so there's there's some well-defined types that you can use and those are also available as part of like the tonic pro I'm trying I forget the name but we have these defined inside the repository Okay and by the way I seem to realize only just now why actually the name Tonic. Yeah? Or you're asking why it's called tonic? Yeah, exactly. I mean, I feel we talked the entire time about Tonic and we still didn't really talk about Elephant in the Room, like why it is called Tonic. So there's actually no real reason why it's called tonic. I think the original kind of connection was because of Prost. Obviously, Prost has been heavily tied to the tonic project since the beginning. And so I think it was kind of related to Prost being like the, I believe it's like the cheersing thing in German. Feel free to correct me on that. But I think that was the connection. The reality is that tower-grpc was not gonna be the right name for what Tonic wanted to be, because we were kind of stepping away from that. And I personally am not a great creative namer of projects, but Carl, on the other hand, is extremely good at naming projects. mean... If you look at the things that he's created, all the project names are 10 out of 10. So actually Tonic exists because I went to Carl, was like, hey, I have this, we were obviously talking a lot about it. I was like, hey, what should I call this? he's like, oh, I was thinking Tonic. was like, okay, sounds good to me. Let's just go with that. There were some ideas that the transport code could be made a bit more generic, sort of like the load shedding, the load balancing and all that could be made more generic and provided as a kind of thick client on top of Hyper and Tower. And the idea was to call that gin. But that never really came to fruition. So the original idea was to have Tonic and Gin as kind of like the two projects, but yeah, that's about it. Okay, well there is a GEN project that's like in Go, which is I believe in HTTP servers. Yeah, and I actually have reserved the crate name gin for a while, which if you have a project that you wanna create with the crate gin, please feel free to reach out. I already actually gave away another project recently for someone, so I'm happy to do that. I'm not using it. Okay, that's very great to hear. And then I also, you talked several times now in our conversation about the web client and I'm always very interested in the technical details. And so you were mentioning that this allows like either via, well, usually it goes via a router if you use it from JavaScript directly, but like, how does it work exactly? Are you sending like WebSocket or you're talking about WebSocket or is it like you are sending via like a fetch API HTTP request? Like how does it work if I want to talk? from within like a native JavaScript browser client to gRPC probably going then over that proxy. Yeah, so gRPC web itself, which Tonic web implements is another spec. The idea was to try to make the gRPC protocol fit into HTTP 1.1. which has a bunch of limitations and the spec tries to kind of cover up those, work with those limitations and provide a slimmer version or slimmer feature set of what gRPC provides. So there's no WebSockets involved at all. It's all HTTP 1.1, meaning that every request takes up its own TCP connection. But the idea originally was to provide the ability for browsers to talk to... gRPC servers. So originally, the idea was that you would have some sort of gRPC web code in JavaScript that you could use in the browser. And then it would speak http one using the fetch protocol. And it would be intercepted by a gRPC web gateway proxy that would sit in front of your server code. And that proxy would essentially translate the http one code into http two code. And there's a couple kind of things here. There's the way that headers are set is slightly different. There's a couple of encoding things that, know, are very minor, but enough to warrant a different protocol. It then converts this into http2 sends that to your server, the server that replies to http2 like as a normal gRPC client would, and then that gets then converted back into http1 and the correct protocol that then the web browser can read. So it's actually not super complicated. What's really cool in Rust is that, and then again, I don't believe any of the other gRPC implementations have a native like... conversion or encoding decoding of this, they all rely on a third party like process to handle the connection as a third party external proxy would do. But what Tonic does instead is we, because we use Tower and Tower is designed to build proxies, we essentially can build a little mini proxy that receives the request coming from Hyper and converts into something that the rest of the Tonic code can read. And then we'll do the same on the other end. And in fact, the client side code that I wrote literally just takes the same logic and just inverses it for writing on the client side. And it's pretty cool, right? It worked really well and it integrates super easily because of the Tower. ecosystem. It's just a tower service really, that's all it is. Yeah and again it speaks to the genius of the tower service I really like it a lot because for me it always reminds me of a function where you have like an input and an output but kind of like abstracting it to something else where you can stack it easily whereas a function is a bit more barebone. I'm not sure how you conceptually think about it. Yeah, mean, that's the whole idea was to kind of make that mental mapping pretty easy. And you're right, like I remember when I finished writing the client side of it, because I've read the website of it a long time ago, but I needed the client side for a project at work a few years ago. And I remember trying to implement it and just. Besides getting the logic correctly for getting the header bits and all that stuff and coding, decoding done correctly, getting it integrated into the tower stack and just getting it working was really easy and everything just made a lot of sense. you're right. It goes to show a lot about how the abstractions that we do have, while they might be complicated, are flexible and do support the things that you do need at the end of the day. Exactly. then so you also mentioned, okay, this is to support HTTP 1.1 because that's kind of like what is also possible in browsers. I mean, even though it can do HTTP 2, but you said it is then one TCP socket or connection per request, but HTTP 1.1 does kind of provide pipelining. can keep it alive. So am I misunderstanding something there? Yeah, so I think the problem is that HPE 1.1 doesn't allow you to multiplex multiple requests. it doesn't let you have, I haven't looked at pipelining in a while and I vaguely remember hearing problems with pipelining. So, you know, don't take my word as full of truth here, but from what I understand is that HPE 1.1 will limit you to have only one request in flight while you wait for the response. There's no multiplexing of it. And I think you could in theory build something on top of it. It's just, the more you build on top of it, the more complicated it's gonna get, the more problems you're gonna have. So the reality is that I think that you need to have one request and response per connection. You don't need to reestablish the handshake probably, so you can keep the connection alive, but you can't have multiple requests in flight. And more specifically, if you have a server-side streaming request. So with HTTP 1.1 or gRPC Web, there's no client-side streaming like you would have in HTTP 2. So that's the only thing that you cannot do, but you can have server-side streaming and support server-side streaming, you need to have that connection available because there's no way for the client and especially the browser to be able to do the multiplexing. So it kind of creates those sorts of problems. I think there's a couple other fundamental problems with HTTP 1.1 that just make it hard to do these things. And there's a reason that HTTP 2 exists. And if you've ever tried to implement HTTP 2, it's pretty hard. So there's a reason that complexity exists. Yeah, and then how does it work from like a client? Like, let's say it's a pure JavaScript client, not your beautiful client that you just described. Is it that you still have to generate JavaScript code based on this other protocol or does it feel more like a regular HTTP fetch client? No, I think, ⁓ under the hood it uses the fetch API, I would imagine, I haven't actually used this, I'll be honest. I would imagine you have some protobuf code that defines your contract for your API. And then the protoc could generate JavaScript for you that you could just include in your code and you call it like any other function, like you would in any other gRPC. Like at end of the day, the gRPC client is supposed to be a very simple, you call a function and it just does, it returns a future that does something at the end of the day, right? does under the hood, could be calling something locally, it could be calling something remote. The idea is that that's completely abstracted away. So when you call gRPC from the web, I imagine that it's probably doing the same thing where you just have a function that you call and it does the magic under the hood, calls fetch, calls whatever it needs to do. and then I write clients, write servers, but most of all I write a of proxies. And let's say someone wants to proxy, it could either be like just purely like transferring between a client and a server or two different peers, or it could actually be like man in the middling proxying. Are there any specific things I need to be aware of when I want to proxy gRPC data? I, that's a good question actually. I believe any sort of HTTP2 proxy will work. because gRPC is just built on top of it. So anything that supports kind of like continuous mapping of streams to the correct location will work. And that's kind of a core concept of HTTP2. So I believe as long as you have a proxy that supports HTTP2, there's nothing else you need to do to get it to work. Yeah, and that's for a pure, like, let's say transport layer proxy, but let's say I want to, like, the middle of it or like anything specific there that I would have to take into consideration because I would actually like that intercept the different data blobs or I don't know, you call them messages, I suppose. Yeah, is there anything there I need to like take care of like maybe something specific to how the flow of GRPC is or how the client and the server interact like kind of like specific that is there. No. I mean, again, as long as the HTTP2 is being forwarded, there's nothing you really need to do. I if you want to do something based off of what's inside the message or the headers, you would need to have some sort of logic to parse that. I think most proxies avoid at all costs decoding and encoding the messages themselves, because usually the messages don't have any routing information, or they shouldn't. Routing information probably should be sent in the headers that the proxy could then parse out and do something with that. So I think I can't think of an example where it really makes sense to decode and encode the messages and then re-encode and decode them. So I don't think that's that big of a use case. No, for sure. It's definitely a niche. was mostly just wondering out of curiosity and I certainly don't have a use for it. And as you said yourself, you also have the fact that, you the data frames, which is the messages, but then you also have the headers, which look a lot like HTTP headers, but they're not headers, as you said. And you said in those headers, you can find a lot of information. Can you give some examples of that, of the kind of information you find in there, both in the, I guess, request and response and also can you talk a bit about is there a difference in how these headers are used between those like unary requests or the stream ones Sure, yeah, so I think in the context of the proxy, the... The benefit of the headers is that they're basic HTTP headers. So any sort of proxy or tool that allows you to, you know, intercept regular HTTP two headers and do something with them can be used with, with a GRPC. So for example, let's say you have a header that kind of gives a hint at what cert, you know, what server it wants to do proxy to, or something like that. Like, you know, maybe it's a regional hit. I want to be routed to this region. Any tool can parse that and read it and work with it because it's just straight up HTTP2. So that's really cool in itself that it just leverages that technology. The way that Bifidin Unary and Streaming uses headers, there isn't that big of a difference. The only big difference is how the status is conveyed. With a unary response, the status is kind of bundled in with the first message. But with a streaming response, the status can come at any point because a stream can end at any point. So there is a slight difference there. The formatting and the way the headers work is generally the same. Okay, very cool. In that sense, I think I covered most of what I wanted to cover. Is there anything that you wish to still plug into the podcast or things that we talked about you want to talk about more in depth or something we missed and you want to like talk about? No, I think we covered everything. Okay, will we also see you on the Tokio conference which we talked about in episode 5 with Carl? I hope so. I hope to be there. Alright, in that case I thank you very much for your time and I wish you the best of luck in the rest of your life. Elizabeth (Plabayo)
1:27:08 | 🔗
Netstack.fm is brought to you by Plabayo building secure, open, and resilient infrastructure with Rust protocols, and purpose. This show is also made possible by Rama, the open source networking framework. Plabayo offers service contracts and welcome sponsorships to keep building and supporting its ecosystem. The theme music of this podcast was composed by DJ Mailbox. If you enjoyed this episode, don't forget to subscribe on your favorite podcast platform and leave a five-star review. It really helps others discover the show. Thanks for tuning in. We'll see you next time for the next handshake.