On this page On this page
In this Protocol Shorts episode of Netstack.fm , Glen talks with Brecht Stamper about Rama, the modular Rust framework for building clients, servers, proxies, and other network systems. Brecht shares how he came to understand Rama’s core idea that everything is a service, and why that simple input to output contract makes it easier to compose complex stacks without constantly inventing new traits or glue code.
They also discuss how Rama helps with testing, including gRPC client and server logic without spinning up real HTTP or TCP infrastructure. The conversation then moves into practical production lessons, such as gRPC latency caused by TCP_NODELAY, the role of OpenTelemetry and logging, and how Rama can provide better abstractions while still allowing low level control.
Later in the episode, Glen and Brecht explore Rama’s evolving extension system, including the move toward append only extensions, clearer separation between request and connection state, and support for complex scenarios such as HTTP/2 streams and protocol upgrades.
00:00 Intro00:48 Introduction to Rama and Brecht's Role02:46 Understanding Services in Rama08:22 Testing and Mocking in Rama10:23 gRPC and TCP Configuration Issues15:35 Evolution of Extensions in Rama24:58 Building with Rama and Rama's Horizon31:52 OutroMusic for this episode was composed by Dj Mailbox. Listen to his music at https://on.soundcloud.com/4MRyPSNj8FZoVGpytj
Elizabeth (Plabayo BV)
0:16 | 🔗
This is netstack.fm, your weekly podcast about networking, Rust, and everything in between. You are listening to episode 38, our protocol short series, recorded on 20th May, 2026. In this episode, we talk with Brecht Stamper about how Rama helps developers build and test complex network systems. More simply. Let's begin. Welcome to another week of Netstable FM. Today will be another protocol shorts and with me is Brecht. He also helps to maintain Rama. We had him as a guest already in episode 23 where we talked around HTTP and using it as an application network bus. Today will be a bit different. So welcome Brecht. Yeah, we were talking recently and it got to me that you are building all kinds of very interesting stuff with Rama. given that, and also the fact that you have to maintain and evolve Rama, I came to realize that there are a couple of interesting things that we could discuss together and enlighten our listeners. So maybe before we dive into technical details. Can you talk a bit about the things you are building with Rama as well as within Rama? Yes I can. The things I'm building with Rama are mostly high throughput, network moving scenarios. It's really vague because it's not a single use case, but the biggest use case is a proxy gateway like you would expect it to be. But there's a lot of side projects I'm also doing with Rama, either related to my work or for work. And open source. On the Rama aspect, I'm mostly trying to improve the developer experience and make everything less magical and more explicit, which sounds boring, it helps in maintaining stuff making Rama easier to use for complex applications because magic was often times causing very subtle bugs which were hard to debug and slowed everything down. Glen is of the same opinion. Rather be a little bit more expressive and avoid magic but make everything clear and really scalable to however complex the application wants to be. Yeah, totally agreed. And that was indeed always the goal, but sometimes, of course, as you're trying to find abstractions or ways that people can compose network services, with the framework, you at times have accidental magic and I'm very glad that you're helping to clean it up. Now, if we go back in time and think about how you were introduced to Rama. I remember that the concept of everything is the service was a bit overwhelming, I can say, or maybe a bit too generic. Can you maybe talk a bit about how you were looking at it and then how you slowly started to understand what it is and how you understand it right now? Yes I So when I was introduced to Rama, I was told everything is a service. Even most of the file names at that point were called "service" in internal codebase which didn't help a lot. So to me, service was like trying to do everything with only one And in the past, everything was always clear. Like this is core storage, this is network layer, this is that. Suddenly everything is a service, which sounded very abstract and wrong. Service had a different meaning for me back then, but I've come to understand that service is not really an abstract idea, it's more of like a function signature that you lock in. But basically the service in Rama is nothing more than this is the input I'm expecting and this is the output I'm gonna get. Which allows you to stack it with other services and other layers which is magical at certain points as long as the input is is expected and the output also matches you can just stack everything and anything together even things you should not stack which is incredibly useful for testing things but when you think of a service in Rama I really think of a contract between input and output signatures nothing more, nothing less, which is a very simple idea, but it allows you to stack until like crazy complex applications and it stays simple, magically simple. So yeah, that's kind of how it evolved for do you remember a when was the moment that came to realize, okay, now do I understand it or did it just come very slowly and you have no idea? one point I understood the idea I kept trying to fight it because I always "For this specific use case let's make a new trait for it that describes my use case better." doing a particular request you could call it do_my_special_request which is a little nicer to expose instead of just serve. But that's opened so many issues over time, like I want to unit test this, okay I have to implement this trait. Well for service, everything and anything implements it, so as long as the contract agrees, it just plugs in the entire Rama codebase and everything around it. Every single time you do something custom, that doesn't happen You need to do a lot of work yourself in the beginning it's hard to vince people - myself included - why that is so powerful. So everyone exploring Rama, encourage you just don't to the Service idea, try to invent your own and you're gonna very quickly realize why it's so nice and how it saves so much time. Yeah. of one reason why I initially liked it many years ago is because, okay, I easily stack things compare it to functions where a function calls another function similar with services, you them. Meaning like first I do something here, then I pass the request on and eventually something will give me back a response or maybe something earlier gave a response because there was an error or whatever. But then another benefit I came to realize quite quickly was the fact that say I a web server and I have now my entire composition so like my middleware and my router my different endpoints so it's like the end system is ready there normally what I would have done in the past is okay that point I need to spawn this say bind it to a socket and I need to test it with like a real HTTP kind of means I'm also testing even really just want to test my business logic. And with the service concept, I could just send give it a request it will give me back the response kind of like how I'm just using a client. don't think that's immediately clear because it kind of ties in what you were saying that is a service and so it's hard to look past the abstraction, but it's quite beautiful the fact that a server and a client looks the same. But I wonder are there other examples like that you can think of that like didn't immediately work clear to you, but fact that everything is a service and the fact that how it composed so nicely and the fact that it's just a simple idea, like that it unlocked things that you couldn't otherwise have done. Maybe something I could have done in the past but that was always annoying in pretty much every other stack when you want to test stuff you have to mock it but mocking stuff always felt like Really hacky where you patch something you return ugly. Remember a few months ago gRPC support was added to Rama which was a huge PR and gRPC is in itself quite complex. It does a lot but you have a gRPC client and then you the service traits on the server side for the actual logic and it was crazy simple to just unit test these things, the client and the server without needing HTTP, without needing TCP, without all of the stuff. If you do this in other languages, you need to spawn a server, you need to do all of that stuff. you could just literally the client ,and instead of giving it an HTTP client do the network stuff, you could just literally pass in the gRPC server, you just made, implementing the business skipping all of the network and you instantly do all of that logic. Just put the server inside the client as the network stack and works, which was again, like one of those moments I realized, wow, this is so clean, so simple. And it allows test these things I want to test here. Of course to test the entire system. allows you to focus on each really clear, really simple. use case is in Orama we have a mock connector, which is basically a fake connector that gives you a stream. Which allows you to do any connection test that will need TCP otherwise, if you implement the test with TCP before, they're always tricky like either spawn them on port 0. You need to wait until you get a port then everything which involves a lot of Overhead is quite slow if you need it for a lot of tests it adds OS Complexity if you don't care about that like if you're implementing HTTP stuff. You don't care about TCP you care about the TCP just get yourself a client a server and connect them with a fake mock connector and focus on what that layer needs to do, which allows you really test every layer on its own very simple bringing in everything and all the things just for HTTP for are some the sound simple compared to in the past doing these things and how hacky it was or how much code you needed just works with Rama as long as input output match Everything just works, is really nice. Yeah, fully talking about gRPC, been using it quite a lot in the recent months. I remember at one point you had issues I believe it was timeouts, which was the end symptom. And your solution was basically the fact that you needed a couple of low level configs on the socket, such as TCP, no delay. Can you talk a bit about that? Because like, okay. First of all, what were you noticing? And then secondly how did you diagnose that? Because that could be quite interesting learn about then finish off by how could a framework like Rama help guide the user better or even just within its API to prevent such issues. Because I don't think most people that will use gRPC will actually know that much about TCP to even think about those things. I noticed the issue because, on all of the gRPC stuff, I have really strict timeouts Which I advise everyone to do if you expect it to be fast make sure it is fast can either crash on slow or, at the very least, log it if it's slow so you know it's And for everything I do I set up open telemetry which gives me metrics, traces, loss to debug these things And quite consistently, I was seeing above 50 milliseconds gRPC, which got me investigating Like this is a really documented issue- it's all over the place - but basically TCP, by design, doesn't send very small packets because it doesn't like which means it buffers them and there's some rules about when it does send them, but gRPC happens to be almost always small messages, which means you send a message, you almost always trigger a timeout unless you're pushing more over the same So you need to set TCP_NODELAY. Most gRPC libraries do this for you, always behind the scenes. In Rama when you use gRPC you provide your own client, so it's not set everywhere because it's useful to not set this value for normal traffic but for gRPC you do want it. that also brings an important thing with Rama. All the building blocks are there we also provide easy abstractions but it's still important that you know about all of the stuff that's there if you want to do low level stuff. So for gRPC I imagine the thing we need to do and it's still on my to do is actually document this but we also have like the easy web client we could also provide a gRPC web which does these tweaks by design or by default so other people don't need to find this the hard way like I But yeah it's really documented it's all over the internet so when I was searching: Why is my gRPC Google just gave me 50 pages with all the same No, it wasn't opens the door but Rama is really you really need to tweak all of the low level stuff as well. TCP Linux stuff, there's a lot But yeah, when you make a new stuff, definitely set up open telemetry or at least logging so you see what's happening under the hood. Yeah, and of course both are integrated within the Rama And I agree, like you can do all the low level stuff, you can control whatever layer of the network you want. That said, as you also ended yourself, we do provide abstractions at various layers. And for most users, I don't think they will need to tweak this. And so I think next to documenting for sure. We need to also provide an abstraction there similar to how for say - users need to do simple API calls. There is the easy web client, like you mentioned. I think it is right to also provide something like that, like a pre-built transport client either there or somehow provide that support within the existing client to make it easier because yeah we only want them to have to worry about the network layer if they really need to of course otherwise doesn't make any sense That said, this is only like a client issue or that's also like on the server. And then next to that you mentioned that most gRPC implementations ship the support built-in like they have like just a single API and you get your client or whatever then how come they still have the issue? as well or least some of them. they don't really have the issue anymore, but at one point the issue was there. Someone opened an issue for it and it got fixed. The places where it still happens is more lower level libraries where they're explicitly documented. Hey, we don't set this for you for these or these reasons, under almost all circumstances, you do want to set this. Some examples of where you don't want to send this if you're sending this over a high latency networks that's pretty far and you don't really care about instant gRPC responses but you do want to send them in batches which is almost never the use case but libraries do give you the option in that case Okay perfect and then we started this episode around the service trade and it went through a very big evolution. I mean we didn't invent the concept to begin with it I believe library if I remember correctly and then it was within Towers in Rust and then we kind of took it from there first tower-sync a library we made and then finally at Rama we have the current version but even then we had in the beginning not just an input but also a which was meant for like all kind of utilities static states we to the conclusion it was the wrong abstraction, we removed Then we still had the fact, okay, we do need dynamic state, we have extensions. yeah, to begin. What are extensions, Brecht ? And where do they live, why do we even want them? Okay, so let's maybe start at where extensions This is very similar if you've worked with other pieces of Rust are mostly when you need dynamic content. is often the case in a lot of different things if you cannot predict that compile time what everyone will need So the way it basically works in most cases is you have a hash map as the keys, you use TypeId of a generic object. When you do TypeId, Rust gives you a unique ID which you can then use to like add runtime, query this in a hash map. Which is also how we did it in the beginning. But the problem we noticed over time with a hash map is that people insert items, they remove items and it becomes really hard to see what's happening under the hood. How it got there, were leaks. different interactions of all that stuff. So trying to fix this issue, which is an issue that happens all over the codebase and that's also really hard to designed a new system where basically extensions become a vector that would be append-only. So you could kind of see as a history of extensions being inserted and you can never remove an extension. You could use logic for example put it to this item is used, you should not use it anymore. But it's always there leaving a paper trail of what happened. So it's also a really nice debug tool to see what happened, where did it happen, in what order. At that point we had a really simple vector which needed mutable reference. That was kind of working. we kind of moved it all over the place so when we went from a request to a connection we just copied everything over same in other places which was working for simple use cases but for the more complex stacks leaked a lot of information by leaking I mean transferred extensions from the request to a connection then when you used a connection pool that connection still contained extensions from the original request under cases happened to introduce bugs. All of that and was really only meant as an intermediate step. So the next evolution we have a structure that is append-only and read-only for the other parts means we should be able to optimize of using a normal got us to the append-only It's basically a data structure that can grow and where you can insert items only access to self, no mutable reference needed. Because we could use the properties of like, we can only insert items at the back and never modify items that are already in there. By changing to reference instead of mutable reference, we could edit extensions from multiple places at once, which sounds like an anti-pattern in Rust, but is actually needed if you happen to have logic scattered in multiple places, which happens a lot for especially for connection user would want to see what the extensions of connections are. For example, is it broken? Is it healthy? But the connection would still need to be able to edit these. So by relying on that append-only structure with only reference, we could do that and it's already in use and it's working great so far. For the people that are already using it, if there's feedback, please open issues or message us on Discord. And we're happy to look at those. But that got us to append-only VEC. latest transition was really splitting up extensions between requests and connections. they're really seen as distinct extensions now. So when a request goes through a connector stack, we don't copy. the request extensions to the connector, we only copy extensions that should become part of the connection. If you say I want to connect only to this IP, this IP now becomes part of that connection because it is fundamentally how the connection is used. Other examples are which TLS version is being used. Once a connection is established, that should be part of the connection. If you reuse that connection again, the same information applies. If you put a retry policy on the request, we don't copy it over to the connection because a follow-up request might not want to use that. So they're really distinct. They also fork at logical places, meaning when you have a connection, extensions are passed through. If you go from TCP to TLS, we reuse the same extension because logically it's still the same However, when you do multiplexing, HTTP2 does it's using a stream, we fork extensions so they have a new storage. They still keep the original extensions as parent so all the stuff that's in there still applies to a stream but a stream has also its own to stream specific storage things. Examples there are it is possible that a stream is broken but the connection is not broken so in that case we insert connection is broken on the stream but not on the connection allows us to do a lot of complex things. and this is also made to work around upgrades because upgrades are complex items. If you do an upgrade on HTTP 1 you kind of replace the HTTP with the upgrade you just did. Nice, this is easy. But if you upgrade over HTTP 2 you're basically nesting a completely new connection inside a stream which could be doing TLS again, which could be doing a lot of other things again. And things inside that connection don't really affect the outer connection. so they should be isolated but they should still have access to the underlying connection because they do still run over that This system allows that. It works "magically" is kind of the word which is something we're removing but here it is needed to like just be transparent about the network but it also gives you all the building blocks to go as crazy as you want with this. Yeah, part of the complexity or the difficult program space is of course because Rama is a modular framework it gives you all these building blocks but you can compose how you want within the bounds of contracts as specified by network specifications of course otherwise you will get other issues But as long as you respect the different specifications or the world operates, you can make up whatever stack you want Rama makes it pretty easy. In contrast to that, you would need to otherwise even write all the network and protocol code yourself, which I don't think anybody does. Instead what most people would do is they either go for like a black-box solution where they can just use a config file or something, if that at all, or they would glue together all kind of random libraries, but of course plenty of mistakes and imperfections get made while trying to glue these things. Instead, Rama has all these building blocks, they work nicely together, you can compose them, but because we cannot predict how the user will compose them, we of course don't want put them into a surprise scenario where they compose it somehow. Things happen with these extensions because how they passed around and now they get into these situations which, like you mentioned, we had. Because you could remove extensions and... maybe then it no longer contain information it should have contained or it shouldn't have and got pretty I'm pretty happy that that's where we are now as I believe right now you can stack without any fear and I certainly didn't see any surprise anymore since since we arrived at this new design not sure how you see it evolving from here Brecht I think the extension stuff is mostly finished, core logic, we do have a lot of other things that are using extensions, mainly around TLS special builders. these boring builders, were designed before this new extension logic was there, so they kind of solved this problem on their own in a thing that worked at that point of time, but it's quite a complex struct, that does quite a lot of weird things first time you see so the way I see it evolving is now rely on the new extension logic, really rely on the features it offers and simplify all the users of the extension logic by relying on what extensions offer. But yeah, that's mainly the extensions set Okay, very cool. And then run off this episode, you've been now not like a maintainer of Rama, you're already that for quite a while. You've been using Rama now already at several companies as well as, stuff that you do in your free time, sometimes related to work sometimes Can you think of any story like, okay, I'm using Rama now here and it really helped me to unlock something that otherwise would be pretty difficult if not impossible to do. is a big word yeah nothing is impossible. It's just it might be impossible in the required time that's where Rama shines so much like I've been surprised so many times where I implemented like Complex proxy stack or a complex network stack so little time and it just worked like very little time and When I do all of these things I always add tests as Adding these tests is so simple with Rama as All of the core stuff it just works which is nice. could do this without Rama. Let's say I stay in the Rust ecosystem. I would have used Tower, Hyper and Tonic probably. But when I combine all pieces there's always so much glue I need to write myself. Like huge amount of glue. Rama solves this by being a monorepo basically that just has one standard way of doing everything. It's applied very consistently everywhere and all pieces just work together. Like if you use gRPC, okay you're kind of in HTTP world, you can use all the HTTP stuff. The same kind of applies in tower world but the tower ecosystem is kind of stuck on an older Rust version which makes this way more painful. It's also way less consistent and core logic is there but all of the stuff that combines it is not and it's often there that the most subtle bugs exist and where you lose a huge amount of time so yeah is just so nice it combines everything Okay, very cool. And where do you see future going? do you see Rama evolving in the future and what do you expect from it yourself, both as a maintainer as well as user of it? In the short term I mostly see Rama evolving on a feature set. Like there's some really nice things that we still need to add to Rama. Perfect example of that is HTTP 3 and Quick that unlocks so many new use cases again. Like once you have that you can go really crazy. Like you can already go really crazy but that's like fundamental In the modern world that's still missing. So short term I really see Rama adding more features or more developer tools to make things easier, make things better to work with, to debug. Recently, Dial 9 was added which again will help a huge amount to debug these crazy OS issues or Tokyo issues where something is stalled that you never expected So short term feature wise, then long term I really see us focusing on speed. performance, throughput and just maxing out hardware easily. the nice thing is, if you just update Rama, you will get all of this for free basically, which has been the case in a lot of places where I use Rama. Every update, I can remove a lot of special workarounds and stuff I have in code just because right now Rama has solved this automatically for me. and it just works probably better than the stuff I have. That doesn't mean I don't need logic anymore, but I can rely way more on the core logic, which also means if I needed an other project, I can just do that without having to reinvent everything. Yeah, exactly. Because anyway, might be able to do it, but it distracts you from the business logic. A lot of subtle things can go wrong. And then everybody's basically solving the same books if they find them at all. Because there are so many ways to run something. I keep getting surprised by it is because right now, Rama is getting used to more and more different scenarios, different platforms, different use cases. The amount of books that come in is very little, but when they come in, they're very interesting always. And it solves Wong and then, like you said, Wong that solved, you do an update and just get it to fix for free, which you wouldn't really get. Otherwise, because you also mentioned, I had this TCP new delay issue. Okay. I had to look it up. You see in your Google results, because you're using Google, like I'm getting 50 pages of people getting the same issue. And that's the kind of thing that you would get for thousands and thousands of different issues. If you need to start. building yourself with these low level components, gluing it yourself together, trying to figure out how to compose this, how to work with this. And that was always a goal to solve that giving someone the to program their own stack because you do want to program don't to resort to some silly YAML file or something like that where you're pseudo programming config language, happen sometimes. But at the same time, you don't wanna to program everything yourself either. So I'm very, very happy that it's working do have another example of something I didn't think about but just noticed that you fixed kind of recently is the system resolver in Linux is just blocking by if you want to do it properly you need to use spawn_blocking in Tokyo but if you have a high-throughput proxy for example that can easily run out of blocking threads But Rama, if you use a system resolver, it has a small cache implemented for you to work around this problem. This is one of these things you don't think about when developing these types of applications that you need to think about the DNS resolver being slow on your system or not slow expensive to call basically. I very much And of course, when things do go wrong or things can be improved, we always accept contributions. Everybody developing network services might have opinions or ideas or might notice things to highlight again, we will just have to fix it once and we all benefit from it. very much a community project also very happy that you are part of it, Brecht, and thank you for on the project today here in the podcast, as well as explanation about how you got here and how you came to understand Elizabeth (Plabayo)
31:53 | 🔗
Netstack.fm is brought to you by Rama, an open source framework for moving and transforming network packets. Rama is built and maintained by Plabayo a company focused on secure, open, and resilient infrastructure with rust, protocols, and purpose. The theme music of this podcast was composed by DJ Mailbox. For more conversations like this, subscribe so you don't miss what's coming next. And if you know someone who could benefit from this episode, share it with them. They might appreciate have experience in protocols, networking, or infrastructure, and want to share your work, your ideas, or experience, we would love to hear from you. Reach out at hello@netstack.fm. Thank you for being here. See you next time for the next handshake.