On this page On this page
You can find a remastered version of this episode as episode 29 (https://netstack.fm/#episode-29 ) for an improved listening experience with better audio quality. Released on the 3rd of March 2026. Please listen to that episode instead of this one.
In this episode, Glen interviews Sean McArthur, the creator and maintainer of the Hyper ecosystem. They discuss Sean's journey in software engineering, the evolution of Rust and asynchronous programming, and the growth of Hyper from its inception at Mozilla to its current status in the web development landscape. Sean shares insights on the creation of hyper, hyper-util, http, headers, the Warp framework, and the challenges of integrating HTTP/3 and QUIC. The conversation also touches on collaboration with cURL, the FFI layer, and Sean's aspirations for the future of Hyper and the broader ecosystem.
Learn more about Sean McArthur, Hyper and Warp:
If you like this podcast you might also like our modular network framework in Rust: https://ramaproxy.org
00:00 00:45 01:48 06:17 07:54 13:20 15:32 17:32 22:40 22:47 26:43 28:54 30:17 31:31 34:35 38:50 40:16 41:23 42:28 43:57
Music for this episode was composed by Dj Mailbox. Listen to his music at https://on.soundcloud.com/4MRyPSNj8FZoVGpytj .
Elizabeth (Plabayo)
0:18 | 🔗
This is netstack.fm, your weekly podcast about networking, Rust, and everything in between. You are listening to episode two, recorded on the 22nd of August, 2025, where Glen has a conversation with Sean McArthur, creator and maintainer of the foundational hyper ecosystem with a rich history. Welcome to the podcast. Today with me in our virtual studio is Sean. Not everybody knows him yet, but today we're going to introduce him. We will also talk a bit about the hyper ecosystem, about the past, the present and so on. so welcome Sean. Yeah, Thanks for inviting me. thank you for accepting it. So before we're to dive into the technicalities, it would be nice if you could introduce yourself a bit, Sure. I've been in software engineering for, 18 years, something like that. I actually started with mobile games, not like iOS, iPhone games, small games you'd play on your flip phones. And very quickly transitioned into web development, did web development for a little while. got into Rust and I've been doing Rust for 11 years I guess just mostly around the and web ecosystem, So that meant that you were in the pre-1.0 days and I think even in those days, Tokio was, pretty much still starting. I think it was there, but it was very young. And even the Async story in Rust was young. how did you experience those days? I did start pre 1.0 back when there was pointer sigils, the add symbol and the tilde and things. But those went away very quickly when I started getting into it. When I got into HTTP at first it was just all synchronous. Well, no, it wasn't synchronous I.O. Because pre was lib green and what was the other one? Lib metal? Maybe there was like threaded IO and then regular IO. So I didn't worry about async IO at first. And then when 1.0 came out, got rid lib green idea and everything was synchronous, which was fine. eventually you want async IO to scale. Tokio wasn't there yet. I think the first thing I used was rotor. It's this. machine basically define callbacks on a trait and return some enums It was fine, very quickly Carl and Alex reached out. They were working on Tokio and that Hyper could make use of it. And can you explain a bit how come they reach out to you? Was it specifically to you or there was like a public call? No, they reached out specifically to me. was working on Hyper and I'd bump into Alex. were both at Mozilla. And I'd gotten Hyper to work in Servo. it seemed like it was the major HTTP implementation at the time. So it was one of the driving reasons to have async I/O. Yeah, there's other reasons, of course, but that's like the... The shiniest one, Very interesting. I wasn't aware that Hyper started as Mozilla. Was one of those official Mozilla projects or was it one of those things you can do a bit of it on your own time and a bit on the company time? Yeah, it was a scratch your own itch thing. It wasn't ever my official job to work on. Okay, on your About page you say that Rust is the least bad option. Can you elaborate a bit on that? I love Rust. I think Rust solves a ton of problems, but I also don't want to be stuck when something better comes along, which it inevitably will. Then yeah, I'll move to me, all these tools are, a means to an end, which is to make better software. And I think Rust does a great job at that. But I look forward to when things are even better, certainly. Yeah, very pragmatic. I'm kind of in same boat. So that being said, some folks, they think there's a hidden language within Rust and others want more and more of Rust. they wanted the scope to grow. Where are you in that camp? I think a lot of complexity to it. There was this idea that, I think it was Steve Klawnick that mentioned it. It was the, what is it, the weirdness budget, strangeness budget. I think Rust has used most of it. Doing even more weirder things alienates people. So also think that the design that we have so far allows for a lot things. I don't think we need to make things more complicated in order to make software better. And then given your background and given your main focus on what you work on with Rust, how do you see the asynchronous story and are you happy of it so far? I'm very happy where it is now versus where it was. you had a lot of callbacks. You didn't have async await. So you had everything was and then, and map, combinators. Also, JavaScript had this for a while. what they call it, callback Things are better now. There's still things about it that are not, like there's parts of the language you can't use if you're using async, which is unfortunate. We recently got async, fn, and trait, but it has caveats that it can't send or sync traits just right. So if you want those and you need to use, you can't use async, fn, and trait, you need to return an impl trait instead. But if you have associated types, then you can't use impl trait for instance. like if you were to take the service trait, and you wanted to say, hey, want to return an async block instead. I want to return an input trait. You can't because you can't put that in the associated type slot. ⁓ Yeah, so that's something you're waiting for, I suppose, or expecting to be Yeah, I that's something where I don't think that it necessarily makes things more complicated. I think that just smooths out the, hey, if you already have async in the language, then it should work where you expect it, I guess. Okay, if we step a bit back of what you were earlier saying in our conversation, you were starting Hyper at Mozilla. Is it also because of that effect that you were at Mozilla that you got into Rust? ⁓ Well, I was already at Mozilla, but I was working on something unrelated to Rust. And the team that I was working on didn't use Rust at all. I was working on Firefox accounts where just everything was in I was just slowly getting tired of the dynamic nature of Javascript and especially writing 50,000, 100,000 lines of code in JavaScript trying to keep that straight. Flow and TypeScript were just coming out, but not enough that we had embraced it. So I was just looking into what new languages were coming out and I was looking at Rust and Go. tried both out and found that I preferred Rust a lot. And it just so happened that if I had questions, I could go and the same company and talk to them at Mozilla All Hands. So that was nice, but I don't think it was what moved me. were you in general experimenting with languages throughout your career? yeah, I've tried quite a yeah, I've done Java originally when I was doing games and then kind PHP and Python and JavaScript. and I felt like it was time to try a different one. I guess I did like that Mozilla backed it. Not, so much because I at Mozilla myself, but more because I felt like they were at the time they were, you know, the reputation was stellar. The, were a fantastic steward of cool internet things. So that was one additional like, oh cool Mozilla is taking care of this site, I can probably trust it. so started and in the meanwhile, we are many years further. That means there's a lot of baggage, a lot of history. can you maybe go a bit through the history, how Hyper evolved from then and how it is now? Sure. it started because when I got into Rust, the first thing I did was I think I made a simple parser for localization or something just to learn some of the language. But then I was like, well, I mean, I make web servers. I want to make a web server. Where's the HTTP stuff? Did you implement those protocols before? No, I've never implemented HTTP before. I used it plenty, mostly just as someone writing web servers, not as someone who cares about the finer details of HTTP 1.1 and 2 and so forth. So that was an additional fun thing to learn, also reminded that many people say HTTP is simple and most implementers of HTTP realize it's certainly not. Yeah, there is a lot of ambiguity in these protocols and these RFCs and lot of details and how the wild is implementing things slightly different, you could say buggy or even wrong, but it could like accept as the standard, et cetera. yeah, have a lot of, oh, this is what the spec says, but then the real world, this is what actually happens. know, have de facto rules that you need to support in order to stay safe. Or, you know, for people to use your implementation, it needs to work with most of the internet. Yeah, Especially if you have a client and a server. okay, so you were started with I want to have a server and there was nothing HTTP. And you started writing that. I suppose that was still just HTTP 1 at that point. Yeah, and very basic HTTP 1 There was the Rust team at Mozilla was also working on a browser, experimental browser called Servo. It got picked back up again not too long ago. That was the next step was to meet with them and say, hey, I've been working on an HTTP library. How do I make it so that Servo can download web pages? And so that pushed the client implementation and I worked with them and that was pretty exciting to do. It also helped me to realize that was kind of the first time I was presented with, okay, they should be specs say this, but browsers have to do this. And how do you balance which one is right? I suppose it's a bit of an impossible balance. how did it move from there? we had Rust 1.0. I don't remember if that happened before fitting in servo or not. then we started working on async stuff. shortly thereafter. and trying to fit futures into hyper, there was hyper zero point eight or nine or so is when we added async support with Tokio. And I was using future 0.1 that was when futures were kind of what a promise is in JavaScript. It was like ⁓ a channel, a little slot of memory, you'd synchronize on sending things to it. But that meant that we allocated with every single future and every combinator also had it to allocate to. I don't remember who it was. It was Alex or Aaron or Carl or all together. They were able to get it to be a trait. And then combinators were basically free. And the expensive part was if you needed to allocate because you needed to share a slot of memory, But you had to pay for that anyways. It wasn't a cost of doing futures. And so they were working on that and reached out. We made hyper start working with future 0.1 That's very interesting. then despite them reaching out quite early to you and you seem to be tied quite closely to Tokio, even though I think Hyper can also be run without Tokio, if I'm not wrong, it is still in its own organization, right? hyper-rs Was it always a deliberate choice or just an historic artifact? Yeah, it's always been a deliberate choice. I just went back and checked and found, yeah, I have some blog posts about async-hyper from 2016. So that's when that started happening. there was a moment where hyper was more closely linked with Tokio As part of Hyper 1.0, which we two years ago, I put forth effort to remove the parts of Tokio that were still in Hyper. But even before then, Hyper was always generic over anything that implemented the Tokio I/O trait which did mean that people were able to hook in other runtimes just fine. They just had to implement those traits, which kind of felt a little weird. So I, I cleaned up, ⁓ You can use hyper with anything and it has been used with all sorts of runtimes Does that mean that some people are also using it with some of these io_uring runtimes? I've seen a couple of blog posts. There was one, what, last night or something. Trying one out. I know the I-O, the way I-O works is just a little different. And that was a consideration that we made in Hyper 1.0 to the traits since they're owned by Hyper. The idea is that we could add new functions that deal with like owned buffers instead of borrow buffers. to be able to integrate with io_uring when that makes sense. you can integrate right now just by copying the buffers, but then you kind of defeat the whole purpose of using io_uring in the first place. At what point of this journey did you realize that Hyper wasn't just a project, but was really something that you might be carrying forward long term? I don't know if there was a light bulb moment. I think it just kind of started to happen, especially right around the beginning of the pandemic. I changed jobs from a startup to working at Amazon. And the reason that I was hired, like what I was asked to continue to work on was hyper. not some internal Amazon software. And I think that's what made me see, this is something that can live outside of any particular company with a bunch of people contributing. that must be pretty fulfilling because by that time you already were working on it quite a long time probably also plenty of your free time so how did you feel about that when you got hired there and did it meet the expectation? Yeah, it was very useful. It was exceptionally useful to be there for some time because I was able to meet with teams inside Amazon that were building really cool things. And they, for the longest while, have been doing things in Java. And Rust was starting to become more more attractive, and they were building new versions of things, new products entirely. and making use of Rust and they're like, hey, so yeah, we're using Tokio, we're using Hyper, we're Tonic or whatever. you know, this is the scale that we're running at and we realized that we wish that it were a little faster in this way, or we wish that it, you know, could handle this less common thing. It was phenomenal to be able to meet people that were using it at such scale. And were you aware before already that they were using Hyper? I wasn't aware to the extent of how it was used. I just, you know, had heard that, like, you you go to RustConf or something and, yeah, you know, I'm at this company and we've been using it for this thing. And like, that's kind of all you hear. Or maybe you get an issue filed on the issue tracker and you're like, ⁓ you check their profile. Well, it looks like they're at that company. Maybe they're using it. Maybe it's just a hobby project, who knows. And then at some point you also created the hyper-util crate At what point did that split off from Hyper itself and why? So I wanted to stabilize Hyper 1.0. There was a Various people had told me, know, like, it being under 1.0 can give a little bit of, oh, I don't know if I should use this in production. I don't know how stable the ecosystem is. Hyper could be an example to others of like, oh, well, if it can stay unstable forever, so can I. So I had quite a bit of push to make things stable. I mean, it makes sense. All the reasons make sense. The reason it hadn't is because there were some parts that just didn't feel right. Some of the designs felt unstable. They felt like it wasn't sure it was the right way to do it. Now, that was some of the things around the client, like the way the client, the higher level client would work. and also the way some of the server listening traits worked. So the way I solved that was I settled on, the core pieces of Hyper, the way it manages just a single connection, that's pretty solid. That doesn't really need to change. That can be Hyper 1.0. And the higher level stuff like the pool and DNS and all that stuff that we're not sure it's stable yet, that can go into Hyper-Util. We did something similar with Tokio. is Tokio 1.0 and then there was a bunch of extra things like codec and the encoder decoder thing. And then it was like, well, if that exactly the way it should be, I don't know. Let's just put it in Tokio Util and let people use it. Yeah, indeed. I mean, I suppose it's also because the entire streaming and async iterator and all those things are still a bit immature and in development Yeah. So that stuff was pulled out into HyperUtil as a kind of exploration grounds. And as things become stable, there's no reason it can't come back into Hyper 1.0. Like, once we've figured out, this is the way to connection pool in Client, and we're never going to change it, then it can just appear in Hyper directly. Yeah, that makes sense. And then you also have other crates like HTTP for most of your basic primitive types. then you have also headers, which is a typed implementation of some headers. How did those came to be? Did they also just spawn out of Hyper or was there a bigger story to it? both of those originally were in hyper itself. as we were working on hyper, I was working with Carl and Alex and we're making things async and, we were trying to build out this ecosystem of futures and, services and HTTP and TLS And we wanted people to be able to say, like what web server frameworks to be able to say hey my application works with a service of HTTP. And I don't care the implementation. It could be hyper or it could be some other implementation. It doesn't matter. Kind of like how how with Java you have Circlets or. What is it in Python you have whiskey? ⁓ That was kind of the idea behind it. So we took some of these primitives out of hyper and turned them into the HTTP crate So that way you could set, could just depend on the HTTP types and it wasn't you depending on hyper. That was the idea behind it. It's been used a lot of different crates, as you said, not just by crates that use Hyper, but also many other stuff. How do you balance between maintainability and moving forward? I feel those crates are getting maintained less than others. some of them, just don't need as much maintenance. Like for instance, the HTTP crate itself. There's not too much that I feel that needs to be added. Maybe like a request method gets registered like query. We can add that. But to those types, there's not much to change. There is the headers create, which so. That originally was something that was part of hyper-directly. We had typed headers, and eventually that got moved out. We put just a regular kind of string-based header map into HTTP, which was more flexible. It meant less breaking changes as types would need to change. And headers became its own thing. The motivation I had behind it when I moved that stuff out of hyper was, well, first of all, let me put it somewhere because we're doing just regular header map and HTTP and, maybe come up with a way to allow new types to be added without breaking people. That was some, that was a problem with the type headers before is that we had added an implementation of something complicated. ⁓ say content location or I don't remember some of the other ones that, know, they'd have a bunch of types of them, web socket extensions, for instance. And, you know, we'd put up something basic and then someone would show up and say, yeah, but it doesn't allow us to do this, this other part of the spec that, you know, you didn't notice. Well, okay, in order to change that, you know, it'd be a breaking change and We didn't want those breaking changes to affect all of hyper. So that was another reason it got moved into another crate. And then added in basic functionality for the simplest of types. And then it struggled to find time in my priority to figure out a way to add in much more complicated types in a way that we don't have those breaking changes like we did before. Hmm. you're definitely right because despite how useful they are, as you grow that library with more and more headers, where each header can have quite a lot of complexity. mean, despite the fact that a lot of different RFCs do share the same way how you encode and decode certain things or put data together, still there's a lot of nuance to it. And I suppose there will also be always things where you need to consider like you want to support those kind of weird maybe incorrect ways but still widely used ways. So I suppose it's a kind of crate which will be like, yeah, not something you can stabilize very soon. But then again, I don't know what the other option is because right now it's one of those crates which I do think has a lot of use. I find a pretty great way to... to deal with HTTP data. I don't think it belongs in the HTTP crate but it's a very neat way where you in a type safe way can access that rich data, imagine if it's not there, then you as a user, need to manually parse. Like let's say you're an Axum user, you would need to manually parse those data types yourself, which can be pretty like error prone. But then, like, I feel it's bit stale now. how do you see that evolving? how do you deal with that stalemate Some of it was making it so that it was easier for other people to implement the header itself. It used to be like a weirder header when it was in Hyper. So now sometimes when people submit a pull request and I really want this complicated header, I think the first step is, OK, ⁓ implement it in your own library or in your own application and you can immediately use And then hopefully, if you use it in more complicated ways, you might refine it yourself. And then eventually, OK, I've been using this for six months or a year or something, and I think it handles all the use cases. That's one of the easiest ways forward is you're not blocked, but it also allows it to be used. at the beginning of our conversation, you were saying that you were interested because you want to like, okay, how do I serve some like website or some, I want to have a service. And I also know that you are the author of your own framework called Warp. Can you tell a bit about that history quickly? what's your philosophy behind Warp, Yeah, so one of the important pieces when I was working on it was I didn't want to make something that was like everything else but just mine, I didn't want to say, this is Axum, but Sean's Axum or something. Did Axum already exist when you started it? No, it didn't. There was some others. There was a rocket and iron and I think Actix and a few others. another reason why I really want to get into it was I'm very interested in making the type system do as much work for us as possible, kind of like the type headers. But even more so, using the type system for routing. I also didn't like how many times in many web frameworks, not just on Rust, in most web frameworks I've used, there's a special thing around a method or the path. The path calls a method, and then everything else is different. And so with warp, everything is a filter, and everything always just extracts a thing. And it doesn't matter if it's a path parameter or a header or the body. They all behave the same. And the type system also allows you to make sure you didn't accidentally hook up the wrong kind of handler because it's enforced through arguments. There was a lot of influence from Scala frameworks, ACA, HTTP, There was Finch that I was taking a lot of inspiration from. so I wanted something that felt more like that, that push as much in the dive system as possible. Okay, given there are all these frameworks on top of Hyper, will that mean that Hyper will always stay focused and slim at its core? Or do you see that maybe one day you also want some kind of richer, easier way to make a server directly with Hyper? I wouldn't want to add anything like routing or any of that, don't think. Maybe adding some sort of accept layer, like there used to be. There used to be a way, like an accept trait, where you could accept connections and turn them into HTTP connections. That moved out to hyper-util because the trait was lacking. I could see something like that coming back. But prefer allowing people to develop different web server frameworks on top of it instead of me trying to pull something in. And warp, is that something that is for you a hobby project, or there's also a professional purpose there? I was that was entirely just for my my interest. No one was asking me to make it. It was some of it was to push how we handled. like H lists and parsing things of different types and pushing them into functions. And it did help. Some of that logic got pulled into Actix and pulled into Axum. And it's now how they have different extractors. A lot of that was explored in warp originally. So I'm glad that it was able to develop some ideas. I still would love to explore more. I think that we're underutilizing. trait and impl trait in particular. It's kind of annoying to use sometimes. So people prefer specific structs. But I think there's advantages that are underexplored. And I use things like warp and a couple of other crates that I don't really talk about as areas to test out these ideas before suggesting that other people should use them. And so while even HTTP1 wasn't as simple, like one should implement this kind of spec and well, collection of specifications, you realize even HTTP1 is not simple. Still in those days, it was still like at least contained within the HTTP realm. But I feel that since we moved to HTTP2 and also HTTP3, you also have quite a lot of... inner connections with other layers of the network stack such as DNS and especially TLS. But as far as I know, things like warp or Axum they don't really deal with those layers or don't really have direct support for them. Despite the fact that for example in TLS you can negotiate protocols with ALPN or in DNS you can have SVCB records or HTTPS records. how do you deal with that? and So I think those sorts of things are rightly not in Hyper because they're HTTP-adjacent, but they're not HTTP itself. And I like that different implementations can exist. There is native TLS, and there's Rustls and there's KTLS. I like that there's that choice. And by me not picking one for Hyper, you can use whichever one you want. That said, I know that many people, show up and they don't want the choice. They don't want to think about it. They're like, can't you just pick a good option for me? And I think that's where reqwest comes in or other libraries at a similar level, where it says, yes, I behave kind of like hyper, but I picked a bunch of things for you that I think are good for 80 % of the people. And if it's not good for your 20 % use case, that's OK. You can go use the building blocks and make your own thing. if we go back to earlier in conversation, you were saying, okay, hyper grew a bit. also more and more because it can web requests for Servo and as Servo, I think a year ago or a bit more got picked up again, are they using still Hyper or what is the story there? I have not checked in on how, on what Servo is doing at all. I just saw the news that it started, they started working on it again. You made some blog posts around it, how you want to make it more modular. At some point, even the building blocks are going to be opinionated. So is there some way you guide yourself in those decisions? Yeah. I mean reqwest, it's becoming more and more the thing that people use as a client, less so hyper, unless they're doing something where they want very fine grain control. So it does need to support more possibilities. But I also feel like it's reqwest's job to make sure that those possibilities are still among the best options to choose from. A lot of my recent work has been when people want most things in requests, but then they want something quite different that most other people wouldn't want. Those people I still want to serve by doing the kind of work I'm doing right now, which is to take a bunch of the pieces that are in request and try and turn them into things that go into hyper-util or a tower-http where someone could then basically rebuild the request stack, but add in those pieces instead. You're right that there is some opinions built into the building blocks, but I think that's where people can express their different opinions instead much more easily. Very cool. as you're also trying find sponsors and people commercial folks that use your project and also pay you perhaps for service contracts, is that more for hyper or is there also reqwest work in there? Yeah, it's all All the accept sponsors, but I also look for customers that are using various repos that I maintain, even ones that I don't. And then I provide advice for how they could do something different. It also allows me to some of the that I have that they're using requests in a certain that helps me to prioritize. Okay, well, this is something that, you know, I have hundreds of feature requests. How do I prioritize which ones which ones are important, but which ones can I definitely say are needed? And when I see a company with a specific customer need, you know, they're, dealing with users on their, in their product. it's easier to see, okay, well, this is something that I can prioritize as opposed to an issue from someone with a username maybe I don't recognize or don't even know how it's going to be used. It's just a lot harder to a around it. That's very reasonable. And then I know you have I suppose just like everybody else, experimental support for H3 and QUIC. how do you see that getting integrated into something like reqwest or even later in Hyper? Because with HTTP1 and HTTP2, those were still kind of like easily combinable. They both live in the TCP layer. You can even have like an "auto" one and once you should combine them with TLS et cetera. Okay. people can figure out how do you see it happening with HTTP 3 added to that as it's living on entirely different sockets. Yeah, that's what's been so complicated about it is that hyper's philosophy so far has been it doesn't want to care where the IO comes we define these read and write traits, which work for HTTP 1 and And you can provide a TCP stream or a Unix domain socket or who knows what. Sure, yeah, whatever you want. can be a vector even. Hmm or Bluetooth or whatever you you know, the Tokio's duplex stream. But with HTTP 3, quick is different. Like, yes, it is kind of reading and writing bytes, but there's also control streams that you have to deal with and additional error codes that mean things. So it's been more complicated to allow Hyper to say, you can provide me a QUIC implementation. I don't care where you got reqwest we do have experimental support because reqwest doesn't have an idea of being generic over the IO. reqwest is already chosen. It's going to use TCP streams. And it can also choose a quic implementation Is that purely based on the version or is there also something where you say we're going to check for a DNS record or something like that? Yeah, it doesn't yet use DNS records. It'd be nice to. Right now, the experimental support, have to tell it that there's what we call HTTP3 prior knowledge. You're basically saying, I know for sure this other end uses HTTP3, so use that instead of H1 or 2. And that's because of the version you were specifying your request? No it's on the client and the client builder, you just specify like http3 prior knowledge is true. and then all your requests will be HTTP 3 basically. I believe that's how it works. I don't recall if we changed that recently. that's more possible in reqwest because we take full control over most of the implementation. With hyper, there is, I put up a two pull requests. They're not ready by any means, but they provide a new set of traits that allow someone else to say to describe their quic implementation. They can say, this is how a quick connection works, and these are the receive and send streams. And then there's a second pull request that is very much work in progress that would take those and then map them to the H3 create, kind of like how we are able to do H1 and H2. others taking a look. It's a lot of work. Yeah, for sure. And talking about a lot of work, when it was announced that Curl wanted to adopt as I think a second or third project, a Rust project to integrate as an alternative stack into it, a lot of people are excited and there was an experimental implementation being prepared using Hyper and... I think both projects learned a lot from it because maybe hyper at some things correctly or different and those got cracked into curl. And I'm sure also this implementation made you work a lot on the FFI layer, et cetera, and improve certain things to pass their test suites. But then after even your call for a champion, it out. And I think now they're also removing that stack. how do you look back at it? and also maybe tell a bit how we go start in the first place. got started in the first place. think it was 2020. I think Security Research Group had to have a lot of current software rewrite portions of it in Rust. And one of the places that they reached out to was Curl and wondering if they could replace their HTTP backend with something written Rust. And the idea was to use Hyper. We worked on that for a few years. Initially, it was a large burst towards the beginning. And then there was this long tail of, there was like 800 tests or so that Curl has. with the Hyper back end, some 95 % of them were passing, but there was 5 % that, you know, they just exercise weird parts of HTTP that Hyper either didn't support or expose, so that cruel can make use of and, but neither of us were really experts in debugging the other person's project. So it kind of languished there. And then additionally, there, there wasn't a, I want to say a customer that was asking for this at the end of the road, like a distribution that was like, Hey, once, once this is working, we would like to ship this. But all that work that was done so far, was it paid work or that was passion work? It was more kind of Yeah, so I think the Internet Security Research Group paid to have some of the work done CURL. And then I used some of my paid time at Amazon because it seemed like, well, that'd be cool once it works, we could turn it on at Amazon too. But then there wasn't a huge, like, OK, but you guys are stuck, and we really want this. then the maintenance burden just kind of just start showing up over and over. CI would break, or it was annoying to have to refactor this and still be able to support the FFI, or something like that. So yeah, at the end of 2024, did get again from curl. But we learned a lot from the process. I learned a lot from working with Daniel. He's a very excellent maintainer. And curl also was improved as it found various things that, ⁓ it supported this, but hyper doesn't. And it turns out that that's actually better that it shouldn't support. So both things got improved. You were mentioning the FFI layer of it mostly for Curl or was there already one before if it Curl, how did dropping that projects from Curl impact the FFI layer of Hyper? So it didn't exist before. It was made entirely to support cURL. Since cURL no longer uses it, there hasn't been any more explicit work on I have heard of a couple of people that have been using it. I know that there is one pull request someone made to try and do the same thing to the server, which I assume is because they wanted to use FFI but the server API. I don't know much more than that other than what's in the pull request. it continues to And could remove it, I suppose, unless there was customer that said, no, no, no, we really need this. And in fact, we'd love to make it stable. I haven't heard of that. So I don't know. And I believe in 2024 you announced that you were gonna work full-time as a maintainer or at least around things like Hyper, which of course was also a bit true already at Amazon, but that you wanted to be more independent, so I suppose you left Amazon. How did that turn out so far? Are you still walking that path and how do you balance that with the work as well as Hyper? Yeah, was in 2023. It's now been two was May 2023. It continues to work out really well. continue to be able to all of the open source things. of the paid work that I with a customer and learning they wish Hyper or reqwest or Tower or something did something a little differently, I go and spend some time doing that. key part there is that it's the open source project. it continues to be a very of similar to when I was at Amazon. Instead of meeting with internal Amazon teams, I'm with at all sorts of different companies. It that I have to do more sales, which fun. know some people love sales. ⁓ That's not me. But also given me a ton of flexibility. So appreciate that. Very well deserved given how much value you're creating in the ecosystem. Now, how do you imagine Hyper to look five years from now? What do you hope it looks like? And in that kind of setting and environment, role do you see yourself playing in that future, if it existed? Oooh, in five years from now. Well, hopefully, HTTP3 is finally fully working. And if there's an HTTP4, hope there is not. I hope we've made it a lot easier people to build opinionated stacks of servers and clients, that we've made it easier for people to reach for the safe option that we've advocated for using the safer choice instead of the common choice, like using the retry budget instead of retry limits or something like that. I hope that we've been able to continue to maintain the security standards that we set. We get a few reports a month. and some of them turn into something I hope that we continue to have collaborators that are willing to spend some time on those kind of stressful things. I kind of hope not that I want to be less involved, if I were hit by a bus could continue on. I know you're accepting more, co-maintainers or contributors. Is it growing your team? If you look at the list, we have, 10 or We've slowly added a few each year what also is true is that oftentimes many who were more involved become less involved. Maybe they change jobs, they're no longer even working on HTTP, or ⁓ they're no longer allowed to really work on open source anymore or something like that. So it probably makes sense to go through and check the list and see if still in being or if we can put them in the, what do call it, emeritus. It's going okay. That's very reassuring. I know you were also pondering an old blog post, how you can encourage more contributors or collaborators. Did you already have some answers on that? It's an ongoing thing that people and myself and in a journal and existing collaborators. within this context, is there something specifically that you would like to see happening from companies or contributors to make the Rust network sure. an increased of ownership and contributing would be healthy for the ecosystem. Certainly. and what I mean by that is an interest in, ⁓ we depend on these things. Maybe we should about their general health and not just, I have one specific bug that I want fixed. would be very healthy for the ecosystem in general. Not just Hyper, futures and all sorts of things. Are you still very heavily involved in Tokio by the way? Okay, then as we near the end of our conversation for which I'm very grateful, there anything you wish to say to your sponsor, future sponsors, maybe your Patreons, maybe people that sponsor you or on retainers? Well, I appreciate it. Thank you for having me. I appreciate all those that sponsor and those that get retainers. Allows me to keep working on it. And I love what I do. And we love what you do Sean. So thank you for coming on this podcast. You were our first guest in our virtual studio I learned a lot about the history of Hyper which is going a lot deeper than I had imagined and I learned a lot about the human behind Hyper because behind every implementation and every protocol is a hard-working human and even more an entire team around it of contributors, some long-term, some short-term. I wish you all the best, Sean Yeah, thank you very much. And you did remind me of, one last thing to say, which is if there's anything that could do is to just make better software It always pains me when friends that are less technical using software that kind of fights with them or hurts them in some way of using insecure patterns or just not as much care. If there's anything we could do, if we could make better software, mean that all the humans in our lives would benefit. I couldn't agree more. Thanks very much Sean, take care. Elizabeth (Plabayo)
43:58 | 🔗
Netstack.fm is brought to you by Plabayo building secure, open and resilient infrastructure with rust, protocols and purpose. This show is also made possible by Rama, the open source networking framework. Plabayo offers service contracts and welcome sponsorships to keep building and supporting its ecosystem. The theme music of this podcast was composed by DJ Mailbox. If you enjoyed this episode, don't forget to subscribe on your favorite podcast platform and leave a five star review. It really helps others discover the show. Thanks for tuning in. We'll see you next time for the next handshake.