On this page On this page
Episode 7 – Rustls with Dirkjan Ochtman.
In this episode, we go through Dirkjan's extensive experience in software development, particularly focusing on Rust, TLS, and QUIC protocols.
We explore Dirkjan his journey working on QUIC implementations to his contributions to Rust TLS and Hickory DNS. The conversation also delves into the ACME protocol and the Instant Domain Search project.
Dirkjan shares insights on the future of Rustls and the importance of community support in open-source projects.
If you like this podcast you might also like our modular network framework in Rust: https://ramaproxy.org
00:00 00:44 02:02 04:18 13:35 16:47 22:59 28:43 32:13 34:40 35:36 50:31 54:50
Music for this episode was composed by Dj Mailbox. Listen to his music at https://on.soundcloud.com/4MRyPSNj8FZoVGpytj
Hello, it's another week of our podcast and today with me is Dirk Jan. Welcome. So not many in the community might not know Dirikjan unless you're using a lot of Rust TLS and other cryptographic libraries, even though he also does more than cryptography, which we will talk about in this podcast episode as well. So maybe before we begin in the technicalities, could you perhaps tell a bit about your origin story? Sure, so I've been a software developer for I don't know some 20, 25 years now. Started professionally in 2007. Co-founded a startup, worked at a scale up after that. Did some work at a bank and then another startup. And since last year I'm doing sort of full-time open source maintenance on a whole bunch of Rust libraries. Yes, your work is quite impressive and I feel it's almost too much to cover in one episode so I have a feeling I'm going to invite you again in the future if you want. But yeah, the most noticeable library I suppose is Rust TLS even though you also I think work on Quinn. But let's first focus on Rust TLS. How did you get started working on that? So in 2018, worked, or I ended my job, I left my job at the Scale Up, where I was mostly managing, and I wanted to do a talk at Rustfest in Paris, sometime at the end of 2018, and I decided to do it on QUIC, the QUIC protocol, which is then in the early RFC draft stages. And so for the talk, I started a QUIC implementation called QUINN, and... That needs a pretty interesting interaction with GLS. It's quite different from the interaction with GLS that TCP has. And so I started to implement this quic API that we needed for, that I needed for Quint's GLS aspects in Rustles. We pre-pronounced it Rustles instead of Rust-TLS. so I started implementing that and then, yeah, sometime later, I also had a contract working for the ISRG, the organization behind Let's Encrypt to improve robustness in Russell's by eliminating panics. And so that's when I got more involved and eventually also became a maintainer for Russell's. Yeah, and you're quite an active maintainer. So I suppose the contract work helps a lot with them. Like how do these people, like how do these companies find you because you do a lot of contract work, a lot of it at a high level with a lot of high stakes. Like how do they find you or do you find them or how that works? So since then I've done a lot of, I mean I've also, a lot of the work I've done is as a volunteer and I now also have some GitHub sponsors and some organizations sponsoring my work via thanks.dev, which is really nice. They sort of sponsor your whole dependency graph. A lot of the work that has been paid for, has been funded was. Viadi at ISRG who invested quite a bit in Russell's. And sometimes I also have other companies that just find me because they need some work in Russell's or in one of the other projects that I work on. And so they look for people who are already working on those and then see if they can fund some feature or some other functional or non-functional improvements. Okay, and so you started working on Russell. That's how you pronounce this, right? Because I always had a rest TLS. Russells. Yeah, I'm gonna try to do it correctly from now on. So you started working on Russells because of your needs of the quic implementation. Can you maybe describe a bit what was missing and what you exactly had to implement? Russell's. So for TCP you just have a byte stream, right? And then sort of the data byte stream is divided into TLS records. then TLS records, inside TLS records there's a handshake, TLS handshake messages. And for QUIC, which is not exactly a byte stream, They did away with the outer layer. So there's no TLS records that divide sort of the byte stream. And instead the handshake happens on a separate QUIC stream because QUIC is built on UDP, but implement streams on top of that. And you can have many streams that don't block each other. And so in QUIC, TLS is just one of the streams. It's effectively a separate stream. called in what's what are called crypto frames. And so that means you need just a difference. You need a sort of just the handshake parts of TLS. and so that means you need like a sort of invasive smaller core of the TLS API to do, quic. you're saying for the handshake phase that's on a separate quic stream because within one h3 connection you have like multiple streams and and and just one of them is purely focused on the initial handshake and also the intermediate handshakes or like not handshakes but like the key updates yes Key updates also happen on the crypto stream. Yeah. So I wasn't talking about H3 yet. I don't have a lot of experience with H3, but just quic which I think is fairly similar to H3 in some ways. But yeah, so there is a separate crypto stream which contains all the, contains the TLS handshake data and then also can contain key update messages. And so that just requires a very different interaction API with the TLS core. And so that's what I implemented in, Russell's. was my first contribution. Okay. Okay, and then that means, it is doing, it has one stream which it uses for what we just discussed and using that information it can then encrypt the actual data streams I mentioned. Yeah, so the way it works in QUIC is that there are header protection keys and packet protection keys. And so on the start of the connection, you have a pretty straightforward, like derived from the connection ID that you pick at random. So you have some level of encryption, but the key is sort of out there or can be derived straightforwardly. And then as the handshake progresses, You derive new keys that can be used for Quic's header production and packet production keys. And so the Quic API in Russell's provides these keys as the handshake progresses and then you feed back in the crypto stream or the big crypto frame data to advance the handshake. And that's because QUIC is always encrypted, right? Like there's no way to use it without encryption. Or am I wrong? Yes, well, so the standard requires that quic connections are encrypted. Quinn exposes a modular encryption API. And so there is a Quinn plain text crate, which provides an implementation of that encryption API that allows you to do quic without encryption, which is useful in cases like data centers or when you have connections that are already protected by wire guard or something like that. Hmm, understand. But that means, if I understand correctly, that when you add TLS, or when you combine TLS with QUIC, then TLS basically just becomes a key provider. Yes, pretty much. And so the QUIC standard is sort of modular in this way. Like there's a separate RFC, I think it's RFC 9001 for TLS or maybe it's 9002, I forget for TLS, for the TLS parts of QUIC. And that's the only way of encrypting QUIC that is standardized, but QUIC itself is sort of modular in the way that it allows other ways of encryption. So there has also been like an experimental implementation that uses noise, the noise protocol for QUIC connection encryption. But yeah, so it's basically just making sure that these keys for both header protection and packet protection are available at the right time. And are the algorithms also specified, the ones that are allowed to be used? is that something the... Well, yeah, those and then some other ones too. You mean like cypher sweets? I mean, ⁓ so yeah, TLS, for quic only allows TLS 1.3. And I think that's the main restriction. And then for the, I think for the header, I forget exactly how it works. There are, there are some, a small set of well-defined cipher suites that are. primarily used for some aspects of the Quic TLS work, but I forget the details there, it's been a while. Okay. Yeah, I understand. mean, you're also dealing with a lot. And so I want to pick up on something quickly, but I don't want to go too deep on it because we have a lot to cover. But you, I said by accident, H3 before, and of course I meant quic. So thank you for that correction. But I do want to have clarity because I'm neither an expert on that topic. As far as I understood, H3 was just built on top of quic in a sense that quic becomes kind of like your transport layer and you can build application layers on top of it and H3 is an example of that. Correct or am I wrong? Yeah. Okay, okay. Just that's good because yeah, for a while I thought you meant that they use an entirely different kind of implementation than QUIC, which I didn't think so. Okay. And so yeah, then you also mentioned of course it all started with QUIN, which was I think the first. That's right. Yeah, that's right. Quic implementation in Rust or am I wrong? Yeah, I mean while I was working on Quinn I found out there's quicker QI UICR which was started sort of in parallel by Benjamin Saunders or releth on github and so when we found out that we were both working on the same thing and and Benjamin had been using Some open SSL stuff for the for the TLS work, which was kind of annoying and there wasn't a great way to do that. And so he wanted to migrate to Russell's anyway. And so we decided to merge our implementations. And in the end, I think what ended up happening is we took the name of Quinn with the implementation of Quicker, and then I did a whole bunch of refactoring in Quicker. And that's what became Quinn as it is today, more or less. Okay, and then of course Rust evolves, the community evolves, the landscape evolves. Now you also have Quiche and you have... forgot what is the... I didn't even hear of that one, but I also know Hyper S one. Maybe they use... It's possible. Actually, I'm not even sure if Hyper S is a quic implementation. I know they have an H3 implementation, but not sure what they use for the quic layer. Neko? as to when, as to when quic, think. Yeah, Hyper does not. So, I worked on, we had a contributor to Quinn who implemented H3 in a project called Quinn H3. And I did some work on getting that supported in Hyper. But Sean, the maintainer of Hyper, wanted to make Hyper abstract over quic implementations. because there are other quic implementations in Rust. And so he wanted Hyper to be able to support some others as well. And that's when basically our H3 contributor went away and worked with Sean on this H3 crate and then a H3 Quinn crate, which adapts Quinn. to implement these traits that are defined in the H3 crate. So that there is now the H3 crate and the Quinn H3 crate, I think is the, or the H3 Quinn crate, which I think is the most used implementation. And there is experimental support for H3 in reqwest using Quinn and H3. Okay, and so pretty much regardless of the language or the ecosystem, I feel that all quic implementations and the things built on top of it are still labeled as experimental. Does that just mean because of how young it is or is there actual production stuff missing that one would want to be solved before calling it no longer experimental? for quic implementations. Yeah, because I usually see them like, yeah, we have quic, yes, we have H3, but it's experimental. And so I always wonder like, how long will it remain like that? Or what does it really mean? I think Quinn is not experimental. It's a pretty high quality implementation of QUIC and it's used in quite a few places. Solana, the blockchain uses it for a whole bunch of stuff. It's used at fly.io for some of the work that they do internally. So I think Quinn is basically a production quality QUIC implementation. think the H3 work in Rust. in the H3 grade is somewhat experimental. think it's supported by request and that is marked as experimental. think there is plans that Sean has announced plans that he wants to start work on marking or moving H3 to be a supported part of Hyper. But that's still in the future at this point. I don't know his plans exactly. Yes, yes, I talked to him recently. I suppose the difficulty is that on a client side, you kind of can control about, okay, I want to talk over H3 or I want to talk over H2 or H1, given some pre-knowledge. But on the server side, the kind of issue that I think Hyper has that now suddenly have to worry about, yeah, maybe something comes in as in... H2 or an H1 and you want to upgrade to H3 and I think that's a new concept for Hyper which before that was kind of like not dealing with having to abstract at such a level before. Yeah, that's quite an extra bit of complexity. Yes, and so I guess it's also because, okay, so with H1 and H2 you were able to peak on your data on the transport layer and see whether it's H1 or H2, or you could use the TLS extension ALPN or ALPS. I think they both exist even though they have different purposes. But I think nowadays you can also more and more use the DNS layer. Is that something, I mean, given your position in the industry, is it used already a lot or is just by a couple of key players? What is used already a lot? the DNS layer like records and all those kind of things for giving pre-knowledge about certain preferences of application layer protocols. don't really know that very well. I assume it's used definitely by stuff like the major browsers and probably also folks like Cloudflare and Fastly and Akamai. I don't know and probably Google and Facebook and parties like that. Again, there's quite a lot of complexity involved in setting all of that up, including your DNS server and then HTTP server. so it's... You probably need some skill before you are going to support that. Or it's like not trivial. Yeah, I mean that's fair enough. But that kind of like brings us to HikroDNS, which you're also a contributor to. Like I don't think you started it, but you're definitely a pretty important contributor. How did that get started? So in 2020 ish, I started working on instant domain search, a domain search, but domain name search engine as a contractor. And one of the things that domain search is known for is like high performance, right? That's why it's called instant. And so it uses DNS resolution as one way of checking if domains are available. And so we had some interesting performance constraints on the DNS resolver. So I used Hickory, which was still called TrustDNS at the time, right? It was renamed later. And so made some changes to make it faster for our use case. And that's when I got involved a little and kept contributing, eventually became a maintainer. And now I also have some work on Hickory DNS that's being sponsored. because let'sencrypt wants to deploy Hickorydian as to replace their unbound instances that they use for domain name validation. And you're involved in that somehow with Let's Encrypt. Yes, so they're paying my company to improve Hickory DNS for their use cases to improve spec compliance and performance and like general stuff. Okay, that's very interesting to know. Now... yeah. There is an open issue that's pinned in the Hickory DNS issue tracker with like, let's encrypt improvements if anyone wants to take a look. Yes, I will definitely do that. And so I like Hickory DNS a lot. I also use it as my main DNS client for my own projects. What I haven't played with yet and which I do want to do in the future is with the server implementation. Have you also worked with that part of Hickory DNS? Definitely. Yeah. I mean, Hickory is a sort of a large project because it's like the base layer library and then there's a resolver implementation on top of that. And then there's a server, which can either be an authoritative name server or recursive name server. So it can do all of these use cases basically. And we, yeah, we support all of that. And so that's quite a lot of code that we want to support. like DNSSEC is another big thing that needs to be. working well for lessen grips, which is a pretty complex standard that a lot of people don't use, so it hasn't been exercised as much. So something that we've been working on as well. Yes, of course DNS used to be like pretty like yeah DNS used to be all about UDP and firing quick connections but nowadays you also have TCP, TLS connections for secure DNS lookups. Has there been any challenges there or any opinions you have regarding that? No, it's nice that Hickory just supports all of it, right? We support over UDP, over TCP, and then over TLS using Russells and then over HTTP 2 using Hyper and then over QUIC, DNS over QUIC is also a thing, and then over HTTP 3. And so we have like sort of all of the protocol support using Russells and QUIN for the QUIC stuff. So they're sort of like... benefits to being involved on all of that and seeing how it works across the stack. Yeah, for sure. I it's one of the areas I want to get definitely stronger at and I feel I only scratched the surface of Hikaru DNS yet, so far at least my experience from the client side has been very pleasant. I have not encountered any issues. I think the only thing was not related to Hikaru DNS, but something that I was just unaware of that. For example, maybe I was using by default some server like Cloudflare and then Eva was deploying to... something in production in a cloud instance, it would no longer resolve my addresses because I would need to use the maybe resolver of the cloud providers so that I can resolve internal domain names. But of course it's nothing to do with Hickory DNS, but I had to think a bit of like, why am I not resolving any of these addresses? And then, yeah, then the other thing I still need to work on is like my, where I prefer things like lookups of A records versus IPv6 records like the how you even say that I wouldn't say AAAA but like how do you pronounce those records? Quadruple A. The first rule you don't talk about quadruple A records. Quadruple A? I don't know actually how you pronounce it, so I don't talk about it, but I just write. Yeah, there is an open issue from the hyper side to support this happy eyeballs resolution mechanism, right? Which I think means you kick off AAA requests and then I know you wait for 50 milliseconds or something and then kick off the A request. So I actually worked on a PR for this, I think last week or something. So trying to implement that. because it was a long-standing feature request for Hickory. I still need to do some testing on that before it gets merged, but would be nice to have. Yeah, interesting. I have something like that, the happy eyeballs in my own network framework called Rama. And then indeed we first tried to give priority to IPv6 records. But then I also noticed you sometimes want to be able to override that because especially if you deploy something on a restricted network or some corporate networks, they often don't allow IPv6 anyway. And so you would needlessly wait on something that would anyway not resolve. And so... It's sadly not something you can just always roll out even though I wish IPv6 was more of a thing. And then another thing I noticed is that in Europe we seem to be much more ahead of IPv6 than in America because there I was working with some customers and my... Systems in my lab and in my office. They were all like IPv6 enabled and I was getting issues which my Colleagues were in America not having and it turned out just because I was allowing IPv6 and it was giving Weird issues with certain firewalls or I don't know other things It's it's Interesting sometimes on the network layer the kind of issues which which I'm sure I don't need to talk to you about So then you also have another fun little project, which is also because you mentioned, and by the way, I want to also talk a bit more about Instant Domain Search later, but it's because you mentioned that you worked there and I also noticed that you have a create called Instant ACME which is, think, only for the client side, recording the ACME protocol. What was the reason that you needed it? Mm-hmm. So for Insta Domain Search, we eventually started a startup called Insta Domains, where we also had a domain registrar and did a site builder. So you could really quickly register a domain and then immediately have a site live on the domain. And of course that needed to be HTTPS supported as well. our challenge was how quickly can we go from you hit submit on paying for your domain. to having a TLS encrypted site up. And so I wrote an Acme client. first surveyed all the queries that were out there and didn't like any of the existing ones. And so I built one myself that we open sourced, which is called Instant Acme. And yeah, that's an Acme client that tries to use things like Tokyo and Russells and Hyper. to very quickly and have a sort of unopinionated, straightforward API to make Acme work for certificate provisioning. Yes, and I realize we didn't really introduce ACME so maybe for the listeners who are not aware of it yet what it is, could you maybe in your own words describe what ACME is about and what it does? Yes, I forget what the acronym expands to. think it's automated certificate something something. So ACME is the protocol that was introduced, I think by the folks at the ISRG that are building Less Encrypt to do automated certificate provisioning for like HTTPS and other kinds of TLS certificates. And that means that you can have a few methods, few different methods to validate that you own a domain with your private key. And then you can use that to basically demonstrate that you own a domain and then let's encrypt or another Acme server provider will give you certificates in return that you can use for some period of time. Yes, and so you have different challenges. You have the HTTP challenge, have the TLS challenge, and you have the DNS challenge. The HTTP one I find the most simple one because you just need to serve something on an HTTP 80 port. The DNS one I only ever needed because I needed like a wildcard certificate, which is only possible with DNS, but that's kind of tricky because I came to realize that most APIs of like domain registers is pretty bare bone or like maybe they have like just a single hard-cored API key giving you access to everything which kind of feels dangerous to put on a server but it seems to be the only one then again I guess you want to be careful when you ever want to give a wildcard because it's quite powerful but then the last one I always wondered why that one is required Yeah. Yeah, I think it's, I mean, it's not required, right? You, as a consumer, you can pick which one you like. I think there are for some CDN providers, the TLS part is the easiest way to do it. So they control the TLS endpoints and they don't want to do it at the HTTP level because that might go back to an origin server, right? ⁓ So that's, I think that's, that's for that use case. Yeah, I understand, why did it... I mean, I'm sure it exists. Okay, yeah, now that makes sense. At InstaDomains we actually had our own DNS server based on Hickory, which had the records sitting in Redis. And so for me, the DNS method was actually quite nice. just stuff some records into Redis and then the DNS server would pick those up. And so it would respond to the ACME provider and that worked pretty neatly. Yeah, if you're a registrar yourself, I suppose it's pretty easy. Yeah. So we, we, we managed to do like this challenge of going from domain purchase to having a site live in like 20 or 30 seconds or something. So that was pretty nice. But I suppose even though you have it in the DNS, I mean, you have your own DNS server and you also control the domains, but like it's in the end, you have no control over what something like Let's Encrypt uses, right? So it still needs to bubble down to other DNS resolvers or am I missing something there? Well, because you serve, so you can update in your own DNS servers. the records, but in the end you have no control over where like, I mean, maybe let's say let's encrypt user's Cloudflare server to look up the DNS records where you prove it. It still needs to bubble down all the way to there, right? No, because so that's what I was kind of getting at earlier is that, lesson crypt has their own unbound recursive resolver, that they use for their Acme provisioning service. And that means it being recursive, I mean, means that they don't talk to just like cloud flare or Google, but they will go talk to ADNs root server and then look at the next level like for.com and then look at the next level for. The DNS servers register for a particular domain. So they always talk to the authoritative authoritative name server for the domain, which in our case is our own name server. interesting. Very interesting, yeah. so they don't use any of the recursive resolvers that are publicly available. Okay, that's very good to know. I didn't know that at all. That's nice. Okay, so we have covered that. Another one that has a lot of use, I noticed, is, and I'm gonna say how I say it, and then you can correct me, but like you have BB8, which I'm not sure what it stands for or how you pronounce it. BB8 is one of the droids from Star Wars and it is named that way I think because an earlier synchronous pooling crate was called R2D2. So Kyle Yui who works on Pernosco and the RR Replay debugger was maintaining or started BB8 I think very early on in the Tokyo 0.1 days. and eventually he stopped maintaining it and I made a few. I was using it for instant domains I think and so I made some contributions and then he just handed it over to me because he didn't want to maintain it anymore. And is it something you still use yourself and do you think it has use everywhere or did certain implementations or database clients in Rust have already catched up and provide now their own nice connection pool? I mean SQLX provides its own but if you use Tokyo Postgres for example then BB8 is a decent option I think. I think Deadpool which is sort of the other option in the in async ecosystem is probably more popular than BB8 but I think BB8 used to be a lot more featureful I think Deadpool has since caught up a bit I don't know so Today I don't have any use for a connection pooling solution. I'm still maintaining it because there's not a lot to maintain so it's not a lot of work. Okay, and then maybe I'm missing something, but I wonder what's so specific about the database that it needs to have even knowledge that the transport or connection is used for database because I like, I would think that a connection pool can just be a connection pool, right? Or is there something specific about databases that you need to be aware of and therefore you need certain like adaptive grates to plug it in. Ummm... There are a few like little details like the BB8 itself is a pretty small crate, right? It has like a pretty generic API for like a connection manager and then you implement that trait. And it has a few like application specific or protocol specific, I guess, methods. So let me check. Yeah, it needs to figure out how to connect, right, which is dependent on the application. And then it can also determine whether the connection is still valid and whether it has broken. So all of those are sort of specific to the application. And so for the fools that want to implement their own connection pool, there some hard lessons you had to learn yourself that you would want to tell your past self? Uh, Oh, I've, I've, I think I've forgotten all of them. They're all in the code there somewhere. Uh, I mean, it's also weird because I took over this existing crate, right? And then over the years I've added a bunch of features and tried to make it more robust and like fix some bugs. But I don't know that I'm have good advice off the top of my head, but, uh, you can read the source code or like, uh, a history of PRs to figure out like, uh, things to watch for. That's already good advice as well. And then as we're all about protocols and the humans behind them, I also noticed that you did some work on the EPP protocol, or at least an implementation of it. I've never heard of this before, so maybe could you talk to me about it as if I was a toddler? Mm-hmm. Yeah, EPP is the extended provisioning protocol, I think, and it's used in the register world. So this is what you use if you want to register or provision a new domain with VeriSign who owns the .com registry. And I think it's used for pretty much all of the registries. So it is an XML based protocol and it uses XML namespaces quite a bit. And there was an earlier create called EPP clients that we used, which was written by someone else. And we've made a whole bunch of contributions to that. Um, but it's still like the way it uses XML names and it was using, think, quic XML, the quic XML create, which does like 30, uh, work to do serialize and serialize XML, but it didn't really have good namespacing support. And so eventually I decided to write a new XML crate actually, which is called Instant XML and sort of rebase EPP clients on top of Instant XML instead to get a much nicer XML API, which understands namespaces better. And so both of those things are also open source and I maintain them now. They don't have as much use, but if I get PRs or features or issues, can so usually have a look. Okay and that fits in your family of grades that you maintain with the instant prefix I suppose Errol to do with the instant search work? Yes. With Instant Domains, yeah. They're all from the Instant Domains era of my career. I basically wrote all of them. Some other colleagues also contributed. And are you still involved with incident domains or not? No, they sponsor my open source work now, but I'm no longer working there. Okay, and so I often feel there is a lot of innovation possible within domain registers because a lot of them feel very ancient or very bad user experience or not made for developers or they start to get lost too much in all kind of other upsell features that they really shouldn't be distracted on. But how was your experience with working for a domain register because... When I think about doing it, I also worry very much about all the paperwork and possible regulations. Is there something special there that you noticed? I mean, getting started with being like a Verizon registrar is a bunch of paperwork. Mostly like being a domain registrar is, it's not a high margin business, right? Like you compete on price because most customers aren't as sensitive to your features or your innovation and we'll look at your sticker price first. And so it's hard, I think, to have a good business in that area and make progress there. I like, I think it's possible and would be interesting to me, but I'm not sure it's an easy business to be in. No, that's for sure. I understand that. And so I want to go a bit again to where we started, which is rustles. And so when I discovered rust 10 years ago, don't think it existed or maybe it was maybe very early days. But then now it is a lot more powerful. And so a couple of years ago, I started doing full time rust again and I was very excited about Russell DLS or Russells because I don't combine the two Russells. But the only thing that I can't like at, yeah, what didn't work for me is the fact that, so there are two, there are a couple of different ways of interpreting safety. And so you have safety because it's in a memory safe language and all these things. And I get that. And you also have the safety in the sense you want to have. You don't want to allow unsafe things by default, like you want to make sure they use the correct algorithms and that's not unique to Russells, but also to nowadays like in the Go std library where they start to deprecate certain APIs or things that you shouldn't do like older versions of TLS or older algorithms. And I think that's the correct... safe approach to allow, but then there are also legit use cases because you're working with maybe ancient systems or with specific needs where you still don't want to like implement your own TLS because that's pretty crazy, where you also wouldn't want to use something like boring SSL because you are again dealing with unsafe codes, where would but you do want to use Russells but you cannot because it doesn't allow what it would call dangerous APIs. Like there are a couple that it does expose via some kind of dangerous feature flag or dangerous API call or whatever, but it's very limited while there are certain use cases where you want a bit more control over the handshake phase, et cetera. so, yeah, I wonder if there's any specific reason besides maybe maintenance burden, why it wouldn't be possible in something like Rustles to also allow that, not as the default, but around some kind of Here are dragons, danger, danger, know what you're doing kind of API surface Yeah. I mean, it's something we discuss regularly, I would say. There are a whole bunch of features and basically it comes down to maintenance capacity, right? And so do we focus our maintenance capacity on features or like aspects of the project that are relevant to almost everyone? Or do we spend a bunch of time on like reviewing code and then maintaining for forever basically, right? Features that are of much more limited use, that are much more niche. And so we've chosen to mostly not implement a bunch of niche features that also reduce safety or security. We have, I mean, we discussed these things, so I'm... curious if you can be more specific about which features you've missed and then we can go into them in a bit more detail perhaps. But yeah, I mean, we try to make good decisions about these things. And also, like in some cases we think like there can be a Russell's fork that does some of the things that people need and that would be an okay solution or people can use other creates to implement some parts outside Russell's. And that would also be an okay way for supporting some of these niche things. Yes, yes, I understand and I do understand it's a mission so it's definitely something I'm hard pressed on and one day I might be forking RustyLS hoping to also keep in sync and contributing back upstream because it's definitely not something I like to do but I also understand your concerns so My personal view on it is that a lot of it you already have because anyway, you need those kind of primitives yourself, but they might just not be exposed. And so I think one solution could also be, but I'm not sure what in general the risk community thinks about that is to allow these as let's say hidden unstable APIs, which you give no promise about that. they won't like break in between versions like it's not like suddenly remove them but because they are more like internal APIs they will evolve with your own create and so they are hidden they know promises so anybody who use that knows that between versions they might need to update their code to to to yet to keep it working because and that would avoid the maintenance burden on your side because you anyway have that code because how else are you dealing with TLS? You need to work on the primitive level yourself. But it would also allow those kind of use cases without compromises. Yeah, I like, it's something that I think I was mentioning that in a, a private discord channel just like last week or the week before that there, like semifor compatibility is like ingrained throughout the Russell's ecosystem. Right. And if you, if we don't promise, uh, API stability for some particular low level API, And then that means that downstream creates that want to depend on that are going to have to pin a particular version, specific version. And that means that creates downstream of that are then prevented from getting upgrades or dependent on getting upgrades, which means if we release a security fix, we are not sure that these downstream of downstream cage will get it because they are now pins to, because there are some unstable API. So I think there's, while it seems attractive, like if you consider it in the grand scheme of things, it has a bunch of issues as well, right? Like if the maintainer of the pinning downstream, Yes, that's a very legit concern. becomes unavailable or uninterested or is working on other things. That means every down, everyone downstream of that suddenly can no longer pull in newer Russells. And it will not be obvious because like all the tooling is built around. If you get a duplicate version because there's a new Semper incompatible version and you'll notice, right? You see it in your cargo log files, but if. there's a newer compatible version which is not being pulled in because there's a pinning thing going on somewhere that is not as noticeable. But yeah, I would be interested to hear the specific use case that you have and then we can discuss in more detail if that could be supported somehow. But one that comes up often is fingerprinting. Yeah, I understand. Yeah, I mean, what I'm out, yeah. Yeah, that's indeed the use case. I have a mind too. Yeah, and so there was a fork that supported this, for us, so one thing we recently did in fact change the entire representation for the client and server hello messages, which would make that a problem, like which would have changed that API. And so now that the change is done, that makes it even harder to support it well, think. So yeah. Hmm. Yeah, it's those kind of use cases. that's why in Rama, the network framework, I... I maintain and develop it. My open source company is we have Russell's and I love it. And I love also, I forgot the name of your colleague, but I think that's the guy who started Russell's. I think so. Yeah. I'm really terrible with names. Something I need to improve urgently, but yeah, there's also a lot of other cryptography libraries that he and that Russell's ecosystem is developing and I like them a lot. And so I think one day I want to also like completely remove my alternative boring SSL fork of Google because like while it is nice it is definitely a headache and like it's the only kind like FFI code that I have in there which like is pretty much always enabled for those kind of use cases which is a bit of a shame. Yeah, I mean, I think there's still something you could do potentially with Russell's, like working around with some of the non-compare or non-stable API and then use an acceptor in the server to sort of parse and then do something different. There's probably stuff that you can do by using the API in creative ways. Okay, that is a challenge noted. One day I might pick up, definitely have a on my plate already. And so, but it's totally understandable. And then you were mentioning about this Semver compatibility issues and this kind of like downstream on downstream. And this reminded me of the entire chaos that followed the removal of the ring cryptographic primitives in many. downstream creates or, well, I started with REST TLS and then some were still using ring as the default while others were using AWS, LC or S or whatever it's called. And then there was a lot of issues because you can only have, and maybe this is just a lot of knowledge missing in folks like myself, but they were then you could get into compiler errors where it would say like, okay, you cannot have those two cryptographic providers together. And so this forced sometimes to have to fork. one downstream of a downstream library which was using it somehow and you had no control over the specific feature of like used by it or by its dependency in rustles and so yeah i mean maybe we missed something and it was never an issue to begin with but like it caused a lot of chaos in the ecosystem the entire switch Yeah, I mean, in retrospect, we might have chosen to not have a defaults crypto provider, right? And that would have forced every downstream to either provide its own features or to basically forward the choice of crypto provider or the downstreams might have chosen the default for you, which would end. have ended up at sort of the same problem or with two crypto providers. Yeah, and that's kind of what happened. Like, some didn't even rely on a default, but explicitly enabled Rust TLS already since long before the switch, even though it was a default. But is it like a... Yeah, so I think the next version of Russell's like 0.24 will probably not have a default crypto provider and we might even choose to move the crypto providers out of the core crate and move them into separate crates. And so then every downstream of Russell's will have to pick explicitly basically, which has issues of its own I think because now that means that if you either don't want to expose that publicly, that dependency on Russell's, because Russell's is still not 1.0, or you are gonna expose it, but you have to do a semphor incompatible bump as soon as Russell's does another semphor incompatible bump. So it's a tricky problem. Hopefully we are working towards 1.0, so. Hopefully at that point we'll have a really stable API and people can put Russell's types, including the crypto provider type in their public API without worrying about it. And that might be the best solution in the long term. But yeah, and this is probably where I miss some knowledge, but I just feel intuitively, I don't see the technical reason why you wouldn't be able to combine crypto providers in the, because usually this happens when, okay, you have maybe some database that you use and you have also REST TLS for your own server, but like in the code pods, they are not even related because they are dealing with separate connections with, they are completely isolated. So I wonder what's the harm in having one thing using AWS and wanting ring and then I also know there's like another pure rust implementation of a crypto provider I forgot the name but it's like Besides ring there is like another one. I think yeah raviola I think yes, so I wonder like how come they cannot just be combined. I must mean there is some global state somewhere or Do you mean Graviola or do you mean? Yeah. I think the reason that they can't be combined is that some of we have this concept of an installed crypto provider and we made it so that that API where you rely on the installed crypto provider panics if both features are enabled in Russell's. And that was a choice that we made while preparing for 0.23 release. And in retrospect, it might've been the wrong choice. I think a lot of people who care about security would not like it when there are two crypto providers in their binary. All right. Because they're sensitive to both the bill time concerns and also like the security, like what's crypto is being used for my application. And if there are two crypto providers, which parts of my application use one and which parts use the other and where, how can they rely on. security if I don't even know what crypto code is running for some particular aspect of my application. So that's why we made it panic and that makes it harder for both to be enabled at the same time. Yeah. I mean, yeah. Yeah. Yeah, I mean, I'm sympathetic to that argument as well, but the sad part is that in most complex, big projects, you eventually come into a situation where you have zero control over the fact that at some point two are used, especially when you deal with databases, servers, all kinds of other intermediate caching, blah, blah. then often the only thing you can do is fork one of these dependencies. And it's pretty sad because in the end you don't really want... to do anything with it, just, your program doesn't even compile. And I wonder if that's ever, I mean, I don't feel that's even in your control because yeah, how would you ever prevent that if they just panic, if the two are detected, which by the way, yeah. Yeah, I mean, I ended up contributing code to a whole lot of upstream libraries, To forward these cargo features so that you cannot, you don't need both at the same time. And so yes, you need to fork if the upstream is not maintained well to the standard that you need. Right. And that is just how open source works. think, right. You contribute back upstream and if the maintainer is not cooperative or doesn't like it, you find another dependency to work with. Yeah, I think that's fair enough and I like it a lot. But yeah, I didn't even know that argument. I thought it was more to do with the fact that let's say if you use some kind of native or C++ dependency for your cryptography, then it might include some shared libraries, which it in any way cannot link to if you have multiple of those shared library objects combined. Yeah, I think that was not the problem in most of these cases, but it might have been a problem early on. Okay. Okay, that's fair enough. then nowadays you are used, now you're fully independent, right? You have your own company and is it like a one person company or it means you have some employees and you all work together on different projects? Mm-hmm. It is a one-person company. I do hire a contractor sometimes for some of the work that I take on. So yeah, it's mostly one person. Okay, and then you mentioned this platform tanks.dev, which I wasn't aware of yet. Did it help you a lot? Mm-hmm. I mean a lot. It's some part of my income, right? It's not the largest part of my income by far, but it and GitHub Sponsor do meaningfully help my income each month. So that is quite nice. Yeah, mean, well deserved. have a lot of high impact on the system. Where do you see Rust TLS going now that it was also adopted as the first project in the Rust Foundation lab or the Rust Labs or I don't know what the name was anymore. Yeah, the Rust Innovation Labs, think the Rust. I mean, that is sort of a legal and financial support structure, right? So we have found a lot of funding to improve Russell's in the past years, and it's grown a lot as a result. So now that performance is at a really great level and we're working on API stability to hopefully get to that 1.0 release. But funding maintenance work while the code is more or less in a good place is harder. And so we hope that bringing Russell's to the Rust Innovation Lab will help with that. For the next year or so, we're focused mainly in terms of the roadmap, the technical roadmap, we're focused mainly on getting to API stability. So that will probably mean getting to a 0.24 release. And then if that works out okay, then we'll move quickly, fairly quickly, I think to a 1.0 release. But there's a bunch of stuff that we want to work out before 0.24 is ready. And there is, I think there's an issue in the Russells. issue tracker on that as well that has a bunch of more details for people who are interested in that. Okay, and do you ever see Russell's being used in as the cryptography layer for browsers? I think that would be nice. servo is using it today. think, that took quite a while before it moved from NSS, I think to Russell's. we'll see where our browsers go. I think, Google's probably pretty committed to boring SSL. so it depends a little bit on where browser engines go and Mozilla has been pretty committed to NSS. I think that's. changing slowly perhaps. But I think it might depend on like, does Servo become a bigger project with more users? Then I think we stand a pretty good chance. And at same time, like we're using it, we're looking more at like the Linux ecosystem. like Ubuntu going with sudo rs and uutils and seeing if they might be interested in. oxidizing some more of their base layer. I Very cool. So I think we covered everything I want to cover today. Is there something you want to plug in the episode or want to still mention before we say goodbye? Yeah, just on that last point, it might be interesting to mention, don't know for people who don't know this is that Russell's also has a OpenSSL compatibility layer today. So there is a project called Russell's OpenSSL Compat, which provides a shared library, which can be used in projects like NGINX and also some other projects. And basically provides the same OpenSSL ABI. while using Russells for the TLS implementation. So I think that's a really interesting way to plug Russells into your existing applications. And is this something you want to say to all your sponsors and supporters? Thanks for all the support and I hope to keep doing open source maintenance for a lot longer. And so I hope we can keep this sustainable. Yes, we hope it for you as well. Thank you very much for appearing on our episode today and I wish you a nice journey ahead of your life. Thanks for doing this podcast. It will be interesting. Elizabeth (Plabayo)
54:50 | 🔗
Netstack.fm is brought to you by Plabayo building secure, open, and resilient infrastructure with Rust protocols, and purpose. This show is also made possible by Rama, the open source networking framework. Plabayo offers service contracts and welcome sponsorships to keep building and supporting its ecosystem. The theme music of this podcast was composed by DJ Mailbox. If you enjoyed this episode, don't forget to subscribe on your favorite podcast platform and leave a five-star review. It really helps others discover the show. Thanks for tuning in. We'll see you next time for the next handshake.