Repository navigation
Description
Currently, the biggest "missing feature" in stdlib ruby URI/DNS resolution supply chain, is IDNA support. addressable, the OS alternative to stdlib uri, has some support for it, which is, I believe, the main reason why it is a transitive dependency from many other gems (It's other feature, uri templates, is just not as compelling).
This is a proposal for a way to solve this.
punycode
IDNA domains are translated to its punycode representation, in order to be used in DNS queries (which require ascii domains). ruby core stdlib does not have a punycode converter, so this is where it should start IMO. For that, I propose: either a new punycode stdlib gem (bundled?), or its functionality to be available as a submodule of URI in the uri stdlib:
# as a bundled gem
require "punycode"
Punycode.encode("l♥️h.ws") #=> "xn--lh-t0xz926h.ws"
Punycode.decode("xn--lh-t0xz926h.ws") #=> "l♥️h.ws"
# as internal functionality
require "uri/punycode"
URI::Punycode.encode(...implementation
addressable, as well as other (mostly abandoned) gems, support the IDNA 2003 standard. You'll find both libidn based extensions, as well as pure ruby ports. This has been since superseded by the IDNA 2008 standard (which essentially supports all the more recent unicode versions, plus some edge cases). While I think that a pure ruby implementation should be entertained at some point, I think that at this point, ruby should do best by adopting the most standardized implementation around, and that's libidn2: it's used by most other network libraries, including curl, and distributed as a package for most (all?) OSes supported by ruby.
Integration of libidn2 can be done by either a C extension, or FFI (I'm the maintainer of idnx, which already FFI's into libidn2 and winnls for windows). The advantage of the latter is that it works OOTB for java. The disadvantage may be performance (?), for which a C extension may be a better fit, but then we'd need to know whether java stdlib contains an equivalent of IDNA conversion supporting IDNA 2008.
This means that libidn2 would become a dependency when building ruby. It could be dealt with, however, as an optional dependency, like openssl is: when available, URI::Punycode is defined, and when it isn't, URI::Punycode is not. most ruby installers could then opportunistically install the package as well, just like it's done already with openssl.
(addressable is aware of its lack of IDNA 2008 support, and is working on it by FFI'ing into libidn2 as well).
API
uri could then transparently handle translation internally. I propose that, beyond the proposal made above, nothing else in the public API changes. Instead URI::Generic would support translation OOTB on building objects:
uri = "https://l♥️h.ws"
uri = URI(uri)
uri.host #=> "l♥️h.ws"
uri.hostname #=> "xn--lh-t0xz926h.ws"
# the example above is inspired in how uri already handles IPv6 addresses
uri = URI("https://[::1]")
uri.host #=> "[::1]", cannot be used in Socket.new(host, port)
uri.hostname #=> "::1", can be used in Socket.new(host, port)This could then be used internally in the resolv library, before issuing the DNS query.
Activity
byroot commentedon Jun 1, 2023
I think Ruby-core would rather avoid depending on more system libraries. It would be preferable to have a pure ruby implementation.
@nobu @hsbt any opinion here?
hsbt commentedon Jun 1, 2023
Agreed, and we don't have ffi gem on
ruby/ruby.HoneyryderChuck commentedon Jun 1, 2023
But there is
fiddle, right? Couldn't it be used for the same purpose (minus JRuby support)? Nevertheless, I'd expect it to be a C extension regardless.I understand the concern, hence why I'd make this an "optional dependency" a la openssl, i.e. it's either there and we have punycode, or we don't and we don't have it. libidn2 should have the same type of platform availability as openssl. And while there is maintenance effort in carrying this dependency forward and one should avoid it at all costs, other initiatives requiring external package dependency are already being considered, so one should balance the tradeoff between maintenance overhead vs. cost of not having the feature.
That being said, I also agree that having a pure ruby punycode implementation would be the best, but there isn't one yet.
byroot commentedon Jun 2, 2023
I know there is precedent for this, but this is very much a last resort thing. When you build Ruby without openssl or libyaml, it's not really "Ruby" given that the vast majority of code out there won't work. So I think we'd rather avoid creating more of these situations.
https://github.com/knu/ruby-domain_name/blob/c64a59027939aa34e1f5f0efc5cb654d73ccb966/lib/domain_name/punycode.rb could be used as a start.
But before all, I think we'd need a 👍 on what the new API should look like, after that work can be done, the spec isn't trivial, but it's not rocket science either and it's easy to unit test.
HoneyryderChuck commentedon Jun 2, 2023
The link of the punycode parser you linked is IDNA 2003 compliant, not 2008. I can't evaluate what's the effort in "upgrading" it. But I agree, let's wait for more input.
byroot commentedon Jun 2, 2023
I know. Hence why I said it could be used as a start.
skryukov commentedon Aug 5, 2023
Hey, I was trying to find an implementation of IDNA 2008 on pure Ruby for my project, and since I couldn't find anything, I wrote a new gem https://github.com/skryukov/uri-idna 🙃
I was inspired by @HoneyryderChuck's idea to put all IDNA-related functionality inside
URI. It would be cool to hear feedback from you ❤️HoneyryderChuck commentedon Aug 7, 2023
@skryukov massive effort! thank you for this 💪
I haven't done yet due diligence,but can you confirm that your library tests against standard conformance testing examples
(like this one)[http://www.unicode.org/reports/tr46/#Conformance_Testing]?
Besides that, integration with the URI lib would be just a matter of hooking into
URI::IDNA.to_ascii(host, uts46_transitional: true)(I guess we'd want transitional mode enabled by default)?I'll defer to @hsbt for the details of potential integration in the ruby standard library.
skryukov commentedon Aug 7, 2023
Hey, @HoneyryderChuck
Yup, it conforms to all tests from UTS46 (the spec file is here). I also manually added some tests for IDNA 2008 rules, so if you know of a full IDNA 2008 testing suite, I would love to give it a spin.
I don't mind changing API and/or defaults to better suite current needs, almost all rules are configurable (here is a list of options).
Also note, that the gem might differ a bit from
libidn2, for example,libidn2'stoUnicodeversion doesn't validate the result:skryukov commentedon Aug 23, 2023
@hsbt is there anything I can do to help this issue going?
hsbt commentedon Sep 24, 2025
If we have a pure ruby implementation for IDNA support. I'm +1 to that into
URIlibrary of ruby.byroot commentedon Sep 24, 2025
I believe this feature would be very valuable, I just had a chat with @hsbt and he's positive as well, he'll talk more with other maintainers soon.
In the meantime I'd like to help getting a mergeable PR together. I think we should go with the pure Ruby solution.
@skryukov do you still wish to help with this? I'd understand if not given the delay. If not I'll try to upstream what I can from https://github.com/skryukov/uri-idna
byroot commentedon Sep 25, 2025
I opened a feature request upstream as requested by Naruse: https://bugs.ruby-lang.org/issues/21617
skryukov commentedon Sep 25, 2025
I'd love to help, and I'll be available in a week.
Also want to highlight https://github.com/y-yagi/uri-whatwg_parser by @y-yagi
byroot commentedon Sep 25, 2025
That's great to hear. I think I could do the work, but you obviously have more context than me.
byroot commentedon Sep 25, 2025
There's probably no rush, the feature request should be discussed Tuesday, hopefully we get a go ahead, little point starting the work before that.
Nice, I will update the feature request.
HoneyryderChuck commentedon Sep 25, 2025
It took some time, but nice to finally see some progress! 💪
Perhaps, in order to help @skryukov navigate the required changes, has there been discussion/agreement on what the public API of the feature should be? Something like what I proposed in the description, or something more similar to
addressable? If not, perhaps you can do it next Tuesday?byroot commentedon Sep 25, 2025
Note that next Tuesday is the week ruby-core office hour, I won't be present there, only hsbt and naruse. If you have suggestions beyond what I proposed on the feature request, please post them there. I however recommend to keep is short and to the point as to not get the discussion too derailed.
You probably have more experience than me on what's really needed, but IMHO we should strive to cater to the most common use cases, it's fine if more advanced one still require a third party gem.
Hence why my feature proposal cover mostly
URI.parseas well as a way to convert from the unicode form to the ASCII one and back.HoneyryderChuck commentedon Sep 25, 2025
just added the proposal in the bug tracker, if the other ruby core members can pick up on it. @skryukov feel welcome to add your thoughts there.
byroot commentedon Apr 16, 2026
I was just reminded of this while triaging my old tickets. This seem to have fallen through the cracks, but from the look of it, this was largely accepted by @nurse https://bugs.ruby-lang.org/issues/21617#note-5
skryukov commentedon Apr 18, 2026
@byroot @y-yagi do you think we can upstream full parser, given all the work @y-yagi already did? or should we start by building just a normalization layer before current parsers (this will allow a lot of IDN domains, but not everything)?
byroot commentedon Apr 18, 2026
I'm not the URI maintainer so it's not my call. But I would suggest starting small to make it easier to review while being clear on the intended followups.
y-yagi commentedon Apr 18, 2026
Considering this comment, I think it's better to start by building just a normalization layer. I think WHATWG Parser can be introduced later.