Repository navigation
Using faster url parser #643
Description
Activity
a pull request or some other request for specific action is likely to get more attention here fwiw
Well I request your opinion before I make it PR ready (it's a normal user module right now)
I like the idea of making
urlfaster, but replacing the whole module frightens me a littleI think there is nothing wrong with using faster parser as it's just an
implementation details as long as API stays the same and all existing tests
passOn Wed, Jan 28, 2015 at 1:39 PM, Vladimir Kurchatkin <
notifications@github.com> wrote:I like the idea of making url faster, but replacing the whole module
frightens me a little—
Reply to this email directly or view it on GitHub
#643 (comment).I would love to see this on io
@xaka It's about subtle things, like data properties vs. accessor properties. It's hard to foresee everything once whole module changes. This requires major version bump at least.
That's a good point. Switching from properties to accessors means breaking
API as things like console.log(...) would work differently and people might
be using the stdout for instance for their own reasons, etc.On Wed, Jan 28, 2015 at 2:36 PM, Vladimir Kurchatkin <
notifications@github.com> wrote:@xaka https://github.com/xaka It's about subtle things, like data
properties vs. accessor properties. It's hard to foresee everything once
whole module changes. This requires major version bump at least.—
Reply to this email directly or view it on GitHub
#643 (comment).this is one of the reasons why i'm trying to get rid of the
util._extend()usage. otherwise, i fear too much would break.Can't we lazily patch the stringify and json transforms to avoid API change/break?
The getters can be made enumerable (but they're still on the prototype) and the Url class can implement
toJSONmethod. So that leavesutil._extend(), which should be internal method?It can also be considered to make the properties just eager data properties, it would still be 11x faster. But often many of the properties are not needed (especially .href?) so it could be a bummer.
Reopening since it was reverted 😄
16 remaining items
Well there is nothing wrong with introducing a warning in a minor release.
The intention is not to follow browsers for the good of writing isomorphic apps.
The intention is to follow url resolution logic like browsers, so that crawlers and other Node.js programs behave reasonably. The url object parsing is based on the browser
locationglobal and<a>html tag fields, but it also has to support being passed tohttp.requestand friends. So, it's not completely identical, and needn't be.A 100% isomorphic browser-style accessor-using auto-updating url object is a great idea. It belongs in npm, not in core.
Will this make it into 4.0.0?
That depends on @petkaantonov right now. See #1650.
Closing given the lack of continued activity and discussion on this. Can reopen if necessary.
@jasnell @petkaantonov testing this using node 7.5.0 on 16.04 (dual core kvm Intel Xeon E3-12xx v2) still shows huge difference
we found about this after cpu profiling our application which showed url.parse takes most of the time and we can't avoid making requests
we have switched to
fast-url-parserfor now
any info on this maybe?node benchmark/nodecore.js
misc/url.js parse(): 40645.099 misc/url.js format(): 36836.169 misc/url.js resolve("../foo/bar?baz=boom"): 38703.878 misc/url.js resolve("foo/bar"): 39136.199 misc/url.js resolve("http://nodejs.org"): 37845.184 misc/url.js resolve("./foo/bar?baz"): 38815.977node benchmark/urlparser.js
misc/url.js parse(): 184957.14 misc/url.js format(): 170904.26 misc/url.js resolve("../foo/bar?baz=boom"): 181324.04 misc/url.js resolve("foo/bar"): 169483.69 misc/url.js resolve("http://nodejs.org"): 180839.33 misc/url.js resolve("./foo/bar?baz"): 174857.71I just hit an issue in production where extremely long referer urls (3000 chars, 5500 chars, and 9000 chars) were causing our node instances to spike CPU usage and we tracked it down to the
urllibrary. @pkoretic yourfast-url-parseris much more performant.Anyone have an update on when this will be merged? I'll switch to
fast-url-parserfor now.Latest node.
$ node -v v12.12.0 toxa@5fb84e50fca4:/tmp/urlparser$ node benchmark/nodecore.js misc/url.js parse(): 68502.876 misc/url.js format(): 60017.421 misc/url.js resolve("../foo/bar?baz=boom"): 59991.764 misc/url.js resolve("foo/bar"): 64876.780 misc/url.js resolve("http://nodejs.org"): 63232.093 misc/url.js resolve("./foo/bar?baz"): 64872.514 toxa@5fb84e50fca4:/tmp/urlparser$ node benchmark/urlparser.js misc/url.js parse(): 170138.65 misc/url.js format(): 191450.64 misc/url.js resolve("../foo/bar?baz=boom"): 208052.98 misc/url.js resolve("foo/bar"): 271166.70 misc/url.js resolve("http://nodejs.org"): 280692.39 misc/url.js resolve("./foo/bar?baz"): 255188.00Still 2.5x:
$ node -v v16.14.0 $ node benchmark/nodecore.js misc/url.js parse(): 78603.333 misc/url.js format(): 72242.340 misc/url.js resolve("../foo/bar?baz=boom"): 71972.167 misc/url.js resolve("foo/bar"): 78404.141 misc/url.js resolve("http://nodejs.org"): 78088.377 misc/url.js resolve("./foo/bar?baz"): 78914.819 $ node benchmark/urlparser.js misc/url.js parse(): 169323.52 misc/url.js format(): 199909.74 misc/url.js resolve("../foo/bar?baz=boom"): 198722.90 misc/url.js resolve("foo/bar"): 226340.86 misc/url.js resolve("http://nodejs.org"): 159880.62 misc/url.js resolve("./foo/bar?baz"): 284006.64
(Original node issue nodejs/node-v0.x-archive#6788)
I have rewritten the url parser module of node core as it was/is a serious bottleneck in some of the techempower benchmarks
Running node's urlparser benchmark using iojs it's still 16x faster when not retrieving properties and 11x faster when retrieving all properties (which are lazy getters in my implementation). In absolute terms the current iojs urlparser throughputs 25k parses per second vs 400k lazy/270k eager parses per second.
formatandresolveare also affected in similar magnitudes.