Issue with Node 4.x when receiving pipelined reqeusts

After updating to Node 4.x from 0.12.x, benchmarking by sending pipelining requests results in an initial spike of requests followed by no requests being processed on those connections, and those connections remaining open even after the load generator client finishes (connection leak). The same behavior does not occur when not pipelining requests, and it doesn't occur on 0.10.x or 0.12.x at all. The issue reproduces on both Linux and Windows.

I'm generating load using "wrk" (from a remote Linux machine) and the app is the benchmark app from the TechEmpower benchmarks.

@DamianEdwards

@DamianEdwards which specific version? v4.2.0? v4.1.2?

I reproduced on both 4.1.2 and 4.2.0

Does it occur on 4.1.1? There were a few changes in 4.1.2 (0504066, b3d9678) that might be causing this.

I'll drop back to 4.1.1 tomorrow and try it out.

@indutny

cc @indutny

@DamianEdwards

@DamianEdwards does removing those lines help?

Actually, latter one should be enough to verify it.

@indutny

@indutny I'm assuming I have to build from source to try out removing that line?

@DamianEdwards

@DamianEdwards yeah. Sorry, I forgot to mention it.

@indutny

@indutny 😄
I'll see what I can do. That change was added in 4.1.2?

Yep.

Found one bug, already:

diff --git a/lib/_http_outgoing.js b/lib/_http_outgoing.js
index 6650c06..366c516 100644
--- a/lib/_http_outgoing.js
+++ b/lib/_http_outgoing.js
@@ -663,6 +663,9 @@ OutgoingMessage.prototype._flush = function() {
       this.output = [];
       this.outputEncodings = [];
       this.outputCallbacks = [];
+      if (this._onPendingData !== null)
+        this._onPendingData(-this.outputSize);
+      this.outputSize = 0;
     }

     if (this.finished) {

Looking for more.

Second part of issue is handling of paused parser in consumed socket. Working on fix.

@DamianEdwards

Full fix:

diff --git a/lib/_http_outgoing.js b/lib/_http_outgoing.js
index 6650c06..366c516 100644
--- a/lib/_http_outgoing.js
+++ b/lib/_http_outgoing.js
@@ -663,6 +663,9 @@ OutgoingMessage.prototype._flush = function() {
       this.output = [];
       this.outputEncodings = [];
       this.outputCallbacks = [];
+      if (this._onPendingData !== null)
+        this._onPendingData(-this.outputSize);
+      this.outputSize = 0;
     }

     if (this.finished) {
diff --git a/lib/_http_server.js b/lib/_http_server.js
index c11d369..ffaa0b8 100644
--- a/lib/_http_server.js
+++ b/lib/_http_server.js
@@ -270,6 +270,7 @@ function connectionListener(socket) {
     // need to pause TCP socket/HTTP parser, and wait until the data will be
     // sent to the client.
     outgoingData += delta;
+    console.log(outgoingData);
     if (socket._paused && outgoingData < socket._writableState.highWaterMark)
       return socketOnDrain();
   }
@@ -405,6 +406,7 @@ function connectionListener(socket) {
     if (socket._paused) {
       // onIncoming paused the socket, we should pause the parser as well
       debug('pause parser');
+      parser.unconsume(socket._handle._externalStream);
       socket.parser.pause();
     }
   }

@DamianEdwards may I ask you to give it a try?

@indutny

Hey guys, I believe we're seeing this issue in production for 4.2.1. I can share the error if that'd be helpful.

I believe @indutny fixed this, right? If so, why isn't this released? This should be considered a hotfix and get released as a 4.2.2 release ASAP

Just curious on whats going on with this given what we're seeing, given its fixed, and given the fix is unreleased

@NawarA

@NawarA ... I'm likely going to be taking a look at spinning out a 4.2.2 early next week.

@jasnell

@jasnell thanks. I had to roll back to iojs to keep production stable. Let me know if I can help with the release

@NawarA

@NawarA ... smoke testing the v4.x branch would be helpful. If you pull the branch, do a fresh build, does it resolve the issue or are there specific commits we still need to land to address the issue?

Will do. To confirm the error and ensure its related to this issue, here's what I'm seeing:

_stream_readable.js:480
  dest.on('unpipe', onunpipe);
       ^
TypeError: dest.on is not a function
    at Gzip.Readable.pipe (_stream_readable.js:480:8)
    at ChildProcess.onMessage (/home/admin_augur_io/Augur/LearningServer/routes/recon-master.js:60:10)
    at emitTwo (events.js:87:13)
    at ChildProcess.emit (events.js:172:7)
    at handleMessage (internal/child_process.js:686:10)
    at Pipe.channel.onread (internal/child_process.js:440:11)

I'll pull the latest master branch, and see if it resolves the issue.

@NawarA

@NawarA this does not look related to this issue

@indutny

Good to know. I thought because it was as stream where piping was failing, and due to me seeing this issue after upgrading, that the issue was related. Thanks for sharing @indutny

@indutny

Pipelining, not piping ;)

On Thursday, October 22, 2015, Nawar Alsafar notifications@github.com
wrote:

Good to know. I thought because it was as stream where piping was failing,
and due to me seeing this issue after upgrading, that the issue was
related. Thanks for sharing @indutny https://github.com/indutny

—
Reply to this email directly or view it on GitHub
#3332 (comment).

http: fix stalled pipeline bug

Navigation Menu

Repository navigation

Issue with Node 4.x when receiving pipelined reqeusts #3332

9 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue with Node 4.x when receiving pipelined reqeusts #3332

Description

Activity

rmg commented on Oct 12, 2015

DamianEdwards commented on Oct 13, 2015

brendanashworth commented on Oct 13, 2015

DamianEdwards commented on Oct 13, 2015

Fishrock123 commented on Oct 13, 2015

indutny commented on Oct 13, 2015

indutny commented on Oct 13, 2015

DamianEdwards commented on Oct 13, 2015

indutny commented on Oct 13, 2015

DamianEdwards commented on Oct 13, 2015

indutny commented on Oct 13, 2015

indutny commented on Oct 13, 2015

indutny commented on Oct 13, 2015

indutny commented on Oct 13, 2015

9 remaining items

NawarA commented on Oct 22, 2015

jasnell commented on Oct 22, 2015

NawarA commented on Oct 22, 2015

jasnell commented on Oct 22, 2015

NawarA commented on Oct 22, 2015

indutny commented on Oct 22, 2015

NawarA commented on Oct 22, 2015

indutny commented on Oct 22, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions