[Bug report] fs.createReadStream set highWaterMark, the same file get different result #35465

Activity

1. Zclhlmgqzc commented on Oct 5, 2020

   

   Zclhlmgqzc

   on Oct 5, 2020

   Last edited by Zclhlmgqzc

   Author

   More actions

   @bnoordhuis
   @trxcllnt

   when bytesRead === 0 why poolFragments.push(thisPool.slice(alignedStart, alignedEnd));

   node-v12.2.0/lib/internal/fs/streams.js

   ReadStream.prototype._read = function(n) {
     if (typeof this.fd !== 'number') {
       return this.once('open', function() {
         this._read(n);
       });
     }
   
     if (this.destroyed)
       return;
   
     if (!pool || pool.length - pool.used < kMinPoolSpace) {
       // Discard the old pool.
       allocNewPool(this.readableHighWaterMark);
     }
   
     // Grab another reference to the pool in the case that while we're
     // in the thread pool another read() finishes up the pool, and
     // allocates a new one.
     const thisPool = pool;
     let toRead = Math.min(pool.length - pool.used, n);
     const start = pool.used;
   
     if (this.pos !== undefined)
       toRead = Math.min(this.end - this.pos + 1, toRead);
     else
       toRead = Math.min(this.end - this.bytesRead + 1, toRead);
   
     // Already read everything we were supposed to read!
     // treat as EOF.
     if (toRead <= 0)
       return this.push(null);
   
     // the actual read.
     fs.read(this.fd, pool, pool.used, toRead, this.pos, (er, bytesRead) => {
       if (er) {
         if (this.autoClose) {
           this.destroy();
         }
         this.emit('error', er);
       } else {
         let b = null;
         // Now that we know how much data we have actually read, re-wind the
         // 'used' field if we can, and otherwise allow the remainder of our
         // reservation to be used as a new pool later.
         if (start + toRead === thisPool.used && thisPool === pool) {
           console.log('newUsed')
           const newUsed = thisPool.used + bytesRead - toRead;
           thisPool.used = roundUpToMultipleOf8(newUsed);
         } else {
           console.log('alignedEnd')
           console.log('bytesRead: ' + bytesRead)
           // Round down to the next lowest multiple of 8 to ensure the new pool
           // fragment start and end positions are aligned to an 8 byte boundary.
           const alignedEnd = (start + toRead) & ~7;
           const alignedStart = roundUpToMultipleOf8(start + bytesRead);
           if (alignedEnd - alignedStart >= kMinPoolSpace) {
             console.log('start: ' + start)
             console.log('toRead: ' + toRead)
             console.log('alignedEnd: ' + alignedEnd)
             console.log('alignedStart: ' + alignedStart)
             poolFragments.push(thisPool.slice(alignedStart, alignedEnd));
           }
         }
   
         if (bytesRead > 0) {
           this.bytesRead += bytesRead;
           b = thisPool.slice(start, start + bytesRead);
         }
   
         this.push(b);
       }
     });

   node test-202.js 
   ./test-file-202
   alignedEnd
   bytesRead: 202
   
   alignedEnd
   bytesRead: 202
   
   alignedEnd
   bytesRead: 0
   start: 0
   toRead: 202
   alignedEnd: 200
   alignedStart: 0
   
   ./test-file-202
   
   newUsed
   
   200
   
   alignedEnd
   bytesRead: 202
   
   alignedEnd
   bytesRead: 2
   start: 0
   toRead: 202
   alignedEnd: 200
   alignedStart: 8
   
   2
   newUsed
   
2. Zclhlmgqzc commented on Oct 6, 2020

   

   Zclhlmgqzc

   on Oct 6, 2020

   Last edited by Zclhlmgqzc

   Author

   More actions

   Works fine
   node-12.2.0/12.18.4/14.13.0
   /lib/internal/fs/streams.js

         if (start + toRead === thisPool.used && thisPool === pool) {
           const newUsed = thisPool.used + bytesRead - toRead;
           thisPool.used = roundUpToMultipleOf8(newUsed);
       - } else {
       + } else if (bytesRead > 0) {

   👍React with 👍1Reacted by Paul Taylor
3. watilde

   added

   fsIssues and PRs related to file-system APIs and the fs module.Issues and PRs related to file-system APIs and the fs module.

   on Oct 10, 2020
4. ronag commented on Oct 10, 2020

   

   ronag

   on Oct 10, 2020

   Last edited by ronag

   Member

   More actions

   @Zclhlmgqzc PR welcome! Do you have the same issue on Node 15/master?
5. Zclhlmgqzc commented on Oct 14, 2020

   

   Zclhlmgqzc

   on Oct 14, 2020

   Last edited by Zclhlmgqzc

   Author

   More actions

   @ronag
   Works fine in Node 15.x
   Because #33981

   👍React with 👍1Reacted by Robert Nagy
6. targos

   added

   v14.x

   on Dec 13, 2020
7. gblock0 commented on Mar 23, 2021

   

   gblock0

   on Mar 23, 2021

   Last edited by gblock0

   More actions

   Hi, is there an update on this? Do you think it'll make it into 12.x or 14.x? I don't have a ton of extra time, but if you give me some direction, I can take a swing at fixing it.

   👍React with 👍1Reacted by Kripanshu Bhargava
8. targos commented on May 3, 2023

   

   targos

   on May 3, 2023

   Member

   More actions

   Closing as v14.x is EoL.
9. targos

   closed this as not plannedon May 3, 2023