streaming / iterative fs.readdir

since we're in ES6 territory now, i'm thinking the sync version should be an iterable

var dirs = fs.readdirIter(__dirname);
for (dir of dirs) {

}

and have the async version be an object stream:

var stream = fs.readdirStream(__dirname);
stream.on('data', dir => )

See: nodejs/node-v0.x-archive#388

Using generators/iterators in io.js where they make sense would be a good addition.

Does this depend on the work in joyent/libuv#1574 to be merged first? I don't think it's in libuv yet.

👍

in the future asynchronous version could return an observable:

for (dir on fs.readdir(__dirname)) {

}

👍

I'm all for the new named versions. We don't want to break compatibility with existing code by changing behaviour of existing functions, but new stuff could work. I'm not sure what the policy is yet on new interfaces that deviate from node.js though.

@iojs/tc Thoughts?

Where else does an interface like this make sense? Would this be a one-off of would there be a flood of requests to add the same style interface to other core APIs?

@rvagg

@rvagg I looked for any other APIs that would look like this but I couldn't find any. I think this may be a one-off.

I think the way forward on this is for someone here to propose a change in a PR and then we'll escalate that to a TC discussion because it would be good to cover the question of whether adopting a new style of API is desirable. If it's too much work to come up with an initial implementation then we could just elevate this issue, it'd just be more of a hypothetical discussion then.

I think the streams version makes sense. I would test the iterator design in userland though. It can apply to any sort of stream, really. And it's easy enough to abstract a stream as an iterator in userland.

The eventual observables thing sounds pretty rad.

@Qard

@Qard how would you create an iterator from a stream? streams are async... iterators are not...

@cjb i don't think it's merged. i'm going to open an issue in libuv now

Just use co? (or something similar)

OK, a few things:

Iterables make no sense here; they are synchronous. (Generators are just iterables with .throw() and .return() methods; they don't make any sense either.)
There are vague designs by one TC39 committee member for adding RxJS-style observables to the language, along with for-on syntax. I doubt that will make the ES2016 train since no implementers have yet expressed interest, but I could be wrong. Regardless, it's very tentative. (There's another competing proposal, async iterables, that might be a better fit for this use case in particular.)
In the meantime, Node.js has an awkward version of observables/async-iterables/etc. already: object-mode streams. That's clearly the right choice for this situation, even if in the long run there becomes a language-level primitive for plural asynchronous values (similar to promises being the primitive for singular asynchronous values, or iterables being the primitive for plural synchronous values).

So I don't think this should be a new style of API. Just an object mode stream is fine.

Whoa, yeah this is a totally bogus suggestion as iterators and generators can't be used for iterating over async. This tripped me up at first as well.

But note that though it's not exactly what you had in mind fs.readFileSync(file, 'utf8') is already iterable by way of a String being iterable, and soon fs.readFileSync(file) will too be iterable as Buffers implement the iterable interface: #525

fs: introduce `opendir()` and `fs.Dir`

This has been released in 12.12.0

What about also making directories sync iterable (as initially suggested)?

I think this could be done by using dir.readSync().

The commit message could be "fs: make directories sync iterable".

@ma11hew28

@ma11hew28 sorry to tell you that Sync can't be iterable as its Sync :) a iterable is a async type

There are sync iterators too. It's entirely possible to make a sync version.

@Qard

@Qard why should i use a iterator for a Sync call that will return after all is in mermory already but ok your right it could exist it can be done. i for my self would suggest for...of as iterate method but ok

@frank-dspeed

Thank you, @frank-dspeed and @Qard, for responding. I'm sorry for not responding promptly. @frank-dspeed, I'm sorry, but I think you misunderstood me. What you suggested is what I meant, as it is the first part of what @jonathanong initially suggested. Ie, if we make directories sync iterable, then you could do something like this:

const fs = require('fs')

const dir = fs.opendirSync('.')
for (const dirent of dir) {
  console.log(dirent.name)
}

instead of something like this:

const fs = require('fs')

const dir = fs.opendirSync('.')
let dirent
while ((dirent = dir.readSync()) !== null) {
  console.log(dirent.name)
}
dir.closeSync()

As for the second sentence of my initial comment on this issue, by "this", I meant "making directories sync iterable". Ie, I think a directory's default sync iterator's next() method could call dir.readSync() to get the directory's next entry.

@frank-dspeed

@frank-dspeed It doesn't have to all be loaded into memory with a sync iterator. If you have a directory with millions of entries in it, a sync iterator could read and return entries one at a time, or in batches, but synchronously.

fs: introduce `opendir()` and `fs.Dir`

unix,win: add uv_fs_{open,read,close}dir()

Navigation Menu

Repository navigation

streaming / iterative fs.readdir #583

98 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

streaming / iterative fs.readdir #583

Description

Activity

novacrazy commented on Jan 24, 2015

cjb commented on Jan 24, 2015

vkurchatkin commented on Jan 24, 2015

hemanth commented on Jan 24, 2015

Qard commented on Jan 24, 2015

rvagg commented on Jan 26, 2015

jonathanong commented on Jan 26, 2015

rvagg commented on Jan 26, 2015

Qard commented on Jan 26, 2015

Fishrock123 commented on Jan 26, 2015

jonathanong commented on Jan 28, 2015

Qard commented on Jan 28, 2015

domenic commented on Jan 28, 2015

timoxley commented on Jan 28, 2015

98 remaining items

Fishrock123 commented on Oct 12, 2019

ma11hew28 commented on Nov 22, 2019

frank-dspeed commented on Nov 24, 2019

Qard commented on Nov 27, 2019

frank-dspeed commented on Nov 27, 2019

ma11hew28 commented on Nov 27, 2019

Qard commented on Nov 28, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions