Skip to content

Avoid repeating npm install on each exercise #235

Closed

Description

@rchavarria

Some people find it tedious and a waste of time to have to install NodeJS packages everytime they want to solve an exercise (take a look at exercism.io#2972).

In that discussion, we reached to a solution: improve the documentation on /docs/INSTALATION.md to guide users to install all dependencies on <EXERCISM_HOME>/ecmascript and add that folder to the PATH system variable. This way, all mandatory tools to run tests will be available on the path and there will be no need to npm install on each exercise.

Activity

  1. tejasbubane commented on Dec 1, 2016

    @tejasbubane
    Member

    Adding PATH (and NODE_PATH as some solutions suggest) did not work for me.

    Changing directory is not and won't ever be supported I suppose: npm/npm#775

  2. tejasbubane commented on Dec 1, 2016

    @tejasbubane
    Member

    Can we switch to yarn? It is crazy fast (more so with cached packages). Here is a comparison of npm and yarn on one exercise:

    $ time npm install
    ....
    npm install  31.45s user 9.02s system 34% cpu 1:58.32 total
    

    PS: Notice the total time there^

    without cache:

    $ time yarn install --force
    ......
    ✨  Done in 32.49s.
    yarn install --force  20.74s user 10.64s system 93% cpu 33.392 total
    

    with cache:

    $ rm -rf node_modules
    $ time yarn install 
    .....
    ✨  Done in 15.03s.
    yarn install  9.37s user 5.12s system 90% cpu 15.977 total
    

    Considerably less frustrating than waiting for npm install to complete for every exercise.

  3. rchavarria commented on Dec 3, 2016

    @rchavarria
    ContributorAuthor

    I see. Maybe, installing dependencies in a different directory that the working directory is not a good idea, based on npm/npm#775.

    Yep, I've heard yarn is much faster installing packages than npm. But I'm a bit afraid about forcing users to install yet another tool. From a novice exercism user, it can be overwhelming.

    Is yarn 100% compatible with npm's package.json file? At least about installing dependencies and running scripts? If so, documentation could be updated with a note about the possibility of using yarn. Does installing yarn require any change? If not, it's just a matter of taste. Exercism users can use whatever tool they want.

  4. tejasbubane commented on Dec 6, 2016

    @tejasbubane
    Member

    Yarn is compatible with npm in terms of pulling packages from npm registry and running scripts, although it does not have all commands/options that npm does.
    ref: https://yarnpkg.com/en/docs/migrating-from-npm I think in that way it is not a drop-in replacement for any project. But for our purposes, installing packages and running tests - it can be treated as npm replacement. I have been using it with ecmascript exercises for a while now and works without a hassle.

    I get that it will be an added step for beginners since npm comes bundled with node. We can mention it as a faster alternative to npm and they can use either of the two.

  5. rchavarria commented on Dec 7, 2016

    @rchavarria
    ContributorAuthor

    Nice.

    You could modify the /SETUP.md file, add some text under the Requirements chapter about the posibility to use yarn as an alternative to npm, and create a PR with your changes. The conversation can continue in that PR.

    @exercism/javascript Any ideas on this npm Vs. yarn subject?

  6. masters3d commented on Mar 7, 2017

    @masters3d
    Contributor

    I am using yarn in the TypeScript track. In addition to make being fast at install it also locks the packages so that everybody is using the exact version.

  7. stale commented on May 6, 2017

    @stale

    This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

  8. joeljuca commented on May 7, 2017

    @joeljuca
    SponsorContributor

    Is there any specific case that we would need specific packages for an exercise? I understand that students should have an easy way to test their solutions, but we solved a similar problem on exercism/xjavascript by using a global installation of Jasmine.

    Since Exercism is supposed to teach a programming language and not libraries, platforms and/or tools, why not use a simpler approach?

  9. matthewmorgan commented on May 8, 2017

    @matthewmorgan
    Contributor

    @joelwallis at the moment (I believe) we need some sort of configuration file in each directory for the test runner (Jest.) I don't think we need an npm install for each exercise anymore. This is something that was needed in the past with the way we had the transpile-test pipeline set up.

    Currently, we're using Jest with the babel-jest plugin to transpile and build just using the jest command.

    If you're interested in tackling it, you would be more than welcome. If I can help in ay way, let me know.

  10. rchavarria commented on May 10, 2017

    @rchavarria
    ContributorAuthor

    I don't think we need an npm install for each exercise anymore.

    Really? It'd be awesome. But, users need to install Jest globally I think. How would be the process right now? (just a few words, not a thorough description 😛 )

  11. matthewmorgan commented on May 10, 2017

    @matthewmorgan
    Contributor

    @rchavarria I think we can have a JSON file in each exercise that points jest at the specfiles in the current directory. That should be enough to make jest work.

    But, I am wondering about the "grains" exercise, which requires the BigInt package. We might need to modify the makefile to still copy that file into the grains exercise dir. I'm not sure.

  12. joeljuca commented on May 11, 2017

    @joeljuca
    SponsorContributor

    I don't know much about Jest. I'll take a closer look this weekend.

  13. tejasbubane commented on May 12, 2017

    @tejasbubane
    Member

    I don't think we need an npm install for each exercise anymore. This is something that was needed in the past with the way we had the transpile-test pipeline set up.

    @matthewmorgan If you mean installing jest globally and running it on each exercise, I don't think that will work. Since we still have more packages than jest itself, like babel-jest, preset-env. That means we will need to install all these packages globally.

    That is, unless you have a better way that I am unaware of.

  14. rchavarria commented on May 21, 2017

    @rchavarria
    ContributorAuthor

    Closing and labeling as won't fix, agreed on #182

  15. aeamaea commented on Nov 22, 2020

    @aeamaea

    It's kinda hokey, but I just move the node modules from the completed exercise to the new one before running npm test for the first time. In all honestly, the base module set could just be cached and moved to each exercise as it's fetched and any deltas would be downloaded by npm anyway. seems to work for me. It's a huge waste and has an unacceptable environmental footprint as well. by the end of the JS track, each user would've download 9.5 gigabytes worth , multiply that with 86000 students (at the time of this writing) and you potentially have about 80 terabytes of node-modules down the wire that didn't really need to be downloaded. I'll leave the carbon footprint calculations for those who are interested in such things.

  16. aeamaea commented on Nov 22, 2020

    @aeamaea

    At 3kg / GB , it's about 3000kg / TB or 240,000 kilograms (half a million pounds) of carbon contributed to the atmosphere, just by the Javacript track on exercism.

    It's probably peanuts compared to all the Netflix streaming that's going on, but I'm going to leave it out here anyway.

    https://www.emergeinteractive.com/insights/detail/does-irresponsible-web-development-contribute-to-global-warming/

  17. SleeplessByte commented on Nov 23, 2020

    @SleeplessByte
    Member

    Peanuts or not, we can all do our part, yes? However, I don't get what you're trying to achieve with your comments. If you have something constructive or actionable, I'd like to hear it.

    In all honestly, the base module set could just be cached and moved to each exercise as it's fetched and any deltas would be downloaded by npm anyway

    "just" is relative and there is nothing simple about that setup for people new to this track. There are other options.

    The JavaScript track is likely completely compatible with yarn 2's implementation of pnp, and to be honest, I was hoping by now that npm tink would be integrated and no longer experimental, but it's not.

    • The reason we chose npm over yarn is pretty arbitrary, but the fact that npm comes with nodejs is a plus. For typescript it's less annoying to have the extra dependency, because typescript is one by itself (via tsc or babel).
    • We probably want to stick with npm for now, until there is better/clearer onboarding for students.
    • npm tink is still experimental.

    For running tests in CLI, we actually do pretty much what you suggest:

    1. it creates a workspace directory
    2. it moves in all the files under test
    3. it has a single copy of node_modules
    4. it runs the tests

    The above is conflict free (because the exercise slugs are unique), and is much fast than symlinking the node_modules directory everywhere. However, the set-up is non-trivial to understand, so it's unlikely we'll introduce it to the students at this time.

  18. aeamaea commented on Nov 23, 2020

    @aeamaea

    I'm new to JS, and it's a general comment, didn't mean to offend anyone. I think being aware of the cost of our day to day choices that seem small in the near term is a "good thing". I mentioned it for that very reason.

    I tried to suggest what I'm doing , though it may not be a generalized solution (and therefore not actionable?) As I said, it's hokey but seems to work for me, for now.

    I like the idea of single copy of node_modules. I'm not sure why it needs to be introduced to the students if the CLI can take care of it behind the scenes. Not trying to start another discussion on what the CLI should/shouldn't do. I'm happy with the CLI as it is.

    P.S. I see the title of the article I posted, that was not my intent, I posted it as a reference to the number I used to do the carbon footprint calculation.

  19. SleeplessByte commented on Nov 23, 2020

    @SleeplessByte
    Member

    I think it's okay to suggest having a better way (aka revisiting this topic) with added benefit of trying to preserve the environment a bit more.

    If you have the time, I wouldn't mind seeing your thoughts in a new issue, and then we/you/I/anyone can do further research to see what we can accomplish without too much pain and suffering for the students.

  20. Marloows commented on Nov 9, 2024

    @Marloows

    Just moved node_modules and package-lock.json one folder up and everything seems magically to work :)

  21. SleeplessByte commented on Nov 9, 2024

    @SleeplessByte
    Member

    It does, and once #2548 lands, you don't need to do that anymore

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    wontfix 🙅‍♀️This won't be applied, fixed, worked on, continued.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions