Skip to content

test: investigate child-process-pass-fd #15656

Closed
Closed
test: investigate child-process-pass-fd#15656
Labels
aixIssues and PRs related to the AIX platform.child_processIssues and PRs related to the child_process subsystem.flaky-testIssues and PRs involving tests that fail intermittently in CI.testIssues and PRs related to Node.js core tests and test infrastructure.

Description

@BridgeAR
BridgeAR
opened on Sep 28, 2017

This failed on a recent CI run.

https://ci.nodejs.org/job/node-test-commit-aix/8817/nodes=aix61-ppc64/console

not ok 1753 sequential/test-child-process-pass-fd
  ---
  duration_ms: 2.0
  severity: fail
  stack: |-
    events.js:182
          throw er; // Unhandled 'error' event
          ^
    
    Error: spawn /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/node EAGAIN
        at _errnoException (util.js:1018:13)
        at Process.ChildProcess._handle.onexit (internal/child_process.js:202:19)
        at onErrorNT (internal/child_process.js:390:16)
        at _combinedTickCallback (internal/process/next_tick.js:138:11)
        at process._tickCallback (internal/process/next_tick.js:180:9)
        at Function.Module.runMain (module.js:643:11)
        at startup (bootstrap_node.js:187:16)
        at bootstrap_node.js:608:3
  ...

Activity

  1. BridgeAR
    added
    aixIssues and PRs related to the AIX platform.
    flaky-testIssues and PRs involving tests that fail intermittently in CI.
    testIssues and PRs related to Node.js core tests and test infrastructure.
    on Sep 28, 2017

  2. mhdawson commented on Sep 28, 2017

    @mhdawson
    mhdawson
    on Sep 28, 2017
    Member

    I think I've see if fail a few times today

  3. mhdawson commented on Sep 28, 2017

    @mhdawson
    mhdawson
    on Sep 28, 2017
    Member

    I don't see any changes to the test recently.

  4. mhdawson commented on Sep 28, 2017

    @mhdawson
    mhdawson
    on Sep 28, 2017
    Member

    Stress test https://ci.nodejs.org/job/node-stress-single-test/1423/

  5. mhdawson commented on Sep 28, 2017

    @mhdawson
    mhdawson
    on Sep 28, 2017
    Member

    @nodejs/platform-aix

  6. mscdex
    added
    child_processIssues and PRs related to the child_process subsystem.
    on Sep 28, 2017

  7. richardlau commented on Sep 29, 2017

    @richardlau
    richardlau
    on Sep 29, 2017
    Member

    Stress test (with corrected RUN_TESTS parameter): https://ci.nodejs.org/job/node-stress-single-test/1425/

  8. mhdawson commented on Sep 29, 2017

    @mhdawson
    mhdawson
    on Sep 29, 2017
    Member

    Even though stress test passed, still seeing failures: https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/8867/console

    just this morning.

  9. BridgeAR commented on Oct 1, 2017

    @BridgeAR
    BridgeAR
    on Oct 1, 2017
    MemberAuthor

    This is the test that I see failing most often currently. It would really be great to get this fixed.

    Ping @nodejs/testing @nodejs/platform-aix

  10. Trott commented on Oct 1, 2017

    @Trott
    Trott
    on Oct 1, 2017
    Member

    Stress test with 10K runs rater than 200: https://ci.nodejs.org/job/node-stress-single-test/1431/

  11. Trott commented on Oct 1, 2017

    @Trott
    Trott
    on Oct 1, 2017
    Member

    This test does spawn 80 copies of itself, so perhaps an EAGAIN is something it should reasonably accommodate (either by ignoring or by retrying).

    Pinging @santigimeno for an opinion on that approach as they wrote the test...

  12. Trott commented on Oct 2, 2017

    @Trott
    Trott
    on Oct 2, 2017
    Member

    Build failure on the stress test. Let's try agian: https://ci.nodejs.org/job/node-stress-single-test/1432/nodes=aix61-ppc64/console

    EDIT: Hmm, I guess this 10K tests will take more than a day to complete. Still, would like to let it run for a few hours at least...

    EDIT AGAIN: ...and the weekend is probably the best time for that so....

  13. Trott commented on Oct 2, 2017

    @Trott
    Trott
    on Oct 2, 2017
    Member

    844 successful runs and no failures. I'm going to cancel the stress test.

  14. Trott commented on Oct 2, 2017

    @Trott
    Trott
    on Oct 2, 2017
    Member

    Small sample size, but I suspect the issue may be host-specific.

    The two failures documented in this issue both were on test-osuosl-aix61-ppc64_be-2. The two successful stress tests were both on test-osuosl-aix61-ppc64_be-1.

    Perhaps not coincidentally, test-osuosl-aix61-ppc64_be-2 is exhibiting more extensive problems right now. https://ci.nodejs.org/computer/test-osuosl-aix61-ppc64_be-2/builds is all red right now. It's not even building, erroring out with this:

    Details 
    Started by upstream project "node-test-commit-aix" build number 8918
originally caused by:
 Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on test-osuosl-aix61-ppc64_be-2 (aix61-ppc64) in workspace /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64
Cloning the remote Git repository
Cloning repository https://github.com/nodejs/node.git
 > git init /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64 # timeout=30
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Could not init /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:717)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:511)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
	at hudson.remoting.UserRequest.perform(UserRequest.java:152)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:332)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:274)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
	at hudson.remoting.Engine$1$1.run(Engine.java:85)
	at java.lang.Thread.run(Thread.java:809)
	at ......remote call to Channel to /140.211.9.100(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
	at hudson.remoting.Channel.call(Channel.java:830)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
	at sun.reflect.GeneratedMethodAccessor539.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
	at com.sun.proxy.$Proxy79.execute(Unknown Source)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1075)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115)
	at hudson.scm.SCM.checkout(SCM.java:496)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1281)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
	at hudson.model.Run.execute(Run.java:1728)
	at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:405)
Caused by: hudson.plugins.git.GitException: Error performing command: git init /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1931)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1892)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1888)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1533)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:715)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:511)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
	at hudson.remoting.UserRequest.perform(UserRequest.java:152)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:332)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:274)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
	at hudson.remoting.Engine$1$1.run(Engine.java:85)
	at java.lang.Thread.run(Thread.java:809)
Caused by: java.io.IOException: Cannot run program "git" (in directory "/home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64"): error=11, Resource temporarily unavailable
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1059)
	at hudson.Proc$LocalProc.<init>(Proc.java:245)
	at hudson.Proc$LocalProc.<init>(Proc.java:214)
	at hudson.Launcher$LocalLauncher.launch(Launcher.java:846)
	at hudson.Launcher$ProcStarter.start(Launcher.java:384)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1920)
	... 16 more
Caused by: java.io.IOException: error=11, Resource temporarily unavailable
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:212)
	at java.lang.ProcessImpl.start(ProcessImpl.java:164)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1040)
	... 21 more
ERROR: Error cloning remote repo 'origin'
Run condition [Always] enabling perform for step [[]]
TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: *.tap
Did not find any matching files. Setting build result to FAILURE.
Checking ^not ok
Jenkins Text Finder: File set '*.tap' is empty
Sending e-mails to: michael_dawson@ca.ibm.com gib@uk.ibm.com Bethany.Griggs@uk.ibm.com
Notifying upstream projects of job completion
Finished: FAILURE

    With any luck, re-imaging the host or whatever will make this all go away.

    In the meantime, I'm going to take the node out of Jenkins.

    I'll open an issue in the build repo, but /cc @nodejs/build in case there's more insight about the specific test.

  15. mhdawson commented on Oct 2, 2017

    @mhdawson
    mhdawson
    on Oct 2, 2017
    Member

    I was going to check be-1 but I see that there is a stress test running there and it seems to be passing ok:

    https://ci.nodejs.org/job/node-stress-single-test/nodes=aix61-ppc64/1433/console

    1247 successful runs so far ...

  16. mhdawson commented on Oct 2, 2017

    @mhdawson
    mhdawson
    on Oct 2, 2017
    Member

    and this was opened/handled to address the failure to build problem that was seen earlier: nodejs/build#900 (comment)

  17. apapirovski commented on Apr 12, 2018

    @apapirovski
    apapirovski
    on Apr 12, 2018
    Contributor

    @Trott @mhdawson any status updates on this? It seems like this should maybe be closed now?

  18. Trott commented on Apr 12, 2018

    @Trott
    Trott
    on Apr 12, 2018
    Member

    Seems likely that it was fixed either via infra fixes or else in d64b0a8. Haven't noticed it in a long time and can't find it in CI. Closing. Feel free to re-open if you think I'm wrong.

  19. Trott
    closed this as completedon Apr 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    aixIssues and PRs related to the AIX platform.child_processIssues and PRs related to the child_process subsystem.flaky-testIssues and PRs involving tests that fail intermittently in CI.testIssues and PRs related to Node.js core tests and test infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions