Official 5.6.0 binaries crash on Ubuntu Trust 14.04 x64

I have an amazon instance running Ubuntu 14.04 Trusty x86_64. When I download the "Linux Binaries" from the nodejs.org homepage, the node binary will crash just trying to open the REPL.

If I run node -v I get 5.6.0. If I run just node I get

node: ../src/cares_wrap.cc:1262: void node::cares_wrap::Initialize(v8::Local<v8::Object>, v8::Local<v8::Value>, v8::Local<v8::Context>): Assertion `(r) == (0)' failed.
Aborted (core dumped)

Does the official v5.5.0 binary work?

@indutny

/cc @indutny

@cjthompson

Can't reproduce with the x86 and x64 binaries on FC23 but f1a0827 is the obvious suspect.

@cjthompson Can you try building from source with and without that commit?

@cjthompson

This is very interesting crash, especially considering that we smoke test the binaries.

@cjthompson may I ask you to share it /etc/resolv.conf with us, if it doesn't contain any private information? Additionally, may I ask you to run it with strace node and gist logs?

Thank you so much for reporting this.

The issue appears to be related to /etc/nsswitch.conf not being world-readable. Can anyone confirm if this makes sense? The 5.5.0 binary on the same server worked just fine.

EDIT: Might be related to this issue: c-ares/c-ares#33

strace log: https://gist.github.com/cjthompson/5570c04744715f96dcb5

it's pretty clear at the bottom of the log there's a call to open /etc/nsswitch.conf and permission is denied at which point node promptly crashes.

@cjthompson

@cjthompson it looks like there were no lookup entries in /etc/resolv.conf and because of this is cares tried to read nsswitch.conf.

@cjthompson

@cjthompson and thank you very much for strace log, it really helped!

At this point I think we've narrowed down the exact cause and the fact that it was a change from 5.5.0 to 5.6.0.

Let me know if there's anything more you need me to debug.

@indutny

@indutny Want to revert for now? I believe the next v5 release is next week and it'd be a shame to have it broken in two subsequent releases. (I'm assuming we won't have a fix upstreamed in that time.)

@bnoordhuis

@bnoordhuis let's revert it until resolving this.

@cjthompson may I ask you to do strace thing with 5.5.0 too? Many thanks in advance.

strace of 5.6.0 with nsswitch.conf permissions set to 644 https://gist.github.com/cjthompson/6243cfc5ff1063928c90

strace of 5.5.0 with nsswitch.conf permissions set to 600
https://gist.github.com/cjthompson/2889a83e10e1b0d3ad2e

@cjthompson

Whoa, finally found the reason why behavior is different:

node/deps/cares/src/ares_init.c

Lines 213 to 230 in 2848f84

    
           status2 = init_by_defaults(channel); 
        
           if (status2 != ARES_SUCCESS) { 
        
             DEBUGF(fprintf(stderr, "Error: init_by_defaults failed: %s\n", 
        
                            ares_strerror(status))); 
        
             if (status == ARES_SUCCESS) 
        
               status = status2; 
        
           } 
        
           /* Generate random key */ 
        
           if (status == ARES_SUCCESS) { 
        
             status = init_id_key(&channel->id_key, ARES_ID_KEY_LEN); 
        
             if (status == ARES_SUCCESS) 
        
               channel->next_id = ares__generate_new_id(&channel->id_key); 
        
             else 
        
               DEBUGF(fprintf(stderr, "Error: init_id_key failed: %s\n", 
        
                              ares_strerror(status))); 
        
           }

Basically, 5.5.0 is swallowing that lookup init error. @cjthompson may I ask you to run this js snippet:

require('dns').reverse('8.8.8.8', function() {console.log(arguments)})

on v5.5.0 with strict permissions on nsswitch.conf (the ones on which v5.6.0 crashes)? I expect that it will either error or (more likely) crash.

@bnoordhuis

@bnoordhuis please disregard my comment about reverting it, looks like this behavior change is actually a pretty important bug fix. At least let's wait for results of dns.reverse testing that I have just asked @cjthompson about.

Node 5.5.0 with nsswitch.conf permissions as 600:

QueryReqWrap {
  bindingName: 'getHostByAddr',
  callback: { [Function: asyncCallback] immediately: true },
  hostname: '8.8.8.8',
  oncomplete: [Function: onresolve] }
{ '0':
   { [Error: getHostByAddr ENOTFOUND 8.8.8.8]
     code: 'ENOTFOUND',
     errno: 'ENOTFOUND',
     syscall: 'getHostByAddr',
     hostname: '8.8.8.8' } }

node 5.5.0 and 5.6.0 with nsswitch.conf set to 644:

QueryReqWrap {
  bindingName: 'getHostByAddr',
  callback: { [Function: asyncCallback] immediately: true },
  hostname: '8.8.8.8',
  oncomplete: [Function: onresolve] }
{ '0': null, '1': [ 'google-public-dns-a.google.com' ] }

@cjthompson

Ah, this is actually better than I expected. Thank you so much, @cjthompson !

@bnoordhuis looks like we will need to revert it after all. Sounds like a plain bug in c-ares. It is caused by c-ares/c-ares@46bb820 , and proposed fix is: c-ares/c-ares#39 .

We can always re-land the c-ares update after c-ares/c-ares#39 will land.

The patch has been landed c-ares/c-ares@e514088 . I'm going to wait for reply from c-ares team with regards to use of c-ares master branch in our repo, and depending on it we will be able to decide what to do next.

@bnoordhuis

Alright, I have a green light from c-ares team. @bnoordhuis I'm going to submit PR for cherry-picking that commit as soon as I will wake up (/me is sleeptyping at the moment).

Should be fixed by #5199

Is this still an issue? Do we need to keep this open?

I'll close. I forgot to after landing the revert.

Navigation Menu

Repository navigation

Official 5.6.0 binaries crash on Ubuntu Trust 14.04 x64 #5185

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	status2 = init_by_defaults(channel);
	if (status2 != ARES_SUCCESS) {
	DEBUGF(fprintf(stderr, "Error: init_by_defaults failed: %s\n",
	ares_strerror(status)));
	if (status == ARES_SUCCESS)
	status = status2;
	}

	/* Generate random key */

	if (status == ARES_SUCCESS) {
	status = init_id_key(&channel->id_key, ARES_ID_KEY_LEN);
	if (status == ARES_SUCCESS)
	channel->next_id = ares__generate_new_id(&channel->id_key);
	else
	DEBUGF(fprintf(stderr, "Error: init_id_key failed: %s\n",
	ares_strerror(status)));
	}

Official 5.6.0 binaries crash on Ubuntu Trust 14.04 x64 #5185

Description

Activity

mscdex commented on Feb 10, 2016

mscdex commented on Feb 10, 2016

bnoordhuis commented on Feb 10, 2016

indutny commented on Feb 10, 2016

cjthompson commented on Feb 10, 2016

cjthompson commented on Feb 10, 2016

indutny commented on Feb 10, 2016

indutny commented on Feb 10, 2016

cjthompson commented on Feb 10, 2016

bnoordhuis commented on Feb 10, 2016

indutny commented on Feb 10, 2016

cjthompson commented on Feb 10, 2016

indutny commented on Feb 10, 2016

indutny commented on Feb 10, 2016

cjthompson commented on Feb 10, 2016

indutny commented on Feb 10, 2016

indutny commented on Feb 11, 2016

indutny commented on Feb 11, 2016

indutny commented on Feb 11, 2016

jasnell commented on Apr 19, 2016

bnoordhuis commented on Apr 19, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions