Repository navigation
Seg Fault while loading (my special) dcp-client-bundle #231
Description
Issue type
Bug
How did you install PythonMonkey?
Source
OS platform and distribution
Linux Ubuntu
Python version (python --version)
Python 3.10.12
PythonMonkey version (pip show pythonmonkey)
will/crypto-module-buildin
Bug Description
Yeah so, @Hamada-Distributed @wesgarland and I were trying to load dcp-client dcp-client-bundle in PythonMonkey so I could deploy a job using the dcp api!
In order to do this, I had to make my own dcp-client-bundle which differs in very significant ways from the dcp-client-bundle shipped with the above repo - in fact, I don't think I can share it on GitHub publicly! Ask me for it in a dm if you work for Distributive (;
I was excited to see I got to the exec readystatechange 😲 ! However, right after that, the process seg faults...
Here's a backtrace:
Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
0x00007ffff5552b98 in js::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, js::AnyInvokeArgs const&, JS::MutableHandle<JS::Value>, js::CallReason) () from /home/will/git/pythonmonkey/python/pythonmonkey/libmozjs-115.so
(gdb) bt
#0 0x00007ffff5552b98 in js::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, js::AnyInvokeArgs const&, JS::MutableHandle<JS::Value>, js::CallReason) () from /home/will/git/pythonmonkey/python/pythonmonkey/libmozjs-115.so
#1 0x00007ffff55e81ee in JS_CallFunctionValue(JSContext*, JS::Handle<JSObject*>, JS::Handle<JS::Value>, JS::HandleValueArray const&, JS::MutableHandle<JS::Value>) () from /home/will/git/pythonmonkey/python/pythonmonkey/libmozjs-115.so
#2 0x00007ffff7ba7942 in callJSFunc (jsCxThisFuncTuple=<optimized out>, args=<optimized out>) at /home/will/git/pythonmonkey/_spidermonkey_install/include/mozjs-115/js/RootingAPI.h:1219
#3 0x00005555556ae138 in ?? ()
#4 0x00005555556a4a7b in _PyObject_MakeTpCall ()
#5 0x00007ffff7b9a07f in eventLoopJobWrapper (jobFn=<optimized out>, _unused__=<optimized out>) at /home/will/git/pythonmonkey/src/PyEventLoop.cc:9
#6 0x00005555556aba3e in ?? ()
#7 0x000055555578abc5 in ?? ()
#8 0x0000555555806572 in ?? ()
#9 0x00005555556a199b in ?? ()
#10 0x00005555556995d7 in _PyEval_EvalFrameDefault ()
#11 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#12 0x000055555569745c in _PyEval_EvalFrameDefault ()
#13 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#14 0x000055555569745c in _PyEval_EvalFrameDefault ()
#15 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#16 0x000055555569745c in _PyEval_EvalFrameDefault ()
#17 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#18 0x000055555569745c in _PyEval_EvalFrameDefault ()
#19 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#20 0x000055555569853c in _PyEval_EvalFrameDefault ()
#21 0x00005555556939c6 in ?? ()
#22 0x0000555555789256 in PyEval_EvalCode ()
#23 0x00005555557b4108 in ?? ()
#24 0x00005555557ad9cb in ?? ()
#25 0x00005555557b3e55 in ?? ()
#26 0x00005555557b3338 in _PyRun_SimpleFileObject ()
#27 0x00005555557b2f83 in _PyRun_AnyFileObject ()
#28 0x00005555557a5a5e in Py_RunMain ()
#29 0x000055555577c02d in Py_BytesMain ()
#30 0x00007ffff7c29d90 in __libc_start_call_main (main=main@entry=0x55555577bff0, argc=argc@entry=2, argv=argv@entry=0x7fffffffde08) at ../sysdeps/nptl/libc_start_call_main.h:58
#31 0x00007ffff7c29e40 in __libc_start_main_impl (main=0x55555577bff0, argc=2, argv=0x7fffffffde08, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffddf8) at ../csu/libc-start.c:392
#32 0x000055555577bf25 in _start ()^ Philippe suggested adding additional flags to setup.sh to help debug this --- I would but I unfortunately don't have the time to look deeper into this right now 😭
Philippe also notes it may be related to the async code which @Xmader has experience with and could provide useful insight into!
TL;DR
- I made a custom dcp-client-bundle.js which you have to ask me for on Slack since I'm not sure if I can post it on a public GitHub repo (it's unminified etc)
- I load it in the script (pasted below in the "Standalone code to reproduce the issue" issue)
- I get a seg fault sometime after I deploy a job and it gets to the "readystatechange" "exec" state!
NOTE:
- you will see lots of weird javascript stacktraces above, they are just console.logs... ignore them.... Just look at the seg fault at the very bottom
Let me know if you run into any issues reproing this and I can help (*:
Standalone code to reproduce the issue
#! /usr/bin/env python3
"""
dcp-client nodejs uses the following modules:
process.mainModulepath.join
path.resolve
path.dirname
path.basename
path.isAbsolute
path.sep
path.parsefs.readFileSync
fs.mkdirSync
!fs.existsSync
fs.promises.mkdir
fs.promises.writeFile
fs.statSync
fs.constants.W_OK
"""
import pythonmonkey as pm
import os
import urllib.request;
import asyncio
def readFileSync(filename, charset) -> str:
"""
Utility function for reading files.
Returns:
str: The contents of the file
"""
filename = os.path.normpath(filename)
with open(filename, "r", encoding=charset) as fileHnd:
return fileHnd.read()
file = readFileSync('/some/path/to/my/special/dcp-client-bundle.js', 'utf-8');
async def initDcpClient():
try:
dcpConfigSrc = urllib.request.urlopen('https://scheduler.distributed.computer/etc/dcp-config.js').read().decode();
pmRequire = pm.createRequire('.', False)
#pm.eval('globalThis')['crypto'] = pmRequire('crypto')
pm.eval('globalThis.window = {}; globalThis.dcpConfig =' + dcpConfigSrc);
pm.eval('delete globalThis.window')
pm.eval('globalThis.dcpConfig.build = "debug";');
dcpClient = pm.eval(file)
pm.eval('globalThis')['dcp'] = dcpClient;
pm.eval(''''use strict';
const job = dcp.compute.for(Array.from('hello world'), function workFn(letter) {
progress();
return letter.toUpperCase();
});
''');
print("about to eval after compoute.for")
pm.eval('''
//console.log(job); // uncomment this line for a fun seg fault : D
console.log("holy crap it didn't crash");
job.on('readystatechange', (state) => console.log(`readystatechange: ${state}`));
job.on('submit', console.log);
job.on('accepted', console.log);
''')
#my_epic_keystore = await pm.new(dcpClient.wallet.Keystore)('{"address":"0x6db720Ec2e7DB70737Cf0CC6986711e653a25E71","crypto":{"ciphertext":"43a8dc28fd6c3a571ec719b6ca86100ccd5e128431fc34dbc1d80875b3bf7aa2","cipherparams":{"iv":"2cbd4e1db754b4839741138d4b33c5a1"},"cipher":"aes-128-ctr","kdf":"scrypt","kdfparams":{"dklen":32,"salt":"be5a8db9714f86146f4f03322d43b182c4faaba1cd95680ffca94ea5b6648419","n":1024,"r":8,"p":1},"mac":"9e4ffb60a33d57db3d25e6b8077afd53a8b194bb68c6b38b90205c53c4d62470"},"version":3}')
job_deploy = pm.eval('''
async function jobDeploy()
{
const theKeystore = await new dcp.wallet.Keystore('{"address":"0x6db720Ec2e7DB70737Cf0CC6986711e653a25E71","crypto":{"ciphertext":"43a8dc28fd6c3a571ec719b6ca86100ccd5e128431fc34dbc1d80875b3bf7aa2","cipherparams":{"iv":"2cbd4e1db754b4839741138d4b33c5a1"},"cipher" :"aes-128-ctr","kdf":"scrypt","kdfparams":{"dklen":32,"salt":"be5a8db9714f86146f4f03322d43b182c4faaba1cd95680ffca94ea5b6648419","n":1024,"r":8,"p":1},"mac":"9e4ffb60a33d57db3d25e6b8077afd53a8b194bb68c6b38b90205c53c4d62470"},"version":3}'); // yes this is a real dcp keystore with real funds on it... go ahead and steal the funds from it; idc, I mean I'd prefer if you don't but whatever... If you don't work at distributive and you're reading this github issue and you tranferred the funds from this account into yours, you're probably cool! Send me an email - let's chat <willkantorpringle@gmail.com>
const theIdKeystore = await new dcp.wallet.IdKeystore('{"address":"0x6db720Ec2e7DB70737Cf0CC6986711e653a25E71","crypto":{"ciphertext":"43a8dc28fd6c3a571ec719b6ca86100ccd5e128431fc34dbc1d80875b3bf7aa2","cipherparams":{"iv":"2cbd4e1db754b4839741138d4b33c5a1"},"cipher" :"aes-128-ctr","kdf":"scrypt","kdfparams":{"dklen":32,"salt":"be5a8db9714f86146f4f03322d43b182c4faaba1cd95680ffca94ea5b6648419","n":1024,"r":8,"p":1},"mac":"9e4ffb60a33d57db3d25e6b8077afd53a8b194bb68c6b38b90205c53c4d62470"},"version":3}');
job.setPaymentAccountKeystore(theKeystore);
dcp.wallet.addId(theIdKeystore);
return job.exec() //.then(results => console.log('wtf', results));
}
jobDeploy''')
my_ret = await job_deploy()
except Exception as error:
print('Error loading bundle:', error)
await pm.wait() # blocks until all asynchronous calls finish
asyncio.run(initDcpClient(), debug=True)Relevant log output or backtrace
Segmentation fault (core dumped)Additional info if applicable
Message me on slack for my secret special `dcp-client-bundle.js`What branch of PythonMonkey were you developing on? (If applicable)
will/crypto-module-buildin
Activity
wiwichips commentedon Jan 31, 2024
^ in the python code, this is the line which causes the segmentation fault
wiwichips commentedon Jan 31, 2024
you may need to
npm i crypto-browserifyin the directory you run the python file in <-- I forget, maybe you don't...wiwichips commentedon Jan 31, 2024
For more information, refer to the slack message on the private Distributive slack which only Distributive developers have access to: https://dstrbtv.slack.com/archives/C03RNFRL4NQ/p1706736863134069
wiwichips commentedon Feb 2, 2024
use branch
will/crypto-module-buildinphilippedistributive commentedon Feb 13, 2024
with the code on branch #234 we get a wrapped up error instead:
Exception in callback JSFunctionProxy.eventLoopJobWrapper()
handle: <TimerHandle when=197397.648137234 JSFunctionProxy.eventLoopJobWrapper() created at /home/philippe/.pyenv/versions/3.11.7/lib/python3.11/asyncio/events.py:80>
source_traceback: Object created at (most recent call last):
File "/home/philippe/Sources/PythonMonkey/main3-pringle-edition.py", line 93, in
asyncio.run(initDcpClient(), debug=True)
File "/home/philippe/.pyenv/versions/3.11.7/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
File "/home/philippe/.pyenv/versions/3.11.7/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
File "/home/philippe/.pyenv/versions/3.11.7/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete
self.run_forever()
File "/home/philippe/.pyenv/versions/3.11.7/lib/python3.11/asyncio/base_events.py", line 607, in run_forever
self._run_once()
File "/home/philippe/.pyenv/versions/3.11.7/lib/python3.11/asyncio/base_events.py", line 1914, in _run_once
handle._run()
File "/home/philippe/.pyenv/versions/3.11.7/lib/python3.11/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
Traceback (most recent call last):
File "/home/philippe/.pyenv/versions/3.11.7/lib/python3.11/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
pythonmonkey.SpiderMonkeyError: Error in file evaluate line 4037 > eval, on line 90:
Error: Assertion failure
This is because of the fix that checks the call result at line 9:
#5 0x00007ffff7b9a07f in eventLoopJobWrapper (jobFn=, unused_=) at /home/will/git/pythonmonkey/src/PyEventLoop.cc:9
and returns null instead of crashing on a null callback result
[-]Mysterious Seg Fault while loading (my special) dcp-client-bundle[/-][+]Seg Fault while loading (my special) dcp-client-bundle[/+]philippedistributive commentedon Feb 15, 2024
I see this
readystatechange: exec
readystatechange: init
readystatechange: preauth
readystatechange: deploying
philippedistributive commentedon Feb 15, 2024
I also see a stack trace being printed:
calling requireNative { '0': '
@kingsds
/socket.io-client' } TRACE webpackNativeBridge$$requireNative@webp
ack://dcp/./src/dcp-client/webpack-native-bridge.js?:27:19
SocketIOTransport@webpack://dcp/./src/protocol-v4/transport/socketio.js?:142:15
a$connectWithTransport/<@webpack://dcp/./src/protocol-v4/connection/connection.js?:783:27
a$connectWithTransport@webpack://dcp/./src/protocol-v4/connection/connection.js?:781:14
a$connectToTargetTransport@webpack://dcp/./src/protocol-v4/connection/connection.js?:817:46
a$_connect@webpack://dcp/./src/protocol-v4/connection/connection.js?:491:32
asyncconnect@webpack://dcp/./src/protocol-v4/connection/connection.js?:401:12
asyncsend@webpack://dcp/./src/protocol-v4/connection/connection.js?:1658:22
request@webpack://dcp/./src/protocol-v4/connection/connection.js?:1700:17
useDeployConnection@webpack://dcp/./src/dcp-client/job/index.js?:1653:49
deployJob@webpack://dcp/./src/dcp-client/job/index.js?:1561:31
asyncexec@webpack://dcp/./src/dcp-client/job/index.js?:1051:20
asyncjobDeploy@evaluate:16:16
top first as usual...
the top is function is webpackNativeBridge$$requireNative in file webpack-native-bridge.js indeed
the bottom also looks good!
philippedistributive commentedon Feb 15, 2024
it eventually fails:
Error loading bundle: Error in file evaluate line 4591 > eval, on line 31, column 32:
Error: There is no native require() on this platform
Stack Trace:
webpackNativeBridge$$requireNative@webpack://dcp/./src/dcp-client/webpack-native-bridge.js?:31:32
SocketIOTransport@webpack://dcp/./src/protocol-v4/transport/socketio.js?:142:15
a$connectWithTransport/<@webpack://dcp/./src/protocol-v4/connection/connection.js?:783:27
a$connectWithTransport@webpack://dcp/./src/protocol-v4/connection/connection.js?:781:14
a$connectToTargetTransport@webpack://dcp/./src/protocol-v4/connection/connection.js?:817:46
a$_connect@webpack://dcp/./src/protocol-v4/connection/connection.js?:491:32
asyncconnect@webpack://dcp/./src/protocol-v4/connection/connection.js?:401:12
asyncsend@webpack://dcp/./src/protocol-v4/connection/connection.js?:1658:22
request@webpack://dcp/./src/protocol-v4/connection/connection.js?:1700:17
useDeployConnection@webpack://dcp/./src/dcp-client/job/index.js?:1653:49
deployJob@webpack://dcp/./src/dcp-client/job/index.js?:1561:31
asyncexec@webpack://dcp/./src/dcp-client/job/index.js?:1051:20
asyncjobDeploy@evaluate:16:16
then the pm error happens
Exception in callback JSFunctionProxy.eventLoopJobWrapper()
OK...if I take in @tom Tang's Socket.io-related fix, I no longer have the pm error
This is the fix to #232 here a001cca
philippedistributive commentedon Feb 15, 2024
so the question is why does nativeRequire succeed so many times and then eventually fails
with the same exact stack trace every time
so the question is why does nativeRequire succeed so many times and then eventually fails (edited)
with the same exact stack trace every time
Bryan Hoang
That error occurs when eval('typeof require === "function"') returns false. Something funky is going on with the module system (i.e., require becoming... undefined?) when it gets to that exact point. Unfortunately I don't have a clue why that would be the case
A potential difference between the successful calls & and erroneous call to requireNative is that the successful ones only occur during module initialization/loading from what I can tell. i.e., at the top level when all the CJS modules are being resolved synchronously. The erroneous call to requireNative occurs after all the modules are resolved/loaded, during a call to a constructor. I know we've ran into issues where that difference mattered in the past. (edited)
philippedistributive commentedon Feb 21, 2024
Note: Will had worked some changes in feature/DCP-4054/make-dcp-client-grok-pythonmonkey, we should continue from there
also some work for https://distributive.atlassian.net/browse/DCP-4054 done in there
philippedistributive commentedon Feb 21, 2024
LOTS more going on in slack
wesgarland commentedon Mar 14, 2024
working now with grok pythonmonkey branch