Repository navigation
Malformed value for primitive Edm.DateTimeOffset type #231
Description
We are using pyodata v1.7.1 the whole time and I'm tried to update to the latest v1.10.0 version. But now I get this error when querying data from the odata service:
Traceback (most recent call last):
File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/model.py", line 534, in from_json
milliseconds_since_epoch = matches.group('milliseconds_since_epoch')
AttributeError: 'NoneType' object has no attribute 'group'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/prefect/engine.py", line 1192, in orchestrate_task_run
result = await run_sync(task.fn, *args, **kwargs)
File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 57, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(call, cancellable=True)
File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/workspace/odatavenv/flow_utilities/prefect/tasks/odata.py", line 60, in fetch_odata
for cnt, batch in enumerate(batches, start=1):
File "/workspace/odatavenv/flow_utilities/api/odata.py", line 114, in query_all
yield list(self.query(skip, top))
File "/workspace/odatavenv/flow_utilities/api/odata.py", line 98, in query
response = self._entity_service().skip(skip).top(top).execute()
File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/service.py", line 349, in execute
return self._call_handler(response)
File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/service.py", line 362, in _call_handler
return self._handler(response)
File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/service.py", line 1509, in get_entities_handler
entity = EntityProxy(self._service, self._entity_set, self._entity_set.entity_type, props)
File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/service.py", line 814, in __init__
self._cache[type_proprty.name] = type_proprty.from_json(proprties[type_proprty.name])
File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/model.py", line 872, in from_json
return self.typ.traits.from_json(value)
File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/model.py", line 537, in from_json
raise PyODataModelError(
pyodata.exceptions.PyODataModelError: Malformed value /Date(1599659339088)/ for primitive Edm.DateTimeOffset type. Expected format is /Date(<ticks>±<offset>)/
Maybe this is related to our instance because they are uncompatible dates (see #143).
Activity
phanak-sap commentedon Sep 9, 2022
Hi sti0,
you created a PR in this library, so you know this is not quite an actionable issue just from information provided so far :)
model.py", line 534with'NoneType' object has no attribute 'group'bonus: I am quite frankly confused by these lines in the stack trace:
you were running async on 1.7.1 version of pyodata? or your code is mostly async, but pyodata was used in sync call, with requests library? Not that this should affect the parsing of DateTimeOffset, just curious.
Also, you may check the new async service initialization feature in 1.10.0 - or was it the true intent behind the version upgrade :) ? It is not documented yet, but from the tests it should be enough of a starting point. You should be able to move to "all http networking is async" in your codebase.
Check fd28f5f
Since I am frankly starting with the whole python async thing, feel free to add any "code review" notes if this feature should be implemented in any better way, perhaps in a new issue. It is so fresh that even backward incompatible changes would be IMHO OK for 1.11.0
phanak-sap commentedon Sep 21, 2022
Hi sti0,
I have tried to reproduce the bug with the info you provided (metadata would be helpful to be sure) and it seems OK to me.
The error
pyodata.exceptions.PyODataModelError: Malformed value /Date(1599659339088)/ for primitive Edm.DateTimeOffset type.is valid for provided input and Edm.DateTimeOffset type.The correct type should be Edm.DateTime for provided input, then is parsed from json to
datetime.datetime(2020, 9, 9, 13, 48, 59, 88000, tzinfo=datetime.timezone.utc). Or the service should return the data according the declared type, Edm.DateTimeOffset, e.g./Date(1599652139000+0060)/.Unless you provide more info or ideally script to reproduce the bug, I am seeing it as Not a bug / Works as designed.
sti0 commentedon Sep 22, 2022
Hi @phanak-sap ,
sorry for the late response. The async calls coming out of our orchestration flows. They have nothing todo with this problem I could reproduce it with a simple script.
First of all here the metadata for one endpoint:
As you can see the
ETagattribute is defined asEdm.DateTimeOffset. This breaks with pyodata 1.10.0.Minimal script:
Error:
In this case I think the Odata Service provides the data in a wrong format (like this one #143 (comment)). Maybe there is any chance to do something like the SCREWED_UP-parameters?
What do you think about?
phanak-sap commentedon Sep 22, 2022
OK, it seems like something to change for Edm.DateTimeOffset parsing of JSON values.
We are currently expecting that the offset part is mandatory and that your service should return
/Date(1659430517461+0000)/for offset set to UTC. I am not able to find in the odata.org specification exactly if the offset part is mandatory, nor in the W3C XML schema. But it seems that at least SAP thinks this should be correct - "As of Release Q1 2019, DateTimeOffset values without timezone information are also accepted in query URLs and edit request payloads. The value will be processed by the server as UTC time."So the fix should be to apply this logic as well - if the offset part is missing, work as if +0000 was provided. But IMHO this should be just for the
from_jsonlogic, theto_jsonshould always contain offset information.Relevant code:
python-pyodata/pyodata/v2/model.py
Line 531 in 5d14908
python-pyodata/tests/test_model_v2.py
Line 749 in 5d14908
Possible quick/ugly workaround:
Seems to me that the literal parsing is correct. Could you try with the call with $value for literal form of the specific entity? IMHO this modification for your script should work (if not, maybe secondary bug :) ):
for d in data.execute().get_value():Longer backgroung explanation:
IMHO it is working on old version, well, just because in 1.7.1 the Edm.DateTimeOffset was effectively "not implemented", with no Traits and in both literal and json format it was just passing the value up trough base class...
links to 1.7.1 tag:
python-pyodata/pyodata/v2/model.py
Line 213 in 5731a4c
python-pyodata/pyodata/v2/model.py
Line 343 in 5731a4c
Now, it is worth going trough the commit ec56995 that implements the Edm.DateTimeOffset as starting point.
https://blogs.sap.com/2017/01/05/date-and-time-in-sap-gateway-foundation/
model: fix edge case for Edm.DateTimeOffset.from_json() without offset
phanak-sap commentedon Sep 23, 2022
Hi @sti0,
check the PR #232 - should this work for you? Also, were you able to use the workaround with the
get_value()?model: fix edge case for Edm.DateTimeOffset.from_json() without offset
model: fix edge case for Edm.DateTimeOffset.from_json() without offset (