You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
comp_new ipc failure due to invalid ipc size with SMP arch option #766
I ran into this issue today where the comp_new ipc failed because of invalid size during the suspend_stress_test. I did not use the chrome suspend_stress_test command but instead used a
custom script to suspend and resume the dsp using a debugfs entry.
The ipc size is 76 which seems to be correct but the config.hdr.size doesnt match the expected size.
@jajanusz@dabekjakub, @ranj063 mentioned a similar bug on this mornings call, can we confirm what the hdr.size is on the DSP and host sides. It could be some corruption at one end....
@dabekjakub some more logs from the changes I made today. I printed the ipc header and payload on both the host and DSP and I can clearly see that all the data is there but the ipc still fails. This is really baffling.
changed the title [-]comp_new ipc failure due to invalid ipc size[/-][+]comp_new ipc failure due to invalid ipc size with suspend_stress_test speed up script[/+]on Dec 21, 2018
@dabekjakub some more logs from the changes I made today. I printed the ipc header and payload on both the host and DSP and I can clearly see that all the data is there but the ipc still fails. This is really baffling.
[ 1686.285314] sof-audio sof-audio: error: failed to restore pipeline after resume -22
@ranj063 can you share the FW patch you used for this log also?
@dabekjakub, @xiulipan made an amazing discovery that this could be related to the multicore support in the FW. We are testing this with xtensa arch option instead of xtensa-smp and it seems to be doing better. Do give it a try and let us know.
changed the title [-]comp_new ipc failure due to invalid ipc size with suspend_stress_test speed up script[/-][+]comp_new ipc failure due to invalid ipc size with suspend_stress_test with SMP arch option[/+]on Dec 21, 2018
I dont want to close this issue yet. We have met IPC errors lately in several different situations. This script is the only way to reproduce it consistently. I will work with @fredoh9 to check if the IPC message is intact at the point when the doorbell is sent to the DSP
@naveen-manohar@sathyap-chrome Closing as we don't have this error on RVPs and probably the problem is gone on your side on kernel w/o MSI. Feel free to reopen this issue, if you still have this problem.
Activity
lgirdwood commentedon Dec 20, 2018
@jajanusz @dabekjakub, @ranj063 mentioned a similar bug on this mornings call, can we confirm what the hdr.size is on the DSP and host sides. It could be some corruption at one end....
ranj063 commentedon Dec 20, 2018
@dabekjakub I have pushed some changed to the kernel to print debug messages with the ipc data before sending the ipc to the same branch as below:
https://github.com/ranj063/linux/tree/debug/suspend_stress_test
ranj063 commentedon Dec 21, 2018
@dabekjakub some more logs from the changes I made today. I printed the ipc header and payload on both the host and DSP and I can clearly see that all the data is there but the ipc still fails. This is really baffling.
logger:
0 2 IPC 96065.156250 125.156250 apl-ipc.c:114 ipc: msg rx -> 0x30010000
0 2 IPC 96070.572917 5.416667 handler.c:1064 ipc: hdr 0x30010000 tx (80) > rx (28)
0 2 IPC 96073.854167 3.281250 handler.c:1067 ipc: pipe 5 comp 31 -> new (type 2)
0 2 IPC 96077.395833 3.541667 handler.c:80 ipc header cmd: 0x30010000 size: 80
driver logs:
1686.285024] sof-audio sof-audio: ipc header cmd: 0x30010000 size: 80
[ 1686.285025] sof-audio sof-audio: ipc comp id: 31 type: 2 pipeline id: 5
[ 1686.285026] sof-audio sof-audio: config header
[ 1686.285028] sof-audio sof-audio: ipc header cmd: 0x0 size: 36
[ 1686.285029] sof-audio sof-audio: periods_sink: 0 periods_source: 2 preload_count: 2 frame_fmt: 2 xrun_action: 0
[ 1686.285031] sof-audio sof-audio: dir: 0 dai_index: 3 type: 3 dmac_config: 0
[ 1686.285055] sof-audio sof-audio: ipc tx: 0x30010000
[ 1686.285265] sof-audio sof-audio: ipc rx: 0x90020000
[ 1686.285282] sof-audio sof-audio: ipc rx done: 0x90020000
[ 1686.285302] sof-audio sof-audio: error: ipc error for 0x30010000 size 0x14
[ 1686.285311] sof-audio sof-audio: error: failed to load widget type 27 with ID: 31
[ 1686.285314] sof-audio sof-audio: error: failed to restore pipeline after resume -22
[-]comp_new ipc failure due to invalid ipc size[/-][+]comp_new ipc failure due to invalid ipc size with suspend_stress_test speed up script[/+]keyonjie commentedon Dec 21, 2018
@ranj063 can you share the FW patch you used for this log also?
ranj063 commentedon Dec 21, 2018
@keyonjie here it is:
https://github.com/ranj063/sof/tree/debug/suspend_stress_test
Please make sure to change the xtensa-build-all.sh to use "xtensa" instead of xtensa-smp for ARCH in apollolake.
dabekjakub commentedon Dec 21, 2018
@ranj063 that is in line with what i observed. i will have to track where the change of size to 0 occurs.
ranj063 commentedon Dec 21, 2018
@dabekjakub, @xiulipan made an amazing discovery that this could be related to the multicore support in the FW. We are testing this with xtensa arch option instead of xtensa-smp and it seems to be doing better. Do give it a try and let us know.
[-]comp_new ipc failure due to invalid ipc size with suspend_stress_test speed up script[/-][+]comp_new ipc failure due to invalid ipc size with suspend_stress_test with SMP arch option[/+]20 remaining items
michalgrodzicki commentedon Feb 13, 2019
@ranj063 I'll asign this issue to you.
naveen-manohar commentedon Apr 2, 2019
Issue observed on CML v4 series during stress cycle
2019-03-28T14:33:40,309989+05:30 sof-audio-pci 0000:00:1f.3: error: ipc error for 0x30010000 size 0x14
2019-03-28T14:33:40,309997+05:30 sof-audio-pci 0000:00:1f.3: error: DSP failed to add widget id 0 type 27 name : SSP1.OUT stream SSP1-Codec reply -22
2019-03-28T14:33:40,309999+05:30 sof-audio-pci 0000:00:1f.3: ASoC: failed to load widget SSP1.OUT
2019-03-28T14:33:40,310001+05:30 sof-audio-pci 0000:00:1f.3: error: tplg component load failed -22
2019-03-28T14:33:40,310005+05:30 sof-audio-pci 0000:00:1f.3: error: failed to load DSP topology -22
cc: @sathyap-chrome
sathyap-chrome commentedon Apr 2, 2019
The issue noted by @naveen-manohar is seen on power cycling stress test on CML.
This is seen 1 / 2 times in 100 power cycles.
lgirdwood commentedon Apr 2, 2019
@naveen-manohar @sathyap-chrome could someone attach dmesg and sof-logger output. Thanks.
naveen-manohar commentedon Apr 2, 2019
failure_dmesg.txt
pass_dmesg.txt
PFA dmesg between working and non-working scenario
tlauda commentedon Apr 17, 2019
@naveen-manohar @sathyap-chrome Could you verify with FW having this change: 95b896b? It should fix wrong IPC data on SMP reported in this issue.
jajanusz commentedon Apr 30, 2019
@naveen-manohar @sathyap-chrome ping
sathyap-chrome commentedon May 6, 2019
We failed with similar error.
The test was done using
Topology : cml_drop_0003 branch
Firmware : master
Signature 1:
2019-04-29T12:59:40,045482+05:30 sof-audio-pci 0000:00:1f.3: error: ipc error for 0x30010000 size 20
2019-04-29T12:59:40,045491+05:30 sof-audio-pci 0000:00:1f.3: error: DSP failed to add widget id 0 type 27 name : SSP0.OUT stream SSP0-Codec reply -22
2019-04-29T12:59:40,045495+05:30 sof-audio-pci 0000:00:1f.3: ASoC: failed to load widget SSP0.OUT
Signature 2:
2019-04-29T04:20:08,655337-07:00 sof-audio-pci 0000:00:1f.3: error: ipc error for 0x30010000 size 20
2019-04-29T04:20:08,655346-07:00 sof-audio-pci 0000:00:1f.3: error: DSP failed to add widget id 0 type 27 name : HDA2.OUT stream iDisp3 reply -22
2019-04-29T04:20:08,655350-07:00 sof-audio-pci 0000:00:1f.3: ASoC: failed to load widget HDA2.OUT
dmesg_cold.txt
dmesg_warm.txt
sathyap-chrome commentedon May 6, 2019
Update: Topology and kernel : SOF V0.3 for CML.
tlauda commentedon Jun 6, 2019
@naveen-manohar @sathyap-chrome Do we still observe such issues with kernel without MSI?
jajanusz commentedon Jun 17, 2019
@naveen-manohar @sathyap-chrome Closing as we don't have this error on RVPs and probably the problem is gone on your side on kernel w/o MSI. Feel free to reopen this issue, if you still have this problem.