You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] IPC timed out when multiple-pipeline on TGLH_RVP_NOCODEC_ZEPHYR #4493
Describe the bug
ipc timed out and IO error , when multiple-pipeline on TGLH_RVP_NOCODEC_ZEPHYR
inner daily 5289?model=TGLH_RVP_NOCODEC_ZEPHYR&testcase=multiple-pipeline-all To Reproduce
TPLG=/lib/firmware/intel/sof-tplg/sof-tgl-h-nocodec.tplg ~/sof-test/test-case/multiple-pipeline.sh -f c -l 50
Environment
Start Time: 2021-07-15 02:10:50 UTC
End Time: 2021-07-15 04:47:20 UTC
Kernel Branch: topic/sof-dev
Kernel Commit: 9b277b80
SOF Branch: main
SOF Commit: 0fa9f7d
Hmm, it seems the occurence rate is less than <1/10. If the multi-pipeline test is run for more than 10 iterations, then you start hitting failures reliably. Curiously, the IPC timeout is preceded by a DSP overload situation:
When the problematic use-case (four concurrent capture streams) is run in normal conditions the DSP load is 30000/34800 ~= 70% , but in the above failing case, same use-case is taking 106% of the DSP.
So something is causing DSP execution to slow down (but not on every iteration of the test case).
Tested with XCC build and the problem is no longer hit. With GCC build, the nocodec pipeline load goes to 90% during the test case and if there are transient spikes in processing, the DSP can no longer catch up. The XCC build has sufficient margin to handle stream start/stop actions on core0 and the problem is not hit.
@lyakh@kv2019i@andyross fyi performance should also align shortly once we move objects into cache region, @ranj063 is working on the coherency issues so that we can use cache alias when needed. @andyross I guess you have plans for cache alisa usage on Zephyr code too ?
@XiaoyunWu6666 This should be fixed now indeed. We do have a new failure that shows up in other tests. You can identify this from IPC timeout for TRIG_STOP. I filed #4680 for this.
Activity
kv2019i commentedon Jul 22, 2021
I could reproduce this now on a TGL system.
kv2019i commentedon Jul 22, 2021
Hmm, it seems the occurence rate is less than <1/10. If the multi-pipeline test is run for more than 10 iterations, then you start hitting failures reliably. Curiously, the IPC timeout is preceded by a DSP overload situation:
68475050 1783204 us, 45 us: ssp_start()
68477892 1783278 us, 74 us: task add 0x9e059d50 0x1fffa180U priority 0 flags 0x0
68482700 1783403 us, 125 us: validate(), ll drift detected, delta = 41508
68522969 1784452 us, 1048 us: OVERRUN in scheduler, missed 1, cycles 40883, cycles-1 32263
68526666 1784548 us, 96 us: validate(), ll drift detected, delta = 43966
68613050 1786798 us, 2249 us: validate(), ll drift detected, delta = 46118
68635382 1787379 us, 581 us: OVERRUN in scheduler, missed 1, cycles 41129, cycles-1 30558
68656862 1787939 us, 559 us: validate(), ll drift detected, delta = 43813
68710141 1789326 us, 1387 us: validate(), ll drift detected, delta = 53279
When the problematic use-case (four concurrent capture streams) is run in normal conditions the DSP load is 30000/34800 ~= 70% , but in the above failing case, same use-case is taking 106% of the DSP.
So something is causing DSP execution to slow down (but not on every iteration of the test case).
kv2019i commentedon Jul 22, 2021
Tested with XCC build and the problem is no longer hit. With GCC build, the nocodec pipeline load goes to 90% during the test case and if there are transient spikes in processing, the DSP can no longer catch up. The XCC build has sufficient margin to handle stream start/stop actions on core0 and the problem is not hit.
XCC build requires recently merged XCC patchset to Zephyr -> zephyrproject-rtos/zephyr#36787
lgirdwood commentedon Jul 23, 2021
@lyakh @kv2019i @andyross fyi performance should also align shortly once we move objects into cache region, @ranj063 is working on the coherency issues so that we can use cache alias when needed. @andyross I guess you have plans for cache alisa usage on Zephyr code too ?
kv2019i commentedon Aug 24, 2021
Potential bugfix: #4673
kv2019i commentedon Aug 25, 2021
@XiaoyunWu6666 This should be fixed now indeed. We do have a new failure that shows up in other tests. You can identify this from IPC timeout for TRIG_STOP. I filed #4680 for this.
XiaoyunWu6666 commentedon Aug 26, 2021
inner daily 6090 shows good result
Close it via #4673