You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] If ChainDMA is enabled FW crashes during multiple-pause-resume test #7084
To Reproduce
Enable ChainDMA for TGL by reverting c3807ae and run on TGL upx-i11:
TPLG=/lib/firmware/intel/avs-tplg/sof-hda-generic.tplg MODEL=TGLU_UP_HDA_IPC4ZPH ~/work/sof-test/test-case/multiple-pause-resume.sh -r 50
Reproduction Rate
Does not always fail immediately, but sometimes takes over ten seconds. However, the FW always crashes well before the test completes.
Expected behavior
multiple-pause-resume test would pass.
Impact
Can not enable ChainDMA when its broken and no multichannel audio without ChainDMA.
Environment
Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
I have executed the test several times and there is no clear pattern to be seen on what happened on FW logs before the crash. However, from the Linux side logs its visible that the crash always happens while handling ChainDMA messages:
Feb 13 15:35:58 jhary kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc timed out for 0xe040000|0x0
...
Feb 13 16:19:46 jhary kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc timed out for 0xe050101|0x0
...
Feb 13 16:26:02 jhary kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc timed out for 0xe070101|0x0
...
Feb 13 22:13:53 jhary kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc timed out for 0xe050000|0x0
This thing is annoyingly getting out of hand. This latest problem was not there couple of weeks back when I was tracking some earlier issue with ChainDMA, and its not a result of a change in the chain dma code, since the only change there is 6009c21 and reverting it does not help with the issue. We can not have ChainDMA enabled because its broken, and because its not enabled, it gets even more broken.
I can of course provide a full Linux side log, or even FW log, but I doubt that will help you to track down the problem. The multiple-pause-resume test is a stress test in nature, and there hundreds of IPC message before the crash happens.
We are interrested in full linux IPC log because we would like to reproduce the exact flow with specific ipc timings and all payloads necessary for module initializations. We would like to identify porential dependencies and detailed FW state before chain dma running. It is also worth to check how many chain DMA IPCs in average is needed to reproduce the problem .... (would be helpful )
So far we haven't encountered reproduction applying clear stress test on logs you provided (only chain dma part below)...
I think in case not send to much data you can filterout all background information except IPCs.
I spent the whole day desperately trying to find the setup that I recall recently passing multiple-pause-resume test, but I could not find it. However, if I go back far enough I can still find a working version. Last version that works is d92f07a, but the next commit 19df10f breaks the pause-resume functionality completely. The pause-resume starts working again after fc501b1 and 7383cbe, but that version crashes in multiple-pause-resume test at the same place as the current main branch. Same happens also, if I cherry-pick relevant commits on top off 19df10f.
It does not look like #7138 fixes the issue. Still trying to root cause what is actually wrong with zephyr_ll.c, but so far no luck. The situation is such that there is two or three HDMI devices banging the zephyr_ll work queue by adding and removing tasks there in tight loop, and eventually something goes wrong, but its pretty hard to say where and how.
It seems that my #7073 somehow gets past the issue by some luck, but maybe a better approach would be not to init() and free the tasks all the time, but instead init the task only once. Then just schedule the task in chain_task_start() and cancel it in chain_task_pause(). The should ease up the stress caused by the multiple-suspend-resume & chain DMA on the zepyhyr_ll queue.
Still it would be so nice to find the actual root cause.
I could not get the schedule() cancel() change to work out of the box... something gets out of sync, not sure what. But I have to drop this now, since there is other work piling up. #7073 is still the only "fix" that solves the chain DMA multiple-pause-resume issue, too bad nobody understands why that works, and more importantly why the other versions do not.
Activity
jsarha commentedon Feb 13, 2023
This thing is annoyingly getting out of hand. This latest problem was not there couple of weeks back when I was tracking some earlier issue with ChainDMA, and its not a result of a change in the chain dma code, since the only change there is 6009c21 and reverting it does not help with the issue. We can not have ChainDMA enabled because its broken, and because its not enabled, it gets even more broken.
jsarha commentedon Feb 14, 2023
This blocks #7073
makarukp commentedon Feb 15, 2023
@jsarha Please provide full IPC logs with payloads (for example for init instance) for local reproduction.
jsarha commentedon Feb 15, 2023
I can of course provide a full Linux side log, or even FW log, but I doubt that will help you to track down the problem. The multiple-pause-resume test is a stress test in nature, and there hundreds of IPC message before the crash happens.
makarukp commentedon Feb 15, 2023
We are interrested in full linux IPC log because we would like to reproduce the exact flow with specific ipc timings and all payloads necessary for module initializations. We would like to identify porential dependencies and detailed FW state before chain dma running. It is also worth to check how many chain DMA IPCs in average is needed to reproduce the problem .... (would be helpful )
So far we haven't encountered reproduction applying clear stress test on logs you provided (only chain dma part below)...
I think in case not send to much data you can filterout all background information except IPCs.
[ 592.571174] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x180: GLB_CHAIN_DMA
[ 592.571893] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
[ 592.672310] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x180: GLB_CHAIN_DMA
[ 592.672814] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
[ 592.773227] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
[ 592.773831] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
[ 592.938600] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
[ 592.939796] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
[ 593.040179] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
[ 593.040561] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
[ 593.041020] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
[ 593.042290] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
[ 593.210475] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
[ 593.211732] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
[ 593.312100] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
[ 593.312571] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
[ 593.412984] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
[ 593.413593] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
[ 593.583423] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
[ 593.584529] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
[ 593.684979] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
[ 593.685762] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
[ 593.686113] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
[ 593.687523] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
[ 593.787914] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe040101|0x0: GLB_CHAIN_DMA
[ 593.788350] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe040000|0x0: GLB_CHAIN_DMA
[ 594.604371] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x180: GLB_CHAIN_DMA
[ 594.605109] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
[ 594.705582] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x180: GLB_CHAIN_DMA
[ 594.706016] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
[ 594.706450] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
[ 594.707745] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
[ 594.786958] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
[ 594.788402] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
[ 594.870971] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
[ 594.872267] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
[ 594.874045] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
[ 594.875247] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
[ 594.957035] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
[ 594.958277] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
[ 594.959983] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
[ 594.962124] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
[ 595.043999] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
[ 595.047072] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
[ 595.048011] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
[ 595.049415] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
[ 595.131977] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
[ 595.133288] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
[ 595.134987] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
[ 595.140212] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
[ 595.140632] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
jsarha commentedon Feb 15, 2023
The full logs are attached:
linux-log.txt
fw-log.txt
jsarha commentedon Feb 15, 2023
I spent the whole day desperately trying to find the setup that I recall recently passing multiple-pause-resume test, but I could not find it. However, if I go back far enough I can still find a working version. Last version that works is d92f07a, but the next commit 19df10f breaks the pause-resume functionality completely. The pause-resume starts working again after fc501b1 and 7383cbe, but that version crashes in multiple-pause-resume test at the same place as the current main branch. Same happens also, if I cherry-pick relevant commits on top off 19df10f.
jsarha commentedon Feb 15, 2023
Made this PR out of curiosity, just to see if the old implementation would still pass the whole CI: #7104
jsarha commentedon Feb 21, 2023
It does not look like #7138 fixes the issue. Still trying to root cause what is actually wrong with zephyr_ll.c, but so far no luck. The situation is such that there is two or three HDMI devices banging the zephyr_ll work queue by adding and removing tasks there in tight loop, and eventually something goes wrong, but its pretty hard to say where and how.
It seems that my #7073 somehow gets past the issue by some luck, but maybe a better approach would be not to init() and free the tasks all the time, but instead init the task only once. Then just schedule the task in chain_task_start() and cancel it in chain_task_pause(). The should ease up the stress caused by the multiple-suspend-resume & chain DMA on the zepyhyr_ll queue.
Still it would be so nice to find the actual root cause.
jsarha commentedon Feb 22, 2023
I could not get the schedule() cancel() change to work out of the box... something gets out of sync, not sure what. But I have to drop this now, since there is other work piling up. #7073 is still the only "fix" that solves the chain DMA multiple-pause-resume issue, too bad nobody understands why that works, and more importantly why the other versions do not.
ll-schedule: fix tasks removed during execution
chain-dma: fix scheduling exception
ll-schedule: fix tasks removed during execution
chain-dma: fix scheduling exception
makarukp commentedon Mar 3, 2023
Correct fix in #7138 and #7189. Issue is resolved.