Skip to content

[BUG] IPC timed out when multiple-pipeline on TGLH_RVP_NOCODEC_ZEPHYR #4493

Closed

Description

@XiaoyunWu6666

Describe the bug
ipc timed out and IO error , when multiple-pipeline on TGLH_RVP_NOCODEC_ZEPHYR
inner daily 5289?model=TGLH_RVP_NOCODEC_ZEPHYR&testcase=multiple-pipeline-all
To Reproduce
TPLG=/lib/firmware/intel/sof-tplg/sof-tgl-h-nocodec.tplg ~/sof-test/test-case/multiple-pipeline.sh -f c -l 50

Environment
Start Time: 2021-07-15 02:10:50 UTC
End Time: 2021-07-15 04:47:20 UTC
Kernel Branch: topic/sof-dev
Kernel Commit: 9b277b80
SOF Branch: main
SOF Commit: 0fa9f7d

log

2021-07-15 03:19:59 UTC [ERROR] Caught kernel log error
===========================>>
[ 4046.379940] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc timed out for 0x60050000 size 12
[ 4046.379964] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ IPC dump start ]------------
[ 4046.379993] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: hda irq intsts 0x00000000 intlctl 0xc000001f rirb 00
[ 4046.380000] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: dsp irq ppsts 0x00000000 adspis 0x00000000
[ 4046.380041] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: host status 0x00000000 dsp status 0x00000000 mask 0x00000003
[ 4046.380047] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ IPC dump end ]------------
[ 4046.380052] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump start ]------------
[ 4046.380091] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: status: fw entered - code 00000005
[ 4046.380184] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: unexpected fault 0x00000000 trace 0x00004000
[ 4046.380191] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump end ]------------
[ 4046.380275] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at soc_component_trigger on 0000:00:1f.3: -110
[ 4046.380285] kernel:  DMIC16kHz: ASoC: trigger FE cmd: 0 failed: -110
<<===========================

Activity

  1. added
    bugSomething isn't working as expected
    ZephyrIssues only observed with Zephyr integrated
    on Jul 15, 2021
  2. kv2019i commented on Jul 22, 2021

    @kv2019i
    Collaborator

    I could reproduce this now on a TGL system.

  3. kv2019i commented on Jul 22, 2021

    @kv2019i
    Collaborator

    Hmm, it seems the occurence rate is less than <1/10. If the multi-pipeline test is run for more than 10 iterations, then you start hitting failures reliably. Curiously, the IPC timeout is preceded by a DSP overload situation:

    68475050 1783204 us, 45 us: ssp_start()
    68477892 1783278 us, 74 us: task add 0x9e059d50 0x1fffa180U priority 0 flags 0x0
    68482700 1783403 us, 125 us: validate(), ll drift detected, delta = 41508
    68522969 1784452 us, 1048 us: OVERRUN in scheduler, missed 1, cycles 40883, cycles-1 32263
    68526666 1784548 us, 96 us: validate(), ll drift detected, delta = 43966
    68613050 1786798 us, 2249 us: validate(), ll drift detected, delta = 46118
    68635382 1787379 us, 581 us: OVERRUN in scheduler, missed 1, cycles 41129, cycles-1 30558
    68656862 1787939 us, 559 us: validate(), ll drift detected, delta = 43813
    68710141 1789326 us, 1387 us: validate(), ll drift detected, delta = 53279

    When the problematic use-case (four concurrent capture streams) is run in normal conditions the DSP load is 30000/34800 ~= 70% , but in the above failing case, same use-case is taking 106% of the DSP.

    So something is causing DSP execution to slow down (but not on every iteration of the test case).

  4. kv2019i commented on Jul 22, 2021

    @kv2019i
    Collaborator

    Tested with XCC build and the problem is no longer hit. With GCC build, the nocodec pipeline load goes to 90% during the test case and if there are transient spikes in processing, the DSP can no longer catch up. The XCC build has sufficient margin to handle stream start/stop actions on core0 and the problem is not hit.

    XCC build requires recently merged XCC patchset to Zephyr -> zephyrproject-rtos/zephyr#36787

  5. lgirdwood commented on Jul 23, 2021

    @lgirdwood
    Member

    @lyakh @kv2019i @andyross fyi performance should also align shortly once we move objects into cache region, @ranj063 is working on the coherency issues so that we can use cache alias when needed. @andyross I guess you have plans for cache alisa usage on Zephyr code too ?

  6. kv2019i commented on Aug 24, 2021

    @kv2019i
    Collaborator

    Potential bugfix: #4673

  7. kv2019i commented on Aug 25, 2021

    @kv2019i
    Collaborator

    @XiaoyunWu6666 This should be fixed now indeed. We do have a new failure that shows up in other tests. You can identify this from IPC timeout for TRIG_STOP. I filed #4680 for this.

  8. XiaoyunWu6666 commented on Aug 26, 2021

    @XiaoyunWu6666
    ContributorAuthor

    inner daily 6090 shows good result
    Close it via #4673

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

TGL-HTGL-H platformZephyrIssues only observed with Zephyr integratedbugSomething isn't working as expected

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions