Repository navigation
Closed
Labels
bugSomething isn't working as expectedSomething isn't working as expected
Milestone
Description
Describe the bug
Cherrytrail playback fails, reboots sometimes gets system to a good state which allows playback but occasionally this will appear in the logs
0 2 DMA 32400464.635417 1.979167 src/drivers/dw/dma.c:442 dw_dma_stop(): dma 1 channel 0 stop
0 1 WAIT 32401809.166667 1344.531250 src/lib/wait.c:96 ewt
0 1 DMA 32401815.989583 6.822917 src/drivers/dw/dma.c:460 dw_dma_stop() error: dma 1 channel 0
or
0 2 SSP 51489200.677083 2.708333 intel/baytrail/ssp.c:572 ssp_trigger()
0 2 SSP 51489203.385417 2.708333 intel/baytrail/ssp.c:521 ssp_start()
0 1 DMA 51490154.375000 950.989563 src/drivers/dw/dma.c:275 dw_dma_start() error: dma 2 channel 1 not ready ena 0x0 status 0x5
0 1 HOST 1.1 51490166.093750 11.718750 src/audio/host.c:796 host_copy() error: dma_copy() failed, ret = 4294967280
0 1 PIPE 51490174.687500 8.593750 src/audio/pipeline.c:801 pipeline_copy() error: ret = -16, start->comp.id = 3, dir = 1
0 1 PIPE 1.10 51490184.843750 10.156250 src/audio/pipeline.c:977 pipeline_task(): xrun recover failed! pipeline will be stopped!
To Reproduce
Load cyan with SOF Tot and attempt to run a speaker test
Reproduction Rate
4 out of 5 reboots approximately
Expected behavior
system is stable
Impact
blocker for kerneluprev for bsw
Environment
- Branch name and commit hash
- SOF 460c41a (master) (with topology: sof-cht-max98090 move DAI definition above media pipeline #1846 and topology: cht-max98090 add missing virtual widgets #1842)
- Kernel 6fc7eb3 (topic/sof-dev)
- Name of the topology file
- sof-cht-max98090.tplg
- Name of the platform(s) on which the bug is observed
- Cherrytrail
Screenshots or console output
localhost ~ # speaker-test -D hw:1,0 -c 2
speaker-test 1.1.6
Playback device is hw:1,0
Stream parameters are 48000Hz, S16_LE, 2 channels
Using 16 octaves of pink noise
Rate set to 48000Hz (requested 48000Hz)
Buffer size range from 96 to 16368
Period size range from 48 to 4080
Using max buffer size 16368
Periods = 4
was set period_size = 3312
was set buffer_size = 16368
0 - Front Left
Write error: -5,Input/output error
xrun_recovery failed: -5,Input/output error
Transfer failed: Input/output error
Relevant dmesg logs
[ 51.864104] sof-audio-acpi 808622A8:00: pcm: open stream 0 dir 0
[ 51.864110] sof-audio-acpi 808622A8:00: period min 192 max 16384 bytes
[ 51.864113] sof-audio-acpi 808622A8:00: period count 2 max 16
[ 51.864116] sof-audio-acpi 808622A8:00: buffer max 65536 bytes
[ 51.864511] sof-audio-acpi 808622A8:00: rate_min: 48000 rate_max: 48000
[ 51.864515] sof-audio-acpi 808622A8:00: channels_min: 2 channels_max: 2
[ 51.866616] sof-audio-acpi 808622A8:00: rate_min: 48000 rate_max: 48000
[ 51.866634] sof-audio-acpi 808622A8:00: channels_min: 2 channels_max: 2
[ 51.866965] sof-audio-acpi 808622A8:00: rate_min: 48000 rate_max: 48000
[ 51.866969] sof-audio-acpi 808622A8:00: channels_min: 2 channels_max: 2
[ 51.866984] sof-audio-acpi 808622A8:00: pcm: hw params stream 0 dir 0
[ 51.866992] sof-audio-acpi 808622A8:00: generating page table for 00000000ab5ef320 size 0xffc0 pages 16
[ 51.866998] sof-audio-acpi 808622A8:00: stream_tag 0
[ 51.867012] sof-audio-acpi 808622A8:00: ipc tx: 0x60010000
[ 51.867824] sof-audio-acpi 808622A8:00: ipc tx succeeded: 0x60010000
[ 51.867850] sof-audio-acpi 808622A8:00: pcm: stream dir 0, posn mailbox offset is 1329664
[ 51.871161] sof-audio-acpi 808622A8:00: pcm: trigger stream 0 dir 0 cmd 1
[ 51.871172] sof-audio-acpi 808622A8:00: ipc tx: 0x60040000
[ 51.871357] sof-audio-acpi 808622A8:00: ipc tx succeeded: 0x60040000
[ 52.375426] sof-audio-acpi 808622A8:00: PCM: stream 0 dir 0 DMA position 0 DAI position 0
[ 52.879599] sof-audio-acpi 808622A8:00: pcm: trigger stream 0 dir 0 cmd 0
[ 52.879625] sof-audio-acpi 808622A8:00: ipc tx: 0x60050000
[ 52.880581] sof-audio-acpi 808622A8:00: ipc tx succeeded: 0x60050000
[ 52.880608] sof-audio-acpi 808622A8:00: ipc tx: 0x60030000
[ 52.880854] sof-audio-acpi 808622A8:00: ipc tx succeeded: 0x60030000
[ 52.881092] sof-audio-acpi 808622A8:00: pcm: free stream 0 dir 0
[ 52.881589] sof-audio-acpi 808622A8:00: pcm: close stream 0 dir 0
Metadata
Metadata
Assignees
Labels
bugSomething isn't working as expectedSomething isn't working as expected
Activity
cujomalainey commentedon Sep 20, 2019
Possibly interesting line that might be causing the DMA to crash, does it really have 65k channels?
paulstelian97 commentedon Sep 20, 2019
Looks like some sort of weird issue, like casting -1 to a uint16_t? I noticed the pattern of using negative channel numbers in dma_get() as called from host. Can you show a few lines before that interesting one?
Relevant code, from
src/audio/host.c(functionhost_params):If it's the call from
dai.cthough there must be a bug (platform data being wrong or not used correctly). But that should cause a permanent, not occasional issue.paulstelian97 commentedon Sep 20, 2019
Found the relevant code for the real issue (line numbers are wrong), the channel stop seems to fail waiting for FIFOs to be empty (function dw_dma_stop, only with CONFIG_DMA_SUSPEND_DRAIN -- which is enabled on CHT and BYT).
Someone who's actually working on that platform?
tlauda commentedon Sep 20, 2019
Please try to increase the timeout. It might be that ocassionally FIFO is not emptied on time. Disabling the channel with FIFO not empty is causing DMA hang and that's why you cannot restart it.
cujomalainey commentedon Sep 20, 2019
Doubling the DMA timeout (for baytrail) did not change the behaviour. It seems to consistently fail on master but on 005-cml-drop it flakes and I was able to get audio working a couple times
cujomalainey commentedon Sep 20, 2019
@paulstelian97 yes I noticed this morning I forgot to update the ldc between builds which caused incorrect line numbers
cujomalainey commentedon Sep 20, 2019
changing all the cherrytrail only dmac configs in
src/platform/baytrail/lib/dma.cto#if 0allows cherrytrail to work. I am going to look closer at the last controllers configlgirdwood commentedon Sep 23, 2019
@cujomalainey that's a good find and interesting, these values are internal DMAC channel priority configurations. They are intended to allow the DMAC to give bandwidth and FIFO priority to some channels over other ones. We do not use these currently in SOF so it should be safe to comment on CHT platform.
cujomalainey commentedon Oct 2, 2019
Continuing DMA failure discussion from #1862
@lgirdwood I logged the SSP state post start and pre stop
See attached for full dump of initial states from set config
bad_after_set_config.txt
good_after_set_config.txt
good_before_set_config.txt
bad_before_set_config.txt
I tried writing the default back to SSSR before set_config with no change in behaviour. I am trying to understand how to determine the DMA alignment from the doc to see if that is the issue. The document is a bit daunting so if can give me some pointers that would be great.
cujomalainey commentedon Oct 4, 2019
fun little tidbit, a suspend/resume cycle (with suspend enabled on byt/cht) appears to fix this bug
lgirdwood commentedon Oct 7, 2019
@cujomalainey one other thing to check is that DMA HW LLI mode is not used on CHT, iirc only SW LLI mode should be used on BYT, CHT and BSW (worth checking dw-dma.c config here). This can cause corrupted audio.
@mengdonglin can someone check all the SSP bits as I'm travelling for next two weeks.
18 remaining items
lgirdwood commentedon Nov 26, 2019
@cujomalainey I think the SSP M/N dividers need an atomic 64bit update OR may need to be disabled prior to updates OR may need a write sequence. I do see an update bit (b61) which would also imply that 32bit writes are supported. i.e write bits 0 - 31, then 32 - 63 and set update bit.
cujomalainey commentedon Nov 26, 2019
I'll try the ordering. Otherwise there is a bug in
shim_read/writeas that is what I was using to set and check.lgirdwood commentedon Nov 26, 2019
@cujomalainey thanks, I'm not 100% sure as we never used these bits on BYT, they are on the other platforms but IO is 32bit instead of 64bit.
cujomalainey commentedon Nov 26, 2019
Ah yes, I definitely made that mistake, also I'm remembering there is a 90% chance I forgot to put the 0x to make it hex
cujomalainey commentedon Nov 27, 2019
SUCCESS!!!
Adding the following writes fixes the bug. Audio playback works and the agent is happy.
Now the question of where to integrate it in SOF? Also I traced the respective writes in depthcharge which are causing the issue.
https://chromium.googlesource.com/chromiumos/platform/depthcharge/+/refs/heads/master/src/drivers/bus/i2s/braswell/braswell.c#607
lgirdwood commentedon Nov 27, 2019
@cujomalainey great :) There is no harm in adding these writes to platform.c init() in order to reset the M/N dividers for all SSP ports to a sane state expected by FW.
@plbossart I guess this same M/N disable logic would be needed for BDW too ?
cujomalainey commentedon Nov 27, 2019
If needed I can run a test on my Samus for BDW
cujomalainey commentedon Nov 28, 2019
Scratch that, there appears to be dai_link issues that are confusing me greatly on Samus with SOF
So I can't test until I wrap my head around why that is happening. I can file another bug if we want to bother fixing samus. I think it would be nice to officially support it for testing purposes.
lgirdwood commentedon Nov 28, 2019
@cujomalainey yeah, can you file bug on that. I've just had a look at bdw-rt5677.c and can see it's mixing DPCM and non DPCM DAI links which is probably rarely tested as all the others are DPCM.
plbossart commentedon Dec 2, 2019
I don't think those are M/N registers, we don't use them on Baytrail. they look like standard dividers to me?
platform: cht: reset ssp clk M/N dividers on boot
youling257 commentedon Dec 16, 2019
with "platform: cht: reset ssp clk M/N dividers on boot" patch, i see these warning.
[ 10.293720] sof-audio-acpi 808622A8:00: error: sink SSP0.IN not found
[ 10.294200] sof-audio-acpi 808622A8:00: ASoC: no sink widget found for SSP0.IN
[ 10.294594] sof-audio-acpi 808622A8:00: ASoC: Failed to add route modem_in -> direct -> SSP0.IN
[ 10.295097] sof-audio-acpi 808622A8:00: error: source SSP0.OUT not found
[ 10.308311] sof-audio-acpi 808622A8:00: ASoC: no source widget found for SSP0.OUT
[ 10.313123] sof-audio-acpi 808622A8:00: ASoC: Failed to add route SSP0.OUT -> direct -> modem_out
I don't want to see.
ranj063 commentedon Dec 16, 2019
@youling257 you'll need edd8d70 to fix those errors
platform: cht: reset ssp clk M/N dividers on boot