Repository navigation
ASoC: SOF: ipc: Rate-limit IPC transmission error messages - #6020
anantxgarg wants to merge 2 commits into
Conversation
|
Can one of the admins verify this patch?
|
| spcm_err(spcm, substream->stream, | ||
| "failed to set hw_params after resume\n"); | ||
| spcm_err_ratelimited(spcm, substream->stream, | ||
| "failed to set hw_params after resume\n"); |
There was a problem hiding this comment.
ASoC still going to fill up the kernel log even if we rate limit.
You can still reproduce the issue with recent kernel and user space?
checkpatch reports several issues, the commit message is misaligned and if possible also align the parameters as well.
There was a problem hiding this comment.
Wrapping fixed.
The issue occurred on kernel 6.8.0-147-generic, PipeWire 1.0.5 where in from what i could determine, after waking up from suspend the the DSP failed the resume handshake which led to STREAM_PCM_PARAMS returning -EINVAL. Pipewire then repeatedly retried snd_pcm_prepare() which produced around 26 GB of log files within a few hours. I havent yet tried to recreate it on newer kernels.
ASoC is still going to fill it up even if we ratelimit in sof, however it would lead to a 75% reduction in error messages. It seems to me a worthwhile change to make.
Further changes upstream to ASoC core will be required to address that issue in particular.
23c4355 to
0823a06
Compare
When an audio DSP fails to properly reinitialize or desynchronizes after a system suspend/resume cycle, subsequent stream parameter setup requests fail with -EINVAL. In response, userspace audio servers (such as PipeWire) attempt to restart the PCM stream immediately in a rapid retry loop. This unthrottled loop can flood system logs with millions of lines, consuming tens of gigabytes and exhausting root filesystems. Mitigate this logging storm by converting dev_err() and spcm_err() to dev_err_ratelimited() and spcm_err_ratelimited() across IPC transmission paths (ipc3_wait_tx_done, ipc4_wait_tx_done) and PCM stream setup paths (sof_pcm_prepare, sof_pcm_hw_params, and sof_pcm_stream_free). Closes: thesofproject/sof#10442 Signed-off-by: Anant Garg <anantxgarg@gmail.com>
When lower-level drivers fail during stream operations, snd_soc_ret() emits an unthrottled dev_err() for all unhandled negative error codes. During repeated userspace retry loops (such as snd_pcm_prepare() retries on failing hardware), this unthrottled logging produces thousands of duplicate error lines, filling system logs. Convert dev_err() in snd_soc_ret() to dev_err_ratelimited() to mitigate log flooding across all ASoC component and DAI operations. Signed-off-by: Anant Garg <anantxgarg@gmail.com>
0823a06 to
1240a67
Compare
|
Hi @ujfalusi I re-tested this running a 7.0 kernel with PipeWire 1.0.5, and the issue still reproduces reliably. During stream prepare retries, it spat out around 700 error lines into dmesg within a few seconds. Roughly a third of those logs were coming straight from snd_soc_ret() in soc-utils.c, so rate-limiting SOF alone wasn't enough to stop the flood. The branch is reworked into two clean commits:
Commit wrapping, parameter alignment, and checkpatch warnings should all be sorted now. Please run CI when you get a chance. |
When an audio DSP fails to properly reinitialise or desynchronises after a system suspend/resume cycle, subsequent stream parameter setup requests fail with -EINVAL. In response, userspace audio servers (such as PipeWire) attempt to restart the PCM stream immediately in a rapid retry loop. This unthrottled loop can flood system logs with millions of lines, consuming tens of gigabytes and exhausting root filesystems.
Mitigate this logging storm by converting dev_err() to dev_err_ratelimited() in ipc3_wait_tx_done() and ipc4_wait_tx_done(), and adding spcm_err_ratelimited() in sof-audio.h for stream preparation error paths.
Closes: thesofproject/sof#10442