Repository navigation
Conversation
| if (unlikely(ret)) | ||
| goto release; | ||
|
|
||
| if (sdev->fw_state == SOF_FW_BOOT_COMPLETE) { |
There was a problem hiding this comment.
if the DSP is not booted at this point then we would not be able to load the library to fw in the first place, this check is redundant.
If the loading crashes the fw, then this would be skipped anyways.
| } | ||
| ret = sof_ipc4_compr_refresh_codec_info(sdev); | ||
| if (ret) | ||
| return ret; |
There was a problem hiding this comment.
Booting up the DSP and re-reading the info will not change anything, the info refresh after a library loading is the only place where it can change.
There was a problem hiding this comment.
So we need an API call here that can do 2 things:
-
Check the kernel info (refreshed after library loads) of what modules are in FW and return with a match/context in order to proceed here.
-
If no info match found, then we look at the FS for a match.
Now for 2) need a way for 3) kernel to know where to look on the FS for modules.
Re 3) I'm thinking our feature/function topology would be like NXP ipc3 decoder where we can have a container module and we could list UUIDs supported for this configuration. This list would allow the kernel to know which UUIDs would support 44.1k S16le MP3 or whicg UUIDs support 48k 24bit AAC etc.
i.e. kernel would know where to find UUID and hence llext module that would align with request.
| for (i = 0; i < codec_info->count; i++) { | ||
| int _dir = SOF_IPC4_CODEC_INFO_GET_DIR(codec_info->items[i]); | ||
| { | ||
| guard(mutex)(&ipc4_data->codec_info_mutex); |
There was a problem hiding this comment.
Use scoped_guard() instead if must
|
|
||
| ret = sof_ipc4_compr_refresh_codec_info(sdev); | ||
| if (ret) | ||
| return ret; |
There was a problem hiding this comment.
this is not needed, just booting up will not change anything in configuration, keep the existing flow.
| if (count) { | ||
| kfree(new_codec_info); | ||
| new_codec_info = kmemdup(tuple->value, tuple->size, GFP_KERNEL); | ||
| if (!new_codec_info) { |
There was a problem hiding this comment.
you expect multiple CODEC_INFO in the sof_info block and ignore all, but the last one?
When a module is loaded to the firmware, they cannot be unloaded, so the last codec_info should contain the cumulative list of all codecs, including if there are cadence codecs in basefw, ffmpeg codecs or any other codecs loaded as library via UUID.
Loading new codecs will not remove the existing ones.
There was a problem hiding this comment.
actually we do need to be able to purge modules. - we need to consider that non core modules can come and go (and this will also include models too).
| } | ||
|
|
||
| if (ipc4_data->sof_info_runtime_supported) { | ||
| ret = sof_ipc4_query_sof_info(sdev); |
There was a problem hiding this comment.
we just parsed the sof_info from fw_config, the separate sof_info_get will query different sof_info than the sof_info we just parsed?
| } | ||
| } | ||
|
|
||
| if (!ret && ipc4_data->sof_info_runtime_supported) |
There was a problem hiding this comment.
right, this code does not know if the libraries are indeed reloaded or it was a NOP due to the fact that on Intel hardware IMR stored the libraries.
There was a problem hiding this comment.
🟡 Changes recommended
The GET_CAPS refresh lacks a runtime-PM reference across firmware boot and capability querying, so it can access powered-down hardware and fail when SOF is suspended.
1 open issue
What changed in this PR
Adds runtime codec capability discovery to the SOF IPC4 driver, complementing the firmware changes in thesofproject/sof#11472 while preserving boot-time discovery for older firmware.
Changes:
- Defines the runtime SOF-info query and capability flag.
- Refreshes codec information after library loads and reloads, with tuple validation and mutex-protected cache replacement.
- Refreshes capabilities before compressed-stream queries and codec selection, and frees cached information at exit.
| File | Description |
|---|---|
sound/soc/sof/ipc4.c |
Initializes the codec-info mutex and frees cached data at exit. |
sound/soc/sof/ipc4-priv.h |
Declares runtime capability state, locking, and the query helper. |
sound/soc/sof/ipc4-loader.c |
Validates SOF-info tuples and refreshes capabilities during firmware and library setup. |
sound/soc/sof/ipc4-compress.c |
Refreshes and locks codec information used by compressed streams. |
include/sound/sof/ipc4/header.h |
Adds runtime SOF-info query and capability constants. |
Review effort: Balanced
| * Ensure the DSP is booted (on-demand boot) and runtime codec | ||
| * capabilities are synchronized before consulting codec_info. | ||
| */ | ||
| ret = snd_sof_boot_dsp_firmware(sdev); |
|
@ujfalusi I'm going to work on 3) below, so we have something similar to NXP in IPC4 - lets see if we can get a simple method for kernel to get UUIDs for needed modules |
cd8df21 to
42d44e9
Compare
|
Addressed review feedback and added module purge + topology format-to-UUID lookup:
|
Add IPC4 runtime SOF info discovery (SOF_IPC4_FW_PARAM_SOF_INFO_GET) and idle library unload (SOF_IPC4_FW_PARAM_SOF_LIBRARY_UNLOAD) gated by SOF_IPC4_SOF_CAPS capability flags, while keeping boot-time FW_CONFIG SOF_IPC4_SOF_CODEC_INFO parsing for backwards compatibility: - Protect cached codec_info with codec_info_mutex and reject duplicate SOF_IPC4_SOF_CODEC_INFO tuples in a single SOF_INFO payload. - Refresh SOF info on library load and non-IMR library reloads (avoiding redundant queries after FW_CONFIG or IMR restore). - Implement gap-safe library ID allocation, sof_ipc4_unload_library(), and automatic purging of idle unloadable UUID libraries when library slots or memory are exhausted. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…d load Parse SOF_TKN_MEDIA_CODEC_* topology tokens on decoder widgets into swidget->media_formats[] mapping codec IDs and stream constraints to priority-ordered candidate module UUIDs: - Allow media_decoder widgets with media_formats[] to defer module binding at topology load time when their default UUID is not resident. - Report both cached firmware codec_info (under scoped_guard mutex) and topology media_formats[] in sof_ipc4_compr_get_caps() without waking or booting a suspended DSP. - In sof_ipc4_compr_set_params(), free any prior prepared stream first, then perform a two-tier lookup across matching media_formats[] (resident module first, else boot DSP and demand-load/purge UUID library) and bind the selected module before pipeline setup. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
42d44e9 to
c49efc5
Compare
|
Still WIP, this example shows a ffmpeg library, mnext is ffmpeg per module. Here is a complete example of the companion feature topology from thesofproject/sof#11472 ( 1. Example Feature Topology (
|
| Topology2 attribute | Token macro | ID | Type |
|---|---|---|---|
codec_id |
SOF_TKN_MEDIA_CODEC_ID |
2200 |
word (u32) |
min_rate |
SOF_TKN_MEDIA_CODEC_MIN_RATE |
2201 |
word (u32) |
max_rate |
SOF_TKN_MEDIA_CODEC_MAX_RATE |
2202 |
word (u32) |
channels |
SOF_TKN_MEDIA_CODEC_CHANNELS |
2203 |
word (u32) |
bit_depth |
SOF_TKN_MEDIA_CODEC_BIT_DEPTH |
2204 |
word (u32) |
module_uuid |
SOF_TKN_MEDIA_CODEC_MODULE_UUID |
2205 |
uuid (guid_t) |
include/components/media_decoder.conf and include/common/media_format.conf define the container widget (DAPM type "decoder") and format entries:
Class.Widget."media_decoder" {
DefineAttribute."index" { type "integer" }
DefineAttribute."instance" { type "integer" }
<include/components/widget-common.conf>
DefineAttribute."cpc" { token_ref "comp.word" }
...
# Default container/fallback UUID (Cadence codec adapter); overridden at runtime
# when a media_format entry selects a specific module UUID.
uuid "43:84:21:d8:f3:5f:4c:4a:b3:88:6c:fe:07:b9:56:aa"
type "decoder"
no_pm "true"
core_id 0
cpc 5000
num_input_pins 1
num_output_pins 1
heap_bytes_requirement 196608
stack_bytes_requirement 16384
}
Class.Base."media_format" {
DefineAttribute."instance" {}
DefineAttribute."codec_id" { token_ref "media_codec.word" }
DefineAttribute."min_rate" { token_ref "media_codec.word" }
DefineAttribute."max_rate" { token_ref "media_codec.word" }
DefineAttribute."channels" { token_ref "media_codec.word" }
DefineAttribute."bit_depth" { token_ref "media_codec.word" }
DefineAttribute."module_uuid" { type "string" token_ref "media_codec.uuid" }
...
min_rate 8000
max_rate 192000
channels 2
bit_depth 16
}
b) Per-Codec UUID & Config Table (platform/intel/compr-media.conf)
Note the byte-order mapping between topology2 LE UUID bytes and the Linux %pUL LLEXT filenames requested from /lib/firmware/intel/sof-ipc4-lib/<platform>/:
MEDIA_DECODE_CADENCE_UUID(43:84:21:d8:f3:5f:4c:4a:b3:88:6c:fe:07:b9:56:aa) →D8218443-5FF3-4A4C-B388-6CFE07B956AA.binMEDIA_DECODE_FFMPEG_UUID(73:af:07:12:c0:f9:65:4f:92:b4:9d:45:17:8f:c9:a8) →1207AF73-F9C0-4F65-92B4-9D45178FC9A8.bin
Define {
COMPR_PIPELINE_SINK "mixout.1.1"
MEDIA_DECODE_CADENCE_UUID "43:84:21:d8:f3:5f:4c:4a:b3:88:6c:fe:07:b9:56:aa"
MEDIA_DECODE_FFMPEG_UUID "73:af:07:12:c0:f9:65:4f:92:b4:9d:45:17:8f:c9:a8"
SND_AUDIOCODEC_PCM 1
SND_AUDIOCODEC_MP3 2
SND_AUDIOCODEC_AAC 6
SND_AUDIOCODEC_VORBIS 9
SND_AUDIOCODEC_FLAC 10
SND_AUDIOCODEC_ALAC 15
SND_AUDIOCODEC_OPUS_RAW 17
}
Object.Pipeline.compr-playback-media [
{
index $COMPR_PIPELINE_ID
Object.Widget.host-copier.1 {
stream_name $COMPR_PCM_NAME
pcm_id $COMPR_PCM_ID
<include/formats/compr_input_audio_formats.conf>
<include/formats/compr_output_audio_formats.conf>
}
Object.Widget.media_decoder.1 {
core_id $COMPR_DECODER_CORE_ID
stack_bytes_requirement $COMPR_FFMPEG_DEC_STACK
heap_bytes_requirement $COMPR_FFMPEG_DEC_HEAP
<include/formats/compr_input_audio_formats.conf>
<include/formats/compr_output_audio_formats.conf>
# CombineArrays expands each Cartesian product of constraints and
# candidate module_uuid entries in priority order.
CombineArrays.Object.Base.media_format [
{
codec_id [ $SND_AUDIOCODEC_MP3 ]
min_rate [ 8000 ]
max_rate [ 48000 ]
channels [ 1 2 ]
bit_depth [ 16 24 32 ]
module_uuid [
$MEDIA_DECODE_FFMPEG_UUID
$MEDIA_DECODE_CADENCE_UUID
]
}
{
codec_id [ $SND_AUDIOCODEC_AAC ]
min_rate [ 8000 ]
max_rate [ 96000 ]
channels [ 1 2 6 8 ]
bit_depth [ 16 24 32 ]
module_uuid [
$MEDIA_DECODE_CADENCE_UUID
$MEDIA_DECODE_FFMPEG_UUID
]
}
{
codec_id [ $SND_AUDIOCODEC_FLAC ]
min_rate [ 8000 ]
max_rate [ 192000 ]
channels [ 1 2 6 8 ]
bit_depth [ 16 24 32 ]
module_uuid [ $MEDIA_DECODE_FFMPEG_UUID ]
}
{
codec_id [ $SND_AUDIOCODEC_ALAC ]
min_rate [ 8000 ]
max_rate [ 192000 ]
channels [ 1 2 6 8 ]
bit_depth [ 16 24 32 ]
module_uuid [ $MEDIA_DECODE_FFMPEG_UUID ]
}
{
codec_id [ $SND_AUDIOCODEC_VORBIS ]
min_rate [ 8000 ]
max_rate [ 192000 ]
channels [ 1 2 ]
bit_depth [ 16 24 32 ]
module_uuid [ $MEDIA_DECODE_CADENCE_UUID ]
}
{
codec_id [ $SND_AUDIOCODEC_OPUS_RAW ]
min_rate [ 8000 ]
max_rate [ 48000 ]
channels [ 1 2 ]
bit_depth [ 16 24 32 ]
module_uuid [ $MEDIA_DECODE_FFMPEG_UUID ]
}
{
codec_id [ $SND_AUDIOCODEC_PCM ]
min_rate [ 8000 ]
max_rate [ 192000 ]
channels [ 1 2 ]
bit_depth [ 16 24 32 ]
module_uuid [ $MEDIA_DECODE_CADENCE_UUID ]
}
]
}
...
}
]
Object.PCM.pcm [
{
name $COMPR_PCM_NAME
id $COMPR_PCM_ID
direction playback
compress "true"
Object.Base.fe_dai.1 { name "$COMPR_PCM_NAME" }
Object.PCM.pcm_caps.1 {
name $COMPR_PCM_NAME
formats 'S32_LE'
rates '8000'
}
}
]
Object.Base.route [
{
source mixin.$COMPR_PIPELINE_ID.1
sink $COMPR_PIPELINE_SINK
}
{
source host-copier.$COMPR_PCM_ID.playback
sink media_decoder.$COMPR_PIPELINE_ID.1
}
]
2. How It Works in the Kernel (sound/soc/sof/)
-
Topology load — token parsing & deferred binding (
topology.c,ipc4-topology.c)sof_widget_parse_tokens()encountersSOF_MEDIA_CODEC_TOKENSon thesnd_soc_dapm_decoderwidget and callssof_parse_media_codec_tokens()to populateswidget->media_formats[](struct sof_media_codec_fmt) andswidget->num_media_formats.sof_ipc4_widget_set_module_info()seesswidget->id == snd_soc_dapm_decoder && swidget->num_media_formatsand only checkssof_ipc4_find_loaded_module_by_uuid()for the widget's default UUID instead of callingsof_ipc4_find_module_by_uuid().- If no decoder LLEXT is resident at topology load time,
swidget->module_infois allowed to stayNULLand no<UUID>.binload is triggered during probe.sof_ipc4_widget_setup_comp_process()skipssof_ipc4_process_bind_module()until stream configuration, andsof_ipc4_prepare_process_module()guards against unbound widgets (-ENODEV).
-
Capability reporting —
GET_CAPS(ipc4-compress.c)sof_ipc4_compr_get_caps()takes the union of:- cached runtime firmware
codec_info(read underscoped_guard(mutex, &ipc4_data->codec_info_mutex)), and - all
codec_ids declared incodec_swidget->media_formats[].
- cached runtime firmware
sof_ipc4_compr_caps_add_codec()deduplicates entries without issuing IPCs or waking/booting a suspended DSP.
-
Stream configuration & 2-tier UUID lookup —
SET_PARAMS(ipc4-compress.c,ipc4-topology.c)sof_ipc4_compr_has_matching_fmt()checkscodec_swidget->media_formats[]against the requestedcodec.id,sample_rate([min_rate, max_rate]),ch_out, andbit_depth(non-zero fields must match).- If the compress stream was already prepared (
spcm->prepared[dir], e.g., switching tracks/codecs afterDRAINwithout closing the device),sof_ipc4_compr_stream_free()tears down the previous pipeline and frees itsm_idainstance first so the previous decoder library becomes idle and eligible for eviction if needed. sof_ipc4_compr_resolve_media_module()then scans matchingmedia_formats[]entries in priority order in two passes:- Tier 1 (resident preference):
sof_ipc4_find_loaded_module_by_uuid()checks already-loaded modules across base firmware andfw_lib_xa. If a matching candidate is already resident (for example, Cadence already loaded for AAC and also listed as secondary candidate for MP3), it is reused directly with zero extra LLEXT loading. - Tier 2 (demand load): if no matching candidate is resident, it calls
snd_sof_boot_dsp_firmware()and scans matching entries again withsof_ipc4_find_module_by_uuid(), which requests and loads<UUID>.binviasof_ipc4_load_library_by_uuid().
- Tier 1 (resident preference):
sof_ipc4_process_bind_module()then bindscodec_swidget->module_info = fw_module, allocates/updatesprocess->ipc_config_data(andbase_config_extwhenSOF_IPC4_MODULE_INIT_CONFIG_TYPE_BASE_CFG_WITH_EXTapplies), updatesprocess->msg(man4_module_entry.id+ LL/DP domain), and updates widget kcontrol module IDs before pipeline setup/prepare.
-
Dynamic library info refresh & idle LLEXT purge (
ipc4-loader.c)- On external library load (gated by
SOF_IPC4_SOF_CAP_RUNTIME_SOF_INFOand skipped duringFW_CONFIG/ IMR restores),sof_ipc4_query_sof_info()issuesSOF_IPC4_FW_PARAM_SOF_INFO_GETand updatesipc4_data->codec_infoundercodec_info_mutex. - When
sof_ipc4_find_module_by_uuid()needs a new<UUID>.binand either library slots (1 .. max_libs_count - 1) are exhausted or DSP memory allocation fails,sof_ipc4_purge_idle_library()scansfw_lib_xafor an idle (ida_is_empty(&fw_lib->modules[i].m_ida)for all modules)unloadablelibrary, invokessof_ipc4_unload_library():- sends
SOF_IPC4_FW_PARAM_SOF_LIBRARY_UNLOADto base FW, - detaches any
swidget->module_infopointing into that library (sof_ipc4_detach_library_widgets()), - erases the
fw_lib_xaentry and releasesfw_lib->sof_fw.fw+ host metadata, - refreshes
codec_infoviasof_ipc4_query_sof_info(),
and reuses the freedlib_idslot to load the newly requested decoder library.
- sends
- On external library load (gated by

Summary
Upgrade IPC4 codec discovery in the Linux SOF driver to support runtime codec queries (
SOF_IPC4_FW_PARAM_SOF_INFO_GET), idle dynamic library unloading/purging (SOF_IPC4_FW_PARAM_SOF_LIBRARY_UNLOAD), and topologymedia_decoderformat-to-UUID lookup with on-demand LLEXT loading — while preserving boot-timeFW_CONFIGfallback for older firmware.Companion SOF firmware PR: thesofproject/sof#11472
Changes
Runtime
SOF_INFOdiscovery & library unload (header.h,ipc4-priv.h,ipc4.c,ipc4-loader.c)SOF_IPC4_FW_PARAM_SOF_INFO_GET = 37,SOF_IPC4_FW_PARAM_SOF_LIBRARY_UNLOAD = 38,SOF_IPC4_SOF_CAP_RUNTIME_SOF_INFO, andSOF_IPC4_SOF_CAP_LIBRARY_UNLOAD.codec_infoupdates/reads withcodec_info_mutexand rejected duplicateSOF_IPC4_SOF_CODEC_INFOtuples in a single payload.SOF_INFOon library load and non-IMR library reloads (avoiding redundant queries afterFW_CONFIGor IMR restore).sof_ipc4_unload_library(), and automatic purging of idle unloadable UUID libraries when slots or memory are exhausted.media_decoderformat-to-UUID lookup & on-demand load (tokens.h,sof-audio.h,topology.c,ipc4-topology.h,ipc4-topology.c,ipc4-compress.c)SOF_TKN_MEDIA_CODEC_*(2200..2205) tokens intoswidget->media_formats[]and allowsmedia_decoderwidgets to defer module binding when their default UUID is not resident at topology load.codec_info(viascoped_guard(mutex, ...)) andmedia_formats[]insof_ipc4_compr_get_caps()without waking or booting a suspended DSP.sof_ipc4_compr_set_params(), tears down any previous prepared stream first (freeing prior LLEXT references for purge), then resolves the best matchingmedia_formats[]module UUID (resident module first, else boot DSP + demand-load/purge viasof_ipc4_find_module_by_uuid()) and binds it viasof_ipc4_process_bind_module().Verification
scripts/checkpatch.pl:0errors,0warnings on both commits.make M=sound/soc/sof: compiled and linked allsound/soc/sofmodules with0warnings and0errors.