Skip to content

Remove DSP after unregistering the machine driver - #863

Merged
plbossart merged 2 commits into
thesofproject:topic/sof-devfrom
ranj063:fix/nocodec-unload
May 10, 2019
Merged

plbossart merged 2 commits into
thesofproject:topic/sof-devfrom
ranj063:fix/nocodec-unload

Conversation

@ranj063

@ranj063 ranj063 commented Apr 23, 2019 •

Copy link
Copy Markdown
Collaborator

Unregistering the machine driver results in the removal of topology components. Some components like the pipeline component, when they are unloaded will attempt to power down the core that it is scheduled on.

snd_sof_remove() disables the DSP and unmaps the DSP BAR from the kernel space. Unnregistering the machine driver after removing the DSP results in a kernel panic during the removal of the pipeline widget in topology. This is because unloading the pipeline widget attempts to power down the core that the pipeline is scheduled on. While attempting the power down, the core tries to read the DSP status and this results in a kernel oops because the DSP BAR has been unmapped already.

So, the suggested fix here is to unregister the machine driver first and then remove the DSP to avoid the situation described above.

Also note that the above situation only happens in the case of nocodec topology. In the case of topologies with HDA pipelines(either iDisp or HDA codec), the above situation is avoided because the topology is unloaded before the DSP BAR is unmapped. This happens because snd_sof_remove() calls
snd_hdac_ext_bus_device_remove() to remove the codec which unregisters the component driver thereby also removing the topology.

Fixes #861

@ranj063
ranj063 requested review from bardliao and plbossart April 23, 2019 17:51

@plbossart plbossart left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get why this pops up now? what are we trying to fix, is this a regression or something that was never tested?

@ranj063

ranj063 commented Apr 23, 2019

Copy link
Copy Markdown
Collaborator Author

I don't get why this pops up now? what are we trying to fix, is this a regression or something that was never tested?

@plbossart I dont understand that either. It is a regression as pointed out in #861 but this should have been caught much earlier.

@plbossart

Copy link
Copy Markdown
Member

I'd like to know the last time this test was run. It should be part of the daily validation.

@ranj063

ranj063 commented Apr 23, 2019

Copy link
Copy Markdown
Collaborator Author

I'd like to know the last time this test was run. It should be part of the daily validation.

The test run on ww16.5 shows the result as a PASS. But I tested with the same kernel commit as in the test and it failed. So I am not sure what went wrong

@stevyan

stevyan commented Apr 24, 2019

Copy link
Copy Markdown

@ranj063 @plbossart @wenqingfu sorry for the test result mistake & confusion, of cause, this case already added to our daily validation, however, there are more dependency relationships for the sof driver modules, we always use 'kmod_scripts/sof_bootloop.sh' to execute the reload case, actually, the script missed the 'Panic' failure checking, hence, it adjusted the test 'PASS' wrongly. I'll correct all the corresponding adjusting condition - 'Panic' in relative test scripts.

@wenqingfu

Copy link
Copy Markdown

This has been clarified as an unfortunate test case failure. The test case has been fixed. Ping for update on patch review.

@ranj063

ranj063 commented Apr 29, 2019

Copy link
Copy Markdown
Collaborator Author

@plbossart this was a missed failure when the change to make the component driver resource managed was implemented. Prior to that, we used to unregister the component driver before calling snd_sof_dsp_remove().

@wenqingfu

Copy link
Copy Markdown

ping... @plbossart

@plbossart

plbossart commented May 3, 2019 •

Copy link
Copy Markdown
Member

we had an explicit code move with commit 6b7db56, and I don't get why we are reverting this, see e.g. the previous comment:

/*
+	 * platform_device_unregister() frees the card and its resources.
+	 * So it should be called after unregistering the comp driver
+	 * so that the card is valid while unregistering comp driver.
+	 */

Besides, there are still issues, e.g. snd_soc_unregister_component(sdev->dev); is still used in the core while we moved to devm_ functions.

In short it's a mess. And I will not merge RFC.

@plbossart

Copy link
Copy Markdown
Member

not to mention that 2 approvals are needed as agreed.

@ranj063

ranj063 commented May 3, 2019

Copy link
Copy Markdown
Collaborator Author

we had an explicit code move with commit 6b7db56, and I don't get why we are reverting this, see e.g. the previous comment:

@plbossart this commit was merged some time before we made changes recently in the core to alter the sequence for unloading the SOF module. So, I wont use this against this PR as we dont even have an unregister_component anymore. The component dev is resource managed now and it is unregistered when the sof device is removed.

Having said that, I am not even asking for merging this at this point but only for some feedback. The fix works but I do not understand why the codec topology doesnt face the same issue. Like I mentioned in my description, if the DSP is removed before we unload the modules, unloading the pipeline widget should fail. This is what we see with the nocodec topology but not with the codec topology. I'll work with @bardliao on Monday to see if we can come up with an explanation for this.

@wenqingfu
wenqingfu requested review from kv2019i and lyakh May 3, 2019 17:10
@ranj063

ranj063 commented May 6, 2019

Copy link
Copy Markdown
Collaborator Author

@plbossart @bardliao I now know the reason it works for the pcm512x topology and doesnt work with the nocodec topology.

In the case of pcm512x, hda_dsp_remove() calls the snd_hdac_ext_bus_device_remove() to remove the codec which unregisters the component driver for the hdmi codec. The component device for the hdmi codec is the same as the component device for the SOF pcm component driver. So unregistering hdmi codec ends up unregistering the SOF component driver thereby removing the topology. This happens during:

snd_hdac_ext_bus_device_remove(bus);

And so the topology is removed before the DSP is removed. And this is followed by unregistering the machine driver which attempts to remove the topology but the topology has already been removed. So all is good.

But in the case of nocodec topology, hda_dsp_remove() removes the DSP first. Then unregistering the machine driver ends up removing topology and this crashes because the DSP has been removed already.

@ranj063
ranj063 force-pushed the fix/nocodec-unload branch from 3fae09d to f8d3fb9 Compare May 6, 2019 06:22
@ranj063 ranj063 changed the title [RFC] Remove DSP after unregistering the machine driver Remove DSP after unregistering the machine driver May 6, 2019
bardliao
bardliao previously approved these changes May 6, 2019

@bardliao bardliao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The explanation is clear, and the solution makes sense to me

Comment thread sound/soc/sof/core.c Outdated
if (!IS_ERR_OR_NULL(pdata->pdev_mach))
platform_device_unregister(pdata->pdev_mach);

/* remove DSP */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add some comments nearby to show the correct sequence of driver removing. You explanation in PR is very nice. Summarize and add to here will make sure others understand that.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xiulipan updated

Comment thread sound/soc/sof/core.c
return 0;

comp_err:
snd_soc_unregister_component(sdev->dev);

@keyonjie keyonjie May 6, 2019 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really helpful for fixing #861 or only changing it while you are at here? My understanding is that we can invoke unregister specifically for resource-managed component device.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup, just while i'm at it. Not relevant for #861. Only the first commit is needed to fix the issue

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is a good idea. This is a .probe() "bottom half," it's often performed in a work context after the actual probing has completed and returned success. Usually when you use managed allocations, it's done within .probe(), so that a failure leads to the .probe() failing and then all such allocations are freed. In this case an allocation failure will not lead to .probe() failing, so, the resource will stay allocated. This isn't bad, it will be freed if the modules are unloaded, but it's also not useful to keep it in the error case. So, maybe better to keep that unregister.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lyakh i havent tried this yet but what happens if you leave the unregister here, the machine driver registration fails and then you try to unload sof device?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ranj063 I haven't tried either. Nothing bad should happen at the resource management level, you have already unregistered the resource, so, no second unregistering will be attempted by the resource management system. But I don't know whether the driver's .release() method would choke because it assumes that the registration has completed successfully.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lyakh i just tried it and nothing bad happened. I am OK to leave it here or remove it. removing it makes more sense to me but if you prefer leaving it here i'm OK with it too.

Comment thread sound/soc/sof/core.c

/* remove DSP */
snd_sof_remove(sdev);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, but still one question unclear to me:

In your commit message it is stated "snd_hdac_ext_bus_device_remove() to remove the codec which unregisters the component driver thereby also removing the topology.", per my understanding, this only remove the codec soc component(not the soc component we register here?), how does it removing the topology.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@keyonjie take a look at __snd_soc_unregister_component() in soc-core.c. It removes the tplg for all components who's dev matches the input argument "dev". Now, in our case both the hdmic codec component device and the SOF component device are the same and thats why snd_hdac_ext_bus_device_remove() results in removing the topology

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, if we share the "dev" in both hdmi codec and SOF component, that explains, thanks.

@plbossart plbossart left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ranj063. I find your commit message verbose and convoluted, with multiple repeats, and leaves open questions.

Suggested edits:
Unregistering the machine driver after removing the DSP results in a kernel panic during the removal
of the pipeline widget in topology, which will attempt to power down the core on which it is scheduled by access DSP registers after snd_sof_remove() disables the DSP and unmaps the DSP BAR.

The suggested fix is to unregister the machine driver
first and then remove the DSP to avoid the situation described
above.

Note that the kernel panic only happens in the case of cases where the HDaudio link is not managed by the hdac library, e.g. no codec or cases where HDMI is not supported with an I2S codec.
When the hdac library is used, snd_sof_remove() calls
snd_hdac_ext_bus_device_remove() to remove the codec which
unregisters the component driver thereby also removing the
topology.

--->>> but in the latter case, unregistering the machine driver first will result in error message "Topology forgot to remove link %d", no??

@ranj063
ranj063 force-pushed the fix/nocodec-unload branch from c69380a to 0df8885 Compare May 6, 2019 21:41
@ranj063
ranj063 requested a review from dbaluta as a code owner May 6, 2019 21:41
@ranj063

ranj063 commented May 6, 2019

Copy link
Copy Markdown
Collaborator Author

but in the latter case, unregistering the machine driver first will result in error message "Topology forgot to remove link %d", no??

@plbossart fixed commit message now. Also, I have not seen the above error in either case. Could you please give me some more details on which platform you see this error on?

ranj063 added 2 commits May 6, 2019 14:55
snd_sof_remove() disables the DSP and unmaps the DSP BAR.
Removing topology after disabling the DSP results in a
kernel panic while unloading the pipeline widget. This is
because pipeline widget unload attempts to power down
the core it is scheduled on by accessing the DSP registers.

So, the suggested fix here is to unregister the machine driver
first to remove the topology and then disable the DSP
to avoid the situation described above.

Note that the kernel panic only happens in cases where the
HDaudio link is not managed by the hdac library,
e.g. no codec or when HDMI is not supported.
When the hdac library is used, snd_sof_remove() calls
snd_hdac_ext_bus_device_remove() to remove the codec which
unregisters the component driver thereby also removing the
topology before the DSP is disabled.

Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
No need to call snd_soc_unregister_component in case of error
because the component device is resource-managed.

Signed-off-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
@ranj063
ranj063 force-pushed the fix/nocodec-unload branch from 0df8885 to 5a1a793 Compare May 6, 2019 21:56
@plbossart

Copy link
Copy Markdown
Member

but in the latter case, unregistering the machine driver first will result in error message "Topology forgot to remove link %d", no??

@plbossart fixed commit message now. Also, I have not seen the above error in either case. Could you please give me some more details on which platform you see this error on?

I haven't seen this error, just checking if we are abusing the topology code by doing a cleanup during the card release instead of at the component release. We've circled on this multiple times and I am not happy to change courses without knowing what I am really signing up for.

@ranj063

ranj063 commented May 7, 2019

Copy link
Copy Markdown
Collaborator Author

I haven't seen this error, just checking if we are abusing the topology code by doing a cleanup during the card release instead of at the component release. We've circled on this multiple times and I am not happy to change courses without knowing what I am really signing up for.

@plbossart we're not abusing the topology code. The only difference between the 2 cases is when the topology is removed. In the codec case, it is removed when the hdmi component is unregistered and in the nocodec it is removed when the machine driver is unregistered. In terms of the flow, they are both exactly the same.

@plbossart

Copy link
Copy Markdown
Member

I haven't seen this error, just checking if we are abusing the topology code by doing a cleanup during the card release instead of at the component release. We've circled on this multiple times and I am not happy to change courses without knowing what I am really signing up for.

@plbossart we're not abusing the topology code. The only difference between the 2 cases is when the topology is removed. In the codec case, it is removed when the hdmi component is unregistered and in the nocodec it is removed when the machine driver is unregistered. In terms of the flow, they are both exactly the same.

Sorry, I don't see how removing an HDMI codec would result in the SOF component being removed as well. This is also convoluted in the commit message below:

When the hdac library is used, snd_sof_remove() calls
(a) snd_hdac_ext_bus_device_remove() to remove the codec which
(b) unregisters the component driver thereby also removing the
topology before the DSP is disabled.

I am missing a link here between (a) and (b)

@ranj063

ranj063 commented May 7, 2019

Copy link
Copy Markdown
Collaborator Author

Sorry, I don't see how removing an HDMI codec would result in the SOF component being removed as well. This is also convoluted in the commit message below:

@plbossart every single person in the comments has asked the same question. So let me clarify it again: If you look at the parent-child hierarchy in SOF, both the SOF component device and the HDMI codec component device are the same (which is the SOF device). And when the HDMI codec is removed, we unregister the component device which calls __snd_soc_unregister_component() in soc-core.c. This function removes the topology for all components whose device matches the input argument device. Since both the HDMI component and the SOF pcm component's have the same dev, the topology gets removed as well. Does that make sense?

Now, I did submit a patch to alsa-dev to remove the topology removal code from unregister_component. But mark preferred to keep it to make the device removal robust.

@plbossart

Copy link
Copy Markdown
Member

Sorry, I don't see how removing an HDMI codec would result in the SOF component being removed as well. This is also convoluted in the commit message below:

@plbossart every single person in the comments has asked the same question. So let me clarify it again: If you look at the parent-child hierarchy in SOF, both the SOF component device and the HDMI codec component device are the same (which is the SOF device). And when the HDMI codec is removed, we unregister the component device which calls __snd_soc_unregister_component() in soc-core.c. This function removes the topology for all components whose device matches the input argument device. Since both the HDMI component and the SOF pcm component's have the same dev, the topology gets removed as well. Does that make sense?

Ah, that almost makes sense then, but I don't recall how these two separate parts of the code end-up using the same device. Any pointers?

Now, I did submit a patch to alsa-dev to remove the topology removal code from unregister_component. But mark preferred to keep it to make the device removal robust.

Yes, and that's fine.

@ranj063

ranj063 commented May 7, 2019

Copy link
Copy Markdown
Collaborator Author

Ah, that almost makes sense then, but I don't recall how these two separate parts of the code end-up using the same device. Any pointers?

@plbossart this is a bit convoluted but what I see the the hdac_device's dev member points to the SOF device. ANd when the HDMI driver regsiters the component driver, this is the device that gets passed to it.

@plbossart

Copy link
Copy Markdown
Member

Ah, that almost makes sense then, but I don't recall how these two separate parts of the code end-up using the same device. Any pointers?

@plbossart this is a bit convoluted but what I see the the hdac_device's dev member points to the SOF device. ANd when the HDMI driver regsiters the component driver, this is the device that gets passed to it.

care to add a pointer for the rest of us? Thanks!

@ranj063

ranj063 commented May 7, 2019

Copy link
Copy Markdown
Collaborator Author

Ah, that almost makes sense then, but I don't recall how these two separate parts of the code end-up using the same device. Any pointers?

@plbossart this is a bit convoluted but what I see the the hdac_device's dev member points to the SOF device. ANd when the HDMI driver regsiters the component driver, this is the device that gets passed to it.

care to add a pointer for the rest of us? Thanks!

@plbossart yes sure, I'm looking. Will add it when I find it.

@ranj063

ranj063 commented May 8, 2019 •

Copy link
Copy Markdown
Collaborator Author

@plbossart @keyonjie @bardliao i was so very wrong in my assumption on why the topology gets removed. But now I know what happens but it is still very convoluted. Let me try anyway. There are 2 reasons for why the topology gets removed when the HDMI codec is removed:

  1. The first one is Libin's patch that adds a device link between the HDMI codec and the machine driver. So when the HDMI codec driver is released, so is the machine driver which ends up removing all the widgets. Probably to do with the DL_FLAG_AUTOREMOVE_CONSUMER. @libinyang something for you to take a look at.
  2. So reverted the patch to see if what happens. Even after I reverted this patch, the topology was removed before the DSP is disabled. And this is because when the HDMI component device is unregistered, it calls snd_soc_unbind_card() which removes the the link components by calling soc_remove_link_components(). And this is where the SOF component topology is removed.

As for 2, I feel like this might be a mistake in the ASoC core. Deleting a component shouldnt unbind the card. This is OK if there is only ever one component associated with the card but in the case of multiple components, it shouldnt happen.

@plbossart

plbossart commented May 8, 2019 •

Copy link
Copy Markdown
Member

@plbossart @keyonjie @bardliao i was so very wrong in my assumption on why the topology gets removed. But now I know what happens but it is still very convoluted. Let me try anyway. There are 2 reasons for why the topology gets removed when the HDMI codec is removed:

  1. The first one is Libin's patch that adds a device link between the HDMI codec and the machine driver. So when the HDMI codec driver is released, so is the machine driver which ends up removing all the widgets. Probably to do with the DL_FLAG_AUTOREMOVE_CONSUMER. @libinyang something for you to take a look at.
  2. So reverted the patch to see if what happens. Even after I reverted this patch, the topology was removed before the DSP is disabled. And this is because when the HDMI component device is unregistered, it calls snd_soc_unbind_card() which removes the the link components by calling soc_remove_link_components(). And this is where the SOF component topology is removed.

As for 2, I feel like this might be a mistake in the ASoC core. Deleting a component shouldnt unbind the card. This is OK if there is only ever one component associated with the card but in the case of multiple components, it shouldnt happen.

Thanks @ranj063. it'd be useful to represent the sequence with a call graph, e.g.

sof_remove
  snd_hdac_blah
    snd_soc_unbind_card

so that we can track the logic. It's very hard with sentences to follow the code.

@ranj063

ranj063 commented May 8, 2019

Copy link
Copy Markdown
Collaborator Author

@plbossart here's the pseudo call trace for the 2 cases I explained in my previous message.

  1. With Libin's patch to add a device link between the HDMI codec driver and machine driver:

snd_hdac_ext_bus_device_remove(ehdaudio0D2) -> bus_remove_device(ehdaudio0D2) -> device_release_driver(ehdaudio0D2) -> device_release_driver(bxt-pcm512x)

The last call to release the machine driver is what removes the topology. I am not sure what happens in the release() callback for the machine driver but this ends up removing the topology. @libinyang is looking into what happens and can add more details.

  1. After reverting Libin's patch:
    snd_hdac_ext_bus_device_remove(ehdaudio0D2) -> snd_soc_unregister_component(ehdaudio0D2) ->
    snd_soc_component_del_unlocked(ehdaudio0D2) -> snd_soc_unbind_card() -> soc_remove_link_components() -> soc_remove_component() for all components -> sof_pcm_remove() -> snd_soc_tplg_component_remove()

This is what I feel is likely a mistake. Unregistering the HDMI codec component, results in unbinding the card. This is fine if there is only ever one component for the card but shouldnt be the case when there are more than 1 like in our case where we have the SOF component and the HDMI codec component.

@ranj063

ranj063 commented May 8, 2019

Copy link
Copy Markdown
Collaborator Author

@libinyang @plbossart I have a bit more information on why the device link removal causes the machine driver to be released as well. I think this is the expected behaviour with device links. The device link ensures that the consumer (m/c drv) cannot be probed before the supplier (hdmi codec). And if the supplier is removed, the consumer should be unbound and released. So this is fine and there are no changes needed. But this will mean that when the m/c driver is released, the topology will be removed as well.

@plbossart plbossart left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a migraine reading all the comments. This looks like a mess but I will have to trust this fix is legit.
I need someone else to approve (only the 3rd PR this morning that is stuck for lack of reviewers).

@bardliao bardliao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks good to me. In fact, I think the HDMI case is happen to work. So, we still need the fix.

@keyonjie keyonjie left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's good to go for me also.

@plbossart
plbossart merged commit f19ab80 into thesofproject:topic/sof-dev May 10, 2019
@ranj063
ranj063 deleted the fix/nocodec-unload branch July 18, 2019 01:18
oder-chiou pushed a commit to oder-chiou/linux-soundwire that referenced this pull request May 5, 2026
…hen look_up_lock_class() got confused

Printing this information will be helpful:

  ------------[ cut here ]------------
  Looking for class "l2tp_sock" with key l2tp_socket_class, but found a different class "slock-AF_INET6" with the same key
  WARNING: CPU: 1 PID: 14195 at kernel/locking/lockdep.c:940 look_up_lock_class+0xcc/0x140
  Modules linked in:
  CPU: 1 PID: 14195 Comm: a.out Not tainted 6.0.0-rc6-dirty thesofproject#863
  Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
  RIP: 0010:look_up_lock_class+0xcc/0x140

Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://lore.kernel.org/r/bd99391e-f787-efe9-5ec6-3c6dc4c587b0@I-love.SAKURA.ne.jp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Regression] Kernel panic when tried to remove sof module (sof_pci_dev) on APL UP^2 nocodec.

8 participants