Skip to content

[RFC] device property: Don't set secondary pointer if it is not NULL - #2624

Closed
bardliao wants to merge 1 commit into
thesofproject:topic/sof-devfrom
bardliao:device-property
Closed

bardliao wants to merge 1 commit into
thesofproject:topic/sof-devfrom
bardliao:device-property

Conversation

@bardliao

Copy link
Copy Markdown
Collaborator

…re node pointer

If we do the following sequence of calls

set_secondary_fwnode(dev, NULL);
set_primary_fwnode(dev, NULL);

The secondary will be set to ERR_PTR(-ENODEV) in set_primary_fwnode(),
and it will not pass the fwnode_is_primary() test of call

set_secondary_fwnode(dev, secondary node);

The idea is that if (dev->fwnode) is not NULL, it must be primary no
matter what secondary is.

Signed-off-by: Bard Liao yung-chuan.liao@linux.intel.com

@plbossart plbossart left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not fully understanding the problem and the fix?

Comment thread drivers/base/core.c Outdated
fwnode->secondary = ERR_PTR(-ENODEV);

if (fwnode_is_primary(dev->fwnode))
if (dev->fwnode)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change may be right, but I don't see how this fixes the timeouts and what the relationship is between pm_runtime issues and device properties?
Also your commit message is hard to follow, can you revisit the wording and explain the sequence that leads to the problem better?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's also not clear to me why this change is only required after @andy-shev changes - Really lost on how Andy's changes introduced the resume timeouts we are seeing and how this patches corrects the behavior.

@andy-shev

andy-shev commented Dec 11, 2020 •

Copy link
Copy Markdown

I guess here is a misunderstanding what primary and secondary means. I tried to describe this in the andy-shev@d5dcce0. Also there may be different cases: primary and secondary are coupled, or they are independent and they (each of them) may be exclusive or shared (in this commit andy-shev@99aed92 i explained case when primary is shared and what is the outcome of breaking the link). Note, primary node is configured by OF or ACPI and by individual drivers by sharing it among devices. Cleaning happens only when device is gone (by device core).

Can we see better explanation what's the problem is?

@andy-shev

andy-shev commented Dec 11, 2020 •

Copy link
Copy Markdown

Okay, we have these cases:

  • primary->secondary->-ENODEV
  • primary->NULL
  • secondary->-ENODEV
  • NULL

Last one is easy.

First one goes like this. set_secondary_fwnode(dev, NULL) converts this to the second case.

Second one is relatively easy. set_secondary_fwnode(dev, NULL) is no op, and set_primary_fwnode(dev, NULL) does this dev->fwnode = fn->secondary; which effectivelly assigns NULL to the dev->fwnode.

Third one is simply check if it's secondary and assign NULL directly to dev->fwnode. set_primary_fwnode() becomes no-op.

@andy-shev

andy-shev commented Dec 11, 2020 •

Copy link
Copy Markdown

The secondary will be set to ERR_PTR(-ENODEV) in set_primary_fwnode(), and it will not pass the fwnode_is_primary() test of call set_secondary_fwnode(dev, secondary node);

The idea is that if (dev->fwnode) is not NULL, it must be primary no matter what secondary is.

And the last is not correct. If you assign secondary without primary, something is not okay.

I guess what you are trying to do is actually to have

  • secondary->-ENODEV

follwed by set_secondary_fwnode(dev, secondary);. No, you can't have two secondary fwnodes in a chain.

@plbossart

Copy link
Copy Markdown
Member

@andy-shev for more context, see #2603 (comment).

What we identified later is that the timeouts happens if we first have a sequence of modprobe/rmmod and remove/re-add devices. We bisected these timeouts to the introduction of timeouts to the two patches you added in v5.10-rc2. Reverting the 2 patches solves the timeout issue.

Here in this PR @bardliao was trying to see if there was a problem with properties, but I am with you, I don't get what the suggested fixes do nor why they would impact SoundWire suspend-resume. The one point I do get is we share the same fwnode between two layers so there's isn't a zero correlation with your changes.

@andy-shev

Copy link
Copy Markdown

@plbossart Can you do a quick test if replacing -ENODEV to NULL in that assignment (...->secondary = ERR_PTR(-ENODEV)) helps?

@plbossart

Copy link
Copy Markdown
Member

@plbossart Can you do a quick test if replacing -ENODEV to NULL in that assignment (...->secondary = ERR_PTR(-ENODEV)) helps?

Sorry, not following @andy-shev. The rest of the code uses IS_ERR, except where it doesn't...

e.g.

static inline bool fwnode_is_primary(struct fwnode_handle *fwnode)
{
	return fwnode && !IS_ERR(fwnode->secondary);
}

void set_secondary_fwnode(struct device *dev, struct fwnode_handle *fwnode)
{
	if (fwnode)
		fwnode->secondary = ERR_PTR(-ENODEV);

static int fwnode_property_read_int_array(const struct fwnode_handle *fwnode,
					  const char *propname,
					  unsigned int elem_size, void *val,
					  size_t nval)
{
	int ret;

	ret = fwnode_call_int_op(fwnode, property_read_int_array, propname,
				 elem_size, val, nval);
	if (ret == -EINVAL && !IS_ERR_OR_NULL(fwnode) &&
	    !IS_ERR_OR_NULL(fwnode->secondary))
...

struct fwnode_handle *device_get_next_child_node(struct device *dev,
	/* When no more children in primary, continue with secondary */
	if (fwnode && !IS_ERR_OR_NULL(fwnode->secondary))

@andy-shev

Copy link
Copy Markdown

Reverting only this (it needs to be done manually because of another patch) andy-shev@d5dcce0. I believe if it helps the problem is exactly what I described: an attempt to assign a secondary node to device which already has one.

@bardliao

Copy link
Copy Markdown
Collaborator Author

Thanks @andy-shev for the explanation.

Reverting only this (it needs to be done manually because of another patch) andy-shev@d5dcce0. I believe if it helps the problem is exactly what I described: an attempt to assign a secondary node to device which already has one.

Yes, it helps.

I guess what you are trying to do is actually to have

  • secondary->-ENODEV

follwed by set_secondary_fwnode(dev, secondary);. No, you can't have two secondary fwnodes in a chain.

You are right. But the secondary here is configured by ACPI and it was primary->secondary->-ENODEV. It became secondary->-ENODEVafter set_secondary_fwnode(dev, NULL); and set_primary_fwnode(dev, NULL); . Below is the trace log that I currently have. I will get more trace and explain it in more detail latter.

[    5.282230] rt711 sdw:1:25d:711:0: set_secondary_fwnode start dev->fwnode 00000000e7090087 dev->fwnode->secondary 0000000000000000 fwnode 0000000088f1f67d
[    5.282232] rt711 sdw:1:25d:711:0: set_secondary_fwnode done dev->fwnode 00000000e7090087 dev->fwnode->secondary 0000000088f1f67d
[   29.475313] rt711 sdw:1:25d:711:0: set_secondary_fwnode start dev->fwnode 00000000e7090087 dev->fwnode->secondary 0000000088f1f67d fwnode 0000000000000000
[   29.475316] rt711 sdw:1:25d:711:0: set_secondary_fwnode done dev->fwnode 00000000e7090087 dev->fwnode->secondary 0000000000000000
[   29.525724] soundwire sdw:1:25d:711:0: set_primary_fwnode start dev->fwnode 00000000e7090087 dev->fwnode->secondary 0000000000000000 fwnode 0000000000000000
[   29.525726] soundwire sdw:1:25d:711:0: set_primary_fwnode done dev->fwnode 0000000000000000
[   31.654916] rt711 sdw:1:25d:711:0: set_secondary_fwnode start dev->fwnode 00000000e7090087 dev->fwnode->secondary ffffffffffffffed fwnode 000000008cd3755f
[   31.654918] rt711 sdw:1:25d:711:0: set_secondary_fwnode done dev->fwnode 000000008cd3755f dev->fwnode->secondary ffffffffffffffed
[   39.836593] rt711 sdw:1:25d:711:0: set_secondary_fwnode start dev->fwnode 00000000e7090087 dev->fwnode->secondary ffffffffffffffed fwnode 000000004a1bf865
[   39.836595] rt711 sdw:1:25d:711:0: set_secondary_fwnode done dev->fwnode 000000004a1bf865 dev->fwnode->secondary ffffffffffffffed

@andy-shev

Copy link
Copy Markdown

I guess what you are trying to do is actually to have

  • secondary->-ENODEV

follwed by set_secondary_fwnode(dev, secondary);. No, you can't have two secondary fwnodes in a chain.

You are right. But the secondary here is configured by ACPI and it was primary->secondary->-ENODEV.

ACPI doesn't set secondary. Something is way wrong in your case if it's true. Custom patches?

It became secondary->-ENODEVafter set_secondary_fwnode(dev, NULL); and set_primary_fwnode(dev, NULL); . Below is the trace log that I currently have. I will get more trace and explain it in more detail latter.

I'm sorry I barely understand what is going on in this log. The rule of thumb is when you provide a trace, provide a code which gives that trace to understand where it's exactly taken and what is the mean of the fields in output.

@plbossart

plbossart commented Dec 15, 2020 •

Copy link
Copy Markdown
Member

@andy-shev @bardliao I added a trace of what I think goes South, see plbossart@76d0f5a

I think when we unload a driver and call set_primary_fwnode(NULL), the fwnode remains secondary (set to ERR_PTR(-NODEV)), so the next time we probe and add properties we end-up overriding the fwnode with the properties.

There's something missing in the initialization, either in the device core or the SoundWire core, so that on probe the previous fwnode->secondary has to become NULL again.

@bardliao bardliao changed the title [RFC] device property: set_secondary_fwnode: don't overwrite primary firmwa… [RFC] device property: Don't set secondary pointer if it is not NULL Dec 15, 2020
When we call set_primary_fwnode(dev, NULL) to remove a primary->NULL
fwnode, "fn->secondary = ERR_PTR(-ENODEV);" will force primary->NULL
become primary->ERR_PTR(-ENODEV) which is not what we expected.
This patch doesn't impact removing primary->secondary->ERR_PTR(-ENODEV)
fwnode.

Signed-off-by: Bard Liao <yung-chuan.liao@linux.intel.com>
@bardliao

Copy link
Copy Markdown
Collaborator Author

@andy-shev @plbossart If we use set_primary_fwnode to remove a primary->NULL fwnode, the primary pointer will end up becoming primary-> ERR_PTR(-ENODEV). And it will be treated as a secondary fwnode.
See the code in set_primary_fwnode below

struct fwnode_handle *fn = dev->fwnode;   <<<< say fn is (A)
...
        } else {
                if (fwnode_is_primary(fn)) {
                        dev->fwnode = fn->secondary; <<<< dev->fwnode = NULL which is expected
                        if (!(parent && fn == parent->fwnode) && fn->secondary)
                                fn->secondary = ERR_PTR(-ENODEV); <<<< fn (A) -> secondary is now ERR_PTR(-ENODEV); which seems to be incorrect. 
                } else {
...

The updated commit suggested a fix. Does it make sense?

@andy-shev andy-shev left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. This makes no sense. NAK.

@andy-shev

Copy link
Copy Markdown

@andy-shev @plbossart If we use set_primary_fwnode to remove a primary->NULL fwnode, the primary pointer will end up becoming primary-> ERR_PTR(-ENODEV). And it will be treated as a secondary fwnode.

The above makes no sense. I think you didn't get how it supposed to work. When you call set_primary_fwnode(dev, NULL); against primary --> NULL chain, it effectively means dev->fwnode = NULL;.

The updated commit suggested a fix. Does it make sense?

No.

As I told there is a mix of terminology and position in the list. struct fwnode_handle defines a pointer to the next element in a single-linked list. The list is restricted to have at least 0 at most 2 elements. Possibilities are:

  • NULL
  • primary --> NULL
  • secondary --> -ENODEV
  • primary --> secondary --> -ENODEV

All other combinations are illegal!

@andy-shev

andy-shev commented Dec 15, 2020 •

Copy link
Copy Markdown

@andy-shev @bardliao I added a trace of what I think goes South, see plbossart@76d0f5a

I think when we unload a driver and call set_primary_fwnode(NULL), the fwnode remains secondary (set to ERR_PTR(-NODEV)), so the next time we probe and add properties we end-up overriding the fwnode with the properties.

There's something missing in the initialization, either in the device core or the SoundWire core, so that on probe the previous fwnode->secondary has to become NULL again.

I will put here your words from above link:

And on the second insert, the dev->fwnode will be modified - really nasty side effect. On a device remove, we really don't want the fwnode secondary type to be persistent....

Hooray! Now you understand the issue with device_add_properties() and why we insist to use swnode directly. Your code is responsible to allocate / register / unregister swnodes at exact moments when it's appropriate from your code perspective.

UPDATE device_*_properties() compounds together alloc+register+bind vs. unregister+release+unbind. And it does not work for the case

  • secondary --> -ENODEV

@bardliao

Copy link
Copy Markdown
Collaborator Author

@andy-shev @plbossart If we use set_primary_fwnode to remove a primary->NULL fwnode, the primary pointer will end up becoming primary-> ERR_PTR(-ENODEV). And it will be treated as a secondary fwnode.

The above makes no sense. I think you didn't get how it supposed to work. When you call set_primary_fwnode(dev, NULL); against primary --> NULL chain, it effectively means dev->fwnode = NULL;.

@andy-shev Yes, dev->fwnode = NULL;, but my point is about the original dev->fwnode pointer which we removed, but will be reused when the driver is loaded again.

[   22.165910] soundwire sdw:3:25d:714:1: bard: set_primary_fwnode start dev->fwnode 0000000094d3ce7a fwnode 0000000000000000 <<<< We call set_primary_fwnode(dev, NULL) to remove a primary->NULL fwnode
[   22.165913] soundwire sdw:3:25d:714:1: bard: set_primary_fwnode !(parent && fn == parent->fwnode)		<<<< fn->secondary is set to ERR_PTR(-ENODEV) with the if (!(parent && fn == parent->fwnode)) condition
[   22.165915] soundwire sdw:3:25d:714:1: bard: set_primary_fwnode parent sdw-master-3 parent->fwnode 00000000b2414cc2 dev->fwnode 0000000000000000 
[   22.165918] soundwire sdw:3:25d:714:1: bard: set_primary_fwnode done dev->fwnode 0000000000000000		<<<< dev->fwnode is NULL which is expected, but at the same time the original fwnode pointer (0000000094d3ce7a) ->secondary is ERR_PTR(-ENODEV)
...
[   24.319843] soundwire sdw:3:25d:714:1: bard: set_primary_fwnode start dev->fwnode 0000000000000000 fwnode 0000000094d3ce7a
[   24.319844] soundwire sdw:3:25d:714:1: bard: set_primary_fwnode done dev->fwnode 0000000094d3ce7a
[   24.319845] soundwire sdw:3:25d:714:1: bard: set_primary_fwnode dev->fwnode->secondary ffffffffffffffed	<<<< We call set_primary_fwnode(dev, fwnode), but it end up with primary->ERR_PTR(-ENODEV)

dmesg.txt
The trace commit is bardliao@ce4a7e4
And the testing tree is https://github.com/bardliao/linux/tree/fwnode-trace

The updated commit suggested a fix. Does it make sense?

No.

As I told there is a mix of terminology and position in the list. struct fwnode_handle defines a pointer to the next element in a single-linked list. The list is restricted to have at least 0 at most 2 elements. Possibilities are:

  • NULL
  • primary --> NULL
  • secondary --> -ENODEV
  • primary --> secondary --> -ENODEV

All other combinations are illegal!

Yes, this PR is exactly trying to avoid the unexpected * primary --> -ENODEV
The issue what I see is that fwnode(A) will become a secondary fwnode if we call set_primary_fwnode in below sequence

  • step 1: set_primary_fwnode(dev, A)
    • result 1: dev->fwnode = A->NULL
  • step 2: set_primary_fwnode(dev, NULL)
    • result 2: dev->fwnode = NULL
  • step 3: set_primary_fwnode(dev, A)
    • result 3: dev->fwnode = A->-ENODEV
      Or maybe we should set A->NULL before step 3?

@plbossart

plbossart commented Dec 15, 2020 •

Copy link
Copy Markdown
Member

Hooray! Now you understand the issue with device_add_properties() and why we insist to use swnode directly. Your code is responsible to allocate / register / unregister swnodes at exact moments when it's appropriate from your code perspective.

UPDATE device_*_properties() compounds together alloc+register+bind vs. unregister+release+unbind. And it does not work for the case

  • secondary --> -ENODEV

@andy-shev we do remove the properties when appropriate without relying on the device core. we did this a while ago. Even if we used swnodes directly, the problem would still exist:

The problem is that when doing an rmmod snd-soc-rt711, ACPI_COMPANION_SET is called with a NULL parameter, so the fwnode is tagged as secondary. Then when we redo a modprobe snd-sof-rt711 and later add properties, the fwnode is not considered as primary, so the dev->fwnode will be overridden by the properties. In other words, we get a different result during the second iteration of our kmod load/unload tests.

I think it's keeping the type of the fwnode that is problematic when dealing with probe/remove. That's what @bardliao suggested forcing the fwnode->secondary= NULL to force the fwnode back to a primary when registering a device.

@plbossart

Copy link
Copy Markdown
Member

let's follow the review with #2625 or #2635

@plbossart plbossart closed this Dec 15, 2020
@andy-shev

Copy link
Copy Markdown

The problem is that when doing an rmmod snd-soc-rt711, ACPI_COMPANION_SET is called with a NULL parameter, so the fwnode is tagged as secondary.

So, from the state of primary --> secondary --> -ENODEV it comes to secondary --> -ENODEV. This is correct.
The word tagget above confuses me. What does it mean?

Then when we redo a modprobe snd-sof-rt711 and later add properties, the fwnode is not considered as primary, so the dev->fwnode will be overridden by the properties. In other words, we get a different result during the second iteration of our kmod load/unload tests.

Right.

I think it's keeping the type of the fwnode that is problematic when dealing with probe/remove. That's what @bardliao suggested forcing the fwnode->secondary= NULL to force the fwnode back to a primary when registering a device.

Okay, but do you agree that this is not the best approach?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants