Skip to content

Add holder operation to kernel generator - #1865

Merged
rok-cesnovar merged 13 commits into
stan-dev:developfrom
bstatcomp:cl_kernel_generator_holder
May 26, 2020
Merged

rok-cesnovar merged 13 commits into
stan-dev:developfrom
bstatcomp:cl_kernel_generator_holder

Conversation

@t4c1

@t4c1 t4c1 commented May 5, 2020 •

Copy link
Copy Markdown
Contributor

Summary

Adds holder operation to kernel generator. This operation is a no-op, but the operation object also owns pointers to some dynamically allocated objects, used in the argument expression.

This is the solution to problems such as #1859 (comment) or #1775 with kernel generator expressions.

Also creates default generate member funtion in operation_cl.

Tests

Holder is tested to keep the owned objects alive.

Side Effects

None.

Release notes

Adds holder operation to kernel generator.

Checklist

@t4c1

t4c1 commented May 5, 2020

Copy link
Copy Markdown
Contributor Author

I named this operation holder_cl as it "holds" other expression so they don't go out of scope. However I'm not a fan of this name, so if anyone has a better suggestion ...

@t4c1 t4c1 changed the title Adds holder operation to kernel generator Add holder operation to kernel generator May 5, 2020
@stan-buildbot

Copy link
Copy Markdown
Contributor

Name Old Result New Result Ratio Performance change( 1 - new / old )
gp_pois_regr/gp_pois_regr.stan 5.27 5.03 1.05 4.59% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.02 0.02 0.98 -2.31% slower
eight_schools/eight_schools.stan 0.09 0.09 0.98 -2.36% slower
gp_regr/gp_regr.stan 0.22 0.23 0.99 -1.43% slower
irt_2pl/irt_2pl.stan 6.44 6.44 1.0 0.02% faster
performance.compilation 85.85 84.64 1.01 1.41% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 7.54 7.54 1.0 -0.07% slower
pkpd/one_comp_mm_elim_abs.stan 20.67 20.05 1.03 3.0% faster
sir/sir.stan 91.18 92.27 0.99 -1.19% slower
gp_regr/gen_gp_data.stan 0.05 0.05 0.99 -0.53% slower
low_dim_gauss_mix/low_dim_gauss_mix.stan 2.98 2.98 1.0 -0.17% slower
pkpd/sim_one_comp_mm_elim_abs.stan 0.31 0.31 1.0 -0.49% slower
arK/arK.stan 1.75 1.75 1.0 0.06% faster
arma/arma.stan 0.77 0.77 1.0 -0.31% slower
garch/garch.stan 0.57 0.57 0.99 -0.65% slower
Mean result: 1.00003665187

Jenkins Console Log
Blue Ocean
Commit hash: 8423a84


Machine information ProductName: Mac OS X ProductVersion: 10.11.6 BuildVersion: 15G22010

CPU:
Intel(R) Xeon(R) CPU E5-1680 v2 @ 3.00GHz

G++:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.6.0
Thread model: posix

Clang:
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.6.0
Thread model: posix

@t4c1
t4c1 requested a review from SteveBronder May 12, 2020 09:28
@SteveBronder

Copy link
Copy Markdown
Collaborator

Is this equivalent to Eigen::Ref<>?

The name is kind of weird, should it just be copy?

@t4c1

t4c1 commented May 13, 2020 •

Copy link
Copy Markdown
Contributor Author

Is this equivalent to Eigen::Ref<>?

Not even similar. This solves a different problem. Right now we can not have functions that use an argument multiple times and return an expression, as they would break for rvalues.

Example:

template <typename T>
auto square(T&& x){
    return elementwise_multiply(x, x);
}

If x is a rvalue it will go out of scope when the function returns and multiplication will reference garbage memory as its arguments. Using const ref argument won't help. We could forward x as one of the arguments, but than we have the same problem with the other one. Forwarding an argument twice results in same problem.

So this is the solution. There are two interfaces. The first one is more practical for local variables and the second one for arguments, such as in this example.
1:

template <typename T>
auto square(T&& x){
    auto* x_heap = new std::remove_reference_t<T>(std::forward<T>(x));
    return holder_cl(elementwise_multiply(*x_heap, *x_heap), x_heap);
}

2:

template <typename T>
auto square(T&& x){
    return make_holder([](const auto& x) { return elementwise_multiply(x, x); }, std::forward<T>(x));
}

The name is kind of weird

I agree. I am open to suggestions once you understand what this is about.

should it just be copy?

No.

@SteveBronder

Copy link
Copy Markdown
Collaborator

This feels kind of onerous to put in the user space, should holder_cl be added inside of functions like elementwise_multiply?

By user space I mean any functions that are not in the kernel generator. Can the generator have a constructor for matrix_cl rvalues to handle this?

@SteveBronder

Copy link
Copy Markdown
Collaborator

Didn't we do something similar to this for matrix_cl?

@t4c1

t4c1 commented May 18, 2020 •

Copy link
Copy Markdown
Contributor Author

This feels kind of onerous to put in the user space, should holder_cl be added inside of functions like elementwise_multiply?

That would be all kernel generator functions than. I don't think we can make it work that way. As I see it, holder must be explicitely constructed to make it work. When one kg operation is being constructed it has no knowledge of whether the same rvalue is also used in another function.

An alternative would be to use dynamic allocations and std::shared_ptr for all expressions everywhere. However I don't like it.

Didn't we do something similar to this for matrix_cl?

I don't think so. What exactley do you have in mind?

@rok-cesnovar

Copy link
Copy Markdown
Member

I think Steve is thinking of this PR: #1421

@t4c1

t4c1 commented May 18, 2020

Copy link
Copy Markdown
Contributor Author

In that PR we also moved some expressions to heap to extend their lifetime. But the reason for that was different. In matrix_cl we wanted to do asynchronous copy and we needed to ensure that the matrix we are copying exists untill copying finishes.

Here we want to pass a rvalue to other function(s) multiple times. Instead we move it to heap and pass lvalue references to it.

@SteveBronder

Copy link
Copy Markdown
Collaborator

That would be all kernel generator functions than. I don't think we can make it work that way. As I see it, holder must be explicitely constructed to make it work. When one kg operation is being constructed it has no knowledge of whether the same rvalue is also used in another function.

This is all very confusing. Correct where I'm wrong below. The problem is with this code

template <typename T>
auto square(T&& x){
    return elementwise_multiply(x, x);
}

x is able to be an rvalue reference to a matrix_cl or kernel generator expression. If that is true and x is passed as both sides of another kernel generator expression such as elementwise_multiply your saying the object x is referencing can fall out of scope leading to picking up the address of garbage memory.

If that's true, x is just a reference to something outside this functions scope. If the issue is x falling out of scope in this function won't we need to wrap holder_cl around every function anyway?

Does Eigen not run into this same issue? Could we have each matrix_cl / expression wait for all it's read/write events before calling the destructor?

@t4c1

t4c1 commented May 18, 2020

Copy link
Copy Markdown
Contributor Author

Exactley.

If the issue is x falling out of scope in this function won't we need to wrap holder_cl around every function anyway?

If a matrix_cl or expression is only used once, it can be moved into whatever expression it is used in. So the function return value will also contain the input to the function.

Does Eigen not run into this same issue?

It does. We will need to do something very similar for many functions working with Eigen expressions, before they can return expressions.

Could we have each matrix_cl / expression wait for all it's read/write events before calling the destructor?

At the time such a function returns no kernel is running yet - function is only constructing part of the expression that will be used in the kernel.

@t4c1

t4c1 commented May 20, 2020

Copy link
Copy Markdown
Contributor Author

@SteveBronder Is anything still unclear here?

@t4c1

t4c1 commented May 22, 2020

Copy link
Copy Markdown
Contributor Author

It seems @SteveBronder is too busy. @rok-cesnovar can you review this?

@rok-cesnovar

Copy link
Copy Markdown
Member

Sure, though I am bad at naming stuff, so no new ideas on how to name this. Will look at this later today.

@rok-cesnovar
rok-cesnovar self-requested a review May 22, 2020 08:42
@SteveBronder

Copy link
Copy Markdown
Collaborator

It does. We will need to do something very similar for many functions working with Eigen expressions, before they can return expressions.

I think I'm just confused on this. Does Eigen have the same holder type or what do they do for this?

@andrjohns

Copy link
Copy Markdown
Collaborator

Eigen doesn't actually handle it, which has caused issues in the past (and kinda still is an issue), there's more information over in #1775

@rok-cesnovar rok-cesnovar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doxygen cleanup mostly. Otherwise looks good.

@t4c1

t4c1 commented May 25, 2020

Copy link
Copy Markdown
Contributor Author

Does Eigen have the same holder type or what do they do for this?

Element-wise functions they tend to implement elemet wise - similarly as we do in apply_scalar_unary, except they also add vectorized implementations. In some other cases they don't handle it at all - they evaluate the expression before returning.

@t4c1

t4c1 commented May 25, 2020

Copy link
Copy Markdown
Contributor Author

Also Eigen is written assuming C++03 standard, so they handle nothing about rvalues and moves. Instead they use simple heuristics to choose whether to store argument expressions by reference or value. Matrices and similar are stored by reference, expressions by value.

@t4c1

t4c1 commented May 25, 2020

Copy link
Copy Markdown
Contributor Author

@andrjohns Do you have any suggestion for a better name for this?

rok-cesnovar
rok-cesnovar previously approved these changes May 25, 2020

@rok-cesnovar rok-cesnovar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good, feel free to discuss the name a bit more. There are two outstanding issues on stan/cmdstan which will not allow any tests to pass right now.

rok-cesnovar
rok-cesnovar previously approved these changes May 26, 2020
@stan-buildbot

Copy link
Copy Markdown
Contributor

Name Old Result New Result Ratio Performance change( 1 - new / old )
gp_pois_regr/gp_pois_regr.stan 3.99 3.97 1.01 0.52% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan 0.02 0.02 0.99 -0.72% slower
eight_schools/eight_schools.stan 0.09 0.09 1.01 0.53% faster
gp_regr/gp_regr.stan 0.19 0.19 0.99 -0.57% slower
irt_2pl/irt_2pl.stan 6.56 6.54 1.0 0.39% faster
performance.compilation 84.67 83.78 1.01 1.05% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan 7.7 7.69 1.0 0.25% faster
pkpd/one_comp_mm_elim_abs.stan 20.93 21.65 0.97 -3.45% slower
sir/sir.stan 93.39 92.49 1.01 0.96% faster
gp_regr/gen_gp_data.stan 0.04 0.04 1.0 -0.45% slower
low_dim_gauss_mix/low_dim_gauss_mix.stan 3.04 3.04 1.0 -0.11% slower
pkpd/sim_one_comp_mm_elim_abs.stan 0.32 0.31 1.02 1.61% faster
arK/arK.stan 2.42 2.43 1.0 -0.32% slower
arma/arma.stan 0.6 0.61 0.98 -1.83% slower
garch/garch.stan 0.52 0.52 1.0 0.47% faster
Mean result: 0.999025617155

Jenkins Console Log
Blue Ocean
Commit hash: 01edbf9


Machine information ProductName: Mac OS X ProductVersion: 10.11.6 BuildVersion: 15G22010

CPU:
Intel(R) Xeon(R) CPU E5-1680 v2 @ 3.00GHz

G++:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.6.0
Thread model: posix

Clang:
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.6.0
Thread model: posix

@rok-cesnovar
rok-cesnovar merged commit 72b7e4a into stan-dev:develop May 26, 2020
@rok-cesnovar
rok-cesnovar deleted the cl_kernel_generator_holder branch May 26, 2020 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants