Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 38 additions & 14 deletions stan/math/opencl/copy.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,15 @@ namespace stan {
namespace math {

/** \ingroup opencl
* Copies the source Eigen matrix to
* the destination matrix that is stored
* on the OpenCL device.
* Copies the source Eigen matrix to the destination matrix that is stored on
* the OpenCL device. If a lvalue matrix is passed to this function the caller
* must make sure that the matrix does not go out of scope before copying is
* complete.
*
* That means `.wait()` must be called on the event associated on copying or
* any other event that requires completion of this event. This can be done by
* calling `.wait_for_write_events()` or `.wait_for_read_write_events()` on
* returned matrix or any matrix that is calculated from that one.
*
* @param src source Eigen matrix
* @return matrix_cl with a copy of the data in the source matrix
Expand All @@ -41,9 +47,14 @@ inline matrix_cl<Mat_scalar> to_matrix_cl(Mat&& src) {
}

/** \ingroup opencl
* Copies the source std::vector to
* the destination matrix that is stored
* on the OpenCL device.
* Copies the source `std::vector` to the destination matrix that is stored on
* the OpenCL device. If a lvalue is passed to this constructor the caller must
* make sure that it does not go out of scope before copying is complete.
*
* That means `.wait()` must be called on the event associated on copying or
* any other event that requires completion of this event. This can be done by
* calling `.wait_for_write_events()` or `.wait_for_read_write_events()` on
* returned matrix or any matrix that is calculated from that one.
*
* @param src source std::vector
* @return matrix_cl with a copy of the data in the source matrix
Expand All @@ -55,9 +66,8 @@ inline matrix_cl<Vec_scalar> to_matrix_cl(Vec&& src) {
}

/** \ingroup opencl
* Copies the source matrix that is stored
* on the OpenCL device to the destination Eigen
* matrix.
* Copies the source matrix that is stored on the OpenCL device to the
* destination Eigen matrix.
*
* @param src source matrix on the OpenCL device
* @return Eigen matrix with a copy of the data in the source matrix
Expand Down Expand Up @@ -93,7 +103,7 @@ inline Eigen::Matrix<T, R, C> from_matrix_cl(const matrix_cl<T>& src) {
}

/** \ingroup opencl
* Packs the flat triagnular matrix on the OpenCL device and
* Packs the flat triangular matrix on the OpenCL device and
* copies it to the std::vector.
*
* @param src the flat triangular source matrix on the OpenCL device
Expand Down Expand Up @@ -129,9 +139,15 @@ inline std::vector<T> packed_copy(const matrix_cl<T>& src) {
}

/** \ingroup opencl
* Copies the packed triangular matrix from
* the source std::vector to an OpenCL buffer and
* unpacks it to a flat matrix on the OpenCL device.
* Copies the packed triangular matrix from the source std::vector to an OpenCL
* buffer and unpacks it to a flat matrix on the OpenCL device. If a lvalue is
* passed to this constructor the caller must make sure that it does not go out
* of scope before copying is complete.
*
* That means `.wait()` must be called on the event associated on copying or
* any other event that requires completion of this event. This can be done by
* calling `.wait_for_write_events()` or `.wait_for_read_write_events()` on
* returned matrix or any matrix that is calculated from that one.
*
* @tparam matrix_view the triangularity of the source matrix
* @param src the packed source std::vector
Expand Down Expand Up @@ -212,7 +228,15 @@ inline T from_matrix_cl_error_code(const matrix_cl<T>& src) {
}

/** \ingroup opencl
* Copy an arithmetic type to the device.
* Copy an arithmetic type to the device. If a lvalue is passed to this
* constructor the caller must make sure that it does not go out of scope before
* copying is complete.
*
* That means `.wait()` must be called on the event associated on copying or
* any other event that requires completion of this event. This can be done by
* calling `.wait_for_write_events()` or `.wait_for_read_write_events()` on
* returned matrix or any matrix that is calculated from that one.
*
* @tparam T An arithmetic type to pass the value from the OpenCL matrix to.
* @param src Arithmetic to receive the matrix_cl value.
* @return A 1x1 matrix on the device.
Expand Down
198 changes: 171 additions & 27 deletions stan/math/opencl/matrix_cl.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -218,10 +218,17 @@ class matrix_cl<T, require_arithmetic_t<T>> {
write_events_(std::move(A.write_events_)),
read_events_(std::move(A.read_events_)) {}

/** \ingroup opencl
/**
* Constructor for the matrix_cl that creates a copy of a std::vector of Eigen
* matrices on the OpenCL device. Each matrix is flattened into one column
* of the resulting matrix_cl.
* of the resulting matrix_cl. If a lvalue is passed to this constructor the
* caller must make sure that the vector does not go out of scope before

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@t4c1, first off, thank you for adding information that there is a requirement on the caller to handle memory.

The documentation isn't quite there yet. It's not clear from reading this when "copying is complete." If I'm trying to understand the requirements on memory management, it would help to be more precise about what this means.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hopefully it is clear now.

* copying is complete.
*
* That means `.wait()` must be called on the event associated on copying or
* any other event that requires completion of this event. This can be done by
* calling `.wait_for_write_events()` or `.wait_for_read_write_events()` on
* this matrix or any matrix that is calculated from this one.
*
* @param A the vector of Eigen matrices
*
Expand Down Expand Up @@ -284,29 +291,88 @@ class matrix_cl<T, require_arithmetic_t<T>> {
}

/** \ingroup opencl
* Constructor for the matrix_cl that
* creates a copy of the Eigen matrix on the OpenCL device.
* Regardless of `partial_view`, whole matrix is stored.
* Constructor for the matrix_cl that creates a copy of the Eigen matrix or
* Eigen expression on the OpenCL device. Regardless of `partial_view`, whole
* matrix is stored. If a lvalue matrix is passed to this constructor the
* caller must make sure that the matrix does not go out of scope before
* copying is complete.
*
* @tparam T type of data in the \c Eigen \c Matrix
* @param A the \c Eigen \c Matrix
* That means `.wait()` must be called on the event associated on copying or
* any other event that requires completion of this event. This can be done by
* calling `.wait_for_write_events()` or `.wait_for_read_write_events()` on
* this matrix or any matrix that is calculated from this one.
*
* @tparam Mat type of \c Eigen \c Matrix or expression
* @param A the \c Eigen \c Matrix or expression
* @param partial_view which part of the matrix is used
*
* @throw <code>std::system_error</code> if the memory on the device could not
* be allocated
*/
template <typename Mat, require_eigen_t<Mat>..., require_same_vt<Mat, T>...>
template <typename Mat, require_eigen_t<Mat>..., require_same_vt<Mat, T>...,
require_not_t<is_eigen_contiguous_map<Mat>>...>
explicit matrix_cl(Mat&& A,
matrix_cl_view partial_view = matrix_cl_view::Entire)
: rows_(A.rows()), cols_(A.cols()), view_(partial_view) {
initialize_buffer<std::is_rvalue_reference<Mat&&>::value
|| !is_eigen_matrix<Mat>::value>(A.eval().data());
using Mat_type = std::decay_t<decltype(A.eval())>;
if (size() == 0) {
return;
}
if (is_eigen_matrix_or_array<Mat>::value
&& std::is_lvalue_reference<Mat>::value) {
initialize_buffer(A.eval().data());
} else {
auto* A_heap = new Mat_type(std::move(A.eval()));
try {
cl::Event e = initialize_buffer(A_heap->data());
Comment thread
syclik marked this conversation as resolved.
Resolved
// We set a callback that will delete the memory once copying is
// complete. This event object does not hold the information about
// callback. OpenCL implementation does. So nothing is lost as event
// goes out of scope.
e.setCallback(CL_COMPLETE, &delete_it<Mat_type>, A_heap);
Comment thread
syclik marked this conversation as resolved.
Resolved
} catch (...) {
delete A_heap;
throw;
}
}
}

/** \ingroup opencl
* Constructor for the matrix_cl that
* creates a copy of a scalar on the OpenCL device.
* Regardless of `partial_view`, whole matrix is stored.
* Constructor for the matrix_cl that creates a copy of the Eigen Map on the
* OpenCL device. Regardless of `partial_view`, whole matrix is stored. The
* caller must make sure that the memory referenced by map is not deleted
* before copying is complete.
*
* That means `.wait()` must be called on the event associated on copying or
* any other event that requires completion of this event. This can be done by
* calling `.wait_for_write_events()` or `.wait_for_read_write_events()` on
* this matrix or any matrix that is calculated from this one.
*
* @tparam T type of data in the \c Eigen \c Matrix
* @param A the \c Eigen \c Map
* @param partial_view which part of the matrix is used
*
* @throw <code>std::system_error</code> if the memory on the device could not
* be allocated
*/
template <typename Map, require_t<is_eigen_contiguous_map<Map>>...,
require_same_vt<Map, T>...>
explicit matrix_cl(Map&& A,
matrix_cl_view partial_view = matrix_cl_view::Entire)
: rows_(A.rows()), cols_(A.cols()), view_(partial_view) {
initialize_buffer(A.data());
}

/** \ingroup opencl
* Constructor for the matrix_cl that creates a copy of a scalar on the OpenCL
* device. Regardless of `partial_view`, whole matrix is stored. If a lvalue
* is passed to this constructor the caller must make sure that it does not go
* out of scope before copying is complete.
*
* That means `.wait()` must be called on the event associated on copying or
* any other event that requires completion of this event. This can be done by
* calling `.wait_for_write_events()` or `.wait_for_read_write_events()` on
* this matrix or any matrix that is calculated from this one.
*
* @param A the scalar
* @param partial_view which part of the matrix is used
Expand All @@ -323,7 +389,14 @@ class matrix_cl<T, require_arithmetic_t<T>> {
}

/** \ingroup opencl
* Construct a matrix_cl of size Nx1 from \c std::vector
* Construct a matrix_cl of size Nx1 from \c std::vector. If a lvalue is
* passed to this constructor the caller must make sure that it does not go
* out of scope before copying is complete.
*
* That means `.wait()` must be called on the event associated on copying or
* any other event that requires completion of this event. This can be done by
* calling `.wait_for_write_events()` or `.wait_for_read_write_events()` on
* this matrix or any matrix that is calculated from this one.
*
* @param A Standard vector
* @param partial_view which part of the matrix is used
Expand All @@ -337,8 +410,15 @@ class matrix_cl<T, require_arithmetic_t<T>> {
matrix_cl_view partial_view = matrix_cl_view::Entire)
: matrix_cl(std::forward<Vec>(A), A.size(), 1) {}

/**
* Construct from \c std::vector with given rows and columns
/** \ingroup opencl
* Construct from \c std::vector with given rows and columns. If a lvalue
* is passed to this constructor the caller must make sure that it does not
* go out of scope before copying is complete.
*
* That means `.wait()` must be called on the event associated on copying or
* any other event that requires completion of this event. This can be done by
* calling `.wait_for_write_events()` or `.wait_for_read_write_events()` on
* this matrix or any matrix that is calculated from this one.
*
* @param A Standard vector
* @param R Number of rows the matrix should have.
Expand All @@ -353,11 +433,17 @@ class matrix_cl<T, require_arithmetic_t<T>> {
explicit matrix_cl(Vec&& A, const int& R, const int& C,
matrix_cl_view partial_view = matrix_cl_view::Entire)
: rows_(R), cols_(C), view_(partial_view) {
initialize_buffer<std::is_rvalue_reference<Vec&&>::value>(A.data());
initialize_buffer_optionally_from_heap(std::forward<Vec>(A));
}

/** \ingroup opencl
* Construct from \c array of doubles with given rows and columns
* Construct from \c array with given rows and columns. The caller
* must make sure that data is not deleted before copying is complete.
*
* That means `.wait()` must be called on the event associated on copying or
* any other event that requires completion of this event. This can be done by
* calling `.wait_for_write_events()` or `.wait_for_read_write_events()` on
* this matrix or any matrix that is calculated from this one.
*
* @param A array of doubles
* @param R Number of rows the matrix should have.
Expand Down Expand Up @@ -392,45 +478,91 @@ class matrix_cl<T, require_arithmetic_t<T>> {
* Copy assignment operator.
*/
matrix_cl<T>& operator=(const matrix_cl<T>& a) {
if (a.size() == 0) {
return *this;
}
this->view_ = a.view();
this->rows_ = a.rows();
this->cols_ = a.cols();
if (a.size() == 0) {
return *this;
}
this->wait_for_read_write_events();
initialize_buffer(a);
return *this;
}

private:
/**
/** \ingroup opencl
* Initializes the OpencL buffer of this matrix by copying the data from given
* buffer. Assumes that size of \c this is already set and matches the
* buffer size.
* buffer size. If \c in_order is false the caller must make sure that data
* is not deleted before copying is complete.
*
* That means `.wait()` must be called on the event associated on copying or
* any other event that requires completion of this event. This can be done by
* calling `.wait_for_write_events()` or `.wait_for_read_write_events()` on
* this matrix or any matrix that is calculated from this one.
*
* @tparam in_order whether copying must be done in order
* @param A pointer to buffer
* @return event for the copy
*/
template <bool in_order = false>
void initialize_buffer(const T* A) {
cl::Event initialize_buffer(const T* A) {
cl::Event transfer_event;
if (size() == 0) {
return;
return transfer_event;
}
cl::Context& ctx = opencl_context.context();
cl::CommandQueue& queue = opencl_context.queue();
try {
buffer_cl_ = cl::Buffer(ctx, CL_MEM_READ_WRITE, sizeof(T) * size());
cl::Event transfer_event;
queue.enqueueWriteBuffer(buffer_cl_,
opencl_context.in_order() || in_order, 0,
sizeof(T) * size(), A, nullptr, &transfer_event);
this->add_write_event(transfer_event);
} catch (const cl::Error& e) {
check_opencl_error("initialize_buffer", e);
}
return transfer_event;
}

/**
/** \ingroup opencl
* Initializes the OpencL buffer of this matrix by copying the data from given
* object. Assumes that size of \c this is already set and matches the
* buffer size. If the object is rvalue (temporary) it is first moved to heap
* and callback is set to delete it after copying to OpenCL device is
* complete. If a lvalue is passed to this function the caller must make
* sure that input object does not go out of scope before copying is complete.
*
* That means `.wait()` must be called on the event associated on copying or
* any other event that requires completion of this event. This can be done by
* calling `.wait_for_write_events()` or `.wait_for_read_write_events()` on
* this matrix or any matrix that is calculated from this one.
*
* @tparam U type of object
* @param obj object
* @return event for the copy
*/
template <typename U>
void initialize_buffer_optionally_from_heap(U&& obj) {
using U_val = std::decay_t<U>;
if (size() == 0) {
return;
}
if (std::is_lvalue_reference<U>::value) {
initialize_buffer(obj.data());
} else {
auto* obj_heap = new U_val(std::move(obj));
try {
cl::Event e = initialize_buffer(obj_heap->data());
e.setCallback(CL_COMPLETE, &delete_it<U_val>, obj_heap);
} catch (...) {
delete obj_heap;
throw;
}
}
}

/** \ingroup opencl
* Initializes the OpencL buffer of this matrix by copying the data from given
* matrix_cl. Assumes that size of \c this is already set and matches the
* size of given matrix.
Expand All @@ -451,6 +583,18 @@ class matrix_cl<T, require_arithmetic_t<T>> {
check_opencl_error("copy (OpenCL)->(OpenCL)", e);
}
}

/** \ingroup opencl
* Deletes the container. Used as a callback for OpenCL event.
* @tparam U type of container
* @param e cl_event handle
* @param status status of event
* @param container container to delete
*/
template <typename U>
static void delete_it(cl_event e, cl_int status, void* container) {
delete static_cast<U*>(container);
}
};

template <typename T>
Expand Down
Loading