Skip to content

Revert "pull request #1407 from bstatcomp/matrix_cl_refactor" - #1428

Merged
SteveBronder merged 3 commits into
developfrom
reverts/PR-1407
Nov 4, 2019
Merged

SteveBronder merged 3 commits into
developfrom
reverts/PR-1407

Conversation

@SteveBronder

@SteveBronder SteveBronder commented Nov 3, 2019 •

Copy link
Copy Markdown
Collaborator

This reverts commit 6723f81, reversing
changes made to 6455b69.

Checklist

  • Math issue deduplicate matrix_cl code #1407

  • Copyright holder: Steve Bronder

  • the basic tests are passing

    • unit tests pass (to run, use: ./runTests.py test/unit)
    • header checks pass, (make test-headers)
    • docs build, (make doxygen)
    • code passes the built in C++ standards checks (make cpplint)
  • the code is written in idiomatic C++ and changes are documented in the doxygen

  • the new changes are tested

…_refactor"

This reverts commit 6723f81, reversing
changes made to 6455b69.
rok-cesnovar
rok-cesnovar previously approved these changes Nov 3, 2019

@rok-cesnovar rok-cesnovar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@SteveBronder

Copy link
Copy Markdown
Collaborator Author

Trying to think of the longer term fix testing wise to catch these things earlier

@SteveBronder

Copy link
Copy Markdown
Collaborator Author

@serban-nicusor-toptal this keeps getting errors like the below on windows

g++ -std=c++1y -m64 -D_REENTRANT -Wall -Wno-unused-function -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-variable -Wno-sign-compare -Wno-unused-local-typedefs     -I lib/tbb_2019_U8/include -O3  -I . -I lib/eigen_3.3.3 -I lib/boost_1.69.0 -I lib/sundials_4.1.0/include -I lib/gtest_1.8.1/include -I lib/gtest_1.8.1 -I lib/gtest_1.8.1/include -I lib/gtest_1.8.1    -D_USE_MATH_DEFINES  -DBOOST_DISABLE_ASSERTS        -c -o test/unit/math/mix/mat/fun/log_mix_test.o test/unit/math/mix/mat/fun/log_mix_test.cpp

cc1plus.exe: out of memory allocating 65536 bytes

Do you know why windows would run out of mem? I think it could be because of too many parallel jobs being run at once. I slammed it down to 4 atm. If that works maybe we should change the jenkins setup so it only runs like half as many possible parallel jobs on windows or something

https://jenkins.mc-stan.org/blue/organizations/jenkins/Math%20Pipeline/detail/reverts%2FPR-1407/2/pipeline

@rok-cesnovar

Copy link
Copy Markdown
Member

That seemed to do the trick, though things did not break all the time. So its hard to say if its a false positive. 4 does seem a bit low. Mind changing that to half and we can use that for some time, if things break again we can think come back to it.

@stan-buildbot

Copy link
Copy Markdown
Contributor

(stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan, 1.01)
(stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan, 0.99)
(stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan, 1.0)
(stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan, 1.01)
(stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan, 1.01)
(stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan, 1.03)
(stat_comp_benchmarks/benchmarks/arK/arK.stan, 1.0)
(performance.compilation, 1.02)
(stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan, 0.98)
(stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan, 1.0)
(stat_comp_benchmarks/benchmarks/sir/sir.stan, 1.02)
(stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan, 1.06)
(stat_comp_benchmarks/benchmarks/garch/garch.stan, 0.99)
(stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan, 0.98)
(stat_comp_benchmarks/benchmarks/arma/arma.stan, 0.99)
Result: 1.00560998985
Commit hash: d4c1f3d

@serban-nicusor-toptal

serban-nicusor-toptal commented Nov 4, 2019 •

Copy link
Copy Markdown
Contributor

Do we need beefier machines for windows because of this? It currently has m5d.12xlarge (48 vCPU, 192 GB RAM).
I looked for a way to allocate more memory for cc1plus.exe but didn't find anything, do you know how can I fix this on Jenkins @rok-cesnovar ? Thanks!

@rok-cesnovar

Copy link
Copy Markdown
Member

Do we need beefier machines for windows because of this? It currently has m5d.12xlarge (48 vCPU, 192 GB RAM).

I think that is plenty enough, I would say its overkill on the RAM.

I looked for a way to allocate more memory for cc1plus.exe but didn't find anything, do you know how can I fix this on Jenkins @rok-cesnovar ? Thanks

Is it possible its that same issue we had before with how much RAM Jenkins can use? That came up for one of the tests some time ago. What is it set at currently? Maybe we should try setting a larger limit? By default its 1GB, the test that failed was usign 3GB or something.

@serban-nicusor-toptal

Copy link
Copy Markdown
Contributor

It's set to 32GB on Jenkins ( java process)

@rok-cesnovar

rok-cesnovar commented Nov 4, 2019 •

Copy link
Copy Markdown
Member

I dont really have an idea on how RAM usage scales with multiple jobs, but I guess its (RAM usage for single core) * jobs. Compiling on single core easily takes up a few GB of RAM. If we multiply that with 24 we would get over that limit. Can we try setting it to 64GB for a test?

@serban-nicusor-toptal

Copy link
Copy Markdown
Contributor

Give it a try, it's on 64 GB now.

@rok-cesnovar

Copy link
Copy Markdown
Member

Thanks!! restarted the tests, lets see.

@rok-cesnovar

Copy link
Copy Markdown
Member

It looks fine now. Can we check if this ran on a EC2 instance?

@rok-cesnovar

Copy link
Copy Markdown
Member

@serban-nicusor-toptal can you tell me which GPUs we use on the Mac machine and which one is used on the Linux machine? Would probably help find the reason for the failures that lead to this PR.

@serban-nicusor-toptal

serban-nicusor-toptal commented Nov 4, 2019 •

Copy link
Copy Markdown
Contributor

It's not running on EC2 now. I need to turn off our windows instance so EC2 can boot up.
To check on what's running either look on the main page, you'll see the job under the machine name or go to Job Console Logs and CTRL + F after 'Running on '.

I will send you an email with the GPU info later when I get home.

@stan-buildbot

Copy link
Copy Markdown
Contributor

(stat_comp_benchmarks/benchmarks/gp_pois_regr/gp_pois_regr.stan, 1.03)
(stat_comp_benchmarks/benchmarks/low_dim_corr_gauss/low_dim_corr_gauss.stan, 0.99)
(stat_comp_benchmarks/benchmarks/irt_2pl/irt_2pl.stan, 0.99)
(stat_comp_benchmarks/benchmarks/pkpd/one_comp_mm_elim_abs.stan, 1.0)
(stat_comp_benchmarks/benchmarks/eight_schools/eight_schools.stan, 0.98)
(stat_comp_benchmarks/benchmarks/gp_regr/gp_regr.stan, 0.96)
(stat_comp_benchmarks/benchmarks/arK/arK.stan, 1.0)
(performance.compilation, 1.03)
(stat_comp_benchmarks/benchmarks/low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan, 0.96)
(stat_comp_benchmarks/benchmarks/low_dim_gauss_mix/low_dim_gauss_mix.stan, 1.01)
(stat_comp_benchmarks/benchmarks/sir/sir.stan, 0.99)
(stat_comp_benchmarks/benchmarks/pkpd/sim_one_comp_mm_elim_abs.stan, 1.01)
(stat_comp_benchmarks/benchmarks/garch/garch.stan, 1.0)
(stat_comp_benchmarks/benchmarks/gp_regr/gen_gp_data.stan, 0.96)
(stat_comp_benchmarks/benchmarks/arma/arma.stan, 0.99)
Result: 0.99330496333
Commit hash: b9b4815

@SteveBronder

Copy link
Copy Markdown
Collaborator Author

@rok-cesnovar can you approve?

@SteveBronder

Copy link
Copy Markdown
Collaborator Author

Or should we bump this back up to -j${env.PARALLEL} on windows then merge?

@rok-cesnovar

Copy link
Copy Markdown
Member

Nah lets try with this for now.

@SteveBronder
SteveBronder merged commit da0e9fd into develop Nov 4, 2019
rok-cesnovar added a commit that referenced this pull request Nov 11, 2019
This reverts commit da0e9fd, reversing
changes made to 025a142.
@rok-cesnovar rok-cesnovar mentioned this pull request Nov 11, 2019
1 of 5 tasks
@SteveBronder
SteveBronder deleted the reverts/PR-1407 branch December 30, 2019 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants