Repository navigation
POT Accounting in CAFs using BNB Retriever module - #211
Conversation
…lls in SRHeader with one entry per event. More ideal is an entry per subrun.
cjbacchus
left a comment
There was a problem hiding this comment.
Looks good, but I have a bunch of small comments about reorganizing this.
| if (pot_handle.isValid()) { | ||
| fSubRunPOT = pot_handle->totgoodpot; | ||
| fTotalPOT += fSubRunPOT; | ||
| fIsRealData = true; |
There was a problem hiding this comment.
This is going to conflict with the recently merged PR that makes this variable fcl configurable.
There was a problem hiding this comment.
I'm sorry I did this in somewhat in a rush and forgot to clean this up. I added this to test things as I thought this was set somewhere when I first wrote this in but I found it wasn't. I will fix this by removing that forced set.
There was a problem hiding this comment.
Ah, the other PR is not merged yet, but you'll want to make sure you don't clash too badly with it #209
There was a problem hiding this comment.
I reviewed that PR after the fact and I think everything is fine. As long as fIsRealData gets set via the fcl before beginSubRun is called then it should work. It looks like that is the case. Would want to confirm after this happens. I've made sure to remove the hardcode I used for testing
There was a problem hiding this comment.
Heads up: That PR no longer uses a fhicl parameter - it was changed yesterday to determine it from the art data memeber (evt.isRealData()). It is no longer set before beginSubrun but in the process step itself 😬
| bnb_info_store.clear(); | ||
| fSubRunPOT = 0; | ||
| art::Handle<std::vector<sbn::BNBSpillInfo> > bnb_spill; | ||
| sr.getByLabel("bnbinfo", bnb_spill); |
There was a problem hiding this comment.
Can we make this label fcl configurable?
| else | ||
| { | ||
| art::Handle<sumdata::POTSummary> pot_handle; | ||
| sr.getByLabel("generator", pot_handle); |
There was a problem hiding this comment.
I guess this wasn't configurable before, but it should have been...
| fTotalPOT += fSubRunPOT; | ||
| fIsRealData = true; | ||
| if(fIsRealData) | ||
| { |
There was a problem hiding this comment.
Can we pull this out into a separate FillBNBPOTInfo function? Maybe even in a new FillExposure.cxx? Hopefully we will soon also have a NuMI version of this, which is going to make beginSubRun very long.
There was a problem hiding this comment.
Sure, that was actually my first thought. I will work towards that
| rec.hdr.fno = fFileNumber; | ||
| rec.hdr.pot = fSubRunPOT; | ||
| if(fIsRealData) | ||
| rec.hdr.bnbinfo = bnb_info_store; |
There was a problem hiding this comment.
This is duplicating all the bnbinfo into every event, right? I think we probably only want to write it once. Not sure if you should put an && fFirstInFile here, or just a .clear() after you've copied it into the record.
There was a problem hiding this comment.
Yes, this is what we discussed before. I pushed this before resolving that. Will add that in too. I can try a fFirstInFile and see if that works that's really what we want
| double fSubRunPOT; | ||
| double fTotalSinglePOT; | ||
| double fTotalEvents; | ||
| std::vector<caf::SRBNBInfo> bnb_info_store; |
There was a problem hiding this comment.
Can we give this a name following the convention? fBNBInfo is probably fine. And a comment like ///< Store detailed BNB info to save into the first StandardRecord of the output file
There was a problem hiding this comment.
Yep, my bad I forgot to follow the convention here
| std::vector<art::Ptr<sbn::BNBSpillInfo> > bnb_spill_info; | ||
| if (bnb_spill.isValid()) { | ||
| art::fill_ptr_vector(bnb_spill_info, bnb_spill); | ||
| for(unsigned int i = 0; i < bnb_spill_info.size(); ++i) |
There was a problem hiding this comment.
You can save yourself a lot of [i] indexing in the loop body using a for-each loop
| for(unsigned int i = 0; i < bnb_spill_info.size(); ++i) | |
| for(const art::Ptr<sbn::BNBSpillInfo>& info: bnb_spill_info) |
|
Okay, update in latest commit that I think addresses all comments from @cjbackhouse. Tested and works as I think intended where there's only one non-zero entry at the firstInFile. Thanks for the review and any other comments I'm happy to address. |
cjbacchus
left a comment
There was a problem hiding this comment.
Looking good. A few minor nits.
| std::vector<art::Ptr<sbn::BNBSpillInfo> > bnb_spill_info; | ||
| if (bnb_spill.isValid()) { | ||
| art::fill_ptr_vector(bnb_spill_info, bnb_spill); |
There was a problem hiding this comment.
| std::vector<art::Ptr<sbn::BNBSpillInfo> > bnb_spill_info; | |
| if (bnb_spill.isValid()) { | |
| art::fill_ptr_vector(bnb_spill_info, bnb_spill); | |
| if (bnb_spill.isValid()) { | |
| std::vector<art::Ptr<sbn::BNBSpillInfo> > bnb_spill_info; | |
| art::fill_ptr_vector(bnb_spill_info, bnb_spill); |
| fRecTree->SetBranchAddress("rec", &prec); | ||
| fRecTree->Fill(); | ||
| fBNBInfo.clear(); | ||
| rec.hdr.pot = 0; |
There was a problem hiding this comment.
Can you move these down two lines, below the srcol stuff? Otherwise you are wiping out the POT information from the version of the StandardRecord that gets saved into the art file. We don't use that for much/anything yet, but that will be an unpleasant surprise one day.
| @@ -0,0 +1,39 @@ | |||
|
|
|||
| #include "FillExposure.h" | |||
There was a problem hiding this comment.
| #include "FillExposure.h" | |
| #include "sbncode/CAFMaker/FillExposure.h" |
There was a problem hiding this comment.
Idle comment about this: I would argue that FillExposure.h and FillExposure.cxx living in the same directory is a requirement. If that's true, isn't actually a better choice to go Bruce's Jacob's way?
Currently, we use to officially recommend Chris's approach anyway.
There was a problem hiding this comment.
I'm not sure what Bruce is doing, but I just followed the convention I see with all the other "Fill" files. I agree Chris's approach seems fine too.
There was a problem hiding this comment.
All of our code uses full paths, except for a few places that don't (lots of places in StandardRecord 😢). If I was in charge I would require full paths everywhere. Been bitten too many times by things being picked up from unexpected places.
| #include "art/Framework/Core/EDProducer.h" | ||
| #include "art/Framework/Core/FileBlock.h" | ||
| #include "art/Framework/Core/ModuleMacros.h" | ||
| #include "art/Framework/Principal/Event.h" | ||
| #include "art/Framework/Principal/Handle.h" | ||
| #include "art/Framework/Principal/SubRun.h" | ||
|
|
||
| #include "sbnanaobj/StandardRecord/StandardRecord.h" | ||
| #include "sbnanaobj/StandardRecord/SRHeader.h" | ||
| #include "sbnanaobj/StandardRecord/SRBNBInfo.h" | ||
| #include "sbnobj/Common/POTAccounting/BNBSpillInfo.h" | ||
| #include <vector> |
There was a problem hiding this comment.
The majority of these includes are unnecessary here.
| Comment("List of suffixes to add to TPC reco tag names (e.g. cryo0 cryo1)") | ||
| }; | ||
|
|
||
| Atom<string> POTDataLabel { |
There was a problem hiding this comment.
Can we call this BNBPOTInfoLabel or similar, to allow ourselves room for a NuMIPOTInfoLabel in future?
|
Those comments resolved as well. Still working as before |
PetrilloAtWork
left a comment
There was a problem hiding this comment.
There is a problem in the logic of beginSubRun() that needs to be addressed.
For the rest, C++ suggestions here and there.
|
|
||
| namespace caf | ||
| { | ||
| void FillExposure(const std::vector<art::Ptr<sbn::BNBSpillInfo> > bnb_spill_info, |
There was a problem hiding this comment.
Avoid copying data ([CF-112]):
| void FillExposure(const std::vector<art::Ptr<sbn::BNBSpillInfo> > bnb_spill_info, | |
| void FillExposure(const std::vector<art::Ptr<sbn::BNBSpillInfo> >& bnb_spill_info, |
(change the header too)
| art::Handle<std::vector<sbn::BNBSpillInfo> > bnb_spill; | ||
| sr.getByLabel(fParams.BNBPOTDataLabel(), bnb_spill); | ||
| if (bnb_spill.isValid()) { | ||
| std::vector<art::Ptr<sbn::BNBSpillInfo> > bnb_spill_info; | ||
| art::fill_ptr_vector(bnb_spill_info, bnb_spill); | ||
| FillExposure(bnb_spill_info, fBNBInfo, fSubRunPOT); | ||
| fTotalPOT += fSubRunPOT; | ||
| } |
There was a problem hiding this comment.
Why are we going through this art::Ptr exercise? Are we expecting art::Assns to appear in the future?
If not, this may be simplified into:
| art::Handle<std::vector<sbn::BNBSpillInfo> > bnb_spill; | |
| sr.getByLabel(fParams.BNBPOTDataLabel(), bnb_spill); | |
| if (bnb_spill.isValid()) { | |
| std::vector<art::Ptr<sbn::BNBSpillInfo> > bnb_spill_info; | |
| art::fill_ptr_vector(bnb_spill_info, bnb_spill); | |
| FillExposure(bnb_spill_info, fBNBInfo, fSubRunPOT); | |
| fTotalPOT += fSubRunPOT; | |
| } | |
| auto bnb_spill = sr.getHandle<std::vector<sbn::BNBSpillInfo>>(fParams.BNBPOTDataLabel()); | |
| if (bnb_spill.isValid()) { | |
| FillExposure(*bnb_spill, fBNBInfo, fSubRunPOT); | |
| fTotalPOT += fSubRunPOT; | |
| } |
and using
void FillExposure(const std::vector<sbn::BNBSpillInfo>& bnb_spill_info, std::vector<caf::SRBNBInfo>& BNBInfo, double& subRunPOT);| single_store.M875BB_spill_time_diff = info->M875BB_spill_time_diff; | ||
| single_store.M876BB_spill_time_diff = info->M876BB_spill_time_diff; | ||
| single_store.MMBTBB_spill_time_diff = info->MMBTBB_spill_time_diff; | ||
| BNBInfo.push_back(single_store); |
There was a problem hiding this comment.
The following is conceptually correct (the only purpose of single_store is to deliver that data elsewhere), and it also actually avoids a few of copies.
| BNBInfo.push_back(single_store); | |
| BNBInfo.push_back(std::move(single_store)); |
| for(const art::Ptr<sbn::BNBSpillInfo>& info: bnb_spill_info) | ||
| { | ||
| caf::SRBNBInfo single_store; | ||
| subRunPOT += info->TOR875; |
There was a problem hiding this comment.
I would rely on the knowledge of sbn::BNBSpillInfo to return the correct POT():
| subRunPOT += info->TOR875; | |
| subRunPOT += info->POT(); |
| { | ||
| for(const art::Ptr<sbn::BNBSpillInfo>& info: bnb_spill_info) | ||
| { | ||
| caf::SRBNBInfo single_store; |
There was a problem hiding this comment.
Personally I would have enclosed the translation into a caf::SRBNBInfo makeSRBNBInfo(sbn::BNBSpillInfo const&) helper function, but it's up to you (the function would have the same content of this loop body, except that return single_store; would take the place of the push_back, and the subRunPOT is out).
The advantages are that it's more evident that the subRunPOT count is separate from that translation, and that things like BNBInfo.push_back(makeSRBNBInfo(*info)); would be implicitly optimal without you having to take extra steps to move instead of copying (because of the return value of makeSRBNBInfo() being a temporary, it is never copied).
| #ifndef CAF_FILLEXPOSURE_H | ||
| #define CAF_FILLEXPOSURE_H | ||
|
|
||
| #include "canvas/Persistency/Common/Ptr.h" |
There was a problem hiding this comment.
Not needed if my suggestion of not using art::Ptr is received.
| #include "canvas/Persistency/Common/Ptr.h" |
| if (pot_handle.isValid()) { | ||
| fSubRunPOT = pot_handle->totgoodpot; | ||
| fTotalPOT += fSubRunPOT; | ||
| if(fIsRealData) |
There was a problem hiding this comment.
I suggest a duck-typing work-around (which still sucks because it may be NuMI instead of BNB etc.):
| if(fIsRealData) | |
| auto const bnb_spill = sr.getHandle<std::vector<sbn::BNBSpillInfo>>(fParams.BNBPOTDataLabel(), bnb_spill); | |
| fIsRealData = bnb_spill.isValid(); | |
| if(fIsRealData) |
(of course then you don't need to retrieve bnb_spill again in the if branch).
Unfortunately, for some mysterious reason art::SubRun::isRealData() is not exposed (thus, I have no idea if it is correctly set, either).
There was a problem hiding this comment.
There's a PR that Chris mentioned #209 that puts this in the fcl so I'm inclined to let it be set there? If not we'll probably want to coordinate that. I'm not sure where else this fIsRealData value may be used in the future.
There was a problem hiding this comment.
(commented elsewhere in this PR just a moment ago that this is not the case any more after discussion at yesterdays SBN Ana Framework meeting. Note I'm not really following this PR that much but see my PR getting mentioned so wanted to clarify)
There was a problem hiding this comment.
Never mind, apparently this is not the case anymore (wow this stuff is hard to follow sometimes...). This is likely a good way to do this in light of fIsRealData being set in the process step so I need to set it in the subRun elsewhere to make this work for POT accounting...everything else should be self-consistent even with that change
There was a problem hiding this comment.
Actually in that pull request I was suggesting that fIsRealData be removed, exactly to avoid bugs like this.
The workaround I suggest here may affect that too:
- we can decide to leave
fIsRealDatadata member be, but then the use inbeginRun()will suffer the same issue as here - we can decide to drop it, and then
fIsRealDatahere becomes a localisRealDatavariable.
There was a problem hiding this comment.
I haven't fully looked at the code (or indeed most of the comments in this PR review outside of the few items I've been following that involved my PR 😅 ), but just throwing in my 2 cents with my current thoughts:
-
fIsRealDatadoesn't seem be being used inbeginRun()at the moment though? Is that going to change with other suggestions or just a general comment? Unless you (@PetrilloAtWork ) are maybe considering how one might use such a data member consistently throughout the module (e.g. later when I have things in theproducestep) or something? -
Why not just get rid of a variable altogether (unless the answer to point 1 demands it) and just do
if ( bnb_spill.isValid() )? -
Could you then also do something similar for NuMI and do
if ( bnb_spill.isValid() || numi_spill.isValid() )(to Gianluca's point about NuMI)
There was a problem hiding this comment.
fIsRealData doesn't seem be being used in beginRun() at the moment though? Is that going to change with other suggestions or just a general comment?
General comment. Until one week ago, fIsRealData was not used even in beginSubRun() and we were happy.
But then we forget the quirks, and six months from now we start using it, and much later we realize that we are missing information for one file per job (the issue is only for the first call of begin[Sub]Run(), after which analyze() kicks in; and those methods are in our situation called on each new input file).
There was a problem hiding this comment.
Unfortunately, for some mysterious reason art::SubRun::isRealData() is not exposed
I made an art issue about this years ago, and it was closed WONTFIX. They say that in principle you could mix data and MC events into a single file. Not sure how this is consistent with including it on Event given the possibility of overlays...
| art::Handle<sumdata::POTSummary> pot_handle; | ||
| sr.getByLabel(fParams.GenLabel(), pot_handle); | ||
|
|
||
| if (pot_handle.isValid()) { |
There was a problem hiding this comment.
Fun with C++ (not particularly endorsing this change):
| art::Handle<sumdata::POTSummary> pot_handle; | |
| sr.getByLabel(fParams.GenLabel(), pot_handle); | |
| if (pot_handle.isValid()) { | |
| if (auto pot_handle = sr.getHandle<sumdata::POTSummary>(fParams.GenLabel())) { |
… and CAFmaker. Output comes in same way as it did previously
|
I think I've addressed everything. |
PetrilloAtWork
left a comment
There was a problem hiding this comment.
I confirm all the necessary and some of the optional suggestions have been implemented.
I have left one comment on one suggestion that hasn't been received, which is fine but I don't know whether it was a deliberate choice or just missed do I pedantically point that out.
Approved regardless of how that part goes.
| single_store.M876BB_spill_time_diff = info->M876BB_spill_time_diff; | ||
| single_store.MMBTBB_spill_time_diff = info->MMBTBB_spill_time_diff; | ||
| BNBInfo.push_back(single_store); | ||
| subRunPOT += info.TOR875; |
There was a problem hiding this comment.
I assume that you considered and declined the suggestion of making this
| subRunPOT += info.TOR875; | |
| subRunPOT += info.POT(); |
which is fine.
There was a problem hiding this comment.
I did make that change, then I tried to put everything in a separate function and forgot to keep that change once I figured everything else out. Have recovered that change again.
|
I guess this goes together with SBNSoftware/sbnanaobj#34 , right? |
|
Comments addressed |
|
trigger build SBNSoftware/sbnanaobj#34 |
|
✔️ CI build for LArSoft Succeeded on slf7 for c7:prof -- details available through the CI dashboard |
|
✔️ CI build for LArSoft Succeeded on slf7 for e20:prof -- details available through the CI dashboard |
|
🚨 For more details about the warning phase, check the ci_tests SBND phase logs parent CI build details are available through the CI dashboard |
|
🚨 For more details about the warning phase, check the ci_tests SBND phase logs parent CI build details are available through the CI dashboard |
|
🚨 For more details about the warning phase, check the ci_tests ICARUS phase logs parent CI build details are available through the CI dashboard |
|
🚨 For more details about the warning phase, check the ci_tests ICARUS phase logs parent CI build details are available through the CI dashboard |
Adding POT Accounting to CAFs usign output of BNBRetriever module. Fills in SRHeader with one entry per event. More ideal is an entry per subrun.