Class to handle walker controls with simple global sum. More...

Inheritance diagram for WalkerControlMPI:

Collaboration diagram for WalkerControlMPI:

Public Member Functions
	WalkerControlMPI (Communicate *comm)
	default constructor More...

int	branch (int iter, MCWalkerConfiguration &W, FullPrecRealType trigger) override
	legacy: perform branch and swap walkers as required More...

void	swapWalkersSimple (MCWalkerConfiguration &W)
	legacy: swap implementation More...

Public Member Functions inherited from WalkerControlBase
	WalkerControlBase (Communicate *c)
	default constructor More...

virtual	~WalkerControlBase ()
	empty destructor to clean up the derived classes More...

void	start ()
	start a block More...

void	setWalkerID (MCWalkerConfiguration &walkers)
	start controller and initialize the IDs of walkers More...

void	measureProperties (int iter)
	take averages and writes to a file More...

void	setTrialEnergy (FullPrecRealType et)
	set the trial energy More...

FullPrecRealType	getValue (int i)
	return a value accumulated during a block More...

FullPrecRealType	getCurrentValue (int i)
	return a current value More...

int	doNotBranch (int iter, MCWalkerConfiguration &W)
	legacy: return global population update properties without branching More...

int	sortWalkers (MCWalkerConfiguration &W)
	legacy: sort Walkers between good and bad and prepare branching More...

int	applyNmaxNmin (int current_population)
	legacy: apply per rank limit Nmax and Nmin More...

int	copyWalkers (MCWalkerConfiguration &W)
	legacy: copy good walkers to W More...

virtual void	reset ()
	reset to accumulate data More...

bool	put (xmlNodePtr cur)

void	setMinMax (int nw_in, int nmax_in)

int	get_n_max () const

int	get_n_min () const

FullPrecRealType	get_target_sigma () const

MCDataType< FullPrecRealType > &	get_ensemble_property ()

void	set_ensemble_property (MCDataType< FullPrecRealType > &ensemble_property)

IndexType	get_num_contexts () const

IndexType	get_method () const

void	set_method (IndexType method)

Public Member Functions inherited from MPIObjectBase
	MPIObjectBase (Communicate *c)
	constructor with communicator More...

int	rank () const
	return the rank of the communicator More...

int	getGroupID () const
	return the group id of the communicator More...

Communicate *	getCommunicator () const
	return myComm More...

Communicate &	getCommRef () const
	return a TEMPORARY reference to Communicate More...

mpi_comm_type	getMPI () const
	return MPI communicator if one wants to use MPI directly More...

bool	is_manager () const
	return true if the rank == 0 More...

const std::string &	getName () const
	return the name More...

void	setName (const std::string &aname)

Static Public Member Functions
static void	determineNewWalkerPopulation (int cur_pop, int num_contexts, int my_context, std::vector< int > &num_per_rank, std::vector< int > &fair_offset, std::vector< int > &minus, std::vector< int > &plus)
	creates the distribution plan More...

Static Public Member Functions inherited from WalkerControlBase
static std::vector< IndexType >	syncFutureWalkersPerRank (Communicate *comm, IndexType n_walkers)

Public Attributes
int	Cur_pop

int	Cur_max

int	Cur_min

TimerList_t	myTimers

IndexType	NumWalkersSent

friend	WalkerControlMPITest

Additional Inherited Members
Public Types inherited from WalkerControlBase
enum	{ ENERGY_INDEX = 0, ENERGY_SQ_INDEX, WALKERSIZE_INDEX, WEIGHT_INDEX, EREF_INDEX, R2ACCEPTED_INDEX, R2PROPOSED_INDEX, FNSIZE_INDEX, RNONESIZE_INDEX, RNSIZE_INDEX, B_ENERGY_INDEX, B_WGT_INDEX, SENTWALKERS_INDEX, LE_MAX }
	An enum to access curData and accumData for reduction. More...

using	Walker_t = MCWalkerConfiguration::Walker_t
	typedef of Walker_t More...

using	FullPrecRealType = QMCTraits::FullPrecRealType
	typedef of FullPrecRealType More...

using	IndexType = QMCTraits::IndexType
	typedef of IndexType More...

Public Types inherited from MPIObjectBase
using	mpi_comm_type = Communicate::mpi_comm_type

Protected Attributes inherited from WalkerControlBase
IndexType	method_
	id for the method More...

IndexType	n_min_
	minimum number of walkers More...

IndexType	n_max_
	maximum number of walkers More...

IndexType	MaxCopy
	maximum copy per walker More...

IndexType	NumWalkers
	current number of walkers per processor More...

FullPrecRealType	trialEnergy
	trial energy energy More...

FullPrecRealType	target_sigma_
	target sigma to limit fluctuations of the trial energy More...

std::vector< int >	NumPerRank
	number of particle per rank More...

std::vector< int >	OffSet
	offset of the particle index More...

std::vector< int >	FairOffSet
	offset of the particle index for a fair distribution More...

std::filesystem::path	dmcFname
	filename for dmc.dat More...

std::unique_ptr< std::ofstream >	dmcStream
	file to save energy histogram More...

IndexType	NumWalkersCreated
	Number of walkers created by this rank. More...

IndexType	MyContext
	context id More...

IndexType	num_contexts_
	number of contexts More...

IndexType	SwapMode
	0 is default More...

std::vector< FullPrecRealType >	accumData
	any accumulated data over a block More...

std::vector< FullPrecRealType >	curData
	any temporary data includes many ridiculous conversions of integral types to and from fp More...

std::vector< std::unique_ptr< Walker_t > >	good_w
	temporary storage for good and bad walkers More...

std::vector< std::unique_ptr< Walker_t > >	bad_w

std::vector< int >	ncopy_w
	temporary storage for copy counters More...

bool	use_nonblocking
	Use non-blocking isend/irecv. More...

MCDataType< FullPrecRealType >	ensemble_property_
	ensemble properties More...

Protected Attributes inherited from MPIObjectBase
Communicate *	myComm
	pointer to Communicate More...

std::string	ClassName
	class Name More...

std::string	myName
	name of the object More...

Detailed Description

Class to handle walker controls with simple global sum.

Base class to handle serial mode with branching only

Definition at line 30 of file WalkerControlMPI.h.

Constructor & Destructor Documentation

◆ WalkerControlMPI()

WalkerControlMPI ( Communicate * c )

default constructor

Parameters

[in] comm can not be null it is not checked.

set SwapMode? SwapMode is set to 1 but what does that mean? This object persists inside the SFNB which also persists The zeroing here will not happen in later QMC sections... This seems problematic in that NumWalkersSent will start at a value of no concern to the current section.

In the new drivers SFNB should throw an except if there is attempted reuse of WalkerController

Definition at line 61 of file WalkerControlMPI.cpp.

References WalkerControlMPI::Cur_max, WalkerControlMPI::Cur_min, WalkerControlMPI::NumWalkersSent, and WalkerControlBase::SwapMode.

     : WalkerControlBase(c), myTimers(getGlobalTimerManager(), DMCMPITimerNames, timer_level_medium)
 {
   NumWalkersSent = 0;
   SwapMode       = 1;
   Cur_min        = 0;
   Cur_max        = 0;
 }

Member Function Documentation

◆ branch()

int branch	(	int	iter,
		MCWalkerConfiguration &	W,
		FullPrecRealType	trigger
	)

overridevirtual

legacy: perform branch and swap walkers as required

Perform branch and swap walkers as required.

It takes 5 steps:

sortWalkers marks good and bad walkers.
allreduce collects the number of good walkers + copies on every rank.
applyNmaxNmin avoids too large or too small global population.
swapWalkersSimple makes a decision of load balancing and send/recv walkers. Receiving side recycles bad walkers' memory first.
copyWalkers generates copies of good walkers. In order to minimize the memory footprint fluctuation the walker copying is placed as the last step. In order to reduce the time for allocating walker memory, this algorithm does not destroy the bad walkers in step 1. All the bad walkers are recycled as much as possible in step 3/4.

Reimplemented from WalkerControlBase.

Definition at line 85 of file WalkerControlMPI.cpp.

 {
   ScopedTimer local_timer(myTimers[DMC_MPI_branch]);
   {
     ScopedTimer local_timer(myTimers[DMC_MPI_prebalance]);
     std::fill(curData.begin(), curData.end(), 0.0);
     sortWalkers(W);
     //use NumWalkersSent from the previous exchange
     curData[SENTWALKERS_INDEX] = NumWalkersSent;
     //update the number of walkers for this rank
     //Causes implicit conversion to FullPrecRealType
     curData[LE_MAX + MyContext] = NumWalkers;
     //{ ScopedTimer local_timer(myTimers[DMC_MPI_imbalance]);
     //}
     {
       ScopedTimer local_timer(myTimers[DMC_MPI_allreduce]);
       myComm->allreduce(curData);
     }
     measureProperties(iter);
     W.EnsembleProperty = ensemble_property_;
     for (int i = 0, j = LE_MAX; i < num_contexts_; i++, j++)
       NumPerRank[i] = static_cast<int>(curData[j]);
     int current_population = std::accumulate(NumPerRank.begin(), NumPerRank.end(), 0);
 
     Cur_pop = applyNmaxNmin(current_population);
   }
   {
     ScopedTimer local_timer(myTimers[DMC_MPI_loadbalance]);
     swapWalkersSimple(W);
   }
   {
     ScopedTimer local_timer(myTimers[DMC_MPI_copyWalkers]);
     copyWalkers(W);
   }
   //set Weight and Multiplicity to default values
   for (auto& walker : W)
   {
     walker->Weight       = 1.0;
     walker->Multiplicity = 1.0;
   }
   //update the walkers offsets
   W.setWalkerOffsets(FairOffSet);
 
   return Cur_pop;
 }

◆ determineNewWalkerPopulation()

void determineNewWalkerPopulation	(	int	cur_pop,
		int	num_contexts,
		int	my_context,
		std::vector< int > &	num_per_rank,
		std::vector< int > &	fair_offset,
		std::vector< int > &	minus,
		std::vector< int > &	plus
	)

static

creates the distribution plan

populates the minus and plus vectors they contain 1 copy of a partition index for each adjustment in population to the context.

Todo:: fix this argument salad

Parameters

[in]	cur_pop_pop	population taking multiplicity into account
[in]	num_contexts	number of MPI processes
[in]	my_context	i.e this processes MPI rank
	[in/out]	num_per_rank as if all walkers were copied out to multiplicity
[out]	fair_offset	running population count at each partition boundary
[out]	minus	list of partition indexes one occurrence for each walker removed
[out]	plus	list of partition indexes one occurrence for each walker added

Definition at line 132 of file WalkerControlMPI.cpp.

References qmcplusplus::app_error(), and FairDivideLow().

Referenced by WalkerControlMPI::swapWalkersSimple(), and qmcplusplus::TEST_CASE().

 {
   FairDivideLow(cur_pop, num_contexts, fair_offset);
   for (int ip = 0; ip < num_contexts; ip++)
   {
     // (FairOffSet[ip + 1] - FairOffSet[ip]) gives the partiion ip walker pop
     int dn = num_per_rank[ip] - (fair_offset[ip + 1] - fair_offset[ip]);
     num_per_rank[ip] -= dn;
     if (dn > 0)
     {
       plus.insert(plus.end(), dn, ip);
     }
     else if (dn < 0)
     {
       minus.insert(minus.end(), -dn, ip);
     }
   }
 #ifndef NDEBUG
   if (plus.size() != minus.size())
   {
     app_error() << "Walker send/recv pattern doesn't match. "
                 << "The send size " << plus.size() << " is not equal to the recv size " << minus.size() << " ."
                 << std::endl;
     throw std::runtime_error("Trying to swap in WalkerControlMPI::swapWalkersSimple with mismatched queues");
   }
 #endif
 }

◆ swapWalkersSimple()

void swapWalkersSimple ( MCWalkerConfiguration & W )

legacy: swap implementation

swap Walkers with Recv/Send or Irecv/Isend

The algorithm ensures that the load per rank can differ only by one walker. Each MPI rank can only send or receive or be silent. The communication is one-dimensional and very local. If multiple copies of a walker need to be sent to the target rank, only send one. The number of copies is communicated ahead via blocking send/recv. Then the walkers are transferred via blocking or non-blocking send/recv. The blocking send/recv may become serialized and worsen load imbalance. Non blocking send/recv algorithm avoids serialization completely.

Definition at line 177 of file WalkerControlMPI.cpp.

Referenced by WalkerControlMPI::branch().

 {
   std::vector<int> minus, plus;
   //legacy code does not modify NumPerRank in this call so we copy NumPerRank
   std::vector<int> num_per_rank(NumPerRank);
   determineNewWalkerPopulation(Cur_pop, num_contexts_, MyContext, num_per_rank, FairOffSet, minus, plus);
 
   if (good_w.empty() && bad_w.empty())
   {
     app_error() << "It should never happen that no walkers, "
                 << "neither good nor bad, exist on a rank. "
                 << "Please report to developers. " << std::endl;
     APP_ABORT("WalkerControlMPI::swapWalkersSimple no existing walker");
   }
 
   Walker_t& wRef(*(good_w.empty() ? bad_w[0] : good_w[0]));
   std::vector<std::unique_ptr<Walker_t>> newW;
   std::vector<int> ncopy_newW;
 #ifdef MCWALKERSET_MPI_DEBUG
   std::array<char, 128> fname;
   if (std::snprintf(fname.data(), fname.size() "test.%d", MyContext) < 0)
     throw std::runtime_error("Error generating filename");
   std::ofstream fout(fname.data(), std::ios::app);
   //fout << NumSwaps << " " << Cur_pop << " ";
   //for(int ic=0; ic<NumContexts; ic++) fout << NumPerRank[ic] << " ";
   //fout << " | ";
   //for(int ic=0; ic<NumContexts; ic++) fout << FairOffSet[ic+1]-FairOffSet[ic] << " ";
   //fout << " | ";
   for (int ic = 0; ic < plus.size(); ic++)
   {
     fout << plus[ic] << " ";
   }
   fout << " | ";
   for (int ic = 0; ic < minus.size(); ic++)
   {
     fout << minus[ic] << " ";
   }
   fout << std::endl;
 #endif
   int nswap = plus.size();
   // sort good walkers by the number of copies
   assert(good_w.size() == ncopy_w.size());
   std::vector<std::pair<int, int>> ncopy_pairs;
   for (int iw = 0; iw < ncopy_w.size(); iw++)
     ncopy_pairs.push_back(std::make_pair(ncopy_w[iw], iw));
   std::sort(ncopy_pairs.begin(), ncopy_pairs.end());
 
   int nsend = 0;
   struct job
   {
     const int walkerID;
     const int target;
     job(int wid, int target_in) : walkerID(wid), target(target_in){};
   };
   std::vector<job> job_list;
   for (int ic = 0; ic < nswap; ic++)
   {
     if (plus[ic] == MyContext)
     {
       // always send the last good walker
       auto& awalker = good_w[ncopy_pairs.back().second];
       // count the possible copies in one send
       int nsentcopy = 0;
 
       for (int id = ic + 1; id < nswap; id++)
         if (plus[ic] == plus[id] && minus[ic] == minus[id] && ncopy_pairs.back().first > 0)
         { // increment copy counter
           ncopy_pairs.back().first--;
           nsentcopy++;
         }
         else
         { // not enough copies to send or not the same send/recv pair
           break;
         }
 
       // send the number of copies to the target
       myComm->comm.send_value(nsentcopy, minus[ic]);
       job_list.push_back(job(ncopy_pairs.back().second, minus[ic]));
 #ifdef MCWALKERSET_MPI_DEBUG
       fout << "rank " << plus[ic] << " sends a walker with " << nsentcopy << " copies to rank " << minus[ic]
            << std::endl;
 #endif
 
       // update counter and cursor
       ++nsend;
       ic += nsentcopy;
 
       // update copy counter
       if (ncopy_pairs.back().first > 0)
       {
         ncopy_pairs.back().first--;
         std::sort(ncopy_pairs.begin(), ncopy_pairs.end());
       }
       else
       {
         ncopy_pairs.pop_back();
         bad_w.push_back(std::make_unique<Walker_t>(*awalker));
       }
     }
     if (minus[ic] == MyContext)
     {
       std::unique_ptr<Walker_t> awalker;
       if (!bad_w.empty())
       {
         awalker = std::move(bad_w.back());
         bad_w.pop_back();
       }
 
       int nsentcopy = 0;
       // recv the number of copies from the target
       myComm->comm.receive_n(&nsentcopy, 1, plus[ic]);
       job_list.push_back(job(newW.size(), plus[ic]));
       if (plus[ic] != plus[ic + nsentcopy] || minus[ic] != minus[ic + nsentcopy])
         APP_ABORT("WalkerControlMPI::swapWalkersSimple send/recv pair checking failed!");
 #ifdef MCWALKERSET_MPI_DEBUG
       fout << "rank " << minus[ic] << " recvs a walker with " << nsentcopy << " copies from rank " << plus[ic]
            << std::endl;
 #endif
 
       // save the new walker
       if (awalker)
       {
         newW.push_back(std::make_unique<Walker_t>(*awalker));
       }
       else
       {
         newW.push_back(nullptr);
       }
       ncopy_newW.push_back(nsentcopy);
       // update cursor
       ic += nsentcopy;
     }
   }
 
   if (nsend > 0)
   {
     std::vector<mpi3::request> requests;
     // mark all walkers not in send
     for (auto jobit = job_list.begin(); jobit != job_list.end(); jobit++)
       good_w[jobit->walkerID]->SendInProgress = false;
     for (auto jobit = job_list.begin(); jobit != job_list.end(); jobit++)
     {
       // pack data and send
       auto& awalker   = good_w[jobit->walkerID];
       size_t byteSize = awalker->byteSize();
       if (!awalker->SendInProgress)
       {
         awalker->updateBuffer();
         awalker->SendInProgress = true;
       }
       if (use_nonblocking)
         requests.push_back(myComm->comm.isend_n(awalker->DataSet.data(), byteSize, jobit->target));
       else
       {
         ScopedTimer local_timer(myTimers[DMC_MPI_send]);
         myComm->comm.send_n(awalker->DataSet.data(), byteSize, jobit->target);
       }
     }
     if (use_nonblocking)
     {
       // wait all the isend
       for (int im = 0; im < requests.size(); im++)
       {
         ScopedTimer local_timer(myTimers[DMC_MPI_send]);
         requests[im].wait();
       }
       requests.clear();
     }
   }
   else
   {
     std::vector<mpi3::request> requests;
     for (auto jobit = job_list.begin(); jobit != job_list.end(); jobit++)
     {
       // recv and unpack data
       auto& awalker = newW[jobit->walkerID];
       if (!awalker)
         awalker = std::make_unique<Walker_t>(wRef);
       size_t byteSize = awalker->byteSize();
       if (use_nonblocking)
         requests.push_back(myComm->comm.ireceive_n(awalker->DataSet.data(), byteSize, jobit->target));
       else
       {
         ScopedTimer local_timer(myTimers[DMC_MPI_recv]);
         myComm->comm.receive_n(awalker->DataSet.data(), byteSize, jobit->target);
         awalker->copyFromBuffer();
       }
     }
     if (use_nonblocking)
     {
       std::vector<bool> not_completed(requests.size(), true);
       bool completed = false;
       while (!completed)
       {
         completed = true;
         for (int im = 0; im < requests.size(); im++)
           if (not_completed[im])
           {
             if (requests[im].completed())
             {
               newW[job_list[im].walkerID]->copyFromBuffer();
               not_completed[im] = false;
             }
             else
               completed = false;
           }
       }
       requests.clear();
     }
   }
   //save the number of walkers sent
   NumWalkersSent = nsend;
   // rebuild good_w and ncopy_w
   std::vector<std::unique_ptr<Walker_t>> good_w_temp(std::move(good_w));
   good_w.resize(ncopy_pairs.size());
   ncopy_w.resize(ncopy_pairs.size());
   for (int iw = 0; iw < ncopy_pairs.size(); iw++)
   {
     good_w[iw]  = std::move(good_w_temp[ncopy_pairs[iw].second]);
     ncopy_w[iw] = ncopy_pairs[iw].first;
   }
   //add walkers from other rank
   if (newW.size())
   {
     good_w.insert(good_w.end(), std::make_move_iterator(newW.begin()), std::make_move_iterator(newW.end()));
     ncopy_w.insert(ncopy_w.end(), ncopy_newW.begin(), ncopy_newW.end());
   }
 
   assert(std::accumulate(ncopy_w.begin(), ncopy_w.end(), ncopy_w.size()) == num_per_rank[MyContext]);
 }

Member Data Documentation

◆ Cur_max

int Cur_max

Definition at line 33 of file WalkerControlMPI.h.

Referenced by WalkerControlMPI::WalkerControlMPI().

◆ Cur_min

int Cur_min

Definition at line 34 of file WalkerControlMPI.h.

Referenced by WalkerControlMPI::WalkerControlMPI().

◆ Cur_pop

int Cur_pop

Definition at line 32 of file WalkerControlMPI.h.

Referenced by WalkerControlMPI::branch(), and WalkerControlMPI::swapWalkersSimple().

◆ myTimers

TimerList_t myTimers

Definition at line 35 of file WalkerControlMPI.h.

Referenced by WalkerControlMPI::branch(), and WalkerControlMPI::swapWalkersSimple().

◆ NumWalkersSent

IndexType NumWalkersSent

Definition at line 39 of file WalkerControlMPI.h.

Referenced by WalkerControlMPI::branch(), WalkerControlMPI::swapWalkersSimple(), and WalkerControlMPI::WalkerControlMPI().

◆ WalkerControlMPITest

friend WalkerControlMPITest

Definition at line 77 of file WalkerControlMPI.h.

The documentation for this struct was generated from the following files:

/home/pk7/projects/qmc/for_cron_doxygen/qmcpack/src/QMCDrivers/DMC/WalkerControlMPI.h
/home/pk7/projects/qmc/for_cron_doxygen/qmcpack/src/QMCDrivers/DMC/WalkerControlMPI.cpp

Public Member Functions

Static Public Member Functions

Public Attributes

Additional Inherited Members