This subroutine (calculate_unbalanced_x) wraps up all the functionality required to calculate the xxf blocksizes and work out if and unbalanced decomposition should be used, and if so the size of the unbalanced blocks.
This routine takes the number of processes to be used and the rank of the calling process and returns the amount of unbalance in the calculate decomposition (in a real variable with a range between 0.0 and 1.0 where 1.0 is a 100% difference between the small and large block size and 0.0 is no difference between the small and large block sizes).
This routine can be called from routines within the gs2_layouts module (specifically init_x_transform_layouts) or from outside this module (specifically it is currently called from ingen to provide suggestions for unbalanced decompositions and process counts). However, this routine relies on init_x_transform_layouts having been called to initialise some of the values it used (those in xxf_lo%) which is why there is a check whether the initialize_x_transform variable has been set to true. If not this routine will not run and will notify the user of the problem.
AJ, November 2011: New code from DCSE project
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| integer, | intent(in) | :: | nproc | |||
| integer, | intent(in) | :: | iproc | |||
| real, | intent(out) | :: | unbalanced_amount |
subroutine calculate_unbalanced_x(nproc, iproc, unbalanced_amount)
implicit none
integer, intent (in) :: nproc, iproc
real, intent (out) :: unbalanced_amount
integer :: level_proc_num, i, j, k, m
! Ensure that the xxf_lo% data has been properly initialized as this
! routine relies on some data from that data structure. If it has not
! then abort this routine.
if(.not. initialized_x_transform) then
write(*,*) 'X Transform data structures not initialized so calculate_unbalanced_x subroutine will not operate correctly'
write(*,*) 'Aborting subroutine calculate_unbalanced_x'
return
end if
! The xxf_lo%blocksize is initialised to the value that
! is used if this unbalanced code is not used (the
! original code).
xxf_lo%blocksize = xxf_lo%ulim_world/nproc + 1
xxf_lo%small_block_balance_factor = 1
xxf_lo%large_block_balance_factor = 1
! Initialise the number of processors/cores used in the
! decomposition as the total number available.
level_proc_num = nproc
! The first factor to be considered is nspec (number
! of species).
k = xxf_lo%nspec
! Calculate the factors of nspec and the number of
! processors/cores we have left from the decomposition.
! Further details of i, m, and j are provided in the
! comments for the subroutine calculate_block_breakdown
! but briefly described i is the remainder left from k/level_proc_num,
! m is the remainder of level_proc_num/k and j is the integer
! result of k/level_proc_num.
call calculate_block_breakdown(k, i, m, j, level_proc_num)
! If j = 0 this means that level_proc_num is larger than k
! the factor we are looking to divide so we have reached the
! point where we can't divide anymore and can now build the
! decomposition.
if(j .eq. 0) then
! If we have got to this point then k is larger than the
! number of processes we have so we can try and split it
! up over the processes.
! If m = 0 then we know that k exactly divides into the
! number of processes we have at this point. This means
! we can eliminate this factor from the decomposition
! and reduce the number of processes we have available
! by that factor. Therefore we divide the number
! of processes we have by k and set the factor we need
! to decompose to 1.
! If m does not equal zero then we keep the current number
! of processes and pass the factor (k) on to the next
! level of the decomposition.
if(m .eq. 0) then
level_proc_num = level_proc_num/k
k = 1
end if
! For the xxf decomposition the next factor to be used
! depends on the layout chosen. If it is the yxels
! decompsoition then the next factor is nlambda, otherwise
! it is negrid.
! Multiple the previous factor by the next factor to be
! be used. If the previous factor divided exactly by the
! processes available then k will be 1 so we will only be
! considering this factor.
select case(layout)
case ('yxels')
k = xxf_lo%nlambda * k
case default
k = xxf_lo%negrid * k
end select
! The next code follows the pattern described above
! moving through all the other factor, i.e.:
! for yxels: negrid*nsign*ntgridtotal*naky
! and for the other layouts: nlambda*nsign*ntgridtotal*naky
! until it gets to a point where j = 0 (i.e. level_proc_num
! is greater than the factors being decomposed), or we
! have run out of factors to be decomposed.
! Once either of those points is reached then the
! code moves on to calculate the blocksizes in the
! else clauses of this code below.
! This part of the code is commented in the first occurence
! of "if(i .ne. 0) then" below.
call calculate_block_breakdown(k, i, m, j, level_proc_num)
if(j .eq. 0) then
if(m .eq. 0) then
level_proc_num = level_proc_num/k
k = 1
end if
select case(layout)
case ('yxels')
k = xxf_lo%negrid * k
case default
k = xxf_lo%nlambda * k
end select
call calculate_block_breakdown(k, i, m, j, level_proc_num)
if(j .eq. 0) then
if(m .eq. 0) then
level_proc_num = level_proc_num/k
k = 1
end if
k = xxf_lo%nsign * k
call calculate_block_breakdown(k, i, m, j, level_proc_num)
if(j .eq. 0) then
if(m .eq. 0) then
level_proc_num = level_proc_num/k
k = 1
end if
k = xxf_lo%ntgridtotal * k
call calculate_block_breakdown(k, i, m, j, level_proc_num)
if(j .eq. 0) then
if(m .eq. 0) then
level_proc_num = level_proc_num/k
k = 1
end if
k = xxf_lo%naky * k
call calculate_block_breakdown(k, i, m, j, level_proc_num)
! At this point we have run out of factors to
! decompose. Now we check whether i is not
! equal to 0. If i is equal to 0 (remember that
! i is the remainder left when k/level_proc_num)
! then this is not an unbalanced decomposition so
! we do not have to calculate the unbalanced decompostion.
! If i is not equal to 0 then k cannot be exactly divided
! by level_proc_num. This means we cannot evenly divide
! the decomposition over the processes we have available so
! we need to find a way to create a different decomposition.
if(i .ne. 0) then
! Calculate the least unbalanced split of k over level_proc_num and also
! how what factor of processes are assigned to each block size.
call calculate_unbalanced_decomposition(k, xxf_lo%small_block_balance_factor, xxf_lo%large_block_balance_factor, xxf_lo%num_small, xxf_lo%num_large, level_proc_num)
! Calculate the actual block sizes using the factors calculated above and the remaining data space to be
! decomposed. In this instance we are at the lowest level of the decomposition so there is nothing left to decompose
! so the remaining data space is 1. For other levels of the decomposition this 1 is replaced by the part
! of the data space that has not been split up yet.
call calculate_block_size(iproc, xxf_lo%num_small, xxf_lo%num_large, xxf_lo%small_block_balance_factor, xxf_lo%large_block_balance_factor, &
1, xxf_lo%blocksize, xxf_lo%small_block_size, xxf_lo%large_block_size, xxf_lo%block_multiple, xxf_lo%llim_proc, xxf_lo%ulim_proc, &
xxf_lo%ulim_alloc)
end if
else
if(i .ne. 0) then
call calculate_unbalanced_decomposition(k, xxf_lo%small_block_balance_factor, xxf_lo%large_block_balance_factor, xxf_lo%num_small, xxf_lo%num_large, level_proc_num)
! Calculate the block sizes using the factor of the decomposition that has not yet been split up, namely
! naky for this level of the decomposition.
call calculate_block_size(iproc, xxf_lo%num_small, xxf_lo%num_large, xxf_lo%small_block_balance_factor, xxf_lo%large_block_balance_factor, &
xxf_lo%naky, xxf_lo%blocksize, xxf_lo%small_block_size, xxf_lo%large_block_size, xxf_lo%block_multiple, xxf_lo%llim_proc, xxf_lo%ulim_proc, &
xxf_lo%ulim_alloc)
end if
end if
else
if(i .ne. 0) then
call calculate_unbalanced_decomposition(k, xxf_lo%small_block_balance_factor, xxf_lo%large_block_balance_factor, xxf_lo%num_small, xxf_lo%num_large, level_proc_num)
! Calculate the block sizes using the factor of the decomposition that has not yet been split up, namely
! naky,ntgridtotal for this level of the decomposition.
call calculate_block_size(iproc, xxf_lo%num_small, xxf_lo%num_large, xxf_lo%small_block_balance_factor, xxf_lo%large_block_balance_factor, &
xxf_lo%ntgridtotal*xxf_lo%naky, xxf_lo%blocksize, xxf_lo%small_block_size, xxf_lo%large_block_size, xxf_lo%block_multiple, &
xxf_lo%llim_proc, xxf_lo%ulim_proc, xxf_lo%ulim_alloc)
end if
end if
else
if(i .ne. 0) then
call calculate_unbalanced_decomposition(k, xxf_lo%small_block_balance_factor, xxf_lo%large_block_balance_factor, xxf_lo%num_small, xxf_lo%num_large, level_proc_num)
! Calculate the block sizes using the factor of the decomposition that has not yet been split up, namely
! naky,ntgridtotal,nsign for this level of the decomposition.
call calculate_block_size(iproc, xxf_lo%num_small, xxf_lo%num_large, xxf_lo%small_block_balance_factor, xxf_lo%large_block_balance_factor, &
xxf_lo%nsign*xxf_lo%ntgridtotal*xxf_lo%naky, xxf_lo%blocksize, xxf_lo%small_block_size, xxf_lo%large_block_size, xxf_lo%block_multiple, &
xxf_lo%llim_proc, xxf_lo%ulim_proc, xxf_lo%ulim_alloc)
end if
end if
else
if(i .ne. 0) then
call calculate_unbalanced_decomposition(k, xxf_lo%small_block_balance_factor, xxf_lo%large_block_balance_factor, xxf_lo%num_small, xxf_lo%num_large, level_proc_num)
select case(layout)
case('yxels')
! Calculate the block sizes using the factor of the decomposition that has not yet been split up, namely
! naky,ntgridtotal,nsign,negrid for this layout and level of the decomposition.
call calculate_block_size(iproc, xxf_lo%num_small, xxf_lo%num_large, xxf_lo%small_block_balance_factor, xxf_lo%large_block_balance_factor, &
xxf_lo%negrid*xxf_lo%nsign*xxf_lo%ntgridtotal*xxf_lo%naky, xxf_lo%blocksize, xxf_lo%small_block_size, xxf_lo%large_block_size, &
xxf_lo%block_multiple, xxf_lo%llim_proc, xxf_lo%ulim_proc, xxf_lo%ulim_alloc)
case default
! Calculate the block sizes using the factor of the decomposition that has not yet been split up, namely
! naky,ntgridtotal,nsign,nlambda for these layouts and this level of the decomposition.
call calculate_block_size(iproc, xxf_lo%num_small, xxf_lo%num_large, xxf_lo%small_block_balance_factor, xxf_lo%large_block_balance_factor, &
xxf_lo%nlambda*xxf_lo%nsign*xxf_lo%ntgridtotal*xxf_lo%naky, xxf_lo%blocksize, xxf_lo%small_block_size, xxf_lo%large_block_size, &
xxf_lo%block_multiple, xxf_lo%llim_proc, xxf_lo%ulim_proc, xxf_lo%ulim_alloc)
end select
end if
end if
else
if(i .ne. 0) then
call calculate_unbalanced_decomposition(k, xxf_lo%small_block_balance_factor, xxf_lo%large_block_balance_factor, xxf_lo%num_small, xxf_lo%num_large, level_proc_num)
! Calculate the block sizes using the factor of the decomposition that has not yet been split up, namely
! naky,ntgridtotal,nsign,nlambda,negrid for this level of the decomposition.
call calculate_block_size(iproc, xxf_lo%num_small, xxf_lo%num_large, xxf_lo%small_block_balance_factor, xxf_lo%large_block_balance_factor, &
xxf_lo%negrid*xxf_lo%nlambda*xxf_lo%nsign*xxf_lo%ntgridtotal*xxf_lo%naky, xxf_lo%blocksize, xxf_lo%small_block_size, &
xxf_lo%large_block_size, xxf_lo%block_multiple, xxf_lo%llim_proc, xxf_lo%ulim_proc, xxf_lo%ulim_alloc)
end if
end if
! If the decomposition has calculated that the large block and small blocks are equal then
! we don't have an unbalanced decomposition so set the unbalanced_amount to 0.
! large_block_balance_factor and small_block_balance_factor are initialised to 1 at
! the start of this decomposition code so if they haven't been changed we already
! have a balanced decomposition.
if(xxf_lo%large_block_balance_factor .eq. 1 .and. xxf_lo%small_block_balance_factor .eq. 1) then
unbalanced_amount = 0
else
! If there is an unbalanced decomposition work out the percentage of
! difference between the two blocks. This is used to ensure that
! we don't create decompositions that have significant differences
! between the two block sizes which would impact the amount of
! computation the different groups of processes have to perform.
unbalanced_amount = real(xxf_lo%large_block_size)/real(xxf_lo%small_block_size)
unbalanced_amount = unbalanced_amount - 1
end if
! If we calculate that there is not an unbalanced decomposition or that the
! amount of unbalance is larger than a integer the user sets in the input file
! called: max_unbalanced_xxf ; then do not use the unbalanced decomposition.
if (unbalanced_amount .gt. max_unbalanced_xxf .or. unbalanced_amount .eq. 0) then
unbalanced_xxf = .false.
end if
end subroutine calculate_unbalanced_x