\section{Error Checking, Flop Counting, Message Passing, and the BLAS}
\label{sec:misc}

In this section we discuss a number of useful, miscellaneous
features available in \BS.
These features include error checking, flop counting,
matrix size and ordering information, message-passing
options, and BLAS options for optimizing processor
performance.

\subsection{Error Checking within \BS}\sindex{error checking}
\BS\ uses an error-checking system based on the two macros
{\tt SETERR()}\findex{SETERR()} and {\tt CHKERR()}\findex{CHKERR()},
which are defined in the file {\tt include/BSdepend.h}.
\findex{include/BSdepend.h}
When \BS\ is compiled with the flag {\tt DEBUG\_ALL}\findex{DEBUG_ALL} 
defined in the file {\tt include/BSdepend.h}, then when an internal error 
occurs (such as a failed {\it malloc()} call),
\BS\ returns to the user, and the error code can be checked 
and a traceback returned.
The traceback includes the routine names and line numbers where
the error occurs; this information can be useful if you
suspect an error in \BS.  
We highly recommend the use of {\tt DEBUG\_ALL}
until you are extremely sure of your code;
even then, it is inexpensive to use {\tt DEBUG\_ALL} with \BS.
The default is that {\tt DEBUG\_ALL} is set to \true\ in
{\tt include/BSdepend.h}.

\subsection{Flop Counting}
\sindex{flop counting}
When the \BS\ is compiled with {\tt MLOG} defined,
flops (floating-point operations) are being logged.
(See \S\ref{sec:install} for more information
about compile-time options.)
You can access the current number of flops 
executed by \BS\ on a particular processor 
with the call to the function\findex{BSlocal_flops()}
\begin{verbatim}
  double BSlocal_flops();
\end{verbatim}
which returns a double.
The total number of flops performed on all
processors can be obtained
by calling the function\findex{BSglobal_flops()}
\begin{verbatim}
  double BSglobal_flops(BSprocinfo *procinfo);
\end{verbatim}
which also returns a double.

The following code segment demonstrates how you
can estimate flop rates for a section of code.

{\small
\begin{verbatim}
  init_flops = BSglobal_flops(procinfo);
  init_time = MPI_Wtime();
  num_iter = BSpar_solve(pA,f_pA,Acomm,rhs,x,residual,procinfo); CHKERR(0);
  total_time = MPI_Wtime() - init_time;
  total_flops = BSglobal_flops(procinfo) - init_flops;
  tmflop = total_flops/(total_time);
\end{verbatim}
}
In this case, {\tt tmflop} would be an estimate of the
combined flop rate for all the processors involved in the
call to {\tt BSpar\_solve()}.

\BS\ also does some rudimentary logging of timing and
flop counts of routines within the package when compiled 
with {\tt MLOG} defined.
By calling the function\findex{BSprint_log()}
\begin{verbatim}
int BSprint_log(BSprocinfo *procinfo);
\end{verbatim}
before the final call to {\tt BSfree\_ctx()}, \findex{BSfree_ctx()}
this logging information can be printed out.

\subsection{Matrix Statistics}
\BS\ includes a number of functions that can be called to 
obtain information about a matrix of {\tt typedef BSpar\_mat}.
\findex{typedef BSpar_mat}
For example, the following two functions may be 
called to determine the matrix 
nonzeros locally assigned to a processor and the total
(global) number of nonzeros in the matrix.
\findex{BSlocal_nnz()}
\findex{BSglobal_nnz()}
\begin{verbatim}
  int BSlocal_nnz(BSpar_mat *);
  int BSglobal_nnz(BSpar_mat *,BSprocinfo *);
\end{verbatim}
The following two functions may be called to obtain
the number of i-nodes locally assigned to a processor 
and the global number of i-nodes.
\findex{BSlocal_num_inodes()}
\findex{BSglobal_num_inodes()}
\begin{verbatim}
  int BSlocal_num_inodes(BSpar_mat *);
  int BSglobal_num_inodes(BSpar_mat *);
\end{verbatim}
The following two functions return the
the number of cliques locally assigned to a processor 
and the global number of cliques.
\findex{BSlocal_num_cliques()}
\findex{BSglobal_num_cliques()}
\begin{verbatim}
  int BSlocal_num_cliques(BSpar_mat *);
  int BSglobal_num_cliques(BSpar_mat *);
\end{verbatim}
The following function returns the number of colors used to
color the graph associated with matrix clique structure.
\begin{verbatim}
  int BSnum_colors(BSpar_mat *);
\end{verbatim}
\findex{BSnum_colors()}

\subsection{Blocking and Nonblocking Messages}
\sindex{blocking messages}
\sindex{nonblocking messages}
\BS\ can be compiled so that it does not use blocking (synchronous)
sends and receives.
(In MPI the blocking send is the function {\tt MPI\_Send()};
the nonblocking (asynchronous) send is the function {\tt MPI\_Isend()}.)
The MPI standard does not require that two processors simultaneously
issuing blocking sends buffer messages and return; therefore deadlock 
can result.
This problem can be avoided by using nonblocking sends, at the
expense of more memory (because the message buffers cannot
be freed until it is certain that the messages have been received).
The IBM SP requires the use of such nonblocking sends.

\BS\ compiles with the blocking or nonblocking send versions based on
whether {\tt NO\_BLOCKING\_SEND}\findex{NO_BLOCKING_SEND} is defined
in the include file {\tt include/BSdepend.h}\findex{include/BSdepend.h}.
If you are concerned about performance or memory usage, you should
experiment with the difference between blocking and nonblocking
sends on your particular architecture.

\subsection{MPI Communicators and Message Number Conflicts}
\sindex{message numbers}
\sindex{MPI communicators}
\BS\ currently does not make a copy of the user's MPI communicator;
instead it uses as a default the current {\tt MPI\_COMM\_WORLD}.
If there is any possibility of a message-passing conflict (for example,
unsatisfied wild card receives from the user's code), the 
communicator can be copied by using {\tt MPI\_Comm\_dup()} and
can be handed to \BS\ by using 
{\tt BSctx\_set\_ps()}.\findex{BSctx_set_ps()}

\BS\ uses message numbers beginning at 10,000.  It uses a significant
but variable number of messages after that.  Currently the number of
messages used is 20+(10000*number\_of\_processors).  
The number of messages needed by \BS\ depends on the problem being
solved, but if the number of messages allocated to it is too small, 
it will detect an error and return accordingly (if {\tt DEBUG\_ALL} is on).
The current setting of 10,000 is generous.
The message numbers
as well as the number of messages can be changed simply by altering 
{\tt include/BSprivate.h}.\findex{include/BSprivate.h}
The avoidance of conflicts in the use of message numbers can
be ensured by copying the communicator as described above.

\subsection{Inline Macros for the BLAS}
\sindex{BLAS}
\sindex{inline macros}
The performance of the vendor-supplied BLAS for small systems
can be disappointing because of the subroutine call overhead,
error checking, and special case handling (overloading) in 
the implementation of these BLAS.
In addition, \BS\ has to gather and scatter noncontiguous
vector data (although not matrix data) to use the dense BLAS.
A feature in \BS\ is special handling of the Level-2 BLAS
for small i-node sizes; \BS\ uses macros to put code inline
for the special BLAS cases required.
The performance improvement can be substantial.
For example, in Table~\ref{tab:blas} we show the processor
performance improvement obtained on the Argonne IBM SP
system (RS/6000-370 processor).

\begin{table} [htb]
\caption{\label{tab:blas}Comparison of the single-processor
performance of \BS\ conjugate gradient and GMRES
iterations, with vendor-supplied, Level-2 BLAS (GEMV and TRMV)
and the new inline macros.
The symmetric systems are solved with the conjugate gradient 
method; the nonsymmetric systems are solved with GMRES.}
\begin{center}
\begin{tabular}{|c|c|c|c|c|} \hline
 Size of     &  Symmetric &  Vendor BLAS & BS95 Macros & Improvement \\
i-node       &  (Y or N)  &  (Mflops)    & (Mflops)    & Ratio \\\hline\hline
       2     &    Y       &  ~3.5        &  ~8.3       & 2.35 \\\hline
       3     &    Y       &  ~6.1        &  12.5       & 2.06 \\\hline
       5     &    Y       &  10.3        &  18.3       & 1.77 \\\hline
       6     &    Y       &  14.7        &  22.3       & 1.52 \\\hline
       7     &    Y       &  17.6        &  23.2       & 1.32 \\\hline
\hline  
       2     &    N       &  ~5.2        &  ~7.1       &  1.37 \\\hline
       3     &    N       &  ~8.0        &  10.6       &  1.33 \\\hline
       5     &    N       &  16.1        &  19.1       &  1.19 \\\hline
\end{tabular}
\end{center}
\end{table}

These macros are defined in the {\tt include/BSmy\_blas.h}.
\findex{include/BSmy_blas.h}
The default is that libraries are compiled with the inline
versions for small i-node sizes (less than 10); otherwise,
the vendor-supplied BLAS are used.
These macros can be turned off by editing this file and
not defining {\tt MY\_BLAS\_DTRMV\_ON} and 
{\tt MY\_BLAS\_DGEMV\_ON}.
Also, the maximum level of unrolling can be changed to 
tune performance for a particular architecture.

\newpage
