gmt-fork/doc/GMT_Appendix_B.tex

%------------------------------------------
%       $Id$
%
%       The GMT Documentation Project
%       Copyright (c) 2000-2012.
%       P. Wessel, W. H. F. Smith, R. Scharroo, and J. Luis
%------------------------------------------
%

\chapter{\gmt\ file formats}
\label{app:B}
\thispagestyle{headings}

\section{Table data} 
\index{Table!format}

These files have \emph{N} records which have \emph{M} fields each.
All programs that handle tables can read multicolumn files.  \GMT\ can
read both ASCII, native binary, and netCDF table data.

\subsection{ASCII tables}
\index{Table!ASCII}
\index{ASCII tables}
\index{Table!multisegment}

\subsubsection{Optional file header records}
The first data record may be preceded by one or more header records.
Any records that begins with '\#' is considered a header or comment
line and are always processed correctly.  If your data file has leading
header records that do \emph{not} start with '\#' then you must
make sure to use the \Opt{h} option and set
the parameter \textbf{IO\_N\_HEADER\_RECS} in the \filename{gmt.conf} file
(\GMT\ default is one header record if \Opt{h} is given; you may also
use \Opt{h}\emph{nrecs} directly).  Fields
within a record must be separated by spaces, tabs, or commas.
Each field can be an integer or floating-point number or a geographic
coordinate string using the [+$|$-]dd[:mm[:ss]][W$|$S$|$N$|$E$|$w$|$s$|$n$|$e] format.
Thus, 12:30:44.5W, 17.5S, 1:00:05, and 200:45E are all valid input strings.
On output, fields will be separated by the character given by the parameter \textbf{IO\_COL\_SEPARATOR},
which by default is a TAB.

\subsubsection{Optional segment header records}
When dealing with time- or (\emph{x,y})-series it is usually
convenient to have each profile in separate files.
However, this may sometimes prove impractical due to large numbers
of profiles.  An example is files of digitized lineations where
the number of individual features may range into the thousands.
One file per feature would in this case be unreasonable and
furthermore clog up the directory. \GMT\ provides a mechanism for
keeping more than one profile in a file.  Such files are called
\emph{multiple segment files} and are identical to the ones
just outlined except that they have segment headers interspersed with
data records that signal the start of a new segment.
The segment headers may be of any format, but all must have the same
character in the first column.  The unique character is by default
'$>$', but you can override that by modifying the \textbf{IO\_SEGMENT\_MARKER}
default setting.  Programs can examine the segment headers to see if they
contain \Opt{D} for a distance value, \Opt{W} and \Opt{G} options for specifying pen and
fill attributes for individual segments, \Opt{Z} to change
color via a CPT file, \Opt{L} for label specifications, or \Opt{T} for
general-purpose text descriptions.  These settings
(and occasionally others) will override the corresponding command line options.
\GMT\ also provides for two special values for \textbf{IO\_SEGMENT\_MARKER} that can
make interoperability with other software packages easier.  Choose the marker \textbf{B}
to have blank lines recognized as segment breaks, or use \textbf{N} to have data records
whose fields equal NaN mean segment breaks (e.g., as used by GNU Octave or Matlab).
When these markers are used then no other segment header will be considered.  Note that
\textbf{IO\_SEGMENT\_MARKER} can be set differently for input and output.
\subsection{Binary tables}
\index{Table!binary}
\index{Binary tables}
\index{Input!binary \Opt{bi}}
\index{Output!binary \Opt{bo}}
\index{\Opt{bi} (select binary input)}
\index{\Opt{bo} (select binary output)}

\GMT\ programs also support native binary tables to speed up input-output
for i/o-intensive tasks like gridding and preprocessing.  Files
may have a header section and the \Opt{h}\emph{n} option can be used to skip the
first \emph{n} bytes.  The data record can be in any format, mixing different data types
and even containing byte-swapped items.  For input, specify \Opt{bi} and append
one or more comma-separated combinations of
\emph{n}\textbf{t}, where \textbf{t} is one of \textbf{c} (signed byte),
\textbf{u} (unsigned byte), \textbf{h} (signed 2-byte int), \textbf{U} (unsigned
2-byte int), \textbf{i} (signed 4-byte int), \textbf{I} (unsigned 4-byte int),
\textbf{l} (signed 8-byte int), \textbf{L} (unsigned 8-byte int), \textbf{f}
(4-byte single-precision float), and \textbf{d} (8-byte double-precision
float).  Append \textbf{w} to any item to force byte-swapping.
Alternatively, append \textbf{+L} or \textbf{+B} to indicate that the entire data file
should be read as little- or big-endian, respectively.
Here, \emph{n} is the consecutive number of each data type in your binary input file; the
total number may exceeds the columns actually needed by the program.  If no
\emph{n} is specified we assume that \textbf{t} applies to all columns and
that \emph{n} is implied by the expectation of the program.
Multiple segment files are allowed and the
segment headers are assumed to be records where all the fields equal 
NaN.

\subsection{NetCDF tables}
\index{Table!netCDF}
\index{NetCDF tables}
\index{Input!binary \Opt{bi}}
\index{\Opt{bi} (select binary input)}

More and more programs are now producing binary data in the netCDF format, and so \GMT\ programs
started to support tabular netCDF data (files containing one or more 1-dimensional arrays) starting with \GMT\ version 4.3.0.
Because of the meta data contained in those files, reading them is much less complex than reading native binary tables,
and even than ASCII tables. \GMT\ programs will read as many 1-dimensional columns as are needed by the program, starting with the first
1-dimensional it can find in the file. To specifically specify which variables are to be read,
append the suffix \textbf{?}\emph{var1}\textbf{/}\emph{var2}\textbf{/}\emph{...} to the netCDF file name
or add the option \Opt{bic}\emph{var1}\textbf{/}\emph{var2}\textbf{/}\emph{...}, where \emph{var1}, \emph{var2}, etc.\
are the names of the variables to be processed. The latter option is particularly practical when more
than one file is read: the \Opt{bic} option will apply to all files.
Currently, \GMT\ only reads, but does not write, netCDF tabular data.

\section{Grid files} 

\subsection{NetCDF files}
\index{grid file!formats!netCDF|(}
\index{grid file!formats!COARDS|(}
\index{grid file!formats!CF-1.0|(}

By default, \GMT\ stores 2-D grids as COARDS-compliant netCDF files.
COARDS (which stands for Cooperative Ocean/Atmosphere Research Data Service) is a convention used by
many agencies distributing gridded data for ocean and atmosphere research. Sticking to
this convention allows \GMT\ to read gridded data provided by other institutes
and other programs. Conversely, other general domain programs
will be able to read grids created by \GMT.
COARDS is a subset of a more extensive convention for netCDF data called CF-1.0 (Climate and
Forecast, version 1.0). Hence, \GMT\ grids are also automatically CF-1.0-compliant.
However, since CF-1.0 has more general application than COARDS, not all CF-1.0 compliant netCDF files
can be read by \GMT.

The netCDF grid file in \GMT\ has several attributes (See Table~\ref{tbl:netcdf-format})
to describe the content. The routine
that deals with netCDF grid files is sufficiently flexible so that grid files slightly deviating
from the standards used by \GMT\ can also be read.

\begin{table}
\centering
\begin{tabular}{|l|l|} \hline
\multicolumn{1}{|c}{\emph{Attribute}}   &       \multicolumn{1}{|c|}{\emph{Description}}        \\ \hline
\multicolumn{2}{|c|}{\emph{Global attributes}} \\ \hline
Conventions			& COARDS, CF-1.0 (optional) \\ \hline
title				& Title (optional) \\ \hline
source				& How file was created (optional) \\ \hline
registration			& 0 for gridline node registration (default), 1 for pixel registration \\ \hline
\multicolumn{2}{|c|}{\emph{$x$- and $y$-variable attributes}} \\ \hline
long\_name			& Coordinate name (default: ``Longitude'' and ``Latitude'') \\ \hline
units				& Unit of the coordinate (default: ``degrees\_east'' and ``degrees\_north'') \\ \hline
actual\_range			& Minimum and maximum $x$ and $y$ of region; if absent \\
(or valid\_range)		& the first and last $x$- and $y$-values are queried \\ \hline
\multicolumn{2}{|c|}{\emph{$z$-variable attributes}} \\ \hline
long\_name			& Name of the variable (default: ``z'') \\ \hline
units				& Unit of the variable (no default) \\ \hline
scale\_factor			& Factor to multiply $z$ with (default: 1) \\ \hline
add\_offset			& Offset to add to scaled $z$ (default: 0) \\ \hline
actual\_range			& Minimum and maximum $z$ (optional) \\ \hline
\_FillValue			& Value associated with missing data points; if absent an\\
(or missing\_value)		& appropriate default value is assumed, depending on data type. \\ \hline
\end{tabular} 
\caption{Attributes of default \gmt\ grid file in COARDS-compliant netCDF format.}
\label{tbl:netcdf-format}
\end{table}

By default, the first 2-dimensional variable in a netCDF file will by read as the $z$ variable
and the coordinate axes $x$ and $y$ will be determined from the dimensions of the $z$ variable.
\GMT\ will recognize whether the $y$ (latitude) variable increases or decreases. Both forms of
data storage are handled appropriately.

For more information on the use of COARDS-compliant netCDF files, and on how to load
multi-dimensional grids, read Section~\ref{sec:netcdf}.

\GMT\ also allows other formats to be read.  In addition to
the default netCDF format it can use binary floating points, short
integers, bytes, and bits, as well as 8-bit Sun raster files (colormap
ignored).   Additional formats may be used by supplying read/write
functions and linking these with the \GMT\ libraries.
The source file \filename{gmt\_customio.c} has the information that
programmers will need to augment \GMT\ to read custom grid files.  We
anticipate that the number of pre-programmed formats will increase as
enterprising users implement what they need.  See Section~\ref{sec:grdformats}
for more information.

\index{grid file!formats!netCDF|)}
\index{grid file!formats!COARDS|)}
\index{grid file!formats!CF-1.0|)}

\subsection{Gridline and Pixel node registration} 

Scanline format means that the data are stored in rows (\emph{y} = constant)
going from the ``top'' ($y = y_{max}$ (north)) to the ``bottom''
($y = y_{min}$ (south)).  Data within each row are ordered from
``left'' ($x = x_{min}$ (west)) to ``right'' ($x = x_{max}$
(east)).  The \emph{registration} signals how the nodes are laid out.
The grid is always defined as the intersections of all \emph{x}
( \( x = x_{min}, x_{min} + x_{inc}, x_{min} + 2 \cdot x_{inc}, \ldots, x_{max} \) )
and \emph{y} ( \( y = y_{min}, y_{min} + y_{inc}, y_{min} + 2 \cdot y_{inc}, \ldots, y_{max} \) )
lines.  The two scenarios differ as to which area each data point represents.
The default node registration in \GMT\ is gridline node registration.  Most
programs can handle both types, and for some programs like \GMTprog{grdimage}
a pixel registered file makes more sense.  Utility programs like
\GMTprog{grdsample} and \GMTprog{grdproject} will allow you to convert
from one format to the other; \GMTprog{grdedit} can make changes to the grid header
and convert a pixel- to a gridline-registered grid, or \emph{vice versa}.
The grid registration is determined by the common \GMT\ \Opt{r} option
(see Section~\ref{sec:grid_registration}).

\subsection{Boundary Conditions for operations on grids} 
\index{grid file!boundary conditions|(}

\GMT\ has the option to specify boundary conditions in some programs
that operate on grids (e.g., \GMTprog{grdsample}, \GMTprog{grdgradient},
\GMTprog{grdtrack}, \GMTprog{nearneighbor}, and
\GMTprog{grdview}, to name a few.  The desired condition can be set with
the common \GMT\ option \Opt{n}; see Section \ref{sec:resample}.  The boundary conditions come into play
when interpolating or computing derivatives near the limits of the
region covered by the grid. The \emph{default} boundary
conditions used are those which are ``natural'' for the boundary
of a minimum curvature interpolating surface.
If the user knows that the data are periodic in \emph{x} (and/or
\emph{y}), or that the data cover a sphere with \emph{x},\emph{y}
representing \emph{longitude},\emph{latitude}, then there are better
choices for the boundary conditions.
Periodic conditions on \emph{x} (and/or \emph{y}) are chosen by
specifying \emph{x} (and/or \emph{y}) as the boundary condition flags;
global spherical cases are specified using the \emph{g} (geographical)
flag.  Behavior of these conditions is as follows:

\begin{description}
\index{grid file!boundary conditions!periodic}

\item[Periodic] conditions on $x$ indicate that the data are
periodic in the distance ($x_{max} - x_{min}$) and thus repeat
values after every $N = (x_{max} - x_{min})/x_{inc}$.  Note that
this implies that in a grid-registered file the values in the first
and last columns are equal, since these are located at $x = x_{min}$
and $x = x_{max}$, and there are $N + 1$ columns in the file.
This is not the case in a pixel-registered file, where there are only
$N$ and the first and last columns are located at
$x_{min} + x_{inc}/2$ and $x_{max} - x_{inc}/2$.  If $y$ is
periodic all the same holds for $y$.

\item[Geographical] conditions indicate the following:
\index{grid file!boundary conditions!geographical}

\begin{enumerate}

\item If $(x_{max} - x_{min}) \geq 360$ and also 180 modulo $x_{inc} = 0$
then a periodic condition is used on $x$ with a period of 360;
else a default condition is used on the $x$ boundaries.

\item If condition 1 is true and also $y_{max} = 90$ then a
``north pole condition'' is used at $y_{max}$, else a default
condition is used there.

\item If condition 1 is true and also $y_{min} = -90$ then a
``south pole condition'' is used at $y_{min}$, else a default
condition is used there.

\end{enumerate} 

``Pole conditions'' use a 180\DS\ phase-shift of the data,
requiring 180 modulo $x_{inc} = 0$.

\item[Default] boundary conditions are
\index{grid file!boundary conditions!default}

\[ \nabla^2 f = \frac{\partial}{\partial n} \nabla^2 f = 0 \] 

on the boundary, where $f(x, y)$ is represented by the values in
the grid file, and $\partial/\partial n$ is the derivative in the direction normal to a
boundary, and

\[ \nabla^2 = \left(\frac{\partial^2}{\partial x^2} + \frac{\partial^2}{\partial y^2}\right) \]

is the two-dimensional Laplacian operator.

\end{description}
\index{grid file!boundary conditions|)}

\subsection{Native binary grid files}
\index{grid file!native binary|(}

The old style native grid file format that was common in earlier version of \GMT\ is still
supported, although the use of netCDF\index{grid file!formats!netCDF} files is strongly recommended.
The file starts with a header of 892 bytes containing a number of attributes defining the content.
The \GMTprog{grdedit} utility program will allow you to edit parts of
the header of an existing grid file.  The attributes listed in Table~\ref{tbl:grdheader}
are contained within the header record in the order given (except the $z$-array which
is not part of the header structure, but makes up the rest of the file). As this header
was designed long before 64-bit architectures became available, the jump from
the first three integers to the subsequent doubles in the structure does not occur on a
16-byte alignment.  While \GMT\ handles the reading of these structures correctly, enterprising
programmers must take care to read this header correctly (see our code for details).

\begin{table}
\centering
\begin{tabular}{|l|l|} \hline
\multicolumn{1}{|c}{\emph{Parameter}}   &       \multicolumn{1}{|c|}{\emph{Description}}        \\ \hline
\textbf{int} \emph{nx}                      &       Number of nodes in the $x$-dimension    \\ \hline
\textbf{int} \emph{ny}                      &       Number of nodes in the y-dimension      \\ \hline
\textbf{int} \emph{registration}    &       0 for grid line registration, 1 for pixel registration  \\ \hline
\textbf{double} \emph{x\_min}               &       Minimum $x$-value of region     \\ \hline
\textbf{double} \emph{x\_max}               &       Maximum $x$-value of region  \\ \hline
\textbf{double} \emph{y\_min}               &       Minimum $y$-value of region  \\ \hline
\textbf{double} \emph{y\_max}               &       Maximum $y$-value of region  \\ \hline
\textbf{double} \emph{z\_min}               &       Minimum $z$-value in data set  \\ \hline
\textbf{double} \emph{z\_max}               &       Maximum $z$-value in data set  \\ \hline
\textbf{double} \emph{x\_inc}               &       Node spacing in $x$-dimension  \\ \hline
\textbf{double} \emph{y\_inc}               &       Node spacing in $y$-dimension  \\ \hline
\textbf{double} \emph{z\_scale\_factor}     &       Factor to multiply $z$-values after read  \\ \hline
\textbf{double} \emph{z\_add\_offset}       &       Offset to add to scaled $z$-values  \\ \hline
\textbf{char} \emph{x\_units}[80]   &       Units of the $x$-dimension      \\ \hline
\textbf{char} \emph{y\_units}[80]   &       Units of the $y$-dimension      \\ \hline
\textbf{char} \emph{z\_units}[80]   &       Units of the $z$-dimension      \\ \hline 
\textbf{char} \emph{title}[80]      &       Descriptive title of the data set       \\ \hline
\textbf{char} \emph{command}[320]   &       Command line that produced the grid file  \\ \hline
\textbf{char} \emph{remark}[160]    &       Any additional comments \\ \hline \hline
\textbf{TYPE} \emph{z}[nx*ny]      &       1-D array with $z$-values in scanline format \\ \hline
\end{tabular}
\caption{\gmt\ grid file header record. \textbf{TYPE} can be \textbf{char}, \textbf{short}, \textbf{int},
\textbf{float}, or {\bf
double}.}
\label{tbl:grdheader}
\end{table}
\index{grid file!native binary|)}

\section{Sun raster files}
\index{Raster file!format}

The Sun raster file format consists of a header followed by a series
of unsigned 1-byte integers that represents the bit-pattern.  Bits
are scanline oriented, and each row must contain an even number of
bytes.  The predefined 1-bit
patterns in \GMT\ have dimensions of 64 by 64, but other sizes will be
accepted when using the \Opt{Gp$|$P} option.  The Sun header structure
is outline in Table~\ref{tbl:sunheader}.

\begin{table}[H]
\centering

\begin{tabular}{|l|l|}  \hline
\multicolumn{1}{|c}{\emph{Parameter}}   &       \multicolumn{1}{|c|}{\emph{Description}}        \\ \hline
\textbf{int} \emph{ras\_magic}      &       Magic number  \\ \hline
\textbf{int} \emph{ras\_width}      &       Width (pixels) of image  \\ \hline
\textbf{int} \emph{ras\_height}     &       Height (pixels) of image  \\ \hline
\textbf{int} \emph{ras\_depth}      &       Depth (1, 8, 24, 32 bits) of pixel  \\ \hline
\textbf{int} \emph{ras\_length}     &       Length (bytes) of image  \\ \hline
\textbf{int} \emph{ras\_type}       &       Type of file; see RT\_* below  \\ \hline
\textbf{int} \emph{ras\_maptype}    &       Type of colormap; see RMT\_* below  \\ \hline
\textbf{int} \emph{ras\_maplength}  &       Length (bytes) of following map  \\ \hline
\end{tabular}

\caption{Structure of a Sun raster file.}
\label{tbl:sunheader}
\end{table} 

After the header, the color map (if \emph{ras\_maptype} is not RMT\_NONE)
follows for \emph{ras\_maplength} bytes, followed by an image of
\emph{ras\_length} bytes.  Some related definitions are given in Table~\ref{tbl:sundef}.  

\begin{table}[H]
\index{Raster file!definitions}
\centering
\begin{tabular}{|l|l|}  \hline

\multicolumn{1}{|c}{\emph{Macro name}}  &       \multicolumn{1}{|c|}{\emph{Description}}        \\ \hline
RAS\_MAGIC      &       0x59a66a95  \\ \hline
RT\_STANDARD    &       1 (Raw pixrect image in 68000 byte order)  \\ \hline
RT\_BYTE\_ENCODED       &       2 (Run-length compression of bytes)  \\ \hline
RT\_FORMAT\_RGB &       3 ([X]RGB instead of [X]BGR)  \\ \hline
RMT\_NONE       &       0 (ras\_maplength is expected to be 0)  \\ \hline
RMT\_EQUAL\_RGB &       1 (red[ras\_maplength/3],green[],blue[])  \\ \hline

\end{tabular}

\caption{Sun macro definitions relevant to raster files.}
\label{tbl:sundef}
\end{table} 

Numerous public-domain programs exist, such as \progname{xv} and
\progname{convert} (in the ImageMagick package), that will translate
between various raster file formats such as tiff, gif, jpeg, and
Sun raster.  Raster patterns may be created with \GMT\ plotting
tools by generating \PS\ plots that can be rasterized
by \progname{ghostscript} and translated into the right raster format.