Importing Data from Text Files

Position, depth and (optionally) hardness information can be imported from text files, where each row contains data for a single position.

File Types

Files with the extension .TXT or .CSV are recognized as text data files by ReefMaster, and are imported via the CSV Import Options window described below. If the required data file does not have a .TXT or .CSV extension, it should be renamed so that it does.

Note that .TXT files can also contain NMEA data logs. ReefMaster first attempts to parse files with a .TXT extension as NMEA data. If valid NMEA data is found, the file is treated as an NMEA data log, but if no valid NMEA data is found, the file is treated as a text data file and imported via the CSV Import Options window.

Data format

·Data within text files must be represented numerically, with separate fields for latitude, longitude, depth and hardness (if available).

·Fields must be separated by a single character; commonly, a comma, but other characters such as SPACE or TAB may also be used.

·Each data point must be represented as a single row within the file.

·The positions of individual fields within each row can be configured during import. The format of individual fields can also be configured.

The CSV File Import Options Window

Once a CSV or text file has been selected for import, the CSV File Import Options Window is displayed.

Note that when more than one text file has been selected for import, the options panel is shown only once. The selected settings are applied to all files in the current import "batch". The preview panel shows rows from the first selected file.

A range of options can be configured in the options panel (1). The preview panel (2) is shown underneath the options panel and contains the first 50 rows the selected file.

Once import settings have been configured, the file(s) can be imported by clicking the OK button (3).

Import Options

Row layout and the way in which individual fields are interpreted can be configured in the import options panel:

1. Depth

Depth is required, and must be numerically. Depths can be represented by either positive or negative numbers; a positive representation increases with depth, whilst a negative representation does the opposite. Depth can be represented in centimetres, metres, feet or fathoms.

2. Coordinates

ReefMaster supports a very wide range of geographic and projected coordinate systems.

Choose either a Geographic or Projected system by clicking on the appropriate button. Major and minor classifications can be selected using the drop-down list boxes.

Note that ReefMaster performs only very simple validation when importing position data. There is no way to automatically determine whether the chosen projection is correct, therefore it is incumbent on the user to ensure that the selected coordinate system is appropriate for the data being imported.

3. Data thinning

Data thinning is the process of reducing the number of imported data points, whilst attempting to retain as much useful information as possible.

Data thinning during text file import works by limiting the number of data points to one per "bin". Data points that are located within

a bin are averaged, and a single data point is imported into ReefMaster that has the position of the bin and the average depth value.

Thinning data in this way is extremely useful when importing very dense data-sets, such as 1 or 2m gridded LIDAR or data-sets that have already been processed into a gridded elevation model. In the latter case in particular, thinning data points by averaging and binning does not usually result in a great loss of information.

It is strongly recommended to thin data imports where the source data is very dense (e.g. less than a 5m grid) or the number of data points would exceed 1-2M data points. A 10m bin size is a good starting point.

4. Row format and column positions

Fields within a data row must be separated by a single character. The separator character can be chosen from a preset list including Comma, Spaec, Tab and Semi-colon. If a character outside of this set is required, select the Specify option and enter the required character into the adjacent field.

If hardness information is present in the file, select the Hardness option and enter the appropriate column index in the adjacent column field. Hardness information is always imported into the Peak SV layer.

The column indices for Latitude, Longitude and Depth should be entered into the appropriate fields.

The Preview Panel

The preview panel shows the first 50 rows of the selected file, and indicates whether or not data can be imported from the file using the current settings:

The preview panel header (1) shows the name of the file that has been selected for import, along with an approximate row count.

The first 50 rows of the file are shown in the body of the preview panel (2). Each field is enclosed within its own rectangle; in the example above, there is just a single rectangle per row, which indicates that fields have not been separated and the separator field is specified incorrectly for this file.

The bottom of the preview panel (3) shows an indication of whether data can be imported from the current file using the specified settings. The OK (import) button is only enabled when this basic validation has passed.

The same file with modified import settings shows each field within its own box, and the import indicator has turned green. This means that it is possible to create valid position and depth data points using the specified import settings, but it does not guarantee that these settings are correct. It is very often possible to import valid data using incorrect projection settings; sometimes this will manifest as a gross error (e.g. if latitude and longitude values have been inverted), but is also common for the error to be much less obvious. Take care when choosing coordinate settings.