text {
    :toc: left
    :toclevels: 5
    :toc-title: Content
    :icons:  font
    :sectnums:
    :sectlinks:
    :experimental:
    :linkcss:
    :imagesdir: ./images
    
    ((= chap_tclcsv_ref "The tclcsv extension"))
    Version {tclcsvversion}

    ((author "Ashok P. Nadkarni"))
     
}


text {
    == Introduction

    The `tclcsv` extension for Tcl provides a fast and flexible means of
    reading and writing text files in _Comma Separated Value_ (CSV) format.

    [NOTE]
    `Tcllib` also has a package `csv` that is capable
    of reading and writing CSV files. It has the advantage of being
    a pure Tcl package but conversely has much lower performance which
    is an issue only for larger files. It is also a little less flexible
    in terms of input syntax.

    The extension requires Tcl version 8.6 or later.
}

text {
    == Downloads and installation

    Prebuilt 32- and 64-bit Windows binaries for Tcl 8.6 and 9.0 are available
    from https://sourceforge.net/projects/tclcsv/files/. Unzip the distribution
    into a directory that is included in your Tcl `auto_path`.
    
    For *ix systems, a TEA-compliant source distribution can be downloaded from
    from the same location.
}

text {
    == General usage

    To use the extension, load it with `package require`:
} script {
    package require tclcsv
} text {

    === Reading data
    
    The package provides two ways to read CSV data from a channel:

    * The ((^ tclcsv_csv_read csv_read))
    command parses CSV data and returns it as a list
    of lists in a single call.
    
    * The ((^ tclcsv_reader reader))
    command which returns a command object which can
    then be used to incrementally return rows from the CSV data.
    
    Both forms take various options that
    indicate the specific ((^ tclcsv_dialects "CSV dialect"))
    as well as options that limit which rows
    of the data are returned.

    [TIP]
    ====
    Although the `tclcsv` commands work with channels, not strings,
    it is easy enough to parse CSV data that is contained in a string.
    Use the `tcl::chan::string` package available as part
    of http://tcl.activestate.com/software/tcllib/[tcllib].
    You can load the package as
} shell {
    package require tcl::chan::string
} text {
    and then create channels from string data
    as shown in the examples in this documentation.
    ====

    === Writing data

    The ((^ tclcsv_csv_write csv_write)) command takes a list of records,
    each of which is itself a list of fields, and
    writes it as CSV-formatted data
    to a specified channel. As for the ((^ tclcsv_csv_read csv_read))
    command, the format can be controlled with options that specify the
    dialect.

    ((=== tclcsv_dialects "CSV dialects"))

    The exact form of CSV data can vary. CSV ''dialects'' may differ
    in terms of delimiter character,
    the use of quotes, treatment of leading whitespace, header lines
    and so on. The ((^ tclcsv_dialect dialect))
    command returns appropriate values
    for options to be passed to the ((^ tclcsv_csv_read csv_read)) and
    ((^ tclcsv_reader reader)) commands to handle well-known dialects
    such as Excel.
    
    In addition, the `tclcsv` package
    provides convenience commands that are primarily intended
    for interactive use when the dialect used for the CSV data is not known.

    * The ((^ tclcsv_sniff sniff)) command uses heuristics to determine
    the format of the CSV data and returns a list of appropriate options
    required for parsing it with the ((^ tclcsv_csv_read csv_read)) or
    ((^ tclcsv_reader reader)) commands.

    * The ((^ tclcsv_sniff_header sniff_header))
    command uses heuristics to determine the types of the columns
    in the CSV data and whether the data is prefixed with a header line.

    ==== Interactive dialect configuration
    
    As an aid for both the programmer as well as the end-user to
    correctly select the various options for a dialect when the exact
    dialect for a file is not known, the package
    also provides the ((^ tclcsv_dialectpicker dialectpicker))
    Tk widget.
    The widget allows the user to set various parameters, such as file
    encoding, delimiter etc. simultaneously previewing the first few
    lines. These settings can be programmatically retrieved and
    passed to one of the CSV read commands to correctly parse the data.
}


text {
    == Command reference
    
    All commands are located in the `tclcsv` namespace.

    === Commands

    ((cmddef tclcsv_csv_read "csv_read ?_OPTIONS_? _CHANNEL_"))

    The command reads data from the specified channel
    (which must not be non-blocking) and returns
    a Tcl list each element of which is a list corresponding to
    one row in the read CSV data. The caller should have appropriately
    positioned the channel read pointer and configured its encoding before
    calling this command.

    The command will normally read all data from
    the channel until EOF is encountered and return the corresponding
    rows. The following options modify this behaviour:

    ((.Table tab_tclcsv_rowopts "Options for filtering"))
    [cols="20,80"]
    |===

    |`-excludefields _FIELDINDICES_`
    |Specifies the list of indices of fields that are not to be included
    in the returned data. The corresponding fields will not be included
    even if they are specified via the `-includefields` option. If unspecified
    or an empty list, fields are included as per the `-includefields` option.
    
    |`-includefields _FIELDINDICES_`
    |Specifies the list of indices of fields that are to be included
    in the returned data unless excluded by the `-excludefields` option.
    Any fields whose indices are not present in _FIELDINDICES_ will
    not be included in the returned rows.
    If unspecified
    or an empty list, all fields are included subject to the `-excludefields`
    option.
    
    |`-nrows _NROWS_`
    |If specified, stops after _NROWS_ rows are read. Note however that
    it does not guarantee that the channel read pointer is placed just beyond
    the last read data.

    |`-skipblanklines _BOOLEAN_`
    |If specified as `true` (default), empty lines are ignored. If `false`
    empty lines are treated as rows with no fields.

    |`-skiplines _LINELIST_`
    |If specified, _LINELIST_ must be a list of integer
    line numbers (first line being at position 0). The corresponding lines
    are skipped and not included in the returned data. The line numbering
    includes commented lines if comments are enabled.

    |`-startline _LINENUM_`
    |If specified, the first _LINENUM_ files of input are ignored.
    Note this includes commented lines if comments are enabled.
    
    |===
}

text {

    The following options collectively specify the ((^ tclcsv_dialects dialect))
    of the CSV data.
    
    ((.Table tab_tclcsv_read_fmtopts "CSV read format options"))
    [cols="20,80"]
    |===

    |`-comment _COMMENTCHAR_`
    |Specifies the character to use as a comment leader. All characters
    on a line after _COMMENTCHAR_ are ignored. _COMMENTCHAR_ must
    be an ASCII character. If _COMMENTCHAR_ is
    the empty string (default), comment recognition is disabled.

    |`-delimiter _DELIMCHAR_`
    |Specifies the delimiter character that separates fields. Defaults
    to the `,` (comma) character.

    |`-doublequote _BOOLEAN_`
    |Controls how the quote character inside a field value is treated.
    If specified as `true` (default), quote characters in a field
    are expected to be represented by doubling them. If `false`,
    they are expected to be preceded with an escape character.

    |`-escape _ESCCHAR_`
    |If specified, any character appearing after _ESCCHAR_ is treated
    as an ordinary character with no special meaning. If unspecified
    or specified as an empty string, the escaping mechanism is disabled.
    _ESCCHAR_ must be an ASCII character or an empty string.

    |`-quote _QUOTECHAR_`
    |Specifies the character used for quoting when a field contains
    special characters such as the delimiter or line terminators.
    If set to the empty string, the input is assumed to have no quoting
    character and special characters, if any, are expected to have used
    the escaping mechanism. Defaults to the double quote character.

    |`-skipleadingspace _BOOLEAN_`
    |If specified as `true`, leading space characters in fields are stripped.
    If `false` (default), it is retained.
    
    |`-terminator _TERMCHAR_`
    |Specifies the character to use to terminate a row. By default,
    each input line is a treated as a row.
    
    |===

    [NOTE]
    The command does not require that all rows have the same number of
    fields. If required, the caller has to check that all returned rows
    have the same number of elements.
}

text {
    ((cmddef tclcsv_csv_write "csv_write ?_OPTIONS_? _CHANNEL_ _ROWS_"))

    The command writes _ROWS_ to the specified channel _CHANNEL_.
    _ROWS_ must be a list each of whose elements is a sublist corresponding
    to a single record.
    The caller should have appropriately
    positioned the channel write pointer and configured its encoding before
    calling this command.

    The CSV ((^ tclcsv_dialects dialect)) used for writing is controlled
    through the options in the table below.
    
    ((.Table tab_tclcsv_write_fmtopts "CSV write format options"))
    [cols="20,80"]
    |===

    |`-delimiter _DELIMCHAR_`
    |Specifies the delimiter character that separates fields. Defaults
    to the `,` (comma) character. Must be an ASCII character.

    |`-doublequote _BOOLEAN_`
    |Controls how the quote character inside a field value is treated.
    If specified as `true` (default), quote characters in a field
    are doubled. If `false`, they are preceded with an escape character.

    |`-escape _ESCCHAR_`
    |If quoting is disabled or the `-doublequote` option is false,
    special characters will be preceded with _ESCCHAR_. By default,
    or if specified as an empty string, the escaping mechanism is disabled.
    _ESCCHAR_ must be an ASCII character or an empty string.

    |`-quote _QUOTECHAR_`
    |Specifies the character used for quoting when a field contains
    special characters such as the delimiter or line terminators.
    Defaults to the double quote character.
    Setting it to the empty string has the same effect as setting
    the `-quoting` option to `none`. In that case, special characters
    will be preceded by the escape character.

    |`-quoting _POLICY_`
    |Controls the policy to use when fields are quoted. If
    _POLICY_ is set to `none`, fields are never
    quoted. Delimiters in the data are escaped instead. This has the
    same effect as setting the `-quote` option to the empty string.
    If set to `all`, every field is enclosed by the quotation character.
    If set to `minimal`, only fields that contain special characters
    like delimiters are enclosed in quotes. Finally, `nonnumeric`
    indicates that all fields that are not interpreted as numbers
    are enclosed in quotes. Numbers include decimals 
    (including decimal fractions) of
    arbitrary length, and floating point numbers. Other formats like
    hexadecimal, are not treated as numbers.
    
    |`-terminator _TERM_`
    |Specifies a string of one or two characters to use to terminate a row.
    This should generally not be specified and it defaults to standard
    `\n` linefeed character. Keep in mind that the channel encoding
    settings can be used to then write end of line in a platform specific
    manner.
    
    |===
}


text {
    ((cmddef tclcsv_dialect "dialect _NAME_ ?_DIRECTION_?"))
    Returns the appropriate values for options 
    for the CSV dialect _NAME_.
    Currently, _NAME_ must be `excel` or `excel-tab` which correspond
    to the CSV formats supported by Excel. The former uses commas
    and the latter tabs. 

    If _DIRECTION_ is `read` (default), returned options are
    suitable for passing to
    ((^ tclcsv_csv_read csv_read)) and
    ((^ tclcsv_reader reader)). If _DIRECTION_ is `write`
    the options are suitable for
    ((^ tclcsv_csv_write csv_write)).
    
} shell {
    tclcsv::dialect excel
}

text {
    ((cmddef tclcsv_reader "reader SUBCOMMAND ?_OPTIONS_?"))

    This command takes one of the two forms shown below.
} syntax {
    reader create _CMDNAME_ ?_OPTIONS_? _CHANNEL_
    reader new ?_OPTIONS_? _CHANNEL_
} text {
    Each form creates a command object that will _incrementally_
    parse CSV data from the specified channel (which must not be non-blocking).
    The caller should have appropriately
    positioned the channel read pointer and configured its encoding before
    calling this command.

    The `reader create` command allows the caller
    to specify the name of this command object whereas `reader new` will
    generate a new unique name. Both return the name of the created command.

    Options are as detailed for the ((^ tclcsv_csv_read csv_read))
    command with the exception of the `-nrows` option which is not relevant
    for this interface.
    
    The methods supported by the reader command objects are detailed below.
    
    ((cmddef tclcsv_reader_destroy "_READER_ destroy" 1))
    Destroys the _READER_ command object. Note that closing the attached
    channel is the caller's responsibility.
    
    ((cmddef tclcsv_reader_eof "_READER_ eof" 1))
    Returns 1 if there are no more rows and 0 otherwise.
    
    ((cmddef tclcsv_reader_next "_READER_ next ?_COUNT_?" 1))
    Returns one or more rows. If _COUNT_ is not specified, the return
    value is a list corresponding to a single row.
    If _COUNT_ is specified, the return value
    is a list of up to _COUNT_ sublists each of which corresponds to a row.
    Fewer than _COUNT_ rows may be returned if that many
    are not available.
    
    Note that `_READER_ next` is _not_ the same as `_READER_ next 1`.
    The former returns a single row, the latter returns a list containing
    a single row.

    When no more rows are available, the method returns an empty list.
    This is not distinguishable from an empty line in the CSV input
    if the `-skipblanklines` option was specified as `false`. The
    ((^ tclcsv_reader_eof eof)) method may be used to distinguish
    the two cases.

    .Example

    The following is an example of parsing using `reader` objects.
} shell {
    set fd [tcl::chan::string { \
        r0c0, r0c1, r0c2
        r1c0, r1c1, r1c2
        r2c0, r2c1, r2c2
        r3c0, r3c1, r3c2
    }]
    set reader [tclcsv::reader new -skipleadingspace 1 $fd]
    $reader next
    $reader next 1; # Note difference in return value from previous command
    $reader next 2
    $reader next
    $reader eof
    $reader destroy
    close $fd
}

text {
    ((cmddef tclcsv_sniff "sniff ?-delimiters _DELIMITERS_? _CHANNEL_"))

    Attempts to guess the format of the data in the channel and returns
    a list of appropriate options to be passed to ((^ tclcsv_csv_read)).
    The command uses heuristics that may not work for all files and
    as such is intended for interactive use.

    The channel must be seekable and the command always returns the
    channel in the same position it was in when the command was called.
    This is true for both normal returns as well as exceptions.

} shell {
    set fd [tcl::chan::string { \
        r0c0, r0c1, r0c2
        r1c0, r1c1, r1c2
        r2c0, r2c1, r2c2
    }]
    set opts [tclcsv::sniff $fd]
    tclcsv::csv_read {*}$opts $fd
    close $fd
} shell {
    set fd [tcl::chan::string { \
        'r0;c0';'r0c1';'r0c2'
        'r1c0'; 'r1c1'; 'r1c2'
        'r2c0'; 'r2c1'; 'r2c2'
    }]
    set opts [tclcsv::sniff $fd]
    tclcsv::csv_read {*}$opts $fd
    close $fd
}
    
text {
    ((cmddef tclcsv_sniff_header "sniff_header ?_OPTIONS_? _CHANNEL_"))

    Attempts to guess whether the CSV data contained in the channel
    includes a header. It also attempts to guess the type of the data
    in each column of the CSV file. _OPTIONS_ specify the CSV dialect
    of the data. See ((^ tab_tclcsv_read_fmtopts)).

    If the data includes a header, the command returns a list with two
    elements, the first of which is a list containing the deduced type
    of each column, and the second element being a list containing
    the header fields for each column. If the command deduces that
    the data does not contain a header, the returned list does not
    contain the second element.

    The deduced type of each column is one of `integer`, `real` or
    `string`. Note that `integer` type check is done as a decimal
    string and thus hexadecimal values are treated as strings
    and values like `08` (invalid octal) are  accepted as valid
    integer values.
    
    The command uses heuristics that may not work for all files and
    as such is intended for interactive use.

    The channel must be seekable and the command always returns the
    channel in the same position it was in when the command was called.
    This is true for both normal returns as well as exceptions.

    The following examples show the return values with or without
    a header being present.
} shell {
    set fd [tcl::chan::string { \
        City, Longitude, Latitude
        New York, 40.7127, 74.0059
        London, 51.5072, 0.1275
    }]
    tclcsv::sniff_header $fd
    close $fd
    
    set fd [tcl::chan::string { \
        New York, 40.7127, 74.0059
        London, 51.5072, 0.1275
    }]
    tclcsv::sniff_header $fd
    close $fd
} text {
    Note that when a header is present, you can use the `-skiplines` option
    to ((^ tclcsv_csv_read csv_read)) to skip the header.
}

text {
    == Widget reference

    The package provides a single widget, `dialectpicker`, for configuring
    the dialect settings used to parse CSV data.

    === Widgets

    ((cmddef tclcsv_dialectpicker "dialectpicker _WIDGET_ ?_OPTIONS_? _DATASOURCE_"))

    The `dialectpicker` widget allows interactive configuration of
    the dialect settings for parsing CSV data from the specified
    file or channel. The widget
    presents controls for the various settings and permits the user
    to modify them and inspect the results of parsing the CSV data
    from the channel using the configured settings.

    _WIDGET_ should be the Tk window path for the widget. This is also
    the return value of the command.
    
    _DATASOURCE_ should be either the path to a file or
    the name of the channel from which the CSV
    data is to be read. In the case of a channel,
    the configuration, including the seek
    position and encoding,
    of the channel is restored to its original when the widget
    is destroyed.

    In addition to Tk, the widget requires the `snit` package,
    available as part of
    http://core.tcl.tk/tcllib/doc/trunk/embedded/index.html[tcllib],
    to be installed.

    An example invocation is shown below.

    .The `dialectpicker` widget
    image::dialectpicker.png[]

    The top half of the widget contains the various settings related
    to parsing of CSV data. The bottom half displays a preview table which is
    updated as these settings are modified by the user.

    On creation, the widget sets the initial values by
    ((^ tclcsv_sniff sniffing)) the channel. These can be overridden
    by specifying options to `dialectpicker` when the
    widget is created. These options are

    ((.Table tab_tclcsv_dialectpicker_opts "dialectpicker options"))
    [cols="20,80"]
    |===
    
    |`-comment _CHAR_`
    |See ((^ tab_tclcsv_read_fmtopts)).

    |`-delimiter _CHAR_`
    |See ((^ tab_tclcsv_read_fmtopts)).

    |`-doublequote _CHAR_`
    |See ((^ tab_tclcsv_read_fmtopts)).

    |`-encoding _ENCODING_`
    |Specifies the character encoding to be used for the channel.
    _ENCODING_ must be one of the names returned by the Tcl
    `encoding names` command.
    
    |`-escape _CHAR_`
    |See ((^ tab_tclcsv_read_fmtopts)).

    |`-headerpresent _BOOLEAN_`
    |Indicates whether the CSV data is expected
    to have a header line. If unspecified, the widget will use
    heuristics to set the initial value.

    |`-quote _CHAR_`
    |See ((^ tab_tclcsv_read_fmtopts)).

    |`-skipleadingspace _CHAR_`
    |See ((^ tab_tclcsv_read_fmtopts)).

    |`-skipblanklines _CHAR_`
    |See ((^ tab_tclcsv_read_fmtopts)).

    |===
    
    
    The current settings for the widget can be retrieved through
    two method calls `encoding` and `dialect`.

    ((cmddef tclcsv_dialectpicker_encoding "_WIDGET_ encoding" 1))

    Returns the character encoding name currently selected in the
    widget.
    
    ((cmddef tclcsv_dialectpicker_dialect "_WIDGET_ dialect" 1))

    Returns a dictionary of CSV dialect options with values as
    set in the widget. The dictionary contains all the options
    shown in ((^ tab_tclcsv_dialectpicker_opts)). If the user
    has deselected any of the `Included` checkboxes for any
    column in the preview pane, the dictionary also includes
    an `-includedfields` option specifying the subset of fields to be
    read from the data. If the `Header is present` checkbox is
    selected, the dictionary includes a `-startline 1` option
    indicating the first line should be skipped when reading data.
    
    The return value can directly be passed to
    the ((^ tclcsv_csv_read csv_read)) and ((^ tclcsv_reader reader))
    commands as options. 

    ==== Example

    The following code uses the `widget::dialog` dialog widget from
    the http://tcl.activestate.com/software/tklib/[tklib] package
    to read CSV data using user-selected settings.
    
} uiscript {
    package require tclcsv
    package require widget::dialog
    widget::dialog .dlg -type okcancel
    tclcsv::dialectpicker .dlg.csv qb.csv
    .dlg setwidget .dlg.csv
    set response [.dlg display]; # User response will "ok" or "cancel"
    if {$response eq "ok"} {
        set fd [open qb.csv]
        set encoding [.dlg.csv encoding]
        chan configure $fd -encoding $encoding; # Note we have to explicitly set encoding prior to calling csv_read
        set opts [.dlg.csv dialect]
        set rows [tclcsv::csv_read {*}$opts $fd]
        close $fd
    }
    destroy .dlg
} 

text {
    == Source code

    The source code is available from its repository at
    https://github.com/apnadkarni/tcl-csv.
}

text {
    == Reporting bugs

    Report any bugs at https://github.com/apnadkarni/tcl-csv/issues.
}

text {
    == License

    See the file `license.terms` in the distribution or in the
    `src` directory in the source repository.
}

text {
    == Acknowledgements

    The core of the CSV parsing code is adapted from the CSV parser
    implemented by the Python http://pandas.pydata.org[pandas] library.

    The hashing code is from https://github.com/attractivechaos/klib[attractivechaos].
}

text {
    == Version history

    .Version 2.4

    * Tcl 9 support.

    .Version 2.3

    * Added csv_write for writing.

    * 40% faster parsing.

    .Version 2.2

    * Modify `dialectpicker` to accept either a file path or a channel

    * Latent support for the http://tarray.sf.net[tarray] package

    .Version 2.1
    
    * Added `dialectpicker` widget
    
    * Added options `-includefields` and `-excludefields`.

    * Tweaks to `sniff_header` to improve type and header heuristics
}
