File Extensions

IOStreams uses the extensions in the file name to determine which streams to apply when reading or writing a file. Multiple extensions are applied in order, so sample.csv.gz.pgp is first decrypted with PGP and then decompressed with GZip when read.

Supported extensions:

The gems above are soft dependencies: IOStreams does not require them for installation, they only need to be added to the Gemfile when the corresponding extension is used.

Compression options

GZip accepts a compression level when writing, from 0 (no compression) to 9 (best compression):

IOStreams.path("sample.csv.gz").option(:gz, level: 9).write(data)

BZip2 passes its options through to bzip2-ffi: block_size (1 to 9) and work_factor (0 to 250) when writing, and small and first_only when reading:

IOStreams.path("sample.csv.bz2").option(:bz2, block_size: 9).write(data)
IOStreams.path("sample.csv.bz2").option(:bz2, small: true).read

Options are strict, so an option that a stream does not accept raises an ArgumentError. See Streams. The exception is BZip2, which ignores any other option and logs a warning. One that neither reading nor writing takes also reports a deprecation warning when it is set, since in v3.0 it will raise an ArgumentError too, while v3.0 ignores an option of the other direction, as BZip2 does.

Reading an Excel Spreadsheet

Each row in the spreadsheet is converted into a CSV line, so the regular :line, :array, and :hash modes apply:

IOStreams.path("spreadsheet.xlsx").each(:hash) do |record|
  p record
end

Notes:

• Its rows are read as CSV, so leave the format of the path unset, or set it to format(:csv).
• Since the underlying creek gem operates on files, when reading from a stream (for example S3 or HTTP) the contents are first downloaded into a temp file.
• Writing xlsx files is not supported.

Character encoding

The encode stream converts the character encoding of the text that the application reads or writes. Set it with encoding, which keeps the streams that the file name implies, and also works for a stream without a file name, or with streams set with stream:

IOStreams.path("sample.csv.gz").
  encoding("UTF-8", cleaner: :printable, replace: "").
  each do |line|
    puts line
  end

The encode stream is applied closest to the application, so above the data is decompressed first, and then converted. Each call to encoding merges its options with those already set, and encoding without any arguments returns the encoding that was set, or nil. An option that the encode stream does not take, or a value that it does not accept, such as an encoding that Ruby does not know, raises ArgumentError when it is set, before any file is written.

encoding replaces option(:encode, ...), stream(:encode, ...) and a file name ending in .encode, which also apply the encode stream. They are deprecated: each still works, and reports a deprecation warning, see Upgrading.

Options:

• encoding: [String|Encoding] The target encoding, for example "UTF-8", "US-ASCII", or "ASCII-8BIT". Default: "UTF-8"

• replace: [String] The character to replace with when a character cannot be converted to the target encoding. Default: nil (raise Encoding::UndefinedConversionError on invalid characters)

• cleaner: [nil|Symbol|Proc] Cleanse the data. Built-in rules:

  • :printable removes all non-printable characters except \r and \n.
  • :replace_non_printable replaces all non-printable characters except \r and \n with the replace value, or an empty string when replace is nil. A Proc can also be supplied to perform custom cleansing; it is called with the data and the replace value after every read or write. Default: nil

Registering a custom extension

To add a new extension, supply its reader and writer classes. Both must implement .open that yields a stream implementing #read or #write respectively. See any of the streams under lib/io_streams for examples.

IOStreams.register_extension(:xls, MyXls::Reader, MyXls::Writer)

Similarly, to support a new storage location, supply a Path class for its URI scheme. See IOStreams::Paths::S3 for an example of what is required.

IOStreams.register_scheme(:gcs, MyGoogleCloudStoragePath)