What is IOStreams?

Contents

• Why IOStreams?
  • One API, whatever the format, compression or encryption
  • One API, wherever the file is stored
  • Constant memory, even for huge files
  • Paths are configuration, not code
• A quick tour
• How it works
• Common tasks
• What IOStreams supports
• Installation
• Next steps

IOStreams is a streaming library for Ruby that makes the format, compression, encryption and storage location of a file transparent to your code. Read and write files of any size, one block at a time, whether they are CSV, JSON, Excel or Parquet, compressed with gzip, zip, bzip2 or Zstandard, encrypted with PGP, and whether they are stored on local disk, AWS S3, an SFTP server, or a web server.

Why IOStreams?

Processing files in Ruby usually means writing different code for every variation. One customer sends a gzip compressed CSV file, another a PGP encrypted file, a third uploads an Excel spreadsheet. The files are on local disk in development, but in AWS S3 in production. Each combination needs its own handling, and reading a large file into memory all at once risks exhausting it.

Consider reading a gzip compressed CSV file from S3, one record at a time, without IOStreams:

require "aws-sdk-s3"
require "zlib"
require "csv"

response = Aws::S3::Client.new.get_object(bucket: "my-bucket", key: "data.csv.gz")
gz       = Zlib::GzipReader.new(response.body)
headers  = nil
gz.each_line do |line|
  row = CSV.parse_line(line)
  if headers.nil?
    headers = row
  else
    record = headers.zip(row).to_h
    # ... process record ...
  end
end

Change that file to plain CSV, to a PGP encrypted file, or move it back to local disk, and this code has to change every time. It also downloads the whole object before the first record is processed. With IOStreams the same code handles all of them, while it streams the file:

IOStreams.path("s3://my-bucket/data.csv.gz").each(:hash) do |record|
  # ... process record ...
end

IOStreams reads the file name, data.csv.gz, infers that it is a gzip compressed CSV file, and builds the pipeline of streams that reads it from S3, decompresses it, and parses it. Point the same code at data.csv, data.xlsx, a local file, or data.csv.pgp with its passphrase set on the path, and nothing else changes.

One API, whatever the format, compression or encryption

IOStreams detects the streams of a file from the extensions of its name and applies them in order, so sample.csv.gz.pgp is decrypted, then decompressed, then parsed as CSV, without a line of special handling. The same code that reads a plain CSV file also reads an Excel spreadsheet, a PGP encrypted JSON file, or a Parquet file. A single background job can process files from many senders, in many formats, with no code for each format.

One API, wherever the file is stored

Local disk, AWS S3, S3 compatible stores, SFTP and HTTP(S) share the same IOStreams.path interface, and the same operations, such as #each_child, #exist?, #delete and #move_to, behave the same way on each of them.

Constant memory, even for huge files

Everything is read and written a block at a time, so a 10 GB compressed file uses about as much memory as a 10 KB one, and the code that processes ten rows processes ten million without changes. Files on S3 and HTTP are read as your code reads them, and S3 objects are uploaded as they are written, without a copy on local disk.

Paths are configuration, not code

A path or url can come from an environment variable or a configuration system, so the exact same application code runs against local files in development and S3 or SFTP in production. With roots, the storage location is set once, at startup. See Paths are configuration.

A quick tour

Write a CSV file, compressed with gzip, a record at a time:

IOStreams.path("people.csv.gz").writer(:hash) do |io|
  io << {"name" => "Jack", "born" => "1990-02-01"}
  io << {"name" => "Jill", "born" => "1992-07-14"}
end

Read it back a record at a time, as a Hash, while it is decompressed:

IOStreams.path("people.csv.gz").each(:hash) { |person| p person }
# {"name" => "Jack", "born" => "1990-02-01"}
# {"name" => "Jill", "born" => "1992-07-14"}

The same code reads s3://my-bucket/people.csv.gz or sftp://sftp.example.org/people.csv.gz instead: only the name changes. See Paths for the gems and credentials of each store.

Describe the columns of the file with a schema, so that each value is read as its type:

people = IOStreams.path("people.csv.gz").schema(
  fields: [{name: "name", type: "string"}, {name: "born", type: "date"}]
)
people.each(:hash) { |person| p person }
# {"name" => "Jack", "born" => #<Date: 1990-02-01 ((2447924j,0s,0n),+0s,2299161j)>}
# {"name" => "Jill", "born" => #<Date: 1992-07-14 ((2448818j,0s,0n),+0s,2299161j)>}

Convert it to another format by copying it to a path whose name says so:

IOStreams.path("people.json").copy_from(people, mode: :hash)

IOStreams.path("people.json").read
# => "{\"name\":\"Jack\",\"born\":\"1990-02-01\"}\n{\"name\":\"Jill\",\"born\":\"1992-07-14\"}\n"

The target can be anywhere, compressed and encrypted, such as JSON on S3, encrypted with PGP for a partner whose public key is in your keyring, see PGP:

IOStreams.path("s3://my-bucket/exports/people.json.gz.pgp").
  option(:pgp, recipient: "partner@example.org").
  copy_from(people, mode: :hash)

Write an Excel spreadsheet from records. A Parquet file is written the same way, with a schema that gives the type of each column, see Parquet files:

IOStreams.path("report.xlsx").writer(:hash) do |io|
  io << {"name" => "Jack", "total" => 12}
  io << {"name" => "Jill", "total" => 20}
end

Rescue a missing file the same way wherever it is stored, so that the path can come from configuration:

begin
  IOStreams.path("missing.csv").each(:hash) { |record| p record }
rescue IOStreams::Errors::NotFound => e
  puts "Nothing to process: #{e.message}"
end
# Nothing to process: No such file or directory @ rb_sysopen - missing.csv

How it works

• A path names a file and the store that holds it, such as IOStreams.path("s3://my-bucket/people.csv.gz"). IOStreams.stream(io) reads or writes an IO that you already have, such as a StringIO, the same way. See Paths.
• Each extension at the end of the file name, such as .gz or .pgp, names a stream that the data passes through, a block at a time. In order, they are the pipeline of the path: reading people.csv.gz.pgp decrypts the data and then decompresses it, and writing it compresses the data and then encrypts it. See Streams from the file name.
• The tabular format of the file, such as .csv, .json or .xlsx, says how its rows and records are stored. See Tabular formats.
• The mode says what your code reads or writes: blocks of bytes (:stream), lines of text (:line), rows, each an Array of the values of a line (:array), or records, each a Hash of those values by column name (:hash). For a CSV file, the names of the columns come from its header row, its first line. See Reading and writing.
• A schema names the columns of a file, gives the type of each one, such as a Date, and says which columns a file can have. See Schema.

Common tasks

• Read a CSV, JSON or Excel file a record at a time: IOStreams.path("people.csv").each(:hash) { |record| p record }, see Reading.
• Write a file compressed with gzip: IOStreams.path("people.csv.gz").writer(:hash) { |io| io << record }, see gzip.
• Convert CSV to JSON: IOStreams.path("people.json").copy_from("people.csv", mode: :hash), see Converting rows and records.
• Upload a file to S3: IOStreams.path("s3://my-bucket/people.csv").copy_from("people.csv"), see Copying and moving.
• Encrypt a file with PGP: IOStreams.path("people.csv.pgp").option(:pgp, recipient: "partner@example.org").copy_from("people.csv"), see PGP.
• Decrypt a PGP file: IOStreams.path("people.csv.pgp").option(:pgp, passphrase: ENV.fetch("PGP_PASSPHRASE")).read, see PGP.
• Read numbers and dates as their types, and skip the other columns: path.schema(fieldsMatch: "partial", fields: [{name: "born", type: "date"}]), see Schema.
• Read a Windows-1252 file as UTF-8: path.encoding("Windows-1252:UTF-8"), see Character encoding.
• List the CSV files in a directory or bucket: IOStreams.path("s3://my-bucket/inbound").each_child("*.csv") { |path| p path }, see Listing files.
• Keep the location of the files in configuration: IOStreams.add_root(:default, ENV.fetch("FILES_ROOT")) at startup, then IOStreams.join("people.csv"), see Configuration.
• Handle a missing file wherever it is stored: rescue IOStreams::Errors::NotFound, see Errors.
• Keep untrusted file names within a directory: IOStreams.add_allowed_path("/var/my_app/uploads"), see Security.

What IOStreams supports

Installation

Add the gem to your Gemfile, for Ruby 3.3 or later:

gem "iostreams"

IOStreams itself needs no other gems. Each feature that needs one loads it when it is first used, so add only the gems of the features that your application uses:

When a gem is missing, the feature that needs it raises LoadError, naming the gem to install.

Next steps

• The tutorial walks through reading and writing files, step by step.
• Reading and writing: lines, rows, records and text encodings.
• Tabular formats: CSV, JSON, Excel, fixed width and Parquet files, and schemas.
• Compression and encryption: gzip, zip, bzip2, Zstandard, PGP and Symmetric Encryption.
• Paths: local files, S3, SFTP and HTTP, and the operations on a path.
• Copying and moving: copying files between stores, and converting them on the way.
• Errors: rescuing a missing file the same way wherever it is stored.
• PGP: encrypting, decrypting, signing and verifying files, and managing keys.
• Configuration: roots, temp files and logging.
• Security: allowed paths, untrusted urls and credentials.
• Upgrading: what changed in v3.0, for applications that use an earlier version.