IOStreams Any size. Any format. Anywhere.

Configuring IOStreams

Paths are configuration

A path or url is configuration, not code. Keep it in an environment variable or a centralized configuration system, and pass it to IOStreams, so that the exact same application code runs against local files in development and a completely different file store in staging or production.

# Development: EXPORT_PATH=~/exports/daily.csv
# Staging:     EXPORT_PATH=sftp://staging.example.org/~/exports/daily.csv
# Production:  EXPORT_PATH=s3://my-bucket/exports/daily.csv
IOStreams.path(ENV.fetch("EXPORT_PATH")).write(report)

This also helps when infrastructure changes. For example, while migrating production from on-premises servers to the cloud, the on-premises instance can keep reading and writing local files while the cloud instance uses S3, both running the same release with only their configuration differing.

The same path written for each file store refers to the equivalent place. For example ~/exports is within the home directory, and sftp://hostname/~/exports is within the login directory on the SFTP server.

To keep code independent of where its files are stored:

When an application uses several locations, roots give them names, and each root can be set from configuration:

IOStreams.add_root(:default, ENV.fetch("FILES_ROOT"))
IOStreams.join("exports", "daily.csv").write(report)

add_root

Roots allow paths to reference a particular root directory, so that all path names are appended to that root. Their primary purpose is to allow the exact same code to run in production and development, yet use completely different data sources in each. For example, in production a root can point to an S3 bucket, while in development it points to the local file system.

Roots are configured via an initializer at startup. IOStreams.join then joins the supplied path elements onto the named root, using the :default root whenever a root is not supplied.

Set the default root for this environment in an initializer:

IOStreams.add_root(:default, "/var/my_app/files")

Now the default root path is available:

IOStreams.root
# => #<IOStreams::Paths::File:/var/my_app/files pipeline={}>
 
IOStreams.root.to_s
# => "/var/my_app/files"

Comparing the final path using path and then join that uses a root path:

IOStreams.path("/var/my_app/files", "my_test_file.txt").to_s
# => "/var/my_app/files/my_test_file.txt"

IOStreams.join("my_test_file.txt").to_s
# => "/var/my_app/files/my_test_file.txt"

Using path:

IOStreams.path("/var/my_app/files", "my_test_file.txt").write("Hello World")

With the default root path configured the above code can be simplified by using join since it resolves to the same path.

IOStreams.join("my_test_file.txt").write("Hello World")

Multiple roots can be setup, for example one for input files, another for output files, another for reports, etc. During development the roots can all point to a common location, while in production they could be completely different S3 buckets.

For example add special paths for downloads and uploads.

IOStreams.add_root(:downloads, "/var/my_app/downloads")
IOStreams.add_root(:uploads, "/var/my_app/uploads")

An example that writes a file into the /var/my_app/downloads directory:

IOStreams.join("my_test_file.txt", root: :downloads).write("Hello World")

The other benefit is that the root paths used in an application are externalized from the code base. That way the roots can be changed to different locations depending on the environment.

We can also change the storage mechanism by changing the root:

IOStreams.add_root(:downloads, "s3://my-app-bucket-name/downloads")
IOStreams.add_root(:uploads, "s3://my-app-bucket-name/uploads")

Now the application will write to S3 and the code does not change at all.

IOStreams.join("my_test_file.txt", root: :downloads).write("Hello World")

To use or query a configured root path:

IOStreams.root(:downloads).to_s
# => "s3://my-app-bucket-name/downloads"

temp_dir

When working with large files the standard temp file system location can be too small to handle downloading large files. For example to decrypt a pgp file from S3, because GnuPG is not streaming capable and only operates on local files.

By default IOStreams looks up the location to store temp files in the following order:

To explicity set the temp file location the following config option can be used:

IOStreams.temp_dir = "/var/really_big_temp"

temp_file

To work with a temporary file directly, IOStreams.temp_file yields a path inside temp_dir and deletes the file when the block completes:

IOStreams.temp_file("export", ".csv") do |path|
  path.write("Hello World")
  # ... use the temp file ...
end
# The temp file has been deleted.

The first argument is a base file name to include in the generated temp file name, and the optional second argument is the file extension.

logger

IOStreams can log debug information, such as the external commands it runs for PGP and SFTP.

When Semantic Logger is loaded it is detected automatically, and IOStreams logs to it without any additional configuration.

To use a different logger, or to log when Semantic Logger is not present, assign any logger that responds to the standard logging methods:

require "logger"
IOStreams.logger = Logger.new($stdout)

To disable logging entirely, set the logger to nil:

IOStreams.logger = nil

enforce_column_restrictions

Applies allowed_columns, required_columns and skip_unknown to every input when reading records, including JSON records, so that renaming an uploaded file from .csv to .json cannot bypass them.

To return to the behavior before v3.0, where they only apply to a header row read from the file, set it to false in an initializer. A warning is then logged when applying them would change the records read:

IOStreams.enforce_column_restrictions = false

Default: true. It defaulted to false before v3.0. See Header options.