IOStreams Any size. Any format. Anywhere.

Errors

Contents

When a path’s storage fails, for example because the file does not exist, the path raises the exception that its storage raised, such as Errno::ENOENT for a local file or Aws::S3::Errors::NoSuchKey for S3. It tags that exception with the kind of failure: IOStreams::Errors::NotFound, PermissionDenied or Unavailable.

Likewise, when a stream cannot read the data of a file, for example because a file named .gz is not gzip data, or a zip file does not contain the file to read, the stream raises the exception of its gem, such as Zlib::GzipFile::Error, tagged with the kind of failure: IOStreams::Errors::Corrupt, MissingEntry or MissingKey.

Rescue the kind of failure, rather than the exception from one storage or one gem, so that the same code keeps working when the path changes, for example from a local file in development to S3 in production, or from .gz to .zip. See Paths are configuration. It also keeps working when IOStreams replaces the gem that reads a stream with a newer or better one, or reads it with another library on another platform, as JRuby reads zip files with Java.

Rescue the kind of failure

path = IOStreams.path(ENV.fetch("IMPORT_PATH"))

begin
  path.each(:hash) { |record| Order.import(record) }
rescue IOStreams::Errors::NotFound => e
  logger.info("Nothing to import yet: #{e.display_name}")
end

The same rescue handles the missing file whether IMPORT_PATH is ~/imports/orders.csv, sftp://sftp.example.org/~/orders.csv, s3://my-bucket/imports/orders.csv or https://example.org/orders.csv.

Rescuing the exception of one storage only works for that storage:

# Avoid: misses a file that does not exist on S3, SFTP or HTTP.
rescue Errno::ENOENT

# Avoid: has to know every storage, and also rescues SFTP and HTTP failures that are not a missing file.
rescue Errno::ENOENT, Aws::S3::Errors::NoSuchKey, IOStreams::Errors::CommunicationsFailure

Rescue a file that cannot be read, whatever its streams

path = IOStreams.path(ENV.fetch("IMPORT_PATH"))

begin
  path.each(:hash) { |record| Order.import(record) }
rescue IOStreams::Errors::Corrupt => e
  logger.error("#{path.display_name} cannot be read, ask for it to be sent again", e)
end

The same rescue handles a file that is damaged, cut short, or not what its file name says at all, whether IMPORT_PATH is orders.csv.gz, orders.csv.bz2, orders.csv.zst, orders.zip, orders.csv.enc, orders.csv.pgp or orders.xlsx.

Rescuing the exception of one gem only works for that stream:

# Avoid: misses a corrupt file once the file name changes to .bz2 or .zip.
rescue Zlib::GzipFile::Error

A key that is missing, or a passphrase that is wrong, is a problem of the configuration rather than of the file, so it raises IOStreams::Errors::MissingKey instead, which sending the file again will not fix:

begin
  path.each(:hash) { |record| Order.import(record) }
rescue IOStreams::Errors::MissingKey => e
  raise(ConfigurationError, "Check the PGP keys and passphrase for #{e.display_name || path.display_name}")
end

The exception tree

The kinds of failure are modules that tag the exception that the storage, or the gem of a stream, raised:

A failure of the storage is never a failure of a stream, so rescuing NotFound does not rescue a zip file without the file to read, which reading again will not fix.

IOStreams raises these classes itself:

What each storage raises

The exception that each storage raises, which is tagged with the kind of failure:

Kind of failure Local file S3 SFTP HTTP
NotFound Errno::ENOENT, or Errno::ENOTDIR below a file Aws::S3::Errors::NoSuchKey, NoSuchBucket, NoSuchVersion, or NotFound from a HEAD request IOStreams::Errors::CommunicationsFailure when reading or writing, or Net::SFTP::StatusException IOStreams::Errors::CommunicationsFailure for 404 Not Found or 410 Gone
PermissionDenied Errno::EACCES or Errno::EPERM Aws::S3::Errors::AccessDenied, InvalidAccessKeyId, SignatureDoesNotMatch, ExpiredToken and the like, Forbidden from a HEAD request, or Aws::Errors::MissingCredentialsError IOStreams::Errors::CommunicationsFailure when reading or writing, Net::SFTP::StatusException, or Net::SSH::AuthenticationFailed IOStreams::Errors::CommunicationsFailure for 401 Unauthorized, 403 Forbidden or 407 Proxy Authentication Required
Unavailable Never Seahorse::Client::NetworkingError when S3 cannot be reached, or Aws::S3::Errors::SlowDown, InternalError, ServiceUnavailable, RequestTimeout, or a 429, 500, 502, 503 or 504 response, once the AWS SDK has retried the request IOStreams::Errors::CommunicationsFailure when reading or writing, a connection error such as Errno::ECONNREFUSED or SocketError, or Net::SSH::Disconnect A connection error such as Errno::ECONNREFUSED, SocketError or Net::ReadTimeout, or IOStreams::Errors::CommunicationsFailure for 408, 429, 500, 502, 503 or 504

A host name that does not resolve, such as a mistyped host, is not Unavailable, see Notes.

SFTP reads and writes files with the sftp program, and uses the net-sftp gem for everything else, such as #each_child, #exist? and #delete.

A failure that is none of these kinds raises its exception without a tag, such as an SFTP host key that does not match, or an HTTPS certificate that is not trusted.

What each stream raises

The exception that each stream raises when it cannot read the data, which is tagged with the kind of failure:

Stream Corrupt MissingEntry MissingKey
.gz, .gzip Zlib::GzipFile::Error, such as its Zlib::GzipFile::CRCError or NoFooter, or Zlib::DataError Never Never
.bz2 Bzip2::FFI::Error::DataError, such as its MagicDataError, or Bzip2::FFI::Error::UnexpectedEofError, or IOStreams::Errors::StreamFailure for empty data Never Never
.zst RuntimeError Never Never
.zip IOStreams::Errors::StreamFailure for a file whose size or CRC32 is not the one that the zip file records. For a file that is not a zip file, or data that cannot be decompressed, Zip::Error from rubyzip on MRI, such as Zip::DecompressionError, or, from rubyzip 3.7, NoMethodError, TypeError or ArgumentError for a zip file whose end record is cut short, and IOStreams::Errors::StreamFailure on JRuby, which reads zip files with Java IOStreams::Errors::StreamFailure Never
.enc OpenSSL::Cipher::CipherError, or the exceptions of .gz for compressed data, or IOStreams::Errors::StreamFailure for empty data Never SymmetricEncryption::CipherError for a version of the cipher that is not configured
.xlsx, .xlsm Zip::Error from rubyzip, which reads the spreadsheet, or Errno::ENOENT for a zip file without the parts of a spreadsheet, or IOStreams::Errors::StreamFailure for a part whose size or CRC32 is not the one that the spreadsheet records Never Never
.pgp, .gpg IOStreams::Pgp::Failure for data that is not PGP data, is cut short, including a signature that is cut short, which gpg itself does not report, or was changed, such as integrity protection or a signature that does not match it, or for data without integrity protection, unless ignore_mdc_error: true is supplied Never IOStreams::Pgp::Failure for a secret key that is not in the keyring, a passphrase that is missing or wrong, or the public key of a signature that is not in the keyring
.parquet RuntimeError from the parquet gem, which raises it for a file that is not Parquet, or is damaged or cut short Never Never

The PGP stream tells why gpg failed from gpg’s status, which does not depend on the locale. A PGP file that was not signed by the signer, or that gpg does not trust, raises IOStreams::Pgp::Failure without a kind of failure, since only PGP checks who signed a file.

Rescue the kind of failure rather than these exceptions: they belong to the gem of the stream, which a newer version of IOStreams can replace with another, and the zip stream already raises different ones on MRI and JRuby.

The exception is kept

The tag does not replace the exception, so its class and attributes stay the same, and code that rescues it by its own class still works:

begin
  IOStreams.path("s3://my-bucket/imports/orders.csv").read
rescue IOStreams::Errors::NotFound => e
  e.class        # => Aws::S3::Errors::NoSuchKey
  e.code         # => "NoSuchKey"
  e.display_name # => "s3://my-bucket/imports/orders.csv"
  e.message      # => "s3://my-bucket/imports/orders.csv: The specified key does not exist."
end

The same holds for the failure of a stream, which keeps the class that the gem raised, as in v2.1:

begin
  IOStreams.path("orders.csv.gz").read
rescue IOStreams::Errors::Corrupt => e
  e.class   # => Zlib::GzipFile::Error
  e.message # => "not in gzip format"
end

Examples

Skip a file that has not arrived yet, and process it on the next run:

def import(path)
  path.each(:hash) { |record| Order.import(record) }
  path.delete
rescue IOStreams::Errors::NotFound
  logger.info("Waiting for #{path.display_name}")
end

Check that a file exists before reading it. #exist? returns false for a file that does not exist on every storage, instead of raising. It still raises PermissionDenied when the storage does not permit the check:

path.read if path.exist?

Likewise #size raises NotFound for a file that does not exist, like File.size, while #size? returns nil, like File.size?:

path.size  # raises IOStreams::Errors::NotFound when the file does not exist
path.size? # => nil when the file does not exist, or is empty

Report a problem with the credentials or permissions, which retrying will not fix:

begin
  IOStreams.path(ENV.fetch("EXPORT_PATH")).write(report)
rescue IOStreams::Errors::PermissionDenied => e
  raise(ConfigurationError, "Check the credentials and permissions for #{e.display_name}")
end

The exception from the storage is the #cause of the ConfigurationError, so it is still logged with it.

Retry when the storage is unavailable, such as a server that is restarting:

attempts = 0
begin
  IOStreams.path(ENV.fetch("EXPORT_PATH")).write(report)
rescue IOStreams::Errors::Unavailable => e
  attempts += 1
  raise if attempts >= 3

  logger.warn("Retrying in #{2**attempts} seconds: #{e.message}")
  sleep(2**attempts)
  retry
end

Writing again is safe: S3 and HTTP only store a file once it has been written completely, a local file that fails part way is removed, and SFTP uploads the whole file again, replacing the part that an interrupted upload left on the server. Retry only on Unavailable, and not on NotFound or PermissionDenied, which retrying will not fix, or on a host name that does not resolve, which is not Unavailable.

Log any failure of the storage, with the path that failed:

begin
  IOStreams.join("exports", "daily.csv").write(report)
rescue IOStreams::Errors::StorageError => e
  logger.error("Export to #{e.display_name} failed", e)
  raise
end

AccessDenied is not a PermissionDenied

IOStreams raises IOStreams::Errors::AccessDenied itself, before it accesses the storage, when a path is not within the allowed paths of the application, see Allowed paths. IOStreams::Errors::PermissionDenied is a failure of the storage, such as a file that the credentials cannot read.

They are kept apart, so that rescuing PermissionDenied, for example to report a configuration problem, does not also rescue an attempt to access a path that the application does not allow, such as a file name supplied by a user.

Notes

Storage registered with IOStreams.register_scheme

A path class registered with IOStreams.register_scheme can tag the failures of its storage the same way, see Failures of the store.

Formats registered with IOStreams.register_extension

A reader registered with IOStreams.register_extension can tag the failures of the library that reads its format the same way, see Failures of the format.