Copying Between Files

File copying can be used to:

• copy from one storage location to another.
• create a decrypted / encrypted copy of an existing file.
• create a decompressed / compressed copy of an existing file.

Examples

Decompress example.csv.gz into example.csv:

source = IOStreams.path("example.csv.gz")
IOStreams.path("example.csv").copy_from(source)

Decrypt a file encrypted with Symmetric Encryption:

source = IOStreams.path("example.csv.enc")
IOStreams.path("example.csv").copy_from(source)

Encrypt a file using PGP encryption so that it can only be read by receiver@example.org.

source = IOStreams.path("example.csv")
target = IOStreams.path("example.csv.pgp")
target.option(:pgp, recipient: "receiver@example.org")
target.copy_from(source)

When the file name does not have file extensions that would allow IOStreams to infer what streams to apply, the streams can be explicitly set using stream.

In this example, the file CUSTOMER_DATA has no extensions, so stream(:enc) tells IOStreams that its contents were encrypted with Symmetric Encryption. The decrypted contents are then PGP encrypted and written to xyz.csv.pgp using the pgp key for receiver@example.org.

input = IOStreams.path("CUSTOMER_DATA").stream(:enc)
IOStreams.path("xyz.csv.pgp").option(:pgp, recipient: "receiver@example.org").copy_from(input)

To copy a file without performing any conversions (ignore file extensions), set convert to false:

input = IOStreams.path("sample.json.zip")
IOStreams.path("sample.copy").copy_from(input, convert: false)

Custom stream conversions can be applied to both the source and the target in a single copy. Here the source is read as binary and the target is PGP encrypted:

source = IOStreams.path("source_file").encoding("BINARY")
IOStreams.path("target_file.pgp").option(:pgp, passphrase: "hello").copy_from(source)

To convert the contents row by row, or record by record, during the copy, supply mode. For example, copy a CSV file into JSON, parsing and rendering each record:

source = IOStreams.path("source_file.csv")
IOStreams.path("target_file.json").copy_from(source, mode: :hash)

Notes:

• mode accepts :line, :array, or :hash, and only applies when convert is true. Any other mode raises ArgumentError before the source or the target is opened.
• Without a mode, a copy into a file of another format, such as people.csv into people.json or people.psv, copies the text of the source as it is, so that the target holds CSV text. This is deprecated, and reports a deprecation warning, since IOStreams 3.0 converts the rows of such a copy into the format of the target, as the :array mode does: supply mode: :array to convert the rows, or mode: :hash when either file is JSON, whose rows are records, or mode: :line to copy the text as it is, which give the same result in both versions. A copy within a format, such as .csv.gz into .csv, from or to a file whose format is not known, such as an IO or a .txt file, or with #raw, is copied as it is in both, and so is a spreadsheet into CSV, since its text is CSV.

When a copy fails

• The source is opened before the target, so a source that does not exist, or cannot be read, raises before the target is changed.
• A copy that raises before the target is opened, such as for a schema file that does not exist, leaves the target as it was.
• When the copy fails part way, the target is not left with part of the file: a local file is removed, and an S3 or SFTP file is only uploaded once the copy has written all of it, so an existing S3 object is kept.
• A move only deletes the source once the copy completed.