# IOStreams > IOStreams is a Ruby gem for streaming I/O that makes file formats, compression, encryption, > and storage mechanisms (local file, S3, SFTP, HTTP) transparent to the application. Files of > any size are processed one block at a time without loading them into memory. The public entry points are `IOStreams.path(...)` and `IOStreams.stream(io)`, which are configured via chained `#stream`/`#option` calls and consumed via `#reader`, `#writer`, `#each`, `#read`, `#write`, `#copy_from`, etc. # What is IOStreams? IOStreams is a streaming library for Ruby that makes compression, encryption, file format, and storage location transparent to your code. Read and write files as if they were plain, local text, whether they are gzip, zip, or PGP encrypted, and whether they live on local disk, AWS S3, SFTP, or are fetched over HTTP. ## Why IOStreams? Processing files in Ruby usually means writing different code for every variation. One customer sends a gzip compressed CSV, another sends a PGP encrypted file, a third uploads an Excel spreadsheet. The files sit on local disk in development, but in AWS S3 in production. Each combination needs its own handling, and reading a large file into memory all at once risks exhausting it. Consider reading a gzip compressed CSV file from S3, one record at a time, _without_ IOStreams: ~~~ruby require "aws-sdk-s3" require "zlib" require "csv" response = Aws::S3::Client.new.get_object(bucket: "my-bucket", key: "data.csv.gz") gz = Zlib::GzipReader.new(response.body) headers = nil gz.each_line do |line| row = CSV.parse_line(line) if headers.nil? headers = row else record = headers.zip(row).to_h # ... process record ... end end ~~~ Switch that file to plain CSV, to PGP encrypted, or move it back to local disk, and this code has to change every time. With IOStreams the same single line handles all of them: ~~~ruby IOStreams.path("s3://my-bucket/data.csv.gz").each(:hash) do |record| # ... process record ... end ~~~ IOStreams reads the file name, `data.csv.gz`, infers that it is a gzip compressed CSV, and assembles the streaming pipeline to fetch, decompress, and parse it. Point the same code at `data.csv`, `data.csv.pgp`, or a local path instead, and nothing else changes. ### One API, whatever the format, compression, or encryption IOStreams detects the file type from its extensions and applies the matching streams in order, so `sample.csv.gz.pgp` is decrypted, then decompressed, then parsed as CSV without a line of special handling. The same code that reads a plain CSV also reads an Excel spreadsheet, a PGP encrypted JSON file, or a pipe separated file. A single background job can ingest files from many senders, in many formats, with no per-format code. ### One API, wherever the file is stored Local disk, AWS S3, Google Cloud Storage, SFTP, and HTTP(S) all share the same `IOStreams.path` interface. The only thing that changes between them is the file name, or nothing at all when you [configure roots](config). ### Constant memory, even for huge files Everything is streamed a block at a time, so a 10 GB compressed file uses about as much memory as a 10 KB one. Scaling up to large files is trivial: the code that processes ten rows processes ten million without changes. ### Configure storage once, switch environments with no code change With [roots](config), the storage location lives in a startup initializer instead of being scattered through the code. Point the `:default` root at the local file system in development and at an S3 bucket in production, and the exact same application code runs in both. A path can also come straight from an environment variable or a configuration system. See [Paths are configuration](config#paths-are-configuration). ### Capabilities * Low memory utilization, even when processing very large files. * Parse JSON, CSV, PSV, or fixed width data on the fly. * Encrypt / Decrypt data on the fly. * Compress / Decompress data on the fly. * Change storage location / mechanism transparently without any code changes. #### File Extensions * Zip * Gzip * BZip2 * PGP (Requires GnuPG) * Xlsx (Reading) * Encryption using [Symmetric Encryption](https://encryption.reidmorrison.com/) #### File Storage * File * AWS S3 * Google Cloud Storage (Using the AWS S3 Client) * SFTP * HTTP(S) (Read with GET, write with PUT) #### File formats * CSV * Fixed width formats * JSON * PSV ## Example usages ### Creating files Write a string to a local file called `sample.txt`: ~~~ruby path = IOStreams.path("sample.txt") path.write("Hello World") ~~~ Write a string to AWS S3, storing in the S3 bucket `sample-bucket`, under the path `demo` with a file name of `sample.txt`. ~~~ruby path = IOStreams.path("s3://sample-bucket/demo/sample.txt") path.write("Hello World") ~~~ Write a string into a compressed file by adding the `.gz` extension to the file name: ~~~ruby path = IOStreams.path("sample.txt.gz") path.write("Hello World") ~~~ Compress and encrypt the data into a PGP encrypted file, called `sample.txt.pgp`: ~~~ruby path = IOStreams.path("sample.txt.pgp") # Recipient that can decrypt this file: path.option(:pgp, recipient: "receiver@example.org") path.write("Hello World") ~~~ Note: GnuPG needs to be installed locally for the above PGP example to work. Write a string to a SFTP server, with a host name of `example.org`, under the path `demo`, with a file name of `sample.txt`. Adds the optional `username` and `password`. ~~~ruby path = IOStreams.path("sftp://example.org/demo/sample.txt", username: "example", password: "topsecret") path.write("Hello World") ~~~ Write a string to AWS S3, storing in the S3 bucket `sample-bucket`, under the path `demo`, encrypted with pgp, with a file name of `sample.txt.pgp`. ~~~ruby path = IOStreams.path("s3://sample-bucket/demo/sample.txt.pgp") path.option(:pgp, recipient: "receiver@example.org") path.write("Hello World") ~~~ ### Reading files Read an entire local file called `sample.txt`, into a string: ~~~ruby path = IOStreams.path("sample.txt") path.read # => "Hello World" ~~~ Read an entire file called `sample.txt`, into a string, from the S3 bucket `sample-bucket`, under the path `demo`: ~~~ruby path = IOStreams.path("s3://sample-bucket/demo/sample.txt") path.read # => "Hello World" ~~~ Read an entire local file called `sample.txt.gz`, and decompress the contents into a string: ~~~ruby path = IOStreams.path("sample.txt.gz") path.read # => "Hello World" ~~~ Read an entire local file called `sample.txt.pgp`, decompress, and decrypt the contents into a string: ~~~ruby path = IOStreams.path("sample.txt.pgp") path.read # => "Hello World" ~~~ Notes: * GnuPG needs to be installed locally for the above PGP example to work. ## Streaming Examples When dealing with large files it is important _not_ to load the entire file into memory. Efficiently read the files data in chunks / lines / records. Read 128 characters at a time from a file: ~~~ruby path = IOStreams.path("sample.txt") path.reader do |io| while (data = io.read(128)) p data end end ~~~ Read one line at a time from the file: ~~~ruby path = IOStreams.path("sample.txt") path.each do |line| puts line end ~~~ Write data to the file. ~~~ruby path = IOStreams.path("sample.txt") path.writer do |io| io << "This" io << " is " io << " one line\n" end ~~~ Write lines to the file. By adding `:line` to `writer`, each write appends a new line character. ~~~ruby path = IOStreams.path("sample.txt") path.writer(:line) do |file| file << "these" file << "are" file << "all" file << "separate" file << "lines" end ~~~ ### Reading CSV Files Example CSV file, `example.csv`: ~~~csv name,address,zip_code Jack,There,1234 Joe,Over There somewhere,1234 ~~~ Read each line from the CSV file as lines of strings: ~~~ruby path = IOStreams.path("example.csv") path.each do |line| p line end ~~~ Output: ~~~ruby "name,address,zip_code" "Jack,There,1234" "Joe,Over There somewhere,1234" ~~~ Read each row from the CSV file as arrays: ~~~ruby path = IOStreams.path("example.csv") path.each(:array) do |array| p array end ~~~ Output: ~~~ruby ["name", "address", "zip_code"] ["Jack", "There", "1234"] ["Joe", "Over There somewhere", "1234"] ~~~ Read each row from a csv file as key-value pairs, where the key is the CSV column header, and the value is the value for that row. ~~~ruby path = IOStreams.path("example.csv") path.each(:hash) do |record| p record end ~~~ Output: ~~~ruby {"name"=>"Jack", "address"=>"There", "zip_code"=>"1234"} {"name"=>"Joe", "address"=>"Over There somewhere", "zip_code"=>"1234"} ~~~ ### Writing CSV Files Write an array (row) at a time to the file. Each array is converted to csv before being written to the file. ~~~ruby IOStreams.path("example.csv").writer(:array) do |io| io << ["name", "address", "zip_code"] io << ["Jack", "There", "1234"] io << ["Joe", "Over There somewhere", 1234] end ~~~ Write a hash (record) at a time to the file. Each hash is converted to csv before being written to the file. The header row is extracted from the first hash write that is performed. ~~~ruby path = IOStreams.path("example.csv") path.writer(:hash) do |stream| stream << {name: "Jack", address: "There", zip_code: 1234} stream << {zip_code: 1234, address: "Over There somewhere", name: "Joe"} end ~~~ This time write the CSV data to a compressed zip file, by adding `.zip` to the file name. ~~~ruby path = IOStreams.path("example.csv.zip") path.writer(:hash) do |stream| stream << {name: "Jack", address: "There", zip_code: 1234} stream << {zip_code: 1234, address: "Over There somewhere", name: "Joe"} end ~~~ Changing the file name to change its compression, encryption, or even whether it is local or remote has no effect on the code reading from or writing to the path. ## PSV Files PSV files are faster than CSV files, since CSV files have complex rules for dealing with embedded quotes and newlines. PSV files in IOStreams follow the following simple rules: * Values are delimited using `|`. * Rows are delimeted with new lines. * Values may _not_ contain `|`, or new lines. Example PSV file, `example.psv`: ~~~csv name|address|zip_code Jack|There|1234 Joe|Over There somewhere|1234 ~~~ ### Reading PSV Files Read each row from a psv file as key-value pairs, where the key is the PSV column header, and the value is the value for that row. ~~~ruby path = IOStreams.path("example.psv") path.each(:hash) do |record| p record end ~~~ Output: ~~~ruby {"name"=>"Jack", "address"=>"There", "zip_code"=>"1234"} {"name"=>"Joe", "address"=>"Over There somewhere", "zip_code"=>"1234"} ~~~ ### Writing PSV Files Write a hash (record) at a time to the file. Each hash is converted to psv before being written to the file. The header row is extracted from the first hash write that is performed. ~~~ruby path = IOStreams.path("example.psv") path.writer(:hash) do |stream| stream << {name: "Jack", address: "There", zip_code: 1234} stream << {zip_code: 1234, address: "Over There somewhere", name: "Joe"} end ~~~ ## Getting Started Start with the [IOStreams tutorial](tutorial) for a great introduction to IOStreams. --- # File / Data Streaming with Ruby If all files were small, they could just be loaded into memory in their entirety. However, multi Gigabytes, or even Terabytes in size, loading them into memory is not feasible. In linux it is common to use pipes to stream data between processes. For example: ~~~ # Count the number of lines in a file that has been compressed with gzip cat abc.gz | gunzip -c | wc -l ~~~ For large files it is critical to be able to read and write these files as streams. Ruby has support for reading and writing files using streams, but has no built-in way of passing one stream through another to support for example compressing the data, encrypting it and then finally writing the result to a file. Several streaming implementations exist for languages such as `C++` and `Java` to chain together several streams, `IOStreams` offers similar features for Ruby. ~~~ruby # Read the first 1024 characters from a compressed file: path = IOStreams.path("hello.gz") path.reader do |io| data = io.read(1024) puts "Read: #{data}" end ~~~ The true power of streams is shown when many streams are chained together to achieve the end result, without holding the entire file in memory, or ideally without needing to create any temporary files to process the stream. ~~~ruby # Create a file that is compressed with GZip and then encrypted with Symmetric Encryption: path = IOStreams.path("hello.gz.enc") path.writer do |io| io << "Hello World" io << "and some more" end ~~~ The power of the above example applies when the data being written starts to exceed hundreds of megabytes, or even gigabytes. By looking at the file name supplied above, IOStreams is able to determine which streams to apply to the data being read or written. For example: * `hello.zip` => Compressed using Zip * `hello.zip.enc` => Compressed using Zip and then encrypted using Symmetric Encryption * `hello.gz.enc` => Compressed using GZip and then encrypted using Symmetric Encryption The objective is that all of these streaming processes are performed used streaming so that only the current portion of the file is loaded into memory as it moves through the entire file. ## Step by Step Install IOStreams gem: ~~~ gem install iostreams --no-doc ~~~ If you want to follow the AWS S3 examples below install the AWS S3 gem: ~~~ gem install aws-sdk-s3 --no-doc ~~~ See [Configuring the AWS SDK for Ruby](https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html) to configure the Ruby AWS library. Open a ruby interactive console: ~~~ irb ~~~ Load iostreams: ~~~ruby require "iostreams" ~~~ Reference a file path to hold CSV data and then for fun lets also compress it with GZip: ~~~ruby path = IOStreams.path("sample/example.csv.gz") # => #{}}> ~~~ The path and file name does not exist yet: ~~~ruby path.exist? # => false ~~~ If the path `sample` does not exist, it is created automatically during the first write. Write CSV data to the file, compressing to GZip as we go. ~~~ruby path.writer do |io| io << "name,login\n" io << "Jack Jones,jjones\n" io << "Jill Smith,jsmith\n" end ~~~ To verify the data written above, read the entire file: ~~~ruby path.read # => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n" ~~~ It would be much easier if we could write the CSV data as hashes and let IOStreams deal with all the details on how to create properly formatted CSV data: ~~~ruby path.writer(:hash) do |io| io << {name: "Jack Jones", login: "jjones"} io << {name: "Jill Smith", login: "jsmith"} end ~~~ Verify the data written by reading the entire file: ~~~ruby path.read # => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n" ~~~ Now lets read the file one line at a time: ~~~ruby path.each do |line| puts line end ~~~ Output: ~~~ name,login Jack Jones,jjones Jill Smith,jsmith ~~~ But who wants to do CSV parsing by hand, lets get IOStreams to do that for us by passing `:array` to `each`: ~~~ruby path.each(:array) do |array| p array end ~~~ Output: ~~~ruby ["name", "login"] ["Jack Jones", "jjones"] ["Jill Smith", "jsmith"] ~~~ That was better, but we really want a hash back where IOStreams takes care of the CSV header: ~~~ruby path.each(:hash) do |hash| p hash end ~~~ Output: ~~~ruby {"name"=>"Jack Jones", "login"=>"jjones"} {"name"=>"Jill Smith", "login"=>"jsmith"} ~~~ As the file gets larger and we reach millions of rows the above code does not have to change at all. And memory utilization stays about the same since each block is read in, decompressed, and parsed from CSV one block at a time. The garbage collector can then free the released blocks from memory. Now lets read a zip file hosted on an HTTP Web Server, displaying the first row as a hash: ~~~ruby IOStreams. path("https://www5.fdic.gov/idasp/Offices2.zip"). option(:zip, entry_file_name: "OFFICES2_ALL.CSV"). each(:hash) do |row| p row # Just show the first line for this tutorial break end ~~~ Output: ~~~ruby {"address"=>"1 Lincoln St. Fl 1", "bkclass"=>"SM", "cbsa"=>"Boston-Cambridge-Newton, MA-NH", "cbsa_div"=>"Boston, MA", "cbsa_div_flg"=>"1", "cbsa_div_no"=>"14454", "cbsa_metro"=>"14460", "cbsa_metro_flg"=>"1", "cbsa_metro_name"=>"Boston-Cambridge-Newton, MA-NH", "cbsa_micro_flg"=>"0", "cbsa_no"=>"14460", "cert"=>"14", "city"=>"Boston", "county"=>"Suffolk", "csa"=>"Boston-Worcester-Providence, MA-RI-NH-CT", "csa_flg"=>"1", "csa_no"=>"148", "estymd"=>"1792-01-01", "fi_uninum"=>"6", "mainoff"=>"1", "name"=>"State Street Bank And Trust Company", "offname"=>"State Street Bank And Trust Company", "offnum"=>nil, "rundate"=>"2020-05-14", "servtype"=>"11", "stalp"=>"MA", "stcnty"=>"25025", "stname"=>"Massachusetts", "uninum"=>"6", "zip"=>"2111"} ~~~ Noticed that it took a while to return the first line? That is because `zip` requires the entire file to be downloaded before it can decompress anything in the file. And HTTP uses a push protocol when reading files, so it is downloaded automatically into a temp file behind the scenes so that we can read it as if it was a local file. ## Same Code - Varying File Types Lets define a method to write data to a file. ~~~ruby def write_lines(file_name) path = IOStreams.path(file_name) path.writer do |io| io << "name,login\n" io << "Jack Jones,jjones\n" io << "Jill Smith,jsmith\n" end end ~~~ Create some sample files to work with ~~~ruby write_lines("sample/example.csv") write_lines("sample/example.csv.gz") ~~~ For PGP files we also need to specify the recipient that can decrypt the file. ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, recipient: "receiver@example.org") write_lines(path) ~~~ `IOStreams.path` takes a string as its argument, it can also accept an existing instance of `IOStreams`. That allows the same method to accept the pgp recipient without having to pass the pgp specific recipient information as an argument to the method. Consider a simple method to display the contents of a file a line at a time prefixed with the line number within the file: ~~~ruby def show_lines(file_name) line_number = 1 path = IOStreams.path(file_name) path.each(:line) do |line| puts "[#{line_number}] #{line}" line_number += 1 end end ~~~ Lets read all of the files created above with the new `show_lines` method: ~~~ruby show_lines("sample/example.csv") show_lines("sample/example.csv.gz") show_lines("sample/example.csv.pgp") ~~~ Noticed how they all returned the exact same output, even though the first file was plain text, the second was compressed with Gzip and the third was encrypted with PGP. They all returned: ~~~ [1] name,login [2] Jack Jones,jjones [3] Jill Smith,jsmith ~~~ Now a program can be developed using IOStreams and then without any code changes is able read plain text, compressed, or encrypted files. ## Same Code - Any File Storage Using the unchanged `write_lines` and `show_lines` methods above, lets use them to read and write from S3. But how is that possible since our program / methods above were only tested against local files? Create the same sample files to work with, but this time on AWS S3 in a bucket name `my-iostreams-bucket` ~~~ruby write_lines("s3://my-iostreams-bucket/sample/example.csv") write_lines("s3://my-iostreams-bucket/sample/example.csv.gz") ~~~ For PGP files we also need to specify the recipient that can decrypt the file. ~~~ruby path = IOStreams.path("s3://my-iostreams-bucket/sample/example.csv.pgp") path.option(:pgp, recipient: "receiver@example.org") write_lines(path) ~~~ The only change to switch to S3 storage was to prefix the file name passed in with `s3://my-iostreams-bucket/`. Lets read all of the files created above with the new `show_lines` method: ~~~ruby show_lines("s3://my-iostreams-bucket/sample/example.csv") show_lines("s3://my-iostreams-bucket/sample/example.csv.gz") show_lines("s3://my-iostreams-bucket/sample/example.csv.pgp") ~~~ Noticed how they all returned the exact same output, even though the first file was plain text, the second was compressed with Gzip and the third was encrypted with PGP. They all returned: ~~~ [1] name,login [2] Jack Jones,jjones [3] Jill Smith,jsmith ~~~ Now a program can be developed using IOStreams and then without any code changes is able to read and write across multiple storage locations. ### Configure storage once, not in every call In the examples above we still wrote the `s3://my-iostreams-bucket/` prefix into every call, which means the code itself decides where files are stored. The real goal is for the application code to say nothing at all about storage, so the same code can run unchanged against local files in development and S3 in production. Roots make that possible. Configure the storage location once in a startup initializer: ~~~ruby # config/initializers/iostreams.rb (development) IOStreams.add_root(:default, "tmp/files") ~~~ Then use `IOStreams.join` instead of `IOStreams.path`, and never mention the storage location again: ~~~ruby def write_lines(file_name) path = IOStreams.join(file_name) path.writer do |io| io << "name,login\n" io << "Jack Jones,jjones\n" io << "Jill Smith,jsmith\n" end end write_lines("sample/example.csv") write_lines("sample/example.csv.gz") ~~~ To run the exact same code against S3 in production, change only the initializer: ~~~ruby # config/initializers/iostreams.rb (production) IOStreams.add_root(:default, "s3://my-iostreams-bucket/files") ~~~ The application code does not change at all. See [Configuring IOStreams](config) for multiple roots, for example separate locations for downloads, uploads, and reports. ## Tabular Files Tabular files are any files that start with a header row and then follows with rows of data with each row on a separate line. For example "example.csv" ~~~ name,login Jack Jones,jjones Jill Smith,jsmith ~~~ The first line contains the header: `name,login` Each subsequent line contains the data delimited by a special character such as `,` in the same order as the header. Another example is PSV (Pipe Separated Files) ~~~ name|login Jack Jones|jjones Jill Smith|jsmith ~~~ Of course these are simple examples and there are lots of rules on how to embed or escape the row or column delimiters. ### Reading Tabular Files When reading these files, IOStreams can handle the complexity of the files format and always return the data as a `hash`, or `array`. Lets create another method along the lines of `show_lines` above: ~~~ruby def show_rows(file_name) line_number = 1 path = IOStreams.path(file_name) path.each(:hash) do |row| puts "[#{line_number}] #{row.inspect}" line_number += 1 end end ~~~ The key difference is that `:hash` is being passed into `each` instead of `:line`. Using the sample files created above: ~~~ruby show_rows("sample/example.csv") ~~~ Outputs: ~~~ [1] {"name"=>"Jack Jones", "login"=>"jjones"} [2] {"name"=>"Jill Smith", "login"=>"jsmith"} ~~~ Notice how only 2 rows are returned, since the header row is not actual data, it is just the definition of the rows that follow. The same method works without changes regardless of where the file was stored, or whether it was encrypted or compressed. ~~~ruby show_rows("s3://my-iostreams-bucket/sample/example.csv") show_rows("s3://my-iostreams-bucket/sample/example.csv.gz") show_rows("s3://my-iostreams-bucket/sample/example.csv.pgp") ~~~ ### Writing Tabular Files Lets define a new method that uses a tabular api to write the data. ~~~ruby def write_tabular(file_name) path = IOStreams.path(file_name) path.writer(:hash) do |io| io << {"name"=>"Jack Jones", "login"=>"jjones"} io << {"name"=>"Jill Smith", "login"=>"jsmith"} end end ~~~ The key difference is that `:hash` is being passed into `writer` to indicate that it will receive hashes instead of raw data. Lets create a sample file, and then read it to compare its contents to the raw writer above. ~~~ruby write_tabular("sample/example.csv") IOStreams.path("sample/example.csv").read # => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n" ~~~ Note how the output file is identical to the one created above. Using `writer(:hash)` makes it easier to develop the application without regard for: - The order of columns - Missing columns - Specialized escaping of values to handle row or column delimiters Note: The first row written determines the column names as well as the order of the elements to be written. To supply the header columns up front, to set the order or to filter out which columns should be written to the target file, pass them to the writer: `path.writer(:hash, columns: ["name", "login"])`. Now lets write the same data into a JSON file, then read it to see what it looks like: ~~~ruby write_tabular("sample/example.json") puts IOStreams.path("sample/example.json").read # => "{\"name\":\"Jack Jones\",\"login\":\"jjones\"}\n{\"name\":\"Jill Smith\",\"login\":\"jsmith\"}\n" ~~~ Using the same `show_rows` method above to display the file line by line ~~~ruby show_rows("sample/example.json") ~~~ Outputs the same data even though the file is now json instead of the previous file that was csv: ~~~ [1] {"name"=>"Jack Jones", "login"=>"jjones"} [2] {"name"=>"Jill Smith", "login"=>"jsmith"} ~~~ The same method works without changes regardless of where the file was stored, or whether it was encrypted or compressed, or whether the format was csv or json. ~~~ruby show_rows("sample/example.csv") show_rows("sample/example.json") show_rows("s3://my-iostreams-bucket/sample/example.csv") show_rows("s3://my-iostreams-bucket/sample/example.json.gz") show_rows("s3://my-iostreams-bucket/sample/example.json.pgp") ~~~ ## Conclusion IOStreams makes it possible to write an application to a common api so that * the file can be accessed anywhere ( at least a local file, AWS S3, HTTP(S) and SFTP for now). * the application does not care if or how the file was compressed. * the application does not care if or how the file was encrypted. * the actual file storage mechanism can be determined at runtime, or per environment. * it is transparent whether the application receives an Excel Spreadsheet, CSV, or PSV formatted file. It just works with hashes when desired. IOStreams is an incredibly powerful streaming library that makes changes to file formats, compression, encryption, or storage mechanism transparent to the application. ## Next Steps Move onto the [IOStreams Path Documentation](path) to see how to create paths that support AWS S3, HTTP(S), or SFTP. Or jump straight into the [IOStreams Streams Documentation](streams) for detailed information on working with streams. Read the [IOStreams PGP Documentation](pgp) for a tutorial on how to work with PGP Encrypted files. --- # Path A path identifies _where_ a file is stored and how to reach it, so that the streaming pipeline knows where to read the data from or write it to. Create a path with `IOStreams.path`, passing the file name, which may also be a URI, followed by any arguments specific to that storage location. IOStreams infers the storage mechanism from the URI scheme, so the same call returns a local file path, an S3 path, an SFTP path, and so on, all sharing the identical interface. IOStreams supports accessing files in the following places: * File * AWS S3 * Google Cloud Storage (Using the AWS S3 Client) * SFTP * HTTP(S) (Read with GET, write with PUT) Are you using another cloud provider and want to add support for your favorite? Checkout the supplied [IOStreams S3 path provider](https://github.com/reidmorrison/iostreams/blob/main/lib/io_streams/paths/s3.rb) for an example of what is required. Pull requests welcome. ### File The simplest case is a file on the local disk: ~~~ruby path = IOStreams.path("somewhere/example.csv") ~~~ A `file://` url is also accepted. It is always absolute, so it has an empty host or `localhost`, and characters such as a space, `?` or `#` are percent-encoded. Supply a relative path without `file://`. ~~~ruby IOStreams.path("file:///home/user/my%20file.csv") # Same as: IOStreams.path("/home/user/my file.csv") ~~~ A file name of `~`, or starting with `~/`, is within the current user's home directory. This matches an [SFTP](#sftp-sftp) url, where `~` is the login directory, so the location of a file can be configured, for example in an environment variable, as a local file in one environment and on an SFTP server in another, without changing the code. A `file://` url can also start with `~`, although that is not part of the `file://` standard: ~~~ruby # The file data/example.csv within the home directory IOStreams.path("~/data/example.csv") # Same as: IOStreams.path("file://~/data/example.csv") IOStreams.path("file:///~/data/example.csv") # The file data/example.csv within the login directory on the SFTP server IOStreams.path("sftp://hostname/~/data/example.csv") ~~~ Only a leading `~` on its own is the home directory, so `~user/a.csv`, `~$Book1.xlsx` and `a/~/b.csv` are unchanged. Use `./~/example.csv` for a directory called `~` in the current directory. In code, `IOStreams.home` is the current user's home directory, `IOStreams.home("username")` is another user's, and `IOStreams.working_path` is the current working directory: ~~~ruby IOStreams.home.join("my file.csv") # Same as, when the home directory is /home/user: IOStreams.path("/home/user/my file.csv") IOStreams.working_path.join("my file.csv") ~~~ #### Optional Arguments: * `:create_path` set to false to stop IOStreams from automatically creating the output directories if they do not exist. Default: true ~~~ruby path = IOStreams.path("somewhere/example.csv.gz", create_path: false) ~~~ ### AWS S3 (s3://) If the supplied file name string includes a URI. For example if AWS is configured locally: ~~~ruby path = IOStreams.path("s3://bucket-name/path/example.csv") ~~~ #### Required Arguments: * url [String] Prefix must be: `s3://`, followed by bucket name, followed by key. Any query string in the url is added to the S3 request parameters, for example `s3://my-bucket-name/file_name.csv?acl=bucket-owner-full-control`. Examples: s3://my-bucket-name/file_name.txt s3://my-bucket-name/some_path/file_name.csv Security warning: do not interpolate an untrusted file name into the url, since a name such as `file.csv?acl=public-read` would set request parameters. Join it onto the path instead, which does not parse it as a query: `IOStreams.path("s3://my-bucket-name/uploads").join(untrusted_name)` #### Optional Arguments: * :access_key_id [String] AWS Access Key Id to use to access this bucket. * :secret_access_key [String] AWS Secret Access Key Id to use to access this bucket. * :region [String] The AWS region to connect to. Default: the region set in the environment variables or credential files. * :client [Aws::S3::Client | Hash] Supply the AWS S3 Client instance to use for this path. Or, when a Hash, build a new client using the hash parameters. ~~~ruby client = Aws::S3::Client.new(endpoint: "https://s3.test.com") path = IOStreams.path("s3://bucket/path/file_name.txt", client: client) # Or, pass the client parameters directly: path = IOStreams.path("s3://bucket/path/file_name.txt", client: {endpoint: "https://s3.test.com"}) ~~~ Writer specific options: * :acl [String] The canned ACL to apply to the object. * :cache_control [String] Specifies caching behavior along the request/reply chain. * :content_disposition [String] Specifies presentational information for the object. * :content_encoding [String] Specifies what content encodings have been applied to the object and thus what decoding mechanisms must be applied to obtain the media-type referenced by the Content-Type header field. * :content_language [String] The language the content is in. * :content_length [Integer] Size of the body in bytes. This parameter is useful when the size of the body cannot be determined automatically. * :content_md5 [String] The base64-encoded 128-bit MD5 digest of the part data. This parameter is auto-populated when using the command from the CLI. This parameted is required if object lock parameters are specified. * :content_type [String] A standard MIME type describing the format of the object data. * :expires [Time,DateTime,Date,Integer,String] The date and time at which the object is no longer cacheable. * :grant_full_control [String] Gives the grantee READ, READ\_ACP, and WRITE\_ACP permissions on the object. * :grant_read [String] Allows grantee to read the object data and its metadata. * :grant_read_acp [String] Allows grantee to read the object ACL. * :grant_write_acp [String] Allows grantee to write the ACL for the applicable object. * :metadata [Hash] A map of metadata to store with the object in S3. * :server_side_encryption [String] The Server-side encryption algorithm used when storing this object in S3 (e.g., AES256, aws:kms). * :storage_class [String] The type of storage to use for the object. Defaults to 'STANDARD'. * :website_redirect_location [String] If the bucket is configured as a website, redirects requests for this object to another object in the same bucket or to an external URL. Amazon S3 stores the value of this header in the object metadata. * :sse_customer_algorithm [String] Specifies the algorithm to use to when encrypting the object (e.g., AES256). * :sse_customer_key [String] Specifies the customer-provided encryption key for Amazon S3 to use in encrypting data. This value is used to store the object and then it is discarded; Amazon does not store the encryption key. The key must be appropriate for use with the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm header. * :sse_customer_key_md5 [String] Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321. Amazon S3 uses this header for a message integrity check to ensure the encryption key was transmitted without error. * :ssekms_key_id [String] Specifies the AWS KMS key ID to use for object encryption. All GET and PUT requests for an object protected by AWS KMS will fail if not made via SSL or using SigV4. Documentation on configuring any of the officially supported AWS SDKs and CLI can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version * :ssekms_encryption_context [String] Specifies the AWS KMS Encryption Context to use for object encryption. The value of this header is a base64-encoded UTF-8 string holding JSON with the encryption context key-value pairs. * :request_payer [String] Confirms that the requester knows that she or he will be charged for the request. Bucket owners need not specify this parameter in their requests. Documentation on downloading objects from requester pays buckets can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html * :tagging [String] The tag-set for the object. The tag-set must be encoded as URL Query parameters. (For example, "Key1=Value1") * :object_lock_mode [String] The object lock mode that you want to apply to this object. * object_lock_retain_until_date: [Time,DateTime,Date,Integer,String] The date and time when you want this object's object lock to expire. * object_lock_legal_hold_status: [String] The Legal Hold status that you want to apply to the specified object. ### SFTP (sftp://) If the supplied file name string includes the `sftp` URI. ~~~ruby path = IOStreams.path("sftp://hostname/path/example.csv") ~~~ IOStreams reads and writes SFTP files by shelling out to the `sftp` command line program, so it must be installed and on the `PATH`. When a password is supplied the `sshpass` program is also required to pass the password to `sftp`. Additionally the `net-sftp` gem must be added to the `Gemfile` to use `each_child`. `each_child` lists the files within the path's directory. `each_child` also needs the `ed25519` gem, and the `bcrypt_pbkdf` gem except on JRuby, when the server's host key or the identity key is an ed25519 key: ~~~ruby gem "net-sftp" gem "bcrypt_pbkdf", platform: :ruby gem "ed25519" ~~~ Reading and writing do not need these gems, since the `sftp` program supports ed25519 keys itself. Without them `each_child` fails as follows: * An ed25519 identity key raises `NotImplementedError`: "unsupported key type `ssh-ed25519'". * A `HostKey` that only contains the ed25519 key raises `Net::SSH::HostKeyUnknown`, even though the key is correct, because the ed25519 entry is skipped. The full output of `ssh-keyscan hostname` also includes the server's other key types, which work without these gems. The path in the url is absolute, so `sftp://hostname/path/example.csv` is `/path/example.csv`, and a url without a path, such as `sftp://hostname`, is the root directory `/`. On many servers the user is confined to their own directory, which is then `/`. To refer to a path within the login directory instead, start the path with `~`, as curl does, for example `sftp://hostname/~` to list the login directory: ~~~ruby # The file data/example.csv within the login directory path = IOStreams.path("sftp://hostname/~/data/example.csv") ~~~ Read a file from a remote sftp server. ~~~ruby IOStreams.path("sftp://example.org/path/file.txt", username: "jbloggs", password: "secret"). reader do |input| puts input.read end ~~~ Raises `IOStreams::Errors::CommunicationsFailure` when the file could not be read or written. Write to a file on a remote sftp server. ~~~ruby IOStreams.path("sftp://example.org/path/file.txt", username: "jbloggs", password: "secret"). writer do |output| output.write('Hello World') end ~~~ Display the contents of a remote file, supplying the username and password in the url. Note that `#to_s` then includes the password, so prefer the `username:` and `password:` arguments above: ~~~ruby IOStreams.path("sftp://jack:OpenSesame@test.com:22/path/file_name.csv").reader do |io| puts io.read end ~~~ Use an identity file instead of a password to authenticate: ~~~ruby path = IOStreams.path("sftp://test.com/path/file_name.csv", username: "jack", ssh_options: {IdentityFile: "~/.ssh/private_key"}) path.reader do |io| puts io.read end ~~~ Pass in the IdentityKey itself instead of a password to authenticate. For example, retrieve the identity key stored in Secret Config: ~~~ruby identity_key = SecretConfig.fetch("suppliers/sftp/identity_key") path = IOStreams.path("sftp://test.com/path/file_name.csv", username: "jack", ssh_options: {IdentityKey: identity_key}) path.reader do |io| puts io.read end ~~~ #### Required Arguments: * url [String] Prefix must be: `sftp://`, followed by host name, followed by file name. Format: "sftp:///" "sftp://username:password@hostname:22/path/file_name" A username and password supplied in the url remain part of it, so `#to_s` returns them, as does any log or error message that includes the path. To keep them out of logs, supply them with the `username:` and `password:` arguments instead. #### Optional Arguments: * username: [String] Name of user to login with. * password: [String] Password for the user. * ssh_options: [Hash] * IdentityFile [String] Path to the local identity (private key) file to authenticate with, instead of a password. * IdentityKey [String] The identity (private key) itself, supplied as a string. Under the covers the key is written to a temp file and then passed as `IdentityFile`. * HostKey [String] The expected SSH host key presented by the remote host, instead of storing it in the `known_hosts` file. It must contain the entire line that would be stored in `known_hosts`, including the hostname, ip address, key type and key value. The easiest way to generate the required value is with `ssh-keyscan hostname`. Under the covers the value is written to a temp file and then passed as `UserKnownHostsFile`. * Any other options supported by ssh_config. `man ssh_config` to see all available options. `each_child` lists files with the `net-sftp` gem instead of the `sftp` program, so it only supports these ssh options: `HostKey`, `IdentityKey`, `IdentityFile`, `UserKnownHostsFile`, `StrictHostKeyChecking`, `ConnectTimeout`, `ServerAliveInterval`, `ServerAliveCountMax` and `LogLevel`. Any other option raises `ArgumentError`. Unlike the `sftp` program, `net-sftp` needs the `ed25519` and `bcrypt_pbkdf` gems to use ed25519 host or identity keys, see above. Notes: * Since the `sftp` program operates on local files, reading from or writing to an SFTP path streams through a local temp file behind the scenes. ### HTTP (http://, https://) Read from a remote file over HTTP or HTTPS using an HTTP GET, and write to one using an HTTP PUT. `exist?` and `size` use an HTTP HEAD, and `delete` uses an HTTP DELETE. ~~~ruby IOStreams.path('https://www5.fdic.gov/idasp/Offices2.zip').read IOStreams.path('https://example.com/upload/report.csv', headers: {"Authorization" => "Bearer token"}).write(data) ~~~ Notes: * Since Net::HTTP download only supports a push stream, the data is streamed into a tempfile first. * Writing also streams into a tempfile first, which is uploaded in a single PUT request once the block completes, so that the server receives its size in the `Content-Length` header. The `Content-Type` is `application/octet-stream` unless supplied with `headers:`. Any 2xx response is treated as success. * Only a `307` or `308` redirect to the same scheme, host and port is followed when writing, and the file is uploaded again to its location. Other redirects change the request into a GET, which would discard the upload, and a redirect to another server would send it the data being uploaded, so they raise `IOStreams::Errors::CommunicationsFailure`. * `exist?` returns `false`, `size` returns `nil`, and `delete` does nothing when the server responds with `404 Not Found` or `410 Gone`. Any other unsuccessful response raises `IOStreams::Errors::CommunicationsFailure`, for example when the server does not support HEAD or DELETE. * `delete` follows redirects the same way as writing. `exist?` and `size` follow redirects the same way as reading. * `move_to` from an HTTP path downloads the file and then deletes it with an HTTP DELETE. `move_to` an HTTP path uploads the file and then deletes the source. * A redirect from `https` to `http` is not followed, when reading or writing, and raises `IOStreams::Errors::CommunicationsFailure`. * Each redirect that is followed is logged at info level via `IOStreams.logger`, without any user name, password or query string. #### Required Arguments: * url [String] Prefix must be: `http://`, or `https://` followed by host name, followed by path and file name. Also supports passing the username and password for basic authentication in the URI. Format: * http://hostname/path/file_name * https://username:password@hostname/path/file_name A username and password supplied in the url remain part of it, so `#to_s` returns them, as does any log or error message that includes the path. To keep them out of logs, supply them with the `username:` and `password:` arguments instead. #### Optional Arguments: * username: [String] When supplied, basic authentication is used with the username and password. * password: [String] Password to use use with basic authentication when the username is supplied. * parameters: [Hash] Query parameters to append to the url as a query string. For example, `parameters: {"type" => "csv"}` appends `?type=csv` to the url. * http_redirect_count: [Integer] Maximum number of http redirects to follow. Set to `0` to disable following redirects entirely. Default: `10` * allow_hosts: [String | Array] Optional allow-list of host names that may be contacted. It is applied both to the supplied url and to every redirect that is followed; a request to any other host raises `IOStreams::Errors::CommunicationsFailure`. Default: `nil` (any host is allowed). * maximum_file_size: [Integer] Optional maximum number of bytes to download. When the response body exceeds this size the download is aborted with an `IOStreams::Errors::CommunicationsFailure`. Only applies when reading: writing to a path with a `maximum_file_size` raises `ArgumentError`. Default: `nil` (no limit). * headers: [Hash] Optional headers to add to every request, when reading and when writing, for example `{"Authorization" => "Bearer token"}`, or `{"Content-Type" => "text/csv"}` when writing. Since any header may hold a credential, such as `X-Api-Key`, none of them are resent when a redirect points at a different scheme, host, or port. Default: `nil` (no additional headers). ~~~ruby path = IOStreams.path("http://hostname/path/example.csv") ~~~ #### Security: untrusted URLs (SSRF) Reading an HTTP(S) path causes the application to issue a request to the host named in the url. When the url, or any part of it, can be influenced by untrusted input, an attacker can point it at internal services or cloud metadata endpoints (Server Side Request Forgery). Because redirect targets are chosen by the remote server, validating only the url that is passed in is not sufficient: a trusted (or compromised) server can redirect the request to an internal address. IOStreams provides a few controls to reduce this exposure: * Restrict which hosts may be contacted, including across redirects: ~~~ruby IOStreams.path("https://supplier.example.com/report.csv", allow_hosts: ["supplier.example.com"]).read ~~~ * Disable redirects entirely for untrusted urls: ~~~ruby IOStreams.path(untrusted_url, http_redirect_count: 0).read ~~~ * Cap the download size to avoid unbounded (denial of service) responses: ~~~ruby IOStreams.path(untrusted_url, maximum_file_size: 50 * 1024 * 1024).read ~~~ Basic authentication credentials, and the supplied `headers:`, are only ever sent to the original host. They are not resent when a redirect points at a different scheme, host, or port, so a redirect cannot leak them to another server. A redirect from `https` to `http` is not followed, so the data cannot be read or changed in transit. When writing or deleting, a redirect is only followed to the same scheme, host and port, so a redirect cannot send the data being uploaded to another server, or delete a file on it. Each redirect that is followed is logged at info level via `IOStreams.logger`. For stronger guarantees, route these downloads through an egress proxy or network policy that blocks private, loopback, and link-local (cloud metadata) addresses. Similarly when using https: ~~~ruby path = IOStreams.path("https://hostname/path/example.csv") ~~~ This time IOStreams inferred that the file lives on an HTTP Server and returns `IOStreams::Paths::HTTP`. ### Path Operations Paths support common file operations, regardless of where the file is stored: ~~~ruby path = IOStreams.path("sample/example.csv") # Does the file exist? path.exist? # => true # Size of the file in bytes. path.size # => 64 # Size of the file in bytes, or nil when it does not exist or is empty, like `File.size?`. path.size? # => 64 # Is it a file, or a directory? path.file? # => true path.directory? # => false # Is it an empty file, or a directory without any children? path.empty? # => false # Delete the file. path.delete # Move the file to another path, returning the target path. path.move_to("sample/moved.csv") # Create the directory path, when it does not already exist. IOStreams.path("sample/data").mkpath ~~~ Inspect the components of a path's file name: ~~~ruby # The last component of the path. IOStreams.path("/home/gumby/work/ruby.rb").basename # => "ruby.rb" # Remove a specific suffix from the file name. IOStreams.path("/home/gumby/work/ruby.rb").basename(".rb") # => "ruby" # Remove any extension by supplying ".*". IOStreams.path("/home/gumby/work/ruby.rb").basename(".*") # => "ruby" # The directory portion of the path. IOStreams.path("a/b/d/test.rb").dirname # => "a/b/d" # The extension, including the leading period. IOStreams.path("a/b/d/test.rb").extname # => ".rb" # The extension, without the leading period. IOStreams.path("a/b/d/test.rb").extension # => "rb" ~~~ Notes: * `basename`, `dirname`, `extname`, and `extension` return `nil` when no file name was set. * A leading period on a dotfile is not treated as an extension, so `.profile` has no extension, while `.profile.sh` has the extension `sh`. * A file name ending in a period, such as `foo.`, returns an empty string for the extension. Iterate over the files in a path using a wildcard pattern: ~~~ruby IOStreams.path("sample").each_child("*.csv") do |child| puts child end # Recursively, including sub-directories: IOStreams.path("sample").each_child("**/*.csv") do |child| puts child end ~~~ `each_child` is also available directly on `IOStreams` when the pattern includes the full path: ~~~ruby IOStreams.each_child("sample/**/*.csv") { |child| puts child } ~~~ Notes: * These operations are supported by File, S3 and SFTP paths. HTTP paths support all of them except `each_child`. HTTP has no directories, so `directory?` is always false and `mkpath` does nothing. * S3 has no directories either, only keys that contain `/`. An S3 path is a directory when any key is within it, such as `a` and `a/b` for the key `a/b/c.csv`, or when a folder object exists for it, with a key ending in `/`, such as the folders created by the S3 console. An S3 directory is `empty?` when only its folder object exists. * `file?`, `directory?` and `empty?` are false when the path does not exist. * By default `each_child` patterns are case-insensitive and hidden files are excluded. Supply `case_sensitive: true` or `hidden: true` to change this behavior. * Supply `directories: true` to also return directories. S3 has no directories, only keys that contain `/`, so the directories within the keys are returned, such as `a` and `a/b` for the key `a/b/c.csv`. * `each_child` returns nothing when the path does not exist, or is a file. A directory below the path that cannot be read is skipped. * S3 and SFTP paths also yield the attributes of each child as the second argument to the block. These attributes are specific to each store, and are not available for local files: * S3 yields the attributes of the object in the listing, such as `:size`, `:last_modified` (a `Time`), `:etag` and `:storage_class`. A directory has empty attributes, unless a folder object exists for it. * SFTP yields the attributes returned by the server, such as `:size`, `:permissions` and `:mtime` (seconds since the epoch). Which attributes are present depends on the server. Code that runs against more than one store, for example where the path is configured, should only use the first argument, and use methods on the path, such as `size`, for anything else. ### Using root paths Roots allow paths to reference a particular root directory, so that all path names are appended to that root. By using `IOStreams.join` instead of `IOStreams.path`, the storage location is no longer embedded in the application code, it is configured once at startup. The primary purpose of roots is to allow the exact same code to run in production and development, yet use completely different data sources in each. For example, in production the root can point to an S3 bucket, while in development it points to the local file system. Roots are configured via an initializer at startup. Multiple roots can be setup, for example one for input files, another for output files, another for reports, etc. During development the roots can all point to a common location, while in production they could be completely different S3 buckets. For example, inside an initializer: ~~~ruby IOStreams.add_root(:default, "tmp/export") IOStreams.add_root(:ftp, "tmp/ftp") ~~~ `:default` is used whenever a root is not supplied when calling `IOStreams.join`: ~~~ruby # Uses the :default root: "tmp/export/sample/example.csv" path = IOStreams.join("sample", "example.csv") # Uses the :ftp root: "tmp/ftp/sample/example.csv" path = IOStreams.join("sample", "example.csv", root: :ftp) ~~~ The following code: ~~~ruby path = IOStreams.path("tmp/export", "sample", "example.csv") path.writer(:line) do |io| io << "Welcome" io << "To IOStreams" end ~~~ Can be reduced to: ~~~ruby path = IOStreams.join("sample", "example.csv") path.writer(:line) do |io| io << "Welcome" io << "To IOStreams" end ~~~ Most importantly the root path information and storage mechanism are externalized from the application code. For example, to make the above code write to S3 in production, change the initializer to: ~~~ruby IOStreams.add_root(:default, "s3://my-app-bucket-name/export") IOStreams.add_root(:ftp, "s3://my-app-ftp-bucket-name/ftp") ~~~ The code calling `IOStreams.join` does not change at all, see [Config](config) for more examples. ### Restricting access with allowed paths Roots make paths easy to build, but they do not stop a path from leaving the root, for example `IOStreams.join("../../etc/passwd")`. When file names come from untrusted input, such as a user or a job's configuration, add allowed paths in an initializer to restrict which paths IOStreams can access: ~~~ruby IOStreams.add_allowed_path("/var/my_app/uploads") IOStreams.add_allowed_path("s3://my-app-bucket-name/export") IOStreams.add_allowed_path("sftp://sftp.example.org/outbound") IOStreams.add_allowed_path("https://reports.example.org/daily") ~~~ Once any allowed path has been added, reading, writing, listing, deleting or otherwise accessing a path that is not within one of them raises `IOStreams::Errors::AccessDenied`: ~~~ruby IOStreams.path("/var/my_app/uploads/file.csv").read # => "..." IOStreams.path("/var/my_app/uploads/../secrets.yml").read # => IOStreams::Errors::AccessDenied # Check without raising: IOStreams.allowed_path?("/etc/passwd") # => false ~~~ Paths are normalized before they are compared: - Local file names are resolved to their real path, so neither `..` nor a symbolic link can be used to leave an allowed path. A relative allowed path is resolved against the current working directory when it is added. - S3 paths must be in the same bucket. Keys containing `.` or `..` segments are denied, since some services that implement the S3 API resolve them. - SFTP and HTTP paths must have the same host and port, and for HTTP the same scheme. `.` and `..` are resolved the way the server resolves them. Every HTTP redirect is checked as well. Notes: - By default no allowed paths are added, and every path is accessible. - `each_child` skips children that are not within the allowed paths, for example a symbolic link to a file elsewhere. - Temp files from `IOStreams.temp_file` are always accessible. - `IOStreams.allowed_paths` returns the normalized allowed paths, and `IOStreams.delete_allowed_path` removes one. - Paths from a scheme added with `IOStreams.register_scheme` are denied, unless its path class implements the private method `#allowed_location`. - A local file could be replaced with a symbolic link after it is checked but before it is opened. Do not allow paths where untrusted users can create files. --- # Streams Once you have a [path](path), you read from and write to it with a small, consistent set of methods. Reading and writing always happen a block, line, or record at a time, so memory use stays low no matter how large the file is. Choose how each chunk is delivered by passing a mode: the default streams raw data, `:line` yields one line at a time, `:array` yields each row as an array, and `:hash` yields each record as a hash keyed by the header row. Read 128 characters at a time from the file: ~~~ruby IOStreams.path("example.csv").reader do |io| while (data = io.read(128)) p data end end ~~~ Read one line at a time from the file: ~~~ruby IOStreams.path("example.csv").each do |line| puts line end ~~~ By default the line delimiter is auto-detected from the file, handling both Windows (`\r\n`) and Linux (`\n`) line endings. To break the file up by something other than its line endings, supply `delimiter`: ~~~ruby IOStreams.path("example.txt").each(:line, delimiter: "|") do |line| puts line end ~~~ When a line can contain embedded newlines, such as a CSV field wrapped in double quotes that spans multiple lines, supply `embedded_within` so those newlines are not treated as line endings: ~~~ruby IOStreams.path("example.csv").each(:line, embedded_within: '"') do |line| puts line end ~~~ Notes: * Newlines embedded within quoted fields are kept on the same line automatically when the resolved tabular format quotes its fields, such as CSV (detected from a `.csv` file name or set explicitly via `.format(:csv)`). `embedded_within` only needs to be supplied for quoted formats that are not auto-detected, or to override the quote character. * A file that is named `.csv` but is actually pipe-delimited can avoid quote parsing by declaring its real format with `.format(:psv)`, or by passing `embedded_within: nil` to disable it explicitly: ~~~ruby IOStreams.path("pipe_delimited.csv").format(:psv).each(:line) do |line| puts line end ~~~ Display each row from the csv file as an array: ~~~ruby IOStreams.path("example.csv").each(:array) do |array| p array end ~~~ Display each row from the csv file as a hash, where the first line in the CSV file is the header: ~~~ruby IOStreams.path("example.csv").each(:hash) do |hash| p hash end ~~~ Write data to the file. ~~~ruby IOStreams.path("abc.txt").writer do |io| io << "This" io << " is " io << " one line\n" end ~~~ Write lines to the file. By adding `:line` to `writer`, each write appends a new line character. ~~~ruby IOStreams.path("example.csv").writer(:line) do |file| file << "these" file << "are" file << "all" file << "separate" file << "lines" end ~~~ Write an array (row) at a time to the file. Each array is converted to csv before being compressed with zip. ~~~ruby IOStreams.path("example.csv").writer(:array) do |io| io << ["name", "address", "zip_code"] io << ["Jack", "There", "1234"] io << ["Joe", "Over There somewhere", 1234] end ~~~ Write a hash (record) at a time to the file. Each hash is converted to csv before being compressed with zip. The header row is extracted from the first hash write that is performed. ~~~ruby IOStreams.path("example.csv").writer(:hash) do |stream| stream << {name: "Jack", address: "There", zip_code: 1234} stream << {zip_code: 1234, address: "Over There somewhere", name: "Joe"} end ~~~ Notes * Any additional keys supplied during subsequent write operations will be ignored since the header row has already been written to the file. * The order of the header and values is determined by the order of the keys supplied during the first write. * The order of keys in the subsequent writes does not matter. Stream into an in-memory buffer, useful for testing. The original filename still needs to be supplied so that the streaming pipeline can still be inferred. ~~~ruby io = StringIO.new IOStreams.stream(io).file_name("example.csv.gz").writer(:hash) do |stream| stream << {name: "Jack", address: "There", zip_code: 1234} stream << {name: "Joe", zip_code: 1234, address: "Over There somewhere"} end puts io.string ~~~ Read a CSV file and write the output to an encrypted file in JSON format. ~~~ruby IOStreams.path("sample.json.enc").writer(:hash) do |output| IOStreams.path("sample.csv").each(:hash) do |record| output << record end end ~~~ Read a zip file hosted on a HTTP Web Server, returning each row as a hash: ~~~ruby IOStreams. path("https://www5.fdic.gov/idasp/Offices2.zip"). option(:zip, entry_file_name: "OFFICES2_ALL.CSV"). each(:hash) do |row| p row end ~~~ Notes: * By default IOStreams will read the first file in the zip file. * To choose a specific file name within the zip file, supply: `entry_file_name` ## Notes * Reading a Zip file requires the entire file to be available locally, so reading from a stream (for example S3 or HTTP) downloads it into a temp file first. Writing Zip is fully streamed, no temp file is required. * When writing, `entry_file_name` sets the name of the file entry within the zip file. It defaults to the file name without the `.zip` extension, so writing to `example.csv.zip` creates an entry named `example.csv`. * Gzip is still recommended over Zip for very large files, since Zip files can only be read via a local file. ## Pipeline If the file is compressed, the pipeline will infer the necessary streams that need to be applied to it: ~~~ruby path = IOStreams.path("somewhere/example.csv.gz") # => #{}}> path.pipeline # => {:gz=>{}} ~~~ The `pipeline` above includes `:gz` to indicate that the file should compressed / decompressed with GZip. `compressed?` and `encrypted?` report whether a stream in the pipeline compresses or encrypts the file, whether it was inferred from the file name or set with `stream`: ~~~ruby IOStreams.path("example.csv.gz.pgp").compressed? # => true IOStreams.path("example.csv.gz.pgp").encrypted? # => true IOStreams.path("tempfile2527").stream(:gz).compressed? # => true ~~~ Compression within an encrypted file, such as PGP or Symmetric Encryption, is not reported, since only the encrypted data records whether it was compressed. #### Option Each path supports several options which can be supplied using the `option` method. Set the options for a stream in the pipeline for this file. Each stream can only be applied once and is uniquely identified by its symbolic name. To see the pipeline of streams that IOStreams would infer: ~~~ruby IOStreams.path("example.pgp").pipeline # => {:pgp=>{}} IOStreams.path("example.gz").pipeline # => {:gz=>{}} IOStreams.path("example.gz.pgp").pipeline # => {:gz=>{}, :pgp=>{}} ~~~ If the relevant stream is not found for this file it is ignored. For example, if the file does not have a pgp extension then the pgp option is ignored. The names of the options are still checked, so a misspelled option raises an `ArgumentError` for any file, see [Options for reading and writing](#options-for-reading-and-writing). ~~~ruby IOStreams.path("example.csv.gz"). option(:pgp, passphrase: "receiver_passphrase"). read ~~~ This is great way to pass in stream specific options for when they are required, and to still support paths that do not use that stream. For example, the same code can support pgp encrypted, Symmetric Encryption encrypted, and plain text files. ~~~ruby IOStreams.path("example.csv.enc"). option(:pgp, passphrase: "receiver_passphrase"). read ~~~ To see the what value was previously set for a particular option: ~~~ruby path = IOStreams.path("example.pgp") path.option(:pgp, passphrase: "receiver_passphrase") path.setting(:pgp) # => {:passphrase=>"receiver_passphrase"} ~~~ #### Options for reading and writing The options for a stream are passed to its reader when reading and to its writer when writing, and the two directions usually accept different options. For example, the PGP writer needs the `recipient` to encrypt for, while the PGP reader needs the `passphrase` for the private key. So that a path can be written and then read with the same options, each direction ignores the options of the other direction that it does not need: ~~~ruby path = IOStreams.path("example.csv.pgp"). option(:pgp, recipient: "receiver@example.org", passphrase: "receiver_passphrase") path.write("name,login\nJack Jones,jjones\n") path.read ~~~ Reading ignores the options that only say how to write the file, such as the gzip compression `level`, or `compress` for `.enc`, whose header records whether the file was compressed. Writing ignores the options that only say how to read the file, such as the PGP `passphrase`. Some options apply to both: `entry_file_name` names the file within a zip file when writing, and chooses the file to read. The PGP `signer` signs the file when writing, and when reading requires that the file was signed by that key, while `import_and_trust_key` imports the recipient's key to encrypt for when writing, and when reading imports the sender's key and requires that the file was signed by it, see [PGP](pgp). An option that neither direction accepts, such as a misspelled one, raises an `ArgumentError` that lists the valid options as soon as it is set, even when the file name does not include that stream. So the same code reports it wherever it runs, rather than only where the path, for example from configuration, includes that extension: ~~~ruby IOStreams.path("example.csv").option(:enc, compres: false) # ArgumentError: Unknown option :compres for a :enc stream. # Valid options: :buffer_size, :version, :compress, :cipher_name, :header, :random_key, :random_iv. ~~~ #### Stream The `stream` method stops IOStreams from inferring the streams for this path and only uses the specified streams. For example when using a filename that does not have the necessary file extensions. In this case the file was compressed with Zip, so tell IOStreams to unzip it: ~~~ruby path = IOStreams.path("tempfile2527") path.stream(:zip) path.read ~~~ The above example could also be written as: ~~~ruby IOStreams.path("tempfile2527"). stream(:zip). read ~~~ Multiple streams can also be specified: ~~~ruby IOStreams.path("tempfile2527"). stream(:zip). stream(:pgp, passphrase: "receiver_passphrase"). read ~~~ Now that IOStreams is not inferring the pipeline from the filename, we can still see the above streams: ~~~ruby IOStreams.path("tempfile2527"). stream(:zip). stream(:pgp, passphrase: "receiver_passphrase"). pipeline # => {:zip=>{}, :pgp=>{:passphrase=>"receiver_passphrase"}} ~~~ In this example the file contains JSON data that was compressed with Zip, and since we want to read each row as a hash: ~~~ruby IOStreams.path("tempfile2527"). stream(:zip). each(:hash, format: :json) do |row| p row end ~~~ Alternatively if the original file name is available it can also be supplied allowing IOStreams to infer the above streams: ~~~ruby IOStreams.path("tempfile2527"). file_name("file.json.zip"). each(:hash) do |row| p row end ~~~ To see the what value was previously set for a particular stream: ~~~ruby path = IOStreams.path("tempfile2527") path.stream(:zip) path.stream(:pgp, passphrase: "receiver_passphrase") path.setting(:pgp) # => {:passphrase=>"receiver_passphrase"} ~~~ To ensure no streams are inferred or applied use stream `:none` ~~~ruby path = IOStreams.path("file.zip") path.stream(:none) path.read ~~~ --- # File Extensions IOStreams uses the extensions in the file name to determine which streams to apply when reading or writing a file. Multiple extensions are applied in order, so `sample.csv.gz.pgp` is first decrypted with PGP and then decompressed with GZip when read. Supported extensions: | Extension | Stream | Read | Write | Required gem / program | |:-----------------|:---------------------|:-----|:------|:----------------------------------| | `.bz2` | BZip2 | Yes | Yes | `bzip2-ffi` | | `.enc` | Symmetric Encryption | Yes | Yes | `symmetric-encryption` | | `.gz`, `.gzip` | GZip | Yes | Yes | None (Ruby standard library) | | `.zip` | Zip | Yes | Yes | `rubyzip` (read), `zip_kit` (write). On JRuby the built-in Java zip support is used for reading. | | `.pgp`, `.gpg` | PGP | Yes | Yes | GnuPG command line program (`gpg`) | | `.xlsx`, `.xlsm` | Excel Spreadsheet | Yes | No | `creek` | The gems above are soft dependencies: IOStreams does not require them for installation, they only need to be added to the `Gemfile` when the corresponding extension is used. ## Compression options GZip accepts a compression `level` when writing, from `0` (no compression) to `9` (best compression): ~~~ruby IOStreams.path("sample.csv.gz").option(:gz, level: 9).write(data) ~~~ BZip2 passes its options through to `bzip2-ffi`: `block_size` (`1` to `9`) and `work_factor` (`0` to `250`) when writing, and `small` and `first_only` when reading: ~~~ruby IOStreams.path("sample.csv.bz2").option(:bz2, block_size: 9).write(data) IOStreams.path("sample.csv.bz2").option(:bz2, small: true).read ~~~ Reading ignores the options for writing, and writing ignores the options for reading, so the same path can be written and then read. An option that neither accepts, such as a misspelled one, raises an `ArgumentError`. See [Options for reading and writing](streams#options-for-reading-and-writing). ## Reading an Excel Spreadsheet Each row in the spreadsheet is converted into a CSV line, so the regular `:line`, `:array`, and `:hash` modes apply: ~~~ruby IOStreams.path("spreadsheet.xlsx").each(:hash) do |record| p record end ~~~ Notes: * Since the underlying `creek` gem operates on files, when reading from a stream (for example S3 or HTTP) the contents are first downloaded into a temp file. * Writing xlsx files is not supported. ## Character encoding The special `:encode` stream converts the character encoding of the data being read or written. It is applied with `option` or `stream` rather than a file name extension: ~~~ruby IOStreams.path("sample.csv.gz"). option(:encode, encoding: "UTF-8", cleaner: :printable, replace: ""). each do |line| puts line end ~~~ Options: * `encoding: [String|Encoding]` The target encoding, for example `"UTF-8"`, `"US-ASCII"`, or `"ASCII-8BIT"`. Data read from a file, or other binary data, is treated as already being in this encoding, so its characters are kept and only invalid characters are replaced or raise an error. Data with another encoding, such as a Ruby string being written, is converted. Default: `"UTF-8"` * `replace: [String]` The character to replace with when a character is invalid, or cannot be converted to the target encoding. Default: nil (raise `Encoding::UndefinedConversionError` on invalid characters) * `cleaner: [nil|Symbol|Proc]` Cleanse the data. Built-in rules: * `:printable` removes all non-printable characters except `\r` and `\n`. * `:replace_non_printable` replaces all non-printable characters except `\r` and `\n` with the `replace` value, or an empty string when `replace` is nil. A Proc can also be supplied to perform custom cleansing; it is called with the data and the `replace` value after every read or write. Default: nil ## Registering a custom extension To add a new extension, register the format that reads and writes it, usually a module that answers `reader_class`, `writer_class`, `compressed?` and `encrypted?`, and extends `IOStreams::StreamFormat`, which checks the options supplied for the stream against those of its reader and writer. The reader and writer classes must implement `.open` that yields a stream implementing `#read` or `#write` respectively. See any of the streams under `lib/io_streams` for examples. ~~~ruby module MyXz extend IOStreams::StreamFormat def self.reader_class MyXz::Reader end def self.writer_class MyXz::Writer end # Whether data in this format is compressed, for `#compressed?` on a path or stream. def self.compressed? true end # Whether data in this format is encrypted, for `#encrypted?` on a path or stream. def self.encrypted? false end end IOStreams.register_extension(:xz, MyXz) ~~~ A format that is neither compressed nor encrypted can instead be registered with just its reader and writer classes: ~~~ruby IOStreams.register_extension(:xls, MyXls::Reader, MyXls::Writer) ~~~ To use a registered format for another extension, register the format that `IOStreams.extensions` returns: ~~~ruby IOStreams.register_extension(:tgz, IOStreams.extensions[:gz]) ~~~ Similarly, to support a new storage location, supply a Path class for its URI scheme. See [IOStreams::Paths::S3](https://github.com/reidmorrison/iostreams/blob/main/lib/io_streams/paths/s3.rb) for an example of what is required. ~~~ruby IOStreams.register_scheme(:gcs, MyGoogleCloudStoragePath) ~~~ When writing fails, the path class is responsible for not leaving an incomplete file behind, for example by writing to a temp file and only uploading it once it is complete, like the S3, SFTP and HTTP paths, or by removing the incomplete file, like the local file path. IOStreams does not delete the target of a failed write or copy. --- # File Formats When reading or writing rows (`:array`) or records (`:hash`), IOStreams converts each line to or from the file's tabular format. The following formats are supported: * `:csv` Comma Separated Values * `:psv` Pipe Separated Values * `:json` One JSON document per line * `:fixed` Fixed width columns * `:array` Each line is already an array of values * `:hash` Each line is already a hash PSV has no way to escape values, so when writing PSV, a `|` within a value is replaced with `:` and a line break with a space, so that a value cannot add columns or records. ## Format inference The format is inferred from the file name when it contains a recognized extension: ~~~ruby IOStreams.path("sample.csv").each(:hash) { |record| p record } IOStreams.path("sample.json").each(:hash) { |record| p record } IOStreams.path("sample.psv").each(:hash) { |record| p record } ~~~ The format extension can appear anywhere in the file name, so `sample.csv.gz` and `sample.json.pgp` are recognized as CSV and JSON respectively. When the file name does not contain a recognized format extension, the format defaults to `:csv`. ## Specifying the format When the file name cannot be used to infer the format, set it explicitly with `format`: ~~~ruby path = IOStreams.path("sample_data") path.format(:json) path.each(:hash) { |record| p record } ~~~ `format` can be chained with the other path methods: ~~~ruby IOStreams.path("sample_data").format(:json).each(:hash) { |record| p record } ~~~ ## Format options Format specific options are supplied with `format_options`. They are passed to the parser for the chosen format. The `:fixed` format requires its file layout to be supplied this way, as shown in the next section. The other formats do not currently take any options. ## Fixed width files Fixed width files have no delimiters; each column is identified by its position within the line. Since the layout cannot be inferred from the file, supply it using `format_options`: ~~~ruby path = IOStreams.path("sample_data") path.format(:fixed) path.format_options( layout: [ {size: 23, key: "name"}, {size: 40, key: "address"}, {size: 5, key: "zip"} ] ) path.each(:hash) { |record| p record } ~~~ Writing a fixed width file uses the same layout to render each record: ~~~ruby path = IOStreams.path("sample_data") path.format(:fixed) path.format_options( layout: [ {size: 23, key: "name"}, {size: 40, key: "address"}, {size: 5, key: "zip"} ] ) path.writer(:hash) do |io| io << {"name" => "Jack Jones", "address" => "Somewhere", "zip" => 12345} end ~~~ Note: The keys in the hashes being written must match the layout `:key` values exactly, including whether they are strings or symbols. Layout column definitions: * `:size` The number of characters this column occupies. The last column may use a size of `:remainder` to take the rest of the line as its value. * `:key` The name for this column. Leave out the key to ignore the column during parsing, and to space fill when rendering. * `:type` `:string` (default), `:integer`, or `:float`. Strings are left justified and space padded, numbers are right justified and zero padded. When writing, line breaks within a string are replaced with a space so that a value cannot add records. Raises `IOStreams::Errors::ValueTooLong` when an `:integer` or `:float` value cannot be rendered in `size` characters. * `:decimals` For `:float` columns, the number of decimal places to render. Default: 2 In addition to `layout`, the `:fixed` format takes one more option: * `truncate: [true|false]` Whether to truncate string values that are longer than their column `:size` when writing. When false, a string value that is too long raises `IOStreams::Errors::ValueTooLong` instead of being truncated. Numeric values are never truncated. Default: true ## Header options When reading or writing records (`:hash`), the following options control the header row: * `columns: [Array]` When reading, supplies the header columns for files that do not include a header row. When writing, sets the columns to write, including their order. Keys not listed in `columns` are ignored during writes. * `cleanse_header: [true|false]` Whether to cleanse the column names read from the header row. Column names are stripped of leading and trailing whitespace, lowercased, and spaces and dashes are converted to underscores, so the header `" First Name "` becomes `"first_name"`. Default: true * `allowed_columns: [Array]` List of columns to allow. Any other columns are ignored when `skip_unknown` is true, otherwise an `IOStreams::Errors::InvalidHeader` exception is raised. Default: nil (allow all columns) * `required_columns: [Array]` List of columns that must be present, otherwise an exception is raised. * `skip_unknown: [true|false]` When true, any columns not present in `allowed_columns` are skipped entirely as if they were not in the file at all. When false, an unknown column raises `IOStreams::Errors::InvalidHeader`. Default: true When reading records, `allowed_columns`, `required_columns` and `skip_unknown` apply to the header row, to the supplied `columns`, and, for formats without a header row such as JSON, to the keys of each record. When either `allowed_columns` or `required_columns` is set, JSON keys are cleansed the same way as a header row, unless `cleanse_header: false` is supplied. When reading rows with `each(:array)`, they apply to the header row and to the supplied `columns`, including with `cleanse_header: false`. The header row is yielded as it was read, and each row still contains every value. To return to the behavior before v3.0, where they only apply to a header row read from the file, and only when `cleanse_header` is true, set `IOStreams.enforce_column_restrictions = false`. This applies to reading both records and rows. A warning is then logged when applying them would change the records or the header row read. Since the format is usually inferred from the file name, renaming an uploaded file from `.csv` to `.json` then bypasses them. Example, reading a headerless CSV file: ~~~ruby path = IOStreams.path("no_header.csv") path.each(:hash, columns: ["name", "address", "zip"]) do |record| p record end ~~~ Example, writing only specific columns in a fixed order: ~~~ruby path = IOStreams.path("sample.csv") path.writer(:hash, columns: ["name", "zip"]) do |io| io << {"name" => "Jack Jones", "address" => "Somewhere", "zip" => 12345} end path.read # => "name,zip\nJack Jones,12345\n" ~~~ Note: Column names are converted to strings, and the keys in the hashes being written may be strings or symbols. --- # Copying Between Files File copying can be used to: * copy from one storage location to another. * create a decrypted / encrypted copy of an existing file. * create a decompressed / compressed copy of an existing file. ## Examples Decompress `example.csv.gz` into `example.csv`: ~~~ruby source = IOStreams.path("example.csv.gz") IOStreams.path("example.csv").copy_from(source) ~~~ Decrypt a file encrypted with Symmetric Encryption: ~~~ruby source = IOStreams.path("example.csv.enc") IOStreams.path("example.csv").copy_from(source) ~~~ Encrypt a file using PGP encryption so that it can only be read by `receiver@example.org`. ~~~ruby source = IOStreams.path("example.csv") target = IOStreams.path("example.csv.pgp") target.option(:pgp, recipient: "receiver@example.org") target.copy_from(source) ~~~ When the file name does not have file extensions that would allow IOStreams to infer what streams to apply, the streams can be explicitly set using `stream`. In this example, the file `CUSTOMER_DATA` has no extensions, so `stream(:enc)` tells IOStreams that its contents were encrypted with Symmetric Encryption. The decrypted contents are then PGP encrypted and written to `xyz.csv.pgp` using the pgp key for `receiver@example.org`. ~~~ruby input = IOStreams.path("CUSTOMER_DATA").stream(:enc) IOStreams.path("xyz.csv.pgp").option(:pgp, recipient: "receiver@example.org").copy_from(input) ~~~ To copy a file _without_ performing any conversions (ignore file extensions), set `convert` to `false`: ~~~ruby input = IOStreams.path("sample.json.zip") IOStreams.path("sample.copy").copy_from(input, convert: false) ~~~ Custom stream conversions can be applied to both the source and the target in a single copy. Here the source is read as binary and the target is PGP encrypted: ~~~ruby source = IOStreams.path("source_file").stream(:encode, encoding: "BINARY") IOStreams.path("target_file.pgp").option(:pgp, passphrase: "hello").copy_from(source) ~~~ To convert the contents row by row, or record by record, during the copy, supply `mode`. For example, copy a CSV file into JSON, parsing and rendering each record: ~~~ruby source = IOStreams.path("source_file.csv") IOStreams.path("target_file.json").copy_from(source, mode: :hash) ~~~ Notes: * `mode` accepts `:line`, `:array`, or `:hash`, and only applies when `convert` is `true`. --- # PGP Encrypted Files and Streams IOStreams encrypts and decrypts PGP data by shelling out to the [GnuPG](https://gnupg.org) command line program, so `gpg` must already be installed and on the `PATH` wherever PGP files are read or written (see [Installation](#installation) below). As with every other stream, all you do is add a `.pgp` extension to the file name and IOStreams handles the rest. GnuPG is required because there is no standard, maintained pure-Ruby PGP library. Calling the `gpg` executable directly is the deliberate approach: it is the reference PGP implementation, is widely installed, and is kept current with the OpenPGP standard. It is also well suited to the large files IOStreams targets, since `gpg` streams the data rather than holding it in memory. IOStreams has been tested against GnuPG v1.4, v2.0.30, v2.2.1, and v2.4.7. Because GnuPG is a command line program, IOStreams parses its output to extract information, so each time GnuPG changes its output the regular expression parsers may need updating for that version. ## Installation Install [GnuPG](https://gnupg.org) Mac OSX via homebrew brew install gnupg Redhat / CentOS / Fedora Linux dnf install gnupg2 Ubuntu / Debian Linux apt-get install gnupg Confirm GnuPG is installed: gpg --version ### Tutorial After installing GnuPG above, install iostreams: Install IOStreams gem: ~~~ gem install iostreams --no-doc ~~~ Open a ruby interactive console: ~~~ irb ~~~ Load iostreams: ~~~ruby require "iostreams" ~~~ Generate a private and public key that we can use as the sample sender of the encrypted files: ~~~ruby IOStreams::Pgp.generate_key(name: "Sender", email: "sender@example.org", passphrase: "sender_passphrase") ~~~ Generate a private and public key that we can use as the sample receiver of the encrypted files: ~~~ruby IOStreams::Pgp.generate_key(name: "Receiver", email: "receiver@example.org", passphrase: "receiver_passphrase") ~~~ By default the above keys are RSA 4096 bit encryption keys. Reference a file path to hold the PGP encrypted data by adding `.pgp` as a file name extension: ~~~ruby path = IOStreams.path("sample/example.csv.pgp") # => #{}}> ~~~ Add the email address for the recipient: ~~~ruby path.option(:pgp, recipient: "receiver@example.org") # => #{:recipient=>"receiver@example.org"}} pipeline={:pgp=>{:recipient=>"receiver@example.org"}}> ~~~ An email address only matches keys with exactly that email address, ignoring case. For example `receiver@example.org` does not match a key for `other_receiver@example.org`. The same applies to `signer`, and to the `email:` argument of `IOStreams::Pgp.list_keys`, `key?`, `export`, `delete_keys` and `set_trust`. To search the way gpg does, supply a value that is not just an email address, such as `*receiver@example.org`. Write data to the PGP file: ~~~ruby path.writer do |io| io << "name,login\n" io << "Jack Jones,jjones\n" io << "Jill Smith,jsmith\n" end ~~~ Lets try to read the file without supplying a passphrase: ~~~ruby IOStreams.path("sample/example.csv.pgp").read # IOStreams::Pgp::Failure # gpg: decryption failed: No secret key ~~~ In order to decrypt the file it needs the passphrase for the receivers private key above: ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, passphrase: "sender_passphrase") path.read # IOStreams::Pgp::Failure # "Receiver " # gpg: public key decryption failed: Bad passphrase # gpg: decryption failed: No secret key ~~~ It failed again because we tried to use the senders passphrase. Since only the receiver can decrypt this file we need to use its passphrase and therefore private key: ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, passphrase: "receiver_passphrase") path.read # => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n" ~~~ ##### Sign the file To prevent a man-in-the-middle attack we can sign the file so that the recipient knows the file came from the sender: ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, recipient: "receiver@example.org", signer: "sender@example.org", signer_passphrase: "sender_passphrase") path.writer do |io| io << "name,login\n" io << "Jack Jones,jjones\n" io << "Jill Smith,jsmith\n" end ~~~ Try reading the pgp encrypted file that is now also signed by the sender: ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, passphrase: "receiver_passphrase") path.read # => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n" ~~~ This time when we read the file the signature is automatically verified. However, this only works if the receiver has already imported the senders public key. ##### Check who signed the file gpg verifies any signature in the file, but on its own it accepts a file that is not signed, or that is signed by any key in the keyring. To require that the file was signed by the sender, supply the `signer` when reading: ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, passphrase: "receiver_passphrase", signer: "sender@example.org") path.read # => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n" ~~~ The `signer` is an email address, key id or fingerprint. An email address only matches keys with exactly that email address, ignoring case. Reading raises `IOStreams::Pgp::Failure` unless the file has a good signature by one of the signer's keys, and gpg fully or ultimately trusts that key, for example after importing it with `IOStreams::Pgp.import_and_trust`, or setting its trust with `IOStreams::Pgp.set_trust`. Instead of importing the sender's public key on every server, supply it with `import_and_trust_key`, see [import_and_trust_key](#import_and_trust_key). When reading, the key is imported, and the file must be signed by it, or by the `signer` when both are supplied: ~~~ruby sender_public_key = SecretConfig.fetch("suppliers/acxiom/pgp/public_key") path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, passphrase: "receiver_passphrase", import_and_trust_key: sender_public_key) path.read ~~~ Since the key itself is supplied, a signature by it is accepted whatever trust gpg places in it. So when reading, the trust of the key is only changed when `import_and_trust_level` is supplied, unlike when writing, where it defaults to `5` (Ultimate). A key that is already trusted, for example to encrypt files for that partner, keeps its trust. Like the integrity and signature checks, this check can only be made once the whole file has been read, so by default `IOStreams::Pgp::Failure` is raised after the contents have been processed. Supply `verify_first: true` to only process the contents once every check has passed, see [Verifying a file before processing it](#verifying-a-file-before-processing-it). ##### Sign without encrypting Sometimes the contents do not need to be kept secret, but the recipient still needs to verify that the file came from the sender and was not tampered with. Set `encrypt: false` to sign the file without encrypting it. In this mode a `signer` is required, and `recipient` / `import_and_trust_key` are ignored: ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, encrypt: false, signer: "sender@example.org", signer_passphrase: "sender_passphrase") path.writer do |io| io << "name,login\n" io << "Jack Jones,jjones\n" io << "Jill Smith,jsmith\n" end ~~~ Because a signed-only file is not encrypted, no passphrase is needed to read it. The signature is still verified automatically when the senders public key has been imported: ~~~ruby IOStreams.path("sample/example.csv.pgp").read # => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n" ~~~ ##### Auto-sign all files To automatically sign every PGP output file, set the global config options to sign with the sender credentials. ~~~ruby IOStreams::Pgp::Writer.default_signer = "sender@example.org" IOStreams::Pgp::Writer.default_signer_passphrase = "sender_passphrase" ~~~ Now the file will be signed automatically without needing to supply the `signer` and `signer_passphrase` on every write. ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, recipient: "receiver@example.org") path.writer do |io| io << "name,login\n" io << "Jack Jones,jjones\n" io << "Jill Smith,jsmith\n" end ~~~ ### Decrypt output files In the examples above the sender cannot decrypt the file because it was generated for the recipient only. ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, passphrase: "sender_passphrase") path.read # IOStreams::Pgp::Failure # "Receiver " # gpg: public key decryption failed: Bad passphrase # gpg: decryption failed: No secret key ~~~ Since the sender cannot decrypt the file how do we know what was actually sent to the recipient? There are 2 options: * Add the sender to the recipient list every time a file is created. * Or, use the global configuration option to add the sender automatically to all PGP output files. Add the sender to the recipient list every time a file is created. ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, recipient: ["receiver@example.org", "sender@example.org"]) path.writer do |io| io << "name,login\n" io << "Jack Jones,jjones\n" io << "Jill Smith,jsmith\n" end ~~~ Now the sender can also read the file: ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, passphrase: "sender_passphrase") path.read # => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n" ~~~ Set the global configuration option to add the sender automatically to all PGP output files. ~~~ruby IOStreams::Pgp::Writer.audit_recipient = "sender@example.org" ~~~ The sender is now automatically added to the recipient list every time a file is created. ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, recipient: "receiver@example.org") path.writer do |io| io << "name,login\n" io << "Jack Jones,jjones\n" io << "Jill Smith,jsmith\n" end ~~~ Now the sender can also read the file: ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, passphrase: "sender_passphrase") path.read # => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n" ~~~ By now you must be getting tired of typing in the senders passphrase, so lets set the config option to remove that requirement: ~~~ruby IOStreams::Pgp::Reader.default_passphrase = "sender_passphrase" ~~~ Try reading the above file again: ~~~ruby IOStreams.path("sample/example.csv.pgp").read # => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n" ~~~ ## Cleanup To remove the test PGP keys created above: ~~~ruby IOStreams::Pgp.delete_keys(email: 'sender@example.org', private: true) IOStreams::Pgp.delete_keys(email: 'receiver@example.org', private: true) ~~~ ### import_and_trust_key The standard approach with PGP is to import public keys into the keystore on every server. Copying the public keys to every server can be tedious, especially when running inside docker containers. An ideal way to manage PGP public keys is in a database, or in Secret Config that stores the keys in the AWS SSM Parameter store. Fetch the recipients public key from a data store. For example Secret Config ~~~ruby public_pgp_key = SecretConfig.fetch("suppliers/acxiom/pgp/public_key") ~~~ When creating the pgp encrypted file let IOStreams import and trust the public key every time so that it does not have to be managed on every production server. The option `import_and_trust_key` takes the public key as a string: ~~~ruby path = IOStreams.join("test/sample.pgp", root: :downloads) path.option(:pgp, import_and_trust_key: public_pgp_key) path.write("Hello World") ​~~~ Now try to read the file: ~~~ruby IOStreams.join("test/sample.pgp", root: :downloads).read ~~~ #### Trust level The key is imported and then marked as trusted. The file is encrypted to the imported key itself, whatever its trust level, so that another key in the keyring with the same email address is never used instead. The trust level can be controlled with the `import_and_trust_level` option: ~~~ruby path = IOStreams.join("test/sample.pgp", root: :downloads) path.option(:pgp, import_and_trust_key: public_pgp_key, import_and_trust_level: 4) path.write("Hello World") ~~~ Or by calling `IOStreams::Pgp.import_and_trust` directly with the `trust_level` argument: ~~~ruby IOStreams::Pgp.import_and_trust(key: public_pgp_key, trust_level: 4) ~~~ The available levels are the same as those used by `IOStreams::Pgp.set_trust`: | Level | Meaning | |:------|:-------------------------| | 1 | Undefined (no opinion) | | 2 | Never (do not trust) | | 3 | Marginal | | 4 | Full | | 5 | Ultimate (default) | > **Security warning** > > Only import and trust keys that were received from a verified, trusted source. > > The default trust level is `5` (Ultimate), which tells GPG to treat the imported key as > if it were one of your own keys: it becomes implicitly valid and can in turn confer > validity on other keys that it has signed. Importing an attacker supplied key at this > level allows that attacker to impersonate other recipients. When a key cannot be fully > verified, supply a lower `trust_level`. The file is still encrypted to it, since IOStreams encrypts to the > imported key itself. #### Compression: The compression used by pgp can be specified by suppling the `:compress` option. The valid values for this option: `:none`, `:zip`, `:zlib`, or `:bzip2`. The default compression is `:zip` for which has the highest compatibility. Most PGP tools now support `:zlib` and is the recommended compression to use when possible. ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, recipient: "receiver@example.org", compress: :zlib) path.write("Hello World") ~~~ The compression level can be adjusted with the `:compress_level` option, where `1` is the fastest and `9` compresses the most: ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, recipient: "receiver@example.org", compress: :zlib, compress_level: 9) path.write("Hello World") ~~~ Default: `6` Compression Performance * Running tests on an Early 2015 Macbook Pro Dual Core with Ruby v2.3.1 ~~~ Input text file: test.log 3.6GB :none: size: 3.6GB write: 52s read: 45s :zip: size: 411MB write: 75s read: 31s :zlib: size: 241MB write: 66s read: 23s ( 756KB Memory ) :bzip2: size: 129MB write: 430s read: 130s ( 5MB Memory ) ~~~ ### Verifying a file before processing it gpg can only check a file's integrity (MDC) and its signature once it has read the whole file. By default IOStreams passes the decrypted contents to your block as gpg decrypts them, so that files of any size can be processed without storing the decrypted contents. When a check fails, `IOStreams::Pgp::Failure` is raised after the block has already processed the contents. For example, with a signed file whose contents were changed after it was signed, every record is processed before the `BAD signature` failure is raised: ~~~ruby IOStreams.path("sample/example.csv.pgp").each(:hash) do |row| # Receives every row, including the changed ones. end # IOStreams::Pgp::Failure: ... gpg: BAD signature from "Sender " ~~~ When the block returns before reading the whole file, for example after reading only the first line, the rest of the file is still decrypted and checked before the block's result is returned. Either do not commit any side effects until the block returns without raising, for example by processing the file within a database transaction, or supply the `verify_first` option: ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, passphrase: "receiver_passphrase", verify_first: true) path.each(:hash) do |row| # Only called once gpg has checked the whole file. end ~~~ With `verify_first`, the whole file is first decrypted into a temporary file that only the current user can read. The block is only called once gpg has successfully checked the file, and the temporary file is deleted afterwards. This requires local disk space for the decrypted contents, and an extra pass over the data. ### Reading legacy files without MDC integrity protection Modern GnuPG refuses to decrypt files that lack MDC (Modification Detection Code) integrity protection, failing with `gpg: decryption forced to fail!`. Some legacy or enterprise systems still produce such files. To read them, supply the `ignore_mdc_error` option: ~~~ruby path = IOStreams.path("sample/example.csv.pgp") path.option(:pgp, passphrase: "receiver_passphrase", ignore_mdc_error: true) path.read ~~~ > **Security warning** > > Only enable `ignore_mdc_error` for files from a trusted source: without MDC the decrypted > contents are not protected against tampering. Note: IOStreams never writes files without MDC, this option only applies when reading. ### PGP FAQ: If you get not trusted errors `gpg --edit-key sender@example.org` Select highest level: 5 ### PGP Limitations * Designed for processing larger files since a process is spawned for each file processed. * For lots of small, in memory files, use the [gpgme](https://github.com/ueno/ruby-gpgme) library. For example to attach pgp files to emails. --- # Configuring IOStreams ## Paths are configuration A path or url is configuration, not code. Keep it in an environment variable or a centralized configuration system, and pass it to IOStreams, so that the exact same application code runs against local files in development and a completely different file store in staging or production. ~~~ruby # Development: EXPORT_PATH=~/exports/daily.csv # Staging: EXPORT_PATH=sftp://staging.example.org/~/exports/daily.csv # Production: EXPORT_PATH=s3://my-bucket/exports/daily.csv IOStreams.path(ENV.fetch("EXPORT_PATH")).write(report) ~~~ This also helps when infrastructure changes. For example, while migrating production from on-premises servers to the cloud, the on-premises instance can keep reading and writing local files while the cloud instance uses S3, both running the same release with only their configuration differing. The same path written for each file store refers to the equivalent place. For example `~/exports` is within the home directory, and `sftp://hostname/~/exports` is within the login directory on the SFTP server. To keep code independent of where its files are stored: * Use the methods on the path, such as `exist?`, `each_child`, `move_to`, `delete`, `reader` and `writer`, rather than passing `path.to_s` to `File` or `Dir`, which only work with local files. * Do not check the scheme of a path to decide what to do with it. * Keep credentials in configuration too, for example the AWS environment variables or credential files for S3, and the ssh config for SFTP. When an application uses several locations, [roots](#add_root) give them names, and each root can be set from configuration: ~~~ruby IOStreams.add_root(:default, ENV.fetch("FILES_ROOT")) IOStreams.join("exports", "daily.csv").write(report) ~~~ ## add_root Roots allow paths to reference a particular root directory, so that all path names are appended to that root. Their primary purpose is to allow the exact same code to run in production and development, yet use completely different data sources in each. For example, in production a root can point to an S3 bucket, while in development it points to the local file system. Roots are configured via an initializer at startup. `IOStreams.join` then joins the supplied path elements onto the named root, using the `:default` root whenever a root is not supplied. Set the default root for this environment in an initializer: ~~~ruby IOStreams.add_root(:default, "/var/my_app/files") ~~~ Now the default root path is available: ~~~ruby IOStreams.root # => # IOStreams.root.to_s # => "/var/my_app/files" ~~~ Comparing the final path using `path` and then `join` that uses a root path: ~~~ruby IOStreams.path("/var/my_app/files", "my_test_file.txt").to_s # => "/var/my_app/files/my_test_file.txt" IOStreams.join("my_test_file.txt").to_s # => "/var/my_app/files/my_test_file.txt" ~~~ Using `path`: ~~~ruby IOStreams.path("/var/my_app/files", "my_test_file.txt").write("Hello World") ~~~ With the default root path configured the above code can be simplified by using `join` since it resolves to the same path. ~~~ruby IOStreams.join("my_test_file.txt").write("Hello World") ~~~ Multiple roots can be setup, for example one for input files, another for output files, another for reports, etc. During development the roots can all point to a common location, while in production they could be completely different S3 buckets. For example add special paths for `downloads` and `uploads`. ~~~ruby IOStreams.add_root(:downloads, "/var/my_app/downloads") IOStreams.add_root(:uploads, "/var/my_app/uploads") ~~~ An example that writes a file into the `/var/my_app/downloads` directory: ~~~ruby IOStreams.join("my_test_file.txt", root: :downloads).write("Hello World") ~~~ The other benefit is that the root paths used in an application are externalized from the code base. That way the roots can be changed to different locations depending on the environment. We can also change the storage mechanism by changing the root: ~~~ruby IOStreams.add_root(:downloads, "s3://my-app-bucket-name/downloads") IOStreams.add_root(:uploads, "s3://my-app-bucket-name/uploads") ~~~ Now the application will write to S3 and the code does not change at all. ~~~ruby IOStreams.join("my_test_file.txt", root: :downloads).write("Hello World") ~~~ To use or query a configured root path: ~~~ruby IOStreams.root(:downloads).to_s # => "s3://my-app-bucket-name/downloads" ~~~ ## temp_dir When working with large files the standard temp file system location can be too small to handle downloading large files. For example to decrypt a pgp file from S3, because GnuPG is not streaming capable and only operates on local files. By default IOStreams looks up the location to store temp files in the following order: * `ENV['TMPDIR']` * `ENV['TMP']` * `ENV['TEMP']` * `Etc.systmpdir` * `/tmp` (if it exists) * Otherwise `.` To explicity set the temp file location the following config option can be used: ~~~ruby IOStreams.temp_dir = "/var/really_big_temp" ~~~ ### temp_file To work with a temporary file directly, `IOStreams.temp_file` yields a path inside `temp_dir` and deletes the file when the block completes: ~~~ruby IOStreams.temp_file("export", ".csv") do |path| path.write("Hello World") # ... use the temp file ... end # The temp file has been deleted. ~~~ The first argument is a base file name to include in the generated temp file name, and the optional second argument is the file extension. ## logger IOStreams can log debug information, such as the external commands it runs for PGP and SFTP. When [Semantic Logger](https://logger.reidmorrison.com) is loaded it is detected automatically, and IOStreams logs to it without any additional configuration. To use a different logger, or to log when Semantic Logger is not present, assign any logger that responds to the standard logging methods: ~~~ruby require "logger" IOStreams.logger = Logger.new($stdout) ~~~ To disable logging entirely, set the logger to `nil`: ~~~ruby IOStreams.logger = nil ~~~ ## enforce_column_restrictions Applies `allowed_columns`, `required_columns` and `skip_unknown` to every input when reading records, including JSON records, so that renaming an uploaded file from `.csv` to `.json` cannot bypass them. To return to the behavior before v3.0, where they only apply to a header row read from the file, set it to false in an initializer. A warning is then logged when applying them would change the records read: ~~~ruby IOStreams.enforce_column_restrictions = false ~~~ Default: true. It defaulted to false before v3.0. See [Header options](formats#header-options). --- # Upgrading IOStreams This page covers the changes that may need updates to your application when upgrading IOStreams, and the security issues to check. For every change in each release, see the [CHANGELOG](https://github.com/reidmorrison/iostreams/blob/main/CHANGELOG.md). ## Upgrading to v3.0 v3.0 is a major release with the breaking changes that were postponed from v2.1. In v2.1 each one logged a warning via `IOStreams.logger` when it would change the result, so check your logs from v2.1 for them before upgrading. It also includes bug fixes that change behavior that existing code may depend on, described at the end of this section. ### Column restrictions apply to every input When reading records, `allowed_columns`, `required_columns` and `skip_unknown` now apply to every input. Previously they were ignored for JSON and `:hash` input, when `columns:` was supplied, and with `cleanse_header: false`. When reading rows with `each(:array)`, they now also apply to the supplied `columns:`, and to the header row with `cleanse_header: false`. When either `allowed_columns` or `required_columns` is set, JSON keys are now cleansed the same way as a header row, for example `"Name"` becomes `"name"`. Unknown keys are skipped, or raise `IOStreams::Errors::InvalidHeader` when `skip_unknown: false`, and a record that is missing a required column raises `IOStreams::Errors::InvalidHeader`. Fix: check that the JSON files you read with these options have the expected keys. See [Header options](formats#header-options). To keep the previous behavior, set `IOStreams.enforce_column_restrictions = false` in an initializer, which defaulted to false in v2.1. Renaming an uploaded file from `.csv` to `.json` then bypasses the restrictions. ### BZip2 options are strict The BZip2 reader and writer now raise `ArgumentError` for an option that neither of them accepts, like every other stream. Previously they ignored it. They accept `autoclose`, `first_only` and `small` when reading, and `autoclose`, `block_size` and `work_factor` when writing, and each ignores the options of the other. Fix: correct or remove the option. See [Options for reading and writing](streams#options-for-reading-and-writing). ### Zip `zip_file_name` is removed The Zip writer no longer accepts `zip_file_name`, which the documentation replaced with `entry_file_name` long ago, and `option(:zip, zip_file_name: "a.csv")` raises `ArgumentError`. Fix: use `entry_file_name`, which names the file within the zip file when writing, and chooses the file to read: `option(:zip, entry_file_name: "a.csv")`. ### PGP `export` without an email or key id `IOStreams::Pgp.export(email: nil)` without a `key_id:` now raises `ArgumentError`, instead of `IOStreams::Pgp::Failure`. ### PGP email addresses match exactly An email address supplied as a PGP `recipient` or `signer`, or as the `email:` of `IOStreams::Pgp.list_keys`, `key?`, `export`, `delete_keys` or `set_trust`, now only matches keys with exactly that email address, ignoring case. gpg treats a bare email address as a search for any user id that contains it, so previously `bob@example.com` also matched `jimbob@example.com`. Fix: supply the exact email address of each key. To search the way gpg does, supply a value that is not just an email address, such as `*example.com`. ### Reading gzip returns every member Reading a gzip file that contains several members, for example concatenated gzip files, now returns the contents of every member, instead of only the first. The stream supplied to the block when reading gzip is no longer a `Zlib::GzipReader`; like the other streams it responds to `#read`, `#readpartial` and `#eof?`. Fix: replace calls such as `#gets` on the gzip stream with `each(:line)`. ### The encode stream validates every read With the `:encode` stream, data read from a file is treated as already being in the requested encoding, so valid characters are kept instead of being replaced, and an invalid character raises `Encoding::UndefinedConversionError` unless `replace:` is supplied. Previously reading lines let invalid characters through without raising. Fix: supply `replace:`, for example `option(:encode, encoding: "UTF-8", replace: "")`, to replace invalid characters instead of raising. ### A `+` in an S3 or SFTP url is kept A `+` in the path of an S3 or SFTP url is now kept, so `IOStreams.path("s3://bucket/a+b.csv")` reads the key `a+b.csv`. Previously it was decoded as a space, and read the key `a b.csv`. Fix: use a space, or `%20`, in the url for a space. ### The path of an HTTP url is decoded `#path` of an `http://` or `https://` path is now decoded, like S3 and SFTP paths, so `IOStreams.path("https://example.com/my%20file.csv").path` is `/my file.csv`, and `#basename` is `my file.csv`. Previously it was `/my%20file.csv`. `#to_s` still returns the url unchanged. A url without a path, such as `https://example.com`, now has the path `/`, where it was empty. Fix: use `#to_s` where the url is needed, and remove any code that decodes the `#path` of an HTTP path. ### A direct copy between S3 paths returns the number of bytes `#copy_from` and `#copy_to` with `convert: false` between two S3 paths now return the number of bytes copied, like every other copy. Previously `#copy_from` returned the response from the S3 `copy_object` call, and `#copy_to` returned the target path. `#move_to` still returns the target path. Fix: use the target path that was supplied to `#copy_to`, instead of its return value. ### Local `each_child` honors `case_sensitive:` `#each_child` and `IOStreams.each_child` on local files now honor `case_sensitive:` on every platform. Previously, on a case-sensitive file system such as on Linux, the default `case_sensitive: false` was ignored unless nothing matched, and on a case-insensitive file system such as on macOS, `case_sensitive: true` was ignored. Fix: on Linux, check that patterns that relied on matching case, such as `"*.csv"` in a directory that also has `DATA.CSV`, return the files you expect, or supply `case_sensitive: true`. ### The file within a written zip file is named after the path Writing to a path such as `example.csv.zip` now names the file within the zip file `example.csv`, as documented, instead of `file`. Fix: if the program that reads the zip file expects the name `file`, supply `option(:zip, entry_file_name: "file")`. ### Hash keys match columns once cleansed When writing a hash with `columns:`, or reading JSON or `:hash` records with `columns:`, a key such as `"First Name"` now supplies the value of the column `first_name`, since they are the same once cleansed like a header row. Previously the column was empty. Fix: if a column must stay empty, remove the matching key from the hash before writing it. ### Writing to SFTP returns the result of the block Writing to an SFTP path, for example with `#writer`, now returns the result of the block, like local and S3 paths, instead of the size of the file written. Fix: if you need the size, return it from the block, for example the total of what `io.write` returns. ### Options that a mode or copy cannot use raise `ArgumentError` `#reader` and `#writer` in the `:stream` mode, which is the default, `#copy_from` and `#copy_to` without a `mode:`, and `#copy_from` and `#copy_to` with `convert: false`, now raise `ArgumentError` for any option, instead of ignoring it. Options for the `:line`, `:array` and `:hash` modes already raised. Fix: remove the option, which had no effect. To configure a stream, use `#option` or `#stream`, for example `path.option(:pgp, passphrase: "secret").reader { |io| io.read }`. ### Misspelled stream options raise when they are set An option supplied to `#option` or `#stream` that neither the reader nor the writer for the stream accepts now raises `ArgumentError` when it is set, even when the file name does not include that stream. Previously it was only checked once the stream was used, so for example `IOStreams.path("data.csv").option(:pgp, recipent: "partner@example.org")` was ignored, and only raised for a `.pgp` file, which could be only in production when the path comes from configuration. Fix: correct or remove the option. See [Options for reading and writing](streams#options-for-reading-and-writing). ### Gzip and `.enc` streams leave your IO open When reading or writing an IO that you supply with `IOStreams.stream(io)`, the `:gz`, `:gzip` and `:enc` streams no longer close `io` when the block finishes, like every other stream. Paths are not affected, since IOStreams opens and closes their files itself. Fix: close the IO yourself once you are done with it, for example a pipe or socket that the other end reads until it is closed. ### Paths compare by their full name `==` and `<=>` now compare the full name of a path, as returned by `#to_s`, instead of only `#path`, so paths in different S3 buckets, on different SFTP or HTTP hosts, or in different stores are no longer equal, even with the same key or file name. A path also equals a `String` of its full name, such as `IOStreams.path("/home/user/a.txt") == "/home/user/a.txt"`. Names are compared as given, so a relative path does not equal its absolute path, and `"a.txt" == path` is false, since a `String` does not compare equal to other objects. Fix: to compare only the file names or keys, compare `#path`, for example `a.path == b.path`. ### `#path` is read-only `Path#path=` is no longer public, so changing the name of an existing path raises `NoMethodError`. A path is a `Hash` key for its location, so changing it in place meant it could no longer be found in a `Hash` or `Set` that held it. Fix: create a new path instead. `#join` and `#directory` return a new path that keeps the settings of the original, such as the S3 client or the SFTP credentials: ```ruby root = IOStreams.path("s3://bucket/data", region: "us-east-1") # Before path = root.join("in.csv") path.path = "data/out.csv" # After path = root.join("out.csv") ``` ### `#builder=` is not public `#builder=` is no longer public on a path or a stream, so calling it raises `NoMethodError`. It takes an internal object, so the only use outside IOStreams was `path.builder = nil`, to remove the streams and options set with `#stream` or `#option`. Fix: create the path again with `IOStreams.path`, which has no streams or options, or use the path that `#join` or `#directory` returns, which does not keep them. ### Paths and streams are copied, not shared `IOStreams.path(path)`, `IOStreams.stream(stream)` and `IOStreams.new(stream)` now return a copy of the path or stream supplied, instead of the same object. `IOStreams.join` without any elements, `IOStreams.root`, `IOStreams.roots`, and `Path#join` without any elements also return copies. The copy keeps the streams and options, and is equal to the original with `==`, but changing it, for example with `#stream` or `#option`, no longer changes the original, or a root path for the whole process. Fix: code that relied on changing the original through the returned object should keep and use the returned object instead: ```ruby # Before: also changed `path` IOStreams.path(path).option(:pgp, passphrase: "secret") path.read # After path = IOStreams.path(path).option(:pgp, passphrase: "secret") path.read ``` To change the streams of a root, supply them each time the root is used, for example `IOStreams.join("a.csv").stream(:none)`. Code that compares with `equal?` should compare with `==`. ### `IOStreams.add_root` returns a frozen path `IOStreams.add_root` now freezes the root path it adds, and returns it, so changing the returned path, for example with `#stream` or `#option`, raises `FrozenError` instead of changing the root for the whole process. Fix: change the copy that `IOStreams.root` or `IOStreams.join` returns instead, each time the root is used: ```ruby IOStreams.add_root(:exports, "s3://bucket/exports") IOStreams.join("a.csv.pgp", root: :exports).option(:pgp, passphrase: "secret").write(data) ``` ### Unknown S3 options raise `ArgumentError` An option supplied to an S3 path, or in the query string of its url, that no S3 request accepts now raises `ArgumentError` when the path is created, as does a request parameter that the path sets itself, such as `key` or `bucket`. Each S3 request is supplied the options that it accepts, so for example `acl` applies when writing and copying, and `request_payer` to every request, including `#exist?`, `#size`, `#delete` and `#each_child`. Fix: correct or remove the option. ### SFTP urls without a path are the root directory `IOStreams.path("sftp://host")` is now the root directory `/`, like `sftp://host/`, so `#each_child` lists `/` instead of the login directory. Previously the children it returned were in `/` regardless, so reading a listed file read the wrong file. Reading, writing and `#join` already used `/`, so for example `IOStreams.path("sftp://host").join("a.csv")` is still `/a.csv`. A path starting with `~` is now within the login directory, as curl does, for example `sftp://host/~/data/a.csv`. Fix: use `sftp://host/~` to list the login directory. On a server that confines users to their own directory, which is then `/`, no change is needed. An allowed path that starts with `sftp://host/~/` now refers to the login directory, not to a directory named `~`. ### SFTP and HTTP urls without a path end with `/` `#to_s` of an SFTP or HTTP url without a path now ends with `/`, like the root of an S3 bucket, `s3://bucket/`. For example `IOStreams.path("sftp://host").to_s` is `sftp://host/`, and `IOStreams.path("https://host?a=1").to_s` is `https://host/?a=1`. Previously the url was returned as supplied, so `sftp://host` and `sftp://host/` were not equal, although they are the same location. Fix: compare paths with paths, such as `path == IOStreams.path("sftp://host")`, or compare with the url ending with `/`. ### SFTP `each_child` is case-insensitive `#each_child` on an SFTP path is now case-insensitive by default, like local and S3 paths, as documented, so `"*.csv"` also returns `DATA.CSV`. Previously SFTP defaulted to `case_sensitive: true`. Fix: supply `case_sensitive: true` where a pattern must match the case of the file names. ### SFTP `each_child` returns nothing for a missing directory `#each_child` on an SFTP path now returns nothing when the path does not exist, or is a file, like local and S3 paths. Previously it raised `Net::SFTP::StatusException`. It also skips a directory below the path that cannot be read, like `Dir.glob`, instead of raising. A path that exists but cannot be read still raises. It now only lists the directories that the pattern can match within, so `"*.csv"` only lists the path's directory. Previously every directory below the path was listed for every pattern, which was slow for a large tree. Fix: where your code rescues `Net::SFTP::StatusException` from `#each_child` to detect a missing directory, check for no children instead. ### S3 `each_child` returns directories `#each_child` on an S3 path with `directories: true` now returns the directories within the keys, like local and SFTP paths, such as `s3://bucket/a` and `s3://bucket/a/b` for the key `a/b/c.csv`. Previously it only returned empty folders created with a key ending in `/`, only with a pattern that included `**`, and with the trailing `/`, such as `s3://bucket/a/`. An empty folder is now returned without the trailing `/`. Fix: where your code supplies `directories: true` on S3 paths, check that it handles the extra directories, or remove the trailing `/` that it expected. ### Other `each_child` changes * An exact name supplied to `#each_child` on an S3 path, such as `each_child("a.csv")`, is now yielded with its attributes, like the children that match a pattern. * `IOStreams.each_child` with a name without any pattern characters, such as `"s3://bucket/data/a.csv"`, now yields the attributes for S3 and SFTP paths, and only yields a directory with `directories: true`. Previously it yielded only the path, and yielded a local directory regardless. * Local `#each_child` now returns `nil`, like S3 and SFTP paths. Previously it returned an internal array of the names that it found. Fix: a block that takes a single argument is unaffected by the attributes. A lambda or method that takes exactly one argument must accept the attributes, as it already had to for patterns. Do not use the return value of `#each_child` with a block; collect the children in the block, or use `#children`. ### A local `~` is the home directory A local file name of `~`, or starting with `~/`, is now within the current user's home directory, so `IOStreams.path("~/data/example.csv")` is `/home/user/data/example.csv`. This matches an SFTP url, where `sftp://hostname/~/data/example.csv` is within the login directory, so a configured path can change between local files and SFTP without changing the code. Previously `~` was a directory called `~` in the current directory, so files written to `~/data/example.csv` are in `./~/data/example.csv`. A warning is logged via `IOStreams.logger` when the current directory contains a directory called `~`. A name such as `~user/a.csv` or `~$Book1.xlsx` is unchanged. Fix: move any files that were written to the `~` directory, or use `./~/data/example.csv` to keep using it. ### A failed copy to S3 does not delete the target `#copy_from` an S3 path no longer deletes the object when the copy fails. S3 only stores an object once its upload completes, so a failed copy never leaves an incomplete object. The delete could instead remove a complete object when only the response to its upload was lost, or an object that another writer created during the copy. Fix: none is needed in most cases. When the copy raises, the object is either unchanged or completely replaced. Code that relied on a failed copy always removing the object should call `#delete` after the error. ### HTTPS downloads are not redirected to HTTP Reading an `https://` path no longer follows a redirect to an `http://` url, and raises `IOStreams::Errors::CommunicationsFailure` instead. Previously the download continued over plain http, where the data could be read or changed in transit. Fix: read the `http://` url directly if plain http is acceptable for that server, or have the server redirect to an `https://` url. ### `compressed?` and `encrypted?` follow the streams `#compressed?` and `#encrypted?` now report whether a stream in the pipeline compresses or encrypts the data, whether it was inferred from the file name or set with `#stream`. Previously they only checked the last extension of the file name: * `IOStreams.path("data.csv.gz.pgp").compressed?` is now `true`. * `IOStreams.path("tempfile2527").stream(:gz).compressed?` is now `true`. * `IOStreams.path("data.csv.gz").stream(:none).compressed?` is now `false`. Compression within a PGP or `.enc` file is still not reported, since only the encrypted data records whether it was compressed. Fix: none is needed, unless code relied on the previous result, for example to decide whether to compress data before it is encrypted. ### Extensions are registered as formats `IOStreams.extensions` now returns the format registered for each built-in extension, such as `IOStreams::Gzip` for `:gz`, instead of an `IOStreams::Extension` struct. A format answers `reader_class` and `writer_class` as before, but not the methods of a struct, such as `to_a`, `to_h`, `==` or `[]`. An extension registered with its reader and writer classes is still an `IOStreams::Extension`, which is now frozen, so its setters, such as `writer_class=`, raise `FrozenError`. Previously the setters changed the registered extension for the whole process. An extension registered with just its reader and writer classes is neither compressed nor encrypted. So after replacing a built-in extension that is, such as `:xlsx`, `:gz` or `:pgp`, with your own reader and writer classes, `#compressed?` or `#encrypted?` is now `false` for its files. Previously both were based on the file name. Fix: use `reader_class` and `writer_class` instead of the methods of a struct, and `IOStreams.register_extension` to change a registered extension. To replace the reader or writer of an extension whose data is compressed or encrypted, register a format that says so, see [Registering a custom extension](extensions#registering-a-custom-extension). ## Upgrading to v2.1 v2.1 is a security release, and is backward compatible except for `IOStreams::Pgp.delete_keys`, described below. Most applications upgrade without any code changes. The changes that would break existing code were postponed to v3.0, and log a warning in v2.1. See [Upgrading to v3.0](#upgrading-to-v30). ### Changes that may need code changes #### PGP `delete_keys` requires an email or key id `IOStreams::Pgp.delete_keys` raises `ArgumentError` unless an `email:` or `key_id:` is supplied. Previously, on GnuPG 2.1 and later, calling it without either deleted every key in the keyring, and with `private: true`, every secret key. This is the only breaking change in v2.1, since that could not be undone. To delete several keys, call it once for each key, for example for each key returned by `IOStreams::Pgp.list_keys`. #### SFTP `#each_child` requires a known host key `#each_child` on an SFTP path now verifies the server's host key, the same way reading and writing already did. Previously it trusted a host key the first time it was seen. If listing files now fails, supply the server's host key with the `HostKey` ssh option, or add it to the `known_hosts` file of the user running the application. See [SFTP](path#sftp-sftp). `#each_child` now also supports the `HostKey`, `IdentityKey` and `IdentityFile` ssh options, and some others. Previously any ssh option made it raise `ArgumentError`. Any ssh option it does not support raises `ArgumentError`. #### Other changes in behavior - SFTP `#each_child` lists the files in the path's directory. Previously it ignored the path and listed the login directory, so `IOStreams.path("sftp://host/data/in").each_child("*.csv")` listed `~/*.csv`, returning paths such as `/a.csv`. A url without a path, such as `sftp://host`, still lists the login directory. - `Path#join` and `IOStreams.join` only return an element unchanged when it is the path itself or is inside it. Previously any element that started with the same characters was returned as-is, so `IOStreams.path("s3://bucket/reports").join("reports_2024.csv")` gave `s3://bucket/reports_2024.csv`, instead of `s3://bucket/reports/reports_2024.csv`. - When writing PSV or fixed width files, a line break within a value is replaced with a space, so that a value can no longer start a separate record. - Internal temp files are created so that only the current user can read them. - When reading lines, newlines within quoted values are kept within the line when the tabular format quotes its values, such as CSV. Previously this depended on whether the file name contained `.csv`, so `.format(:psv)` on a `.csv` file still joined quoted lines, and `.format(:csv)` on a stream without a file name did not. - An `ArgumentError` for an option that a stream does not accept now says which direction the option belongs to, or lists the valid options. Previously it was Ruby's `unknown keyword` error. ### Security checklist When upgrading, review how your application uses IOStreams for the following documented security issues. Each one depends on how the application is written, so upgrading alone does not fix it. #### File names from untrusted input If a user, a job, or any other untrusted input can supply a file name, it can also supply `..` or an absolute path, and read or overwrite any file that the process can access. `IOStreams.join` and root paths do not prevent this. New in v2.1, add **allowed paths** in an initializer, so that accessing any other path raises `IOStreams::Errors::AccessDenied`: ~~~ruby IOStreams.add_allowed_path("/var/my_app/uploads") IOStreams.add_allowed_path("s3://my-app-bucket-name/export") ~~~ Once any allowed path is added, every path IOStreams accesses in the process must be within one of them, so add every location that your application reads or writes. See [Restricting access with allowed paths](path#restricting-access-with-allowed-paths). #### Untrusted names in S3 urls Any query string in an S3 url is added to the S3 request parameters. Do not interpolate an untrusted file name into the url, since a name such as `file.csv?acl=public-read` would set request parameters. Join it onto the path instead. See [AWS S3](path#aws-s3-s3). ~~~ruby # Unsafe IOStreams.path("s3://my-bucket-name/uploads/#{untrusted_name}") # Safe IOStreams.path("s3://my-bucket-name/uploads").join(untrusted_name) ~~~ #### Credentials in HTTP and SFTP urls A username and password supplied in an HTTP or SFTP url remain part of it, so `#to_s` returns them, as does any log or error message that includes the path. Supply them with the `username:` and `password:` arguments instead: ~~~ruby # Returns the password from #to_s IOStreams.path("sftp://jack:secret@sftp.example.org/file.csv") # Does not IOStreams.path("sftp://sftp.example.org/file.csv", username: "jack", password: "secret") ~~~ #### Untrusted HTTP urls When any part of an HTTP url comes from untrusted input, an attacker can point it at internal services or cloud metadata endpoints (Server Side Request Forgery), either directly or with a redirect. Use `allow_hosts`, `http_redirect_count` and `maximum_file_size`, or allowed paths, which also check every redirect. See [Security: untrusted URLs (SSRF)](path#security-untrusted-urls-ssrf). #### Processing PGP files before they are verified By default IOStreams passes the decrypted contents of a PGP file to your block as gpg decrypts it. gpg can only check the file's integrity and signature once it has read the whole file, so a tampered file is only rejected after your block has processed its contents. Either do not commit any side effects until the block returns, for example by using a database transaction, or supply the `verify_first` option, new in v2.1. See [Verifying a file before processing it](pgp#verifying-a-file-before-processing-it). ~~~ruby path.option(:pgp, passphrase: "receiver_passphrase", verify_first: true) ~~~ #### Importing and trusting PGP keys `import_and_trust` and the writer's `import_and_trust_key` option trust a key at the Ultimate level by default, so only use them with keys received from a verified, trusted source. When a key cannot be fully verified, supply a lower `trust_level:` to `import_and_trust`, or a lower `import_and_trust_level` option to the writer. See [Trust level](pgp#trust-level). In v2.1, `import_and_trust_key` encrypts to the imported key's fingerprint on GnuPG 2.1 and later. Previously it encrypted to the key's email address, so if your keyring holds another key with the same email address, that key could have been used instead. Check your keyring for duplicate email addresses. #### PGP files without integrity protection Only enable `ignore_mdc_error` for files from a trusted source, since without MDC the decrypted contents are not protected against tampering. See [Reading legacy files without MDC integrity protection](pgp#reading-legacy-files-without-mdc-integrity-protection). #### Column restrictions on uploaded files In v2.1, by default, `allowed_columns` and `required_columns` are ignored for JSON input. Since the format is usually inferred from the file name, renaming an upload from `.csv` to `.json` bypasses them. If your application relies on them to restrict which columns an upload can set, apply them to every input in an initializer, new in v2.1, and check whether any uploads were affected: ~~~ruby IOStreams.enforce_column_restrictions = true ~~~ See [Column restrictions apply to every input](#column-restrictions-apply-to-every-input) for what it changes. ## Upgrading to v2.0 v2.0 is a major release with breaking changes: - **Ruby 3.2 or later is required.** - **Writing Zip files requires the `zip_kit` gem.** The retired `zip_tricks` gem has been replaced by its successor, `zip_kit`. If your application writes Zip files, replace `gem "zip_tricks"` with `gem "zip_kit"` in your Gemfile. Reading Zip files is unaffected. - **The deprecated pre-v1.6 API has been removed**, including the `IOStreams::Deprecated` mix-in. Move any code still using it to the `IOStreams.path` and `IOStreams.stream` API. - **The deprecated PGP writer `compression:` option has been removed.** Use `compress:` instead, available since v1.11.0. - **`IOStreams::Pgp.logger` and `IOStreams::Pgp.logger=` have been removed.** Logging is configured for the whole library with `IOStreams.logger=`. See [logger](config#logger). ## Upgrading from before v1.6 The pre-v1.6 API was deprecated in v1.6 and removed in v2.0. Upgrade to v1.11 first, move all code to the `IOStreams.path` and `IOStreams.stream` API, and then upgrade to v2.