IOStreams Any size. Any format. Anywhere.

PGP Encrypted Files and Streams

IOStreams encrypts and decrypts PGP data by shelling out to the GnuPG command line program, so gpg must already be installed and on the PATH wherever PGP files are read or written (see Installation below). As with every other stream, all you do is add a .pgp extension to the file name and IOStreams handles the rest.

GnuPG is required because there is no standard, maintained pure-Ruby PGP library. Calling the gpg executable directly is the deliberate approach: it is the reference PGP implementation, is widely installed, and is kept current with the OpenPGP standard. It is also well suited to the large files IOStreams targets, since gpg streams the data rather than holding it in memory.

IOStreams has been tested against GnuPG v1.4, v2.0.30, v2.2.1, and v2.4.7. Because GnuPG is a command line program, IOStreams parses its output to extract information, so each time GnuPG changes its output the regular expression parsers may need updating for that version.

Installation

Install GnuPG

Mac OSX via homebrew

brew install gnupg

Redhat / CentOS / Fedora Linux

dnf install gnupg2

Ubuntu / Debian Linux

apt-get install gnupg

Confirm GnuPG is installed:

gpg --version

Tutorial

After installing GnuPG above, install iostreams:

Install IOStreams gem:

gem install iostreams --no-doc

Open a ruby interactive console:

irb

Load iostreams:

require "iostreams"

Generate a private and public key that we can use as the sample sender of the encrypted files:

IOStreams::Pgp.generate_key(name: "Sender", email: "sender@example.org", passphrase: "sender_passphrase")

Generate a private and public key that we can use as the sample receiver of the encrypted files:

IOStreams::Pgp.generate_key(name: "Receiver", email: "receiver@example.org", passphrase: "receiver_passphrase")

By default the above keys are RSA 4096 bit encryption keys.

Reference a file path to hold the PGP encrypted data by adding .pgp as a file name extension:

path = IOStreams.path("sample/example.csv.pgp")
# => #<IOStreams::Paths::File:sample/example.csv.pgp pipeline={:pgp=>{}}>

Add the email address for the recipient:

path.option(:pgp, recipient: "receiver@example.org")
# => #<IOStreams::Paths::File:example.csv.pgp @options={:pgp=>{:recipient=>"receiver@example.org"}} pipeline={:pgp=>{:recipient=>"receiver@example.org"}}>

An email address only matches keys with exactly that email address, ignoring case. For example receiver@example.org does not match a key for other_receiver@example.org. The same applies to signer, and to the email: argument of IOStreams::Pgp.list_keys, key?, export, delete_keys and set_trust. To search the way gpg does, supply a value that is not just an email address, such as *receiver@example.org.

Write data to the PGP file:

path.writer do |io|
  io << "name,login\n"
  io << "Jack Jones,jjones\n"
  io << "Jill Smith,jsmith\n"
end

Lets try to read the file without supplying a passphrase:

IOStreams.path("sample/example.csv.pgp").read
# IOStreams::Pgp::Failure
#  gpg: decryption failed: No secret key

In order to decrypt the file it needs the passphrase for the receivers private key above:

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, passphrase: "sender_passphrase")
path.read
# IOStreams::Pgp::Failure
#        "Receiver <receiver@example.org>"
#  gpg: public key decryption failed: Bad passphrase
#  gpg: decryption failed: No secret key

It failed again because we tried to use the senders passphrase. Since only the receiver can decrypt this file we need to use its passphrase and therefore private key:

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, passphrase: "receiver_passphrase")
path.read
# => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n"
Sign the file

To prevent a man-in-the-middle attack we can sign the file so that the recipient knows the file came from the sender:

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, recipient: "receiver@example.org", signer: "sender@example.org", signer_passphrase: "sender_passphrase")
path.writer do |io|
  io << "name,login\n"
  io << "Jack Jones,jjones\n"
  io << "Jill Smith,jsmith\n"
end

Try reading the pgp encrypted file that is now also signed by the sender:

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, passphrase: "receiver_passphrase")
path.read
# => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n"

This time when we read the file the signature is automatically verified. However, this only works if the receiver has already imported the senders public key.

Check who signed the file

gpg verifies any signature in the file, but on its own it accepts a file that is not signed, or that is signed by any key in the keyring. To require that the file was signed by the sender, supply the signer when reading:

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, passphrase: "receiver_passphrase", signer: "sender@example.org")
path.read
# => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n"

The signer is an email address, key id or fingerprint. An email address only matches keys with exactly that email address, ignoring case. Reading raises IOStreams::Pgp::Failure unless the file has a good signature by one of the signer’s keys, and gpg fully or ultimately trusts that key, for example after importing it with IOStreams::Pgp.import_and_trust, or setting its trust with IOStreams::Pgp.set_trust.

Instead of importing the sender’s public key on every server, supply it with import_and_trust_key, see import_and_trust_key. When reading, the key is imported, and the file must be signed by it, or by the signer when both are supplied:

sender_public_key = SecretConfig.fetch("suppliers/acxiom/pgp/public_key")

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, passphrase: "receiver_passphrase", import_and_trust_key: sender_public_key)
path.read

Since the key itself is supplied, a signature by it is accepted whatever trust gpg places in it. So when reading, the trust of the key is only changed when import_and_trust_level is supplied, unlike when writing, where it defaults to 5 (Ultimate). A key that is already trusted, for example to encrypt files for that partner, keeps its trust.

Like the integrity and signature checks, this check can only be made once the whole file has been read, so by default IOStreams::Pgp::Failure is raised after the contents have been processed. Supply verify_first: true to only process the contents once every check has passed, see Verifying a file before processing it.

Sign without encrypting

Sometimes the contents do not need to be kept secret, but the recipient still needs to verify that the file came from the sender and was not tampered with. Set encrypt: false to sign the file without encrypting it. In this mode a signer is required, and recipient / import_and_trust_key are ignored:

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, encrypt: false, signer: "sender@example.org", signer_passphrase: "sender_passphrase")
path.writer do |io|
  io << "name,login\n"
  io << "Jack Jones,jjones\n"
  io << "Jill Smith,jsmith\n"
end

Because a signed-only file is not encrypted, no passphrase is needed to read it. The signature is still verified automatically when the senders public key has been imported:

IOStreams.path("sample/example.csv.pgp").read
# => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n"
Auto-sign all files

To automatically sign every PGP output file, set the global config options to sign with the sender credentials.

IOStreams::Pgp::Writer.default_signer            = "sender@example.org"
IOStreams::Pgp::Writer.default_signer_passphrase = "sender_passphrase"

Now the file will be signed automatically without needing to supply the signer and signer_passphrase on every write.

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, recipient: "receiver@example.org")
path.writer do |io|
  io << "name,login\n"
  io << "Jack Jones,jjones\n"
  io << "Jill Smith,jsmith\n"
end

Decrypt output files

In the examples above the sender cannot decrypt the file because it was generated for the recipient only.

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, passphrase: "sender_passphrase")
path.read
# IOStreams::Pgp::Failure
#        "Receiver <receiver@example.org>"
#  gpg: public key decryption failed: Bad passphrase
#  gpg: decryption failed: No secret key

Since the sender cannot decrypt the file how do we know what was actually sent to the recipient?

There are 2 options:

Add the sender to the recipient list every time a file is created.

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, recipient: ["receiver@example.org", "sender@example.org"])
path.writer do |io|
  io << "name,login\n"
  io << "Jack Jones,jjones\n"
  io << "Jill Smith,jsmith\n"
end

Now the sender can also read the file:

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, passphrase: "sender_passphrase")
path.read
# => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n"

Set the global configuration option to add the sender automatically to all PGP output files.

IOStreams::Pgp::Writer.audit_recipient = "sender@example.org"

The sender is now automatically added to the recipient list every time a file is created.

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, recipient: "receiver@example.org")
path.writer do |io|
  io << "name,login\n"
  io << "Jack Jones,jjones\n"
  io << "Jill Smith,jsmith\n"
end

Now the sender can also read the file:

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, passphrase: "sender_passphrase")
path.read
# => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n"

By now you must be getting tired of typing in the senders passphrase, so lets set the config option to remove that requirement:

IOStreams::Pgp::Reader.default_passphrase = "sender_passphrase"

Try reading the above file again:

IOStreams.path("sample/example.csv.pgp").read
# => "name,login\nJack Jones,jjones\nJill Smith,jsmith\n"

Cleanup

To remove the test PGP keys created above:

IOStreams::Pgp.delete_keys(email: 'sender@example.org', private: true)
IOStreams::Pgp.delete_keys(email: 'receiver@example.org', private: true)

import_and_trust_key

The standard approach with PGP is to import public keys into the keystore on every server. Copying the public keys to every server can be tedious, especially when running inside docker containers.

An ideal way to manage PGP public keys is in a database, or in Secret Config that stores the keys in the AWS SSM Parameter store.

Fetch the recipients public key from a data store. For example Secret Config

public_pgp_key = SecretConfig.fetch("suppliers/acxiom/pgp/public_key")

When creating the pgp encrypted file let IOStreams import and trust the public key every time so that it does not have to be managed on every production server. The option import_and_trust_key takes the public key as a string:

path = IOStreams.join("test/sample.pgp", root: :downloads)
path.option(:pgp, import_and_trust_key: public_pgp_key)
path.write("Hello World")
​~~~

Now try to read the file:
~~~ruby
IOStreams.join("test/sample.pgp", root: :downloads).read

Trust level

The key is imported and then marked as trusted. The file is encrypted to the imported key itself, whatever its trust level, so that another key in the keyring with the same email address is never used instead. The trust level can be controlled with the import_and_trust_level option:

path = IOStreams.join("test/sample.pgp", root: :downloads)
path.option(:pgp, import_and_trust_key: public_pgp_key, import_and_trust_level: 4)
path.write("Hello World")

Or by calling IOStreams::Pgp.import_and_trust directly with the trust_level argument:

IOStreams::Pgp.import_and_trust(key: public_pgp_key, trust_level: 4)

The available levels are the same as those used by IOStreams::Pgp.set_trust:

Level Meaning
1 Undefined (no opinion)
2 Never (do not trust)
3 Marginal
4 Full
5 Ultimate (default)

Security warning

Only import and trust keys that were received from a verified, trusted source.

The default trust level is 5 (Ultimate), which tells GPG to treat the imported key as if it were one of your own keys: it becomes implicitly valid and can in turn confer validity on other keys that it has signed. Importing an attacker supplied key at this level allows that attacker to impersonate other recipients. When a key cannot be fully verified, supply a lower trust_level. The file is still encrypted to it, since IOStreams encrypts to the imported key itself.

Compression:

The compression used by pgp can be specified by suppling the :compress option.

The valid values for this option: :none, :zip, :zlib, or :bzip2.

The default compression is :zip for which has the highest compatibility.

Most PGP tools now support :zlib and is the recommended compression to use when possible.

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, recipient: "receiver@example.org", compress: :zlib)
path.write("Hello World")

The compression level can be adjusted with the :compress_level option, where 1 is the fastest and 9 compresses the most:

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, recipient: "receiver@example.org", compress: :zlib, compress_level: 9)
path.write("Hello World")

Default: 6

Compression Performance

Verifying a file before processing it

gpg can only check a file’s integrity (MDC) and its signature once it has read the whole file. By default IOStreams passes the decrypted contents to your block as gpg decrypts them, so that files of any size can be processed without storing the decrypted contents. When a check fails, IOStreams::Pgp::Failure is raised after the block has already processed the contents.

For example, with a signed file whose contents were changed after it was signed, every record is processed before the BAD signature failure is raised:

IOStreams.path("sample/example.csv.pgp").each(:hash) do |row|
  # Receives every row, including the changed ones.
end
# IOStreams::Pgp::Failure: ... gpg: BAD signature from "Sender <sender@example.org>"

When the block returns before reading the whole file, for example after reading only the first line, the rest of the file is still decrypted and checked before the block’s result is returned.

Either do not commit any side effects until the block returns without raising, for example by processing the file within a database transaction, or supply the verify_first option:

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, passphrase: "receiver_passphrase", verify_first: true)
path.each(:hash) do |row|
  # Only called once gpg has checked the whole file.
end

With verify_first, the whole file is first decrypted into a temporary file that only the current user can read. The block is only called once gpg has successfully checked the file, and the temporary file is deleted afterwards. This requires local disk space for the decrypted contents, and an extra pass over the data.

Reading legacy files without MDC integrity protection

Modern GnuPG refuses to decrypt files that lack MDC (Modification Detection Code) integrity protection, failing with gpg: decryption forced to fail!. Some legacy or enterprise systems still produce such files. To read them, supply the ignore_mdc_error option:

path = IOStreams.path("sample/example.csv.pgp")
path.option(:pgp, passphrase: "receiver_passphrase", ignore_mdc_error: true)
path.read

Security warning

Only enable ignore_mdc_error for files from a trusted source: without MDC the decrypted contents are not protected against tampering.

Note: IOStreams never writes files without MDC, this option only applies when reading.

PGP FAQ:

If you get not trusted errors

gpg --edit-key sender@example.org

Select highest level: 5

PGP Limitations