File Formats

When reading or writing rows (:array) or records (:hash), IOStreams converts each line to or from the file’s tabular format. The following formats are supported:

• :csv Comma Separated Values
• :psv Pipe Separated Values
• :json One JSON document per line
• :fixed Fixed width columns
• :array Each line is already an array of values
• :hash Each line is already a hash

PSV has no way to escape values, so when writing PSV, a | within a value is replaced with : and a line break with a space, so that a value cannot add columns or records.

Each value is read as text, unless a schema gives the type of its field, such as an Integer or a Date.

Format inference

The format is inferred from the file name when it contains a recognized extension:

IOStreams.path("sample.csv").each(:hash) { |record| p record }
IOStreams.path("sample.json").each(:hash) { |record| p record }
IOStreams.path("sample.psv").each(:hash) { |record| p record }

The format extension can appear anywhere in the file name, so sample.csv.gz and sample.json.pgp are recognized as CSV and JSON respectively.

When the file name does not contain a recognized format extension, the format defaults to :csv.

Specifying the format

When the file name cannot be used to infer the format, set it explicitly with format:

path = IOStreams.path("sample_data")
path.format(:json)
path.each(:hash) { |record| p record }

format can be chained with the other path methods:

IOStreams.path("sample_data").format(:json).each(:hash) { |record| p record }

IOStreams.tabular_formats returns the names of the formats that format accepts, such as :csv, :json and :fixed, for example to check a format held in configuration:

IOStreams.tabular_formats.include?(:json)
# => true

Format options

Format specific options are supplied with format_options. They are passed to the parser for the chosen format. Only the :fixed format takes any: the layout of its file, as shown in the next section.

format_options is deprecated, and reports a deprecation warning, see Seeing deprecation warnings: give the width of each field of a fixed width file in a schema instead, see Moving a layout to a schema. It still works as before.

Fixed width files

Fixed width files have no delimiters; each column is identified by its position within the line. Since the layout cannot be inferred from the file, give the width of each field in a schema, which also gives the type of each field. The layout of the deprecated format_options, below, still works as before, see Moving a layout to a schema:

path = IOStreams.path("sample_data")
path.format(:fixed)
path.format_options(
  layout: [
    {size: 23, key: "name"},
    {size: 40, key: "address"},
    {size: 5,  key: "zip"}
  ]
)
path.each(:hash) { |record| p record }

Writing a fixed width file uses the same layout to render each record:

path = IOStreams.path("sample_data")
path.format(:fixed)
path.format_options(
  layout: [
    {size: 23, key: "name"},
    {size: 40, key: "address"},
    {size: 5,  key: "zip"}
  ]
)
path.writer(:hash) do |io|
  io << {"name" => "Jack Jones", "address" => "Somewhere", "zip" => 12345}
end

Note: The keys in the hashes being written must match the layout :key values exactly, including whether they are strings or symbols.

A layout does not take the columns of the rows written from a header row, such as the IOStreams::Row::HeaderRow that each(:array) yields with a schema, or that copy_from(source, mode: :array) hands it from a source with a schema, since it does not match them to its keys, so it raises IOStreams::Errors::InvalidHeader, rather than write blank values. Give the target a schema instead, which matches them to its fields.

Layout column definitions:

• :size The number of characters this column occupies. The last column may use a size of :remainder to take the rest of the line as its value.
• :key The name for this column. Leave out the key to ignore the column during parsing, and to space fill when rendering.
• :type :string (default), :integer, or :float. Strings are left justified and space padded, numbers are right justified and zero padded. When writing, line breaks within a string are replaced with a space so that a value cannot add records. Raises IOStreams::Errors::ValueTooLong when an :integer or :float value cannot be rendered in size characters.
• :decimals For :float columns, the number of decimal places to render. Default: 2

In addition to layout, the :fixed format takes one more option:

• truncate: [true|false] Whether to truncate string values that are longer than their column :size when writing. When false, a string value that is too long raises IOStreams::Errors::ValueTooLong instead of being truncated. Numeric values are never truncated. Default: true

Header options

When reading or writing records (:hash), the following options control the header row. They are deprecated, and report a deprecation warning, as does cleanse_header, even when they are nil: a schema describes the columns of the file on its path instead, see Moving the column options to a schema. They still work as before without a schema. A writer ignores allowed_columns, required_columns and skip_unknown, which only apply when reading.

With a schema, which describes the columns of the file instead, columns, allowed_columns and required_columns raise ArgumentError unless they are nil, and so does cleanse_header, even when it is nil. skip_unknown only applies to allowed_columns, so it is accepted. A nil one, and skip_unknown, have no effect.

• columns: [Array<String>] When reading, supplies the header columns for files that do not include a header row. When writing, sets the columns to write, including their order. Keys not listed in columns are ignored during writes.

• cleanse_header: [true|false] Whether to cleanse the column names read from the header row. Column names are stripped of leading and trailing whitespace, lowercased, and spaces and dashes are converted to underscores, so the header " First Name " becomes "first_name". Default: true

Without a schema, two columns of a header row that are the same key of a record, such as a,a,b, or A and a, which are cleansed to the same key, give each record the value of only the last of them. This is deprecated, and reports a deprecation warning, since IOStreams 3.0 raises IOStreams::Errors::InvalidHeader for them, rather than lose the others: rename the columns, or read the rows with each(:array), which holds every column.

• allowed_columns: [Array<String>] List of columns to allow. Any other columns are ignored when skip_unknown is true, otherwise an IOStreams::Errors::InvalidHeader exception is raised. Default: nil (allow all columns)

• required_columns: [Array<String>] List of columns that must be present, otherwise an exception is raised.

• skip_unknown: [true|false] When true, any columns not present in allowed_columns are skipped entirely as if they were not in the file at all. When false, an unknown column raises IOStreams::Errors::InvalidHeader. Default: true

By default, when reading records, allowed_columns, required_columns and skip_unknown only apply to a header row read from the file, and only when cleanse_header is true. They are ignored for JSON and :hash input, when columns are supplied, and with cleanse_header: false, and a warning is logged when applying them would change the records read.

Since the format is usually inferred from the file name, renaming an uploaded file from .csv to .json bypasses them. A schema applies to every input instead, including the keys of each JSON record. IOStreams.enforce_column_restrictions = true applies the options to every input too, but it is deprecated with them, see Moving the column options to a schema.

Example, reading a headerless CSV file:

path = IOStreams.path("no_header.csv")
path.each(:hash, columns: ["name", "address", "zip"]) do |record|
  p record
end

Example, writing only specific columns in a fixed order:

path = IOStreams.path("sample.csv")
path.writer(:hash, columns: ["name", "zip"]) do |io|
  io << {"name" => "Jack Jones", "address" => "Somewhere", "zip" => 12345}
end
path.read
# => "name,zip\nJack Jones,12345\n"

Note: Column names are converted to strings, and the keys in the hashes being written may be strings or symbols.

Schema

A schema gives the fields of the rows and records of a file, and the type of each one, so that reading them returns each value as its type, such as an Integer or a Date, rather than as text, and writing them writes each value in the text form of its type. It applies to every format, and gives the position of each field of a fixed width file. Its fields name the keys of the records read, and say which columns a file can have and must have, so that the keys do not depend on how cleansing changes a column, such as Amount ($) and Amount (€), which both cleanse to amount_.

A schema is a Frictionless Table Schema, set on the path with #schema, either as a Hash:

# Read the id and joined columns of the file as their types, and skip its other columns.
path = IOStreams.path("people.csv").schema(
  fieldsMatch: "partial",
  fields:      [
    {name: "id",     type: "integer"},
    {name: "joined", type: "date"}
  ]
)
path.each(:hash) do |person|
  p person
end
# => {"id" => 42, "joined" => #<Date: 2026-10-13 ((2461327j,0s,0n),+0s,2299161j)>}

Or as the path or url of a JSON file that holds it, in any store that IOStreams reads, so that the schema can be held in configuration, like the path of the file:

IOStreams.path(ENV["PEOPLE_PATH"]).schema(ENV["PEOPLE_SCHEMA"]).each(:hash) do |person|
  p person
end

The columns of the file are matched to the fields, and the keys of each record read are the names of its fields, see Matching columns to fields. fieldsMatch: "partial" above reads the fields that the file has, and skips its other columns. Without fieldsMatch, as in Frictionless, the columns of the file must be the fields, in order, which is what a file written with the schema has:

path = IOStreams.path("people.csv").schema(
  fields: [
    {name: "id",     type: "integer"},
    {name: "pay",    type: "number", decimals: 2},
    {name: "joined", type: "date"},
    {name: "active", type: "boolean"}
  ]
)
path.writer(:hash) do |io|
  io << {"id" => 42, "pay" => BigDecimal("12.5"), "joined" => Date.new(2026, 10, 13), "active" => true}
end
path.read
# => "id,pay,joined,active\n42,12.50,2026-10-13,true\n"

path.each(:hash) do |person|
  p person
end
# => {"id" => 42, "pay" => 0.125e2, "joined" => #<Date: 2026-10-13 ((2461327j,0s,0n),+0s,2299161j)>, "active" => true}

A schema in a Hash is checked when it is set. A schema file is read the first time that a reader or writer needs it, so that setting it does not read the file, and the copies of a path read it once. #schema returns the schema that was set. Without a schema, the rows and records of a file are read and written as before.

Like #format, the schema describes the file, so it is set on the path, rather than supplied to each call of #each, #reader or #writer, which raises ArgumentError for schema:. Since it describes the columns of the file, the header options that describe them without a schema raise ArgumentError with it, and so does the layout of a fixed width file in #format_options, when a reader or writer of the path is created.

The text of a file read with a schema is UTF-8, so that its values are UTF-8 strings, and so are its lines read with each(:line), and text that is not valid UTF-8 raises IOStreams::Errors::InvalidEncoding, which is an Encoding::UndefinedConversionError, naming the line that it is on, which its #line_number returns. An encoding set with encoding, such as encoding("BINARY"), is used instead, without checking the text, as it is without a schema, and raw reads the data as it is stored. Binary text written to rows and records with a schema, such as the values of a file without one that is copied, is written as UTF-8, and raises IOStreams::Errors::InvalidEncoding when it is not valid UTF-8, unless an encoding is set, which writes it instead, as it does without a schema. Lines written with writer(:line) are written as they are.

Types

A field without a type is any, as in Frictionless.

Empty text is a missing value, read as nil in every type, including a string or any field, as the default missingValues of Frictionless, [""], says, so that a missing value is nil whether the file leaves it out, such as JSON’s null, or holds empty text, such as "" in CSV or JSON. Text of only whitespace is not missing: a string or any field keeps it as the text, and any other type raises IOStreams::Errors::InvalidValue for it, as it does for N/A. The leading and trailing whitespace of any other text is removed before it is read as a type other than string or any, so that ` 42 ` is the integer 42. Writing works the same way, so that empty text is written as a missing value, which CSV writes as an empty value, such as ,21,, rather than a quoted "", and JSON as null, and text of only whitespace raises for a type other than string or any.

A spreadsheet read with a schema holds the number of each number cell, to the 15 significant digits that Excel displays, or every digit of a whole number below 2^53, which a Float holds exactly, such as the card number 4111111111111111, without its number format, such as 0.125 for 12.5%, in plain digits, such as 0.0000001, and without a decimal point for a whole number, such as 2134, or 0.3 for 0.30000000000000004, in its lines and in its rows and records, so that a string field reads that text, and a number field those digits. A cell of a date holds its date in ISO 8601, such as 2026-10-25, one of a date and time the time of its clock in UTC, such as 2026-10-25T09:30:00Z, since Excel has no time zone, and one of a time of day the time of its clock, such as 09:30:00, so that a date or datetime field reads them. Which cells are dates and times is decided by their number formats, as v3.0 decides it: Excel’s own formats of a date, a time of day, or both, and a custom format of only a year, month or day, such as yyyy-mm-dd or dd/mm/yyyy, which is a date, of only hours, minutes or seconds, such as hh:mm:ss, a time of day, or of both, such as dd/mm/yyyy hh:mm, a date and time, while quoted text, such as the d of 0.00" days", or a color, such as [Red]0.00, does not make a number a date. Without a schema, its cells are read as before, such as 2134.0, as creek reads them, which takes every custom format with a letter of a date or a time, such as yyyy-mm-dd or 0.00" days", as a date and time.

A Hash or an Array of an object or array field, such as one of the :hash format, is converted to what JSON holds of it, so that it is the same as the value that its JSON text reads: its keys are Strings, such as "theme" for :theme, a Symbol within it is its text, and a BigDecimal, Date, Time or DateTime within it is the text that IOStreams writes for it to JSON, such as "12.5", see Values of a field without a type. Any other value is kept as it is, so that a value of another class, such as a Rational, can differ from the value that its JSON text reads. Hashes and Arrays nested more than 100 deep, as JSON reads them, raise IOStreams::Errors::InvalidValue, as does a Float within them that is not finite, such as Float::NAN, which JSON cannot hold, when it is written, in every format, naming the field, also in an application that uses ActiveSupport, as Rails does, which would write it as null. Frictionless keeps a dict as it is, and reads a boolean only from text, and true or false, so this conversion, and reading the number 1 or 0 as a boolean, are what IOStreams adds.

A value that is not one of the type of its field, such as N/A for an integer, raises IOStreams::Errors::InvalidValue, naming the field and the value, both when reading and when writing. So does a date or datetime that does not exist, such as 2026-02-30.

JSON writes integers, numbers and booleans as JSON numbers and booleans, with all of the digits of a number, and the digits after the decimal point that its decimals give, such as 12.50, or none for a whole number without decimals, such as 42, the same as CSV, an object or an array as itself, and every other value as text, such as a date, including in an application that uses ActiveSupport, as Rails does, which replaces how Ruby writes JSON. Reading JSON keeps all of the digits of a number too, rather than reading it through a Float, rounded to its decimals when it has them, while the numbers of the other fields are a Float, as they are without a schema. The :hash and :array formats keep the values themselves, apart from a number with decimals, which is rounded to them, both when reading and when writing.

A number field needs the bigdecimal gem, which is a bundled gem from Ruby 3.4, so add gem "bigdecimal" to the Gemfile, or set floatNumber: true to read a Float instead.

Matching columns to fields

Each column of the header row, or key of a JSON record, is matched to the field:

1. Whose name is the column, such as email for the column email, whether the name is a String or a Symbol.
2. Whose aliases include the column, such as ["DOB", "Birth Date"] for the field date_of_birth.
3. Whose name or alias is the same as the column once both are compared without their case, spaces and punctuation, and with the forms of a character that Unicode treats as the same made the same. So the field first_name matches the columns First Name, FIRST.NAME and firstname, the field email matches E-mail, and id matches the full width ＩＤ that some spreadsheets have.
4. Whose name or alias is the same as the column once both are cleansed as cleanse_header cleanses a header row, see Header options, and then compared the same way. Unlike cleanse_header without a schema, which keeps only the ASCII letters and digits, this cleansing keeps the letters and digits of every language, as IOStreams 3.0 cleanses a header row, so that a field named as a header row is cleansed matches it in both. It drops the characters that Unicode otherwise makes digits, such as ², ½ and ①, so that a field named as the cleansed column, such as area_m for Area (m²), matches it, as area_m2 does.

A column that is the same as several fields in step 3, or as none, is the field whose name or alias is the column once cleansed, if there is one. So a schema whose fields are named as a header row is cleansed finds every column that the cleansing named, such as the fields e_mail and email, which are the same once compared loosely, for the columns E-mail and Email.

The columns are matched once, when the header row is read, or once for each set of keys of the JSON records, which are usually the same for every record, rather than for each value.

The keys of the records read are the names of their fields as they are given, so a field named with a Symbol, such as name: :id, returns the key :id, whatever the column of the file is called, unless keyType forms them otherwise, see Keys of the records read. For a format with a header row, such as CSV, each(:array) returns a header row of the keys of the fields of its columns, and then the values of those fields of each row, in the order of the fields. A file without a header row, such as JSON or fixed width, or with a schema whose header is false, returns the keys of the fields as its header row, and then the values of every field of each row, in order, nil for a key that a JSON record does not have. The header row is an IOStreams::Row::HeaderRow, an Array that says that it is the header row, so that code that reads the rows can skip it with next if row.is_a?(IOStreams::Row::HeaderRow). A column that the schema does not list is skipped, or raises, see fieldsMatch.

A column that could be either of two fields raises IOStreams::Errors::InvalidHeader, rather than guess, such as amount for the fields Amount ($) and Amount (€), which match their own columns exactly. So do two columns that are the same field, such as email and Email, rather than lose the value of one of them.

When writing, the header row is the names of the fields, in order, and the keys of each record written are matched to the fields in the same way. writer(:array) writes that header row itself, before the rows, and takes each row written as the values of the fields, in order, whatever the format:

path = IOStreams.path("people.csv").schema(
  fields: [
    {name: "name"},
    {name: "age", type: "integer"}
  ]
)
path.writer(:array) do |io|
  io << ["Jack", 21]
end
path.read
# => "name,age\nJack,21\n"

A header row written to it, such as the IOStreams::Row::HeaderRow that each(:array) yields with a schema, gives the columns of the rows after it instead, without a line of its own, and its columns are matched to the fields in the same way, so that the values of each row after it can be in any order, and the rows read from another file with a schema can be written as they are, by a loop over each(:array). A field that the header row does not have is written as empty. copy_from(source, mode: :array) hands the target the header row of the source in the same way, also when the source has no schema, and skips the blank rows of the source.

So a file can have readable column names, while the code uses its own keys:

report = IOStreams.path("report.csv").schema(
  fields: [
    {name: "First Name"},
    {name: "Date of Birth", type: "date"}
  ]
)
report.writer(:hash) do |io|
  io << {first_name: "Jack", date_of_birth: Date.new(2001, 2, 3)}
end
report.read
# => "First Name,Date of Birth\nJack,2001-02-03\n"

Keys of the records read

keyType, which IOStreams adds to the properties of a schema, forms the key of each field in the records read from its name:

So a file can have readable column names, which are the names of the fields, while the code reads the keys that it needs:

path = IOStreams.path("people.csv").schema(
  fieldsMatch: "partial",
  keyType:     "cleansedSymbol",
  fields:      [
    {name: "First Name"},
    {name: "Date of Birth", type: "date"}
  ]
)
path.each(:hash) do |person|
  p person
end
# "First Name,Date of Birth\nJack,2001-02-03\n" => {first_name: "Jack", date_of_birth: #<Date: 2001-02-03 ...>}

path.writer(:hash) do |io|
  io << {first_name: "Jill", date_of_birth: Date.new(2002, 3, 4)}
end
# => "First Name,Date of Birth\nJill,2002-03-04\n"

The header row written is still the names of the fields, and a record written can be keyed by the keys of the records read, as well as by the names, so that the records read can be written back. A schema held as JSON, or in a database such as MongoDB, has String names, so keyType: "symbol" keeps the Symbol keys that a schema in Ruby has, such as the schema of a layout, see IOStreams::Schema.from_layout in Moving a layout to a schema. Two fields with the same key, such as First Name and first-name with cleansed, raise ArgumentError, since a record could not hold both, and so does a key that is the name or an alias of another field.

fieldsMatch

fieldsMatch says how the columns of the file must match the fields, as Frictionless defines it:

A file that does not match raises IOStreams::Errors::InvalidHeader, which lists each problem, such as the columns that are not fields, and the fields that are missing. The columns without a name at the end of the header row, such as the empty column that a spreadsheet exports after the last one, or after a comma at the end of each line, are not columns, so a value under one is a value after the last column, see below. A column without a name between other columns is a column that the schema does not list.

partial is the easiest to start with, and suits a schema that only gives the type of some of the columns. A schema for files that are received from elsewhere, such as uploads, is usually stricter: subset accepts files with more columns than the code needs, and equal or exact accept only the columns that the schema lists.

The keys of each JSON record are checked the same way, in any order. The columns of a file without a header row are the fields in order, so header: false only accepts exact, and a fixed width file always has every field.

A row read that has a value after the last column of its header row, or after the last field of a file without one, such as when a value has a comma that is not quoted, raises IOStreams::Errors::InvalidHeader too, unless fieldsMatch skips it, rather than lose the value. A blank value there, such as after a comma at the end of each line, is not data, so it is ignored.

A file that is written always has every field, so writing only checks the keys of each record, the columns of the header row of the rows written, and the values of each row after the last field or column: subset and partial skip them, and the others raise IOStreams::Errors::InvalidHeader. A field that a record does not supply is written as empty, or as null in JSON.

Keeping the other columns

extraColumns: "keep", which IOStreams adds to the properties of a schema, keeps the columns that subset and partial would skip, after the fields, in the order of the columns, each by the key that keyType forms from its column, and with its value as it is. So a file can need some columns, and the code still read every other column that it has:

path = IOStreams.path("upload.csv").schema(
  fieldsMatch:  "subset",
  extraColumns: "keep",
  keyType:      "cleansed",
  fields:       [{name: "zip_code", constraints: {required: true}, allowBlank: true}]
)
path.each(:hash) do |record|
  p record
end
# "Zip Code,First Name,Notes\n12345,Jack,hi\n" => {"zip_code" => "12345", "first_name" => "Jack", "notes" => "hi"}

With subset, a schema can then have no fields, to read every column of every file, keyed as keyType forms them, or as the columns are written without it. The keys of each JSON record are kept the same way, so its rows, which have a header row of the keys of the fields, cannot be read with each(:array), which raises ArgumentError. A fixed width file has only its fields, so it ignores extraColumns. Two columns kept with the same key, such as Notes and notes with keyType: "cleansed", raise IOStreams::Errors::InvalidHeader, rather than lose one of them, and a column without a name is not kept. Writing writes only the fields, whichever it is. The default, "skip", leaves them out, and any other fieldsMatch raises ArgumentError for "keep", since it does not accept them.

Required fields

A field whose constraints are required, as Frictionless defines it, must have a value: a header row, or JSON record, without the field raises IOStreams::Errors::InvalidHeader, whatever fieldsMatch is, and a value that is nil, or empty text, raises IOStreams::Errors::InvalidValue, both when reading and when writing, including a blank record or row written, such as {}, which is otherwise written as a blank line. A blank row read, a line that is empty or only whitespace, or whose every value is nil or blank text, such as a line of CSV of only commas, is nil instead, as a blank line is. Text that is only spaces is a value for a string or any field of a row that has other values, as in Frictionless, while any other type, such as an integer, raises IOStreams::Errors::InvalidValue for it, see Types.

# An upload, whose columns may be in any order, with names that people typed, and columns that the code ignores.
schema = {
  fieldsMatch: "subset",
  fields:      [
    {name: "first_name",    constraints: {required: true}},
    {name: "email",         aliases: ["E-mail Address"], constraints: {required: true}},
    {name: "date_of_birth", type: "date", aliases: ["DOB", "Birth Date"]}
  ]
}
IOStreams.path("upload.csv").schema(schema).each(:hash) do |contact|
  p contact
end
# "E-mail Address,FIRST NAME,DOB,Notes" =>
# {"first_name" => "Jack", "email" => "j@x.com", "date_of_birth" => #<Date: 2001-02-03 ((2451944j,0s,0n),+0s,2299161j)>}

A file can need a column whose values can be blank, such as a middle name that must be in every upload, but that many people do not have. Give its field allowBlank: true with required: a header row, or JSON record, without the field still raises, while a value that is nil, or empty text, is read and written as nil:

schema = {
  fieldsMatch: "partial",
  fields:      [
    {name: "first_name",  constraints: {required: true}},
    {name: "middle_name", constraints: {required: true}, allowBlank: true}
  ]
}
IOStreams.path("upload.csv").schema(schema).each(:hash) do |person|
  p person
end
# "First Name,Middle Name\nJack,\n" => {"first_name" => "Jack", "middle_name" => nil}
# "First Name\nJack\n"               => IOStreams::Errors::InvalidHeader, missing the required field "middle_name"

allowBlank is a property of the field that IOStreams adds, rather than a constraint, since Frictionless defines required as true or false, and a tool reports an error for a constraint that it does not support, while it accepts other properties of a field. A field that is not required raises ArgumentError for it, since its values can be blank anyway.

IOStreams does not support the other constraints of Frictionless yet, such as maxLength, so they raise ArgumentError, rather than appear to check values that they do not.

Schema properties

• fields The fields of the file, in the order of its columns, which can be empty with extraColumns: "keep". Required.
• fieldsMatch How the columns of the file must match the fields, see fieldsMatch. Default: exact
• header false when the file has no header row, as in a Frictionless Table Dialect. The columns of the file are then the fields, in order: reading takes every line as a row, after the names of the fields as the header row that each(:array) yields, and writing writes no header row, and the fields in order. A format without a header row, such as JSON or fixed width, ignores it. Default: true
• width The length of every line of a fixed width file, see Fixed width files with a schema.
• keyType How the keys of the records read are formed from the names of the fields: string, cleansed, symbol or cleansedSymbol, see Keys of the records read. IOStreams adds this property. Default: the names as they are given.
• extraColumns keep to keep the columns that the schema does not list, which subset and partial skip, see Keeping the other columns. IOStreams adds this property. Default: skip

$schema, title and description describe the schema to people and have no effect.

# A CSV file without a header row.
path = IOStreams.path("people.csv").schema(
  header: false,
  fields: [
    {name: "name"},
    {name: "age", type: "integer"}
  ]
)
path.each(:hash) do |person|
  p person
end
# => {"name" => "Jack", "age" => 21}

A column of a file without a header row that the code does not read still needs a field, which holds its place, so give it skip: true, and the records read leave it out:

path = IOStreams.path("people.csv").schema(
  header: false,
  fields: [
    {name: "name"},
    {name: "unused", skip: true},
    {name: "age", type: "integer"}
  ]
)
path.each(:hash) do |person|
  p person
end
# "Jack,x,21\n" => {"name" => "Jack", "age" => 21}

The rows read leave it out too, and writing writes an empty value in its column, from records, and from rows of the values of the other fields, in order, as the rows read have them.

Since the schema is set on the path, every reader of the path reads the file the same way, including #copy_from and #copy_to, whose options only apply to the target.

Field properties

• name The key of the field in each record, and its column in the header row written. Required.
• type See Types. Default: any
• constraints {required: true} for a field that must have a value, see Required fields.
• format For a date or a datetime, the strptime pattern of its text, such as "%d/%m/%Y" or "%Y%m%d", in the syntax of Python’s strptime, as Frictionless defines it: %Y reads exactly 4 digits, so that 31/10/26 raises IOStreams::Errors::InvalidValue with %d/%m/%Y, as do %G and the year of a directive that reads several, such as %F and %c, while %y reads exactly 2 digits, the time of a date or a datetime must be on a clock, so that the hour 24 of %H, such as 24:00, and a leap second, 60 of %S, raise, rather than read the next day or the second before it, or be ignored by a date, and %f reads 1 to 6 digits of microseconds, such as 5 for half a second, and writes 6, such as 500000. Nothing is read on the day that it is read, as Ruby’s strptime reads the parts of a date that the text leaves out: a pattern that reads a day of the month, %d or %e, without a year, %Y, %y or %G, such as %m/%d, raises ArgumentError when the schema is created, as Python’s strptime does from 3.15, since it cannot read February 29: add a year to it, and to the text. So does one that reads an ISO 8601 week without its year, %G, its week, %V, and a weekday, such as %u, which give its date together. Text without a year, such as that of %H:%M, %m %H:%M or %I:%M %p, is on 1900-01-01, the year, month and day that Python gives the parts that it leaves out, such as 1900-07-01 09:30 for 07 09:30 with %m %H:%M, the day of the year of %j counts from 1900-01-01, and a weekday or a week, such as %a or %U, is ignored. A pattern that starts with fmt:, as Data Package v0 wrote it, such as "fmt:%d/%m/%Y", has that prefix removed, so that "fmt:default" is the default form. Ruby’s own directives, such as %N, %L and %s, also work, but other Frictionless tools do not read them, and %s reads whole seconds, ignoring a fraction after it, such as that of %s.%f. Default: ISO 8601.
• trueValues and falseValues For a boolean, the text that is true and false. Default: ["true", "True", "TRUE", "1"] and ["false", "False", "FALSE", "0"]
• floatNumber For a number, true to read a Float rather than a BigDecimal, as frictionless-py does.
• aliases The other names of the column of the field, see Matching columns to fields. IOStreams adds this property.
• allowBlank For a field whose constraints are required, true to accept a value that is blank, nil or empty text, while the file must still have its column, see Required fields. Text of only whitespace is not blank, so it is still read as the type of the field, see Types. IOStreams adds this property.
• decimals For a number, the number of digits after the decimal point, rounding a BigDecimal half up, and a Float, with floatNumber: true, as Ruby’s format writes it, as the layouts of fixed width files always have, which rounds a Float whose text ends in 5 to the even digit, whatever its binary value, such as 0.12 for 0.125, 0.02 for 0.025 and 2.68 for 2.675 with decimals: 2, and 2.0 for 2.5 with decimals: 0. It rounds the number read too, such as the BigDecimal 12.35 for 12.345 with decimals: 2, also a number that is not text, such as one of JSON, and the number written to the :hash and :array formats, so that a record read holds the number that is written. A number that rounds to zero is zero, rather than -0.0. IOStreams adds this property.
• skip true for a field that only holds the place of its column, such as a column of a file without a header row that the code does not read: reading leaves it out of the rows and records, and writing writes an empty value in its column, or leaves it out of a JSON record. It takes only the properties that place its column, name, aliases, width and start. IOStreams adds this property.
• width, start and truncate The position of the field in a fixed width file, see Fixed width files with a schema. IOStreams adds these properties. The other formats ignore them, so that one schema can describe both a fixed width file and the CSV file that it is converted to.

title, description, example and rdfType describe a field to people and have no effect, as do the properties of other tools, whose names include a :, such as "mytool:owner". Any other property raises ArgumentError, as do the parts of Frictionless that IOStreams does not support yet, such as missingValues, primaryKey, the constraints other than required, and the time type, so that a schema never appears to check a value that it does not.

Values of a field without a type

A value of a field without a type, any, is written as follows:

JSON writes a BigDecimal, Date or Time as that text, within an object or an array too, so that a BigDecimal keeps all of its digits, and writes every other value as itself, such as a number or true. The :hash and :array formats keep the values themselves. To write a value in another form, such as a date as 13/10/2026, give its field a type and a format in the schema.

A file without a schema writes each value as before, such as a BigDecimal as its to_s, 0.125e2, including the values that a copy reads from a file with a schema.

Fixed width files with a schema

The schema gives the width of each field of a fixed width file, in the order of the line, in place of the layout of format_options:

path = IOStreams.path("sample_data").format(:fixed).schema(
  fields: [
    {name: "name",    type: "string",  width: 23},
    {name: "address", type: "string",  width: 40},
    {name: "zip",     type: "integer", width: 5}
  ]
)
path.each(:hash) do |record|
  p record
end

path.writer(:hash) do |io|
  io << {"name" => "Jack Jones", "address" => "Somewhere", "zip" => 12345}
end

A fixed width file has no header row, so each(:array) yields the names of the fields as its header row, and then the values of the fields of each line, in order. writer(:array) takes rows of the values of the fields, in order, and takes the header row that each(:array) yields as the header row of the rows after it.

The properties that place each field:

• width The number of characters of the field. The last field may have a width of "remainder" to take the rest of the line.
• start The position of the first character of the field, counting from 1, as the specifications of fixed width files do. It skips the characters after the field before it, such as a filler. Default: right after the field before it.
• truncate For a string, true to write only the first width characters of a longer value. Without it, a value that is too long raises IOStreams::Errors::ValueTooLong. A number is never truncated, and raises instead.

The width at the top level of the schema is the length of every line, when the line ends after the last field, such as after a filler, for example {"fields": [...], "width": 94}. A width of "remainder" accepts lines of any length, ignoring what follows the last field. Otherwise every line ends where the last field ends. A line of another length raises IOStreams::Errors::InvalidLineLength.

When reading, each value is the text of its field without its leading and trailing spaces, converted to the type of its field, so that a field of only spaces is empty text, a missing value, nil, see Types. When writing, a number is right justified and zero filled, such as 000042 for 42 without decimals, or 042.00 with decimals: 2, and a missing number is written as zero, as the specifications of most fixed width files require. A number that is not finite, such as NaN or -INF, is space filled instead, so that it can be read back. Any other value is left justified and space filled, with each line break replaced with a space, so that a value cannot add records. The characters that no field covers, such as a filler, are spaces.

Schemas in IOStreams 2.2 and 3.0

A schema reads and writes the rows and records of a file in IOStreams 2.2 as IOStreams 3.0 does, so that code that uses one works the same way after an upgrade, apart from these differences, which IOStreams 3.0 makes to every file:

• The text of a file read with a schema is UTF-8, see Schema, but IOStreams 2.2 does not remove the byte order mark at the start of a UTF-8 file, which is still part of the first column of the header row, which is still matched to its field, see Matching columns to fields, and leaves the line endings as they are. An encoding set on the encode stream decodes the text as it does without a schema, and its text is not checked.
• The positions of the fields of a fixed width file count the characters of its UTF-8 text, while IOStreams 3.0 reads a fixed width file as ASCII, so that a character that is not ASCII raises. Set the encoding of a file that is not ASCII with #encoding, so that both read it the same way: encoding("UTF-8") for UTF-8 text whose sizes count characters, encoding("BINARY") to read binary strings, whose positions count bytes, or the name of a single-byte code page, such as encoding("ISO-8859-1") for Latin-1, whose values are then strings in that encoding. IOStreams 3.0 also takes encoding("ISO-8859-1:UTF-8"), to read UTF-8 strings.
• A file without a schema is read and written as before: its each(:array) yields its header row as an Array, rather than an IOStreams::Row::HeaderRow, so a loop that writes its rows to a writer with a schema writes the header row as a row of values, while copy_from(source, mode: :array) hands it over as the header row. A copy between two files without a schema copies each row as before.
• Parquet files are only in IOStreams 3.0, which removes the layout of format_options, while IOStreams 2.2 reads and writes it with the layout, as before, and IOStreams::Schema.from_layout returns its schema, see Moving a layout to a schema.