The method is supposed to return an equivalent token stream with all newlines being a single :space token.

Also, group and line tokens have to be cut into lines. The implementation of the HTML encoder's :line_numbers => :inline features is a good example of this technique.

Definition:

p(. A Token stream of correctly split lines is a stream where every line break is a single :space token that is not part of any block or line token.

Tokens#split_into_lines must yield such a stream while retaining the token kind integrity for everything else:

• Every character must have the same token kind and block/line token nesting before and after the split.
• The resulting token stream must contain as little additional new tokens as possible.
• Splitting tokens is allowed only if they contain line breaks.

The token nesting may have to be repeated at the beginning of the new line. The method should be implemented in a way that calling it again is a non-op.

The resulting token stream must also be valid:

• Every token value must be a non-empty string or a valid token action (open, close, begin_line, or end_line).
  ** Tokens may contain line breaks.
• Every token must have a valid token kind (as defined in the token_kinds.rb list).
  ** nil is not a valid token kind.
• Every open action must have a companion close action of the same kind.
• Every begin_line action must have a companion end_line action of the same kind.
  ** Line tokens must not be nested.
  ** Tokens between begin_line and end_line must not contain line breaks.

The method optimize (which combines tokens of the same kind) should not create multi-line tokens. In other words, tokens.optimize.split_into_lines == tokens.optimize.split_into_lines.optimize should be true for all token streams.

I guess we need this for three reasons:

• The Terminal encoders sometimes messes up colors when using pagers.
• The HTML encoder with list and div outputs without tables might be improved by this.
• Redmine/Chili project uses its own line numbering, and breaks up the code into lines before highlighting, which might not be necessary when CodeRay can provide line-wise output.

See also: Redmine issue 7495

From Redmine: http://odd-eyed-code.org/issues/222