FIO02-J. Keep track of bytes read and account for character encoding while reading data

According to the Java API [API 06] for the class InputStream, with an array of bytes b as the parameter, the read(b) method:

Reads some number of bytes from the input stream and stores them into the buffer array b. The number of bytes actually read is returned as an integer. The number of bytes read is, at most, equal to the length of b.

Note that the read() methods return as soon as they find available input data. By default, none of them guarantees that all the requested bytes will be read. It is left to the programmer to check the number of bytes read and call the read() method again as required. Ignoring the result returned by the read() methods is a violation of EXP02-J. Do not ignore values returned by methods.

Multibyte encodings such as UTF-8 are used for character sets that require more than one byte to uniquely identify each constituting character. For example, the Japanese encoding Shift-JIS (shown below), supports multibyte encoding wherein the maximum character length is two bytes (one leading and one trailing byte).

The trailing byte ranges overlap the range of both the single byte and lead byte characters. This can cause issues because if a multibyte character is separated between buffer boundaries, it will be interpreted differently, as defined by its composing bytes [Phillips 05].

A third issue is caused because of the behavior of the String class constructor with respect to the default encoding. See FIO03-J. Specify the character encoding while performing file or network IO for more details on this issue.

Noncompliant Code Example

This noncompliant code example attempts to read a specific number of bytes from a FileInputStream but suffers from a few pitfalls. The objective is to read 1024 bytes and return them as a String. Unfortunately, this may not happen because of the general contract of the read() methods.

public static String readBytes(FileInputStream in) throws IOException {
  String str = "";
  byte[] data = new byte[1024];
  while (in.read(data) > -1) {
    str += new String(data);
  }
  return str;
}

A second issue involves multibyte character encoding. It is possible for the read() method to read data from the stream terminating the String buffer str with the leading byte of a multibyte character and in the next iteration reading the trailing bytes. This is because when the bytes are concatenated to str, the multibyte encoding information is lost as it does not extend across buffer boundaries.

Finally, the buffer str contains data represented by the default encoding of the system as no specific encoding is specified in the call to the String class constructor.

Compliant Solution (1)

This compliant solution accounts for the total number of bytes read (and adjusts the remaining bytes' offset) so that the required data is fully read. It also specifies the String encoding explicitly to facilitate portability across systems that use different default encodings.

public static String readBytes(FileInputStream in) throws IOException {
  int offset = 0;
  int bytesRead = 0;
  byte[] data = new byte[1024];
  while (true) { 
    bytesRead += in.read(data, offset, data.length - offset);
    if (bytesRead == -1 || offset >= data.length)
      break;
    offset += bytesRead;
  }
  String str = new String(data, "UTF-8");
  return str;
}

The size of the data byte buffer depends on the maximum number of bytes required to write an encoded character. For example, UTF-8 encoded data requires a maximum of three bytes to denote one character. Although it seems counter intuitive, any character above U+FFFF requires a maximum of four bytes. However, such a sequence is split into two separate char values of two bytes each as Java internally uses UTF-16 for representing a char. Consequently, the buffer size should be four times the size of a typical byte sequence.

Compliant Solution (2)

The no argument and one argument readFully() methods of the DataInputStream class can be used to read all the requested data. An EOFException is thrown if the end of input is detected before the required number of bytes have been read, and an IOException is thrown if some other input/output error occurs. The exception handler decides the way forward.

public static String readBytes(FileInputStream fis) throws IOException {
  byte[] data = new byte[1024];
  DataInputStream dis = new DataInputStream(fis);
  dis.readFully(data);
  String str = new String(data,"UTF-8");
  return str;
}

Risk Assessment

Failure to comply with this guideline can result in the wrong number of bytes being read or character sequences being interpreted incorrectly.

Automated Detection

TODO

Related Vulnerabilities

Search for vulnerabilities resulting from the violation of this rule on the CERT website.

References

[[API 06]] Class InputStream, DataInputStream
[[Phillips 05]]
[[Harold 99]] Chapter 7: Data Streams, Reading Byte Arrays
[[Chess 07]] 8.1 Handling Errors with Return Codes
[[MITRE 09]] CWE ID 135 "Incorrect Calculation of Multi-Byte String Length"

FIO01-J. Do not let Runtime.exec() fail or block indefinitely      09. Input Output (FIO)      FIO03-J. Specify the character encoding while performing file or network IO