Add non-default noiseShape support to dropout op #1782

syt123450 · 2019-06-11T11:28:02Z

This PR is a followup PR for tfjs-core#1343 which added dropout op, dropout feature initially requested in tfjs#117. This PR cleans up a TODO, implements non default noise shape. The implementation of noiseShape aligns with TensorFlow Python's dropout API.

The noiseShape feature would benefit further dropout related development, for example, make tf.layers.dropout support noiseShape configuration.

Relative PR:

Make dropout layer support non default noiseShape and seed tensorflow/tfjs-layers#556

Reference:

To see the logs from the Cloud Build CI, please join either
our discussion
or announcement mailing list.

This change is

nsthorat

Reviewable status: 0 of 1 approvals obtained (waiting on @syt123450)

src/ops/dropout.ts, line 112 at r1 (raw file):

export const dropout = op({dropout_});
export const getNoiseShape = op({getNoiseShape_});

I don't see this op in tensorflow it is just a helper method. If you need it from layers then we can put it in a utility, wdyt?

…dropout

syt123450

Hi @nsthorat , thanks for your suggestions! As you said, getNoiseShape should be a utility function for dropout op, I have made the corresponding change. Could you please review the PR again?

Reviewable status: 0 of 1 approvals obtained

src/ops/dropout.ts, line 112 at r1 (raw file):

Previously, nsthorat (Nikhil Thorat) wrote…

I don't see this op in tensorflow it is just a helper method. If you need it from layers then we can put it in a utility, wdyt?

Done. Some of my survey and thoughts:

Inspired by your suggestion, I made getNoiseShape to be a private utility function for dropout, which will not be exported to tfjs-layers. I found that _get_noise_shape is a helper function, which has only two usages in TensorFlow python package, one is in tf.nn.dropout, another is in tf.keras.layers.Dropout.

The tf.keras.layers.Dropout uses this utility function to normalize the noise_shape: tensorflow/python/keras/layers/core.py#L141-L147

def _get_noise_shape(self, inputs):
  # Subclasses of `Dropout` may implement `_get_noise_shape(self, inputs)`,
  # which will override `self.noise_shape`, and allows for custom noise
  # shapes with dynamically sized inputs.
  if self.noise_shape is None:
    return self.noise_shape
  return nn_ops._get_noise_shape(inputs, self.noise_shape)  # pylint: disable=protected-access

While Keras directly normalizes the noise_shape: keras/layers/core.py#L110-L117

def _get_noise_shape(self, inputs):
  if self.noise_shape is None:
    return self.noise_shape

  symbolic_shape = K.shape(inputs)
  noise_shape = [symbolic_shape[axis] if shape is None else shape
    for axis, shape in enumerate(self.noise_shape)]
  return tuple(noise_shape)

Now tfjs-layers works in Keras way which directly normalizes the noiseShape: tfjs-layers/blob/master/src/layers/core.ts#L88-L99

private getNoiseShape(input: Tensor): Shape {
  if (this.noiseShape == null) {
    return this.noiseShape;
  }
  const inputShape = input.shape;
  const noiseShape: Shape = [];
  for (let i = 0; i < this.noiseShape.length; ++i) {
    noiseShape.push(
      this.noiseShape[i] == null ? inputShape[i] : this.noiseShape[i]);
  }
  return noiseShape;
}

In tf.keras, the parameters pass to _get_noise_shape, would pass to tf.nn.dropout again through backend, which means that although tf.keras and Keras implement it in the different ways, they will get the same result.

And as getNoiseShape is not so widely used as other utility function, it seems that there is no need to export it from tfjs-core which can make core's api clean.

…dropout

caisq · 2019-08-08T23:55:33Z

Just as a heads-up: tfjs-layers PR tensorflow/tfjs-layers#556 is blocked on this one.

dsmilkov

Great work! Just a few small comments

Reviewed 5 of 5 files at r2.
Reviewable status: 0 of 1 approvals obtained (waiting on @syt123450)

src/ops/dropout.ts, line 40 at r2 (raw file):

 * @param rate A float in the range [0, 1). The probability that each element
 *   of x is discarded.
 * @param noiseShape A 1-D Tensor of type int32, representing the shape for

say "an array of numbers" (instead of 1d tensor)

src/ops/dropout_util.ts, line 26 at r2 (raw file):

 * Normalize noise shape based on provided tensor and noise shape.
 *
 * ```js

since this won't get surfaces in the API, no need for a js code snippet.

src/ops/dropout_util.ts, line 34 at r2 (raw file):

 *
 * @param x Tensor or TensorLike.
 * @param noiseShape A 1-D Tensor of type int32, representing the shape for

@param noiseShape The shape for the randomly generated keep/drop flags, as an array of numbers.

src/ops/dropout_util.ts, line 39 at r2 (raw file):

 */
export function getNoiseShape(
    x: Tensor|TensorLike, noiseShape?: number[]): number[] {

since this is an internal utility, let's have it always take a Tensor and remove 'convertToTensor' below.

src/ops/dropout_util.ts, line 51 at r2 (raw file):

    const newDimension: number[] = [];
    for (let i = 0; i < $x.shape.length; i++) {
      if (noiseShape[i] == null && $x.shape[i] != null) {

is this true in TF python as well? That noiseShape can have null values, which will be replaced with the shape of x? If yes, let's mention that in the jsdoc of the dropout method.

…dropout

syt123450

Hi @dsmilkov , thanks for the review! I have made the changes accordingly, could you please take a look again?

Reviewable status: 0 of 1 approvals obtained (waiting on @dsmilkov)

src/ops/dropout.ts, line 40 at r2 (raw file):