error when running tutorial examples with adanet about adanet

[cweill]

Are you using the new TPUEstimator? Do you have a copy of the tutorial you can share that has this problem?

[sdwldchl]

In simple_dnn.py, I added two lines of codes to output the iteration_step:

def build_subnetwork_train_op(self, subnetwork, loss, var_list, labels,
                                                  iteration_step, summary, previous_ensemble):
"""See adanet.subnetwork.Builder."""
with tf.name_scope(""):       # output iteration_step
  summary.scalar("iteration_step", iteration_step)
#NOTE: The adanet.Estimator increments the global step.
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
  return self._optimizer.minimize(loss=loss, var_list=var_list)

In the Customizing AdaNet example, I modified the input_fn function:

def input_fn(partition, training, batch_size):
"""Generate an input_fn for the Estimator."""
  def _input_fn():
    if partition == "train":
     dataset = tf.data.Dataset.from_generator(
     generator(x_train, y_train), (tf.float32, tf.int32), ((28, 28), ()))
   else:
    dataset = tf.data.Dataset.from_generator(
    generator(x_test, y_test), (tf.float32, tf.int32), ((28, 28), ()))

# We call repeat after shuffling, rather than before, to prevent separate
# epochs from blending together.
  if training:
    dataset = dataset.shuffle(10 * batch_size, seed=RANDOM_SEED).repeat()

  dataset = dataset.map(preprocess_image).batch(batch_size)
  return dataset.repeat()
return _input_fn

Also, argument train_distribution was explicitly specified in configuration for estimator:

devices = ["/device:GPU:0", "/device:GPU:1"]
distribution = tf.contrib.distribute.MirroredStrategy(devices=devices)
config = tf.estimator.RunConfig(
    save_checkpoints_steps=50000,
    save_summary_steps=50000,
    train_distribute=distribution,
    tf_random_seed=RANDOM_SEED)

No other changes were made. Thanks in advance.

[sdwldchl]

Are you using the new TPUEstimator? Do you have a copy of the tutorial you can share that has this problem?

In simple_dnn.py, I added two lines of codes to output the iteration_step:

def build_subnetwork_train_op(self, subnetwork, loss, var_list, labels,
                                                  iteration_step, summary, previous_ensemble):
"""See adanet.subnetwork.Builder."""
with tf.name_scope(""):       # output iteration_step
  summary.scalar("iteration_step", iteration_step)
#NOTE: The adanet.Estimator increments the global step.
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
  return self._optimizer.minimize(loss=loss, var_list=var_list)

In the Customizing AdaNet example, I modified the input_fn function:

def input_fn(partition, training, batch_size):
"""Generate an input_fn for the Estimator."""
  def _input_fn():
    if partition == "train":
     dataset = tf.data.Dataset.from_generator(
     generator(x_train, y_train), (tf.float32, tf.int32), ((28, 28), ()))
   else:
    dataset = tf.data.Dataset.from_generator(
    generator(x_test, y_test), (tf.float32, tf.int32), ((28, 28), ()))

# We call repeat after shuffling, rather than before, to prevent separate
# epochs from blending together.
  if training:
    dataset = dataset.shuffle(10 * batch_size, seed=RANDOM_SEED).repeat()

  dataset = dataset.map(preprocess_image).batch(batch_size)
  return dataset.repeat()
return _input_fn

Also, argument train_distribution was explicitly specified in configuration for estimator:

devices = ["/device:GPU:0", "/device:GPU:1"]
distribution = tf.contrib.distribute.MirroredStrategy(devices=devices)
config = tf.estimator.RunConfig(
    save_checkpoints_steps=50000,
    save_summary_steps=50000,
    train_distribute=distribution,
    tf_random_seed=RANDOM_SEED)

No other changes were made. Thanks in advance.

[cweill]

I see you are using tf.contrib.distribute.MirroredStrategy(devices=devices). I wonder if that could be the issue. I'll have a look.

[futurely]

All methods of Summary should add an argument , collections=None.

https://github.com/tensorflow/adanet/blob/master/adanet/core/summary.py#L38