Question about RandAugment Implementation

[psellcam]

In the paper you mention that flips are applied with 50 percent probability. Is it also the case that each randaugment sample is also applied with 50 percent probability or are two randaugment choices always applied to each unlabelled image?

The reason I ask is that in the main pytorch reimplementation of your work there is a 50 percent chance for each randaugment policy to be applied but I didn't see this in your paper

Many Thanks

[carlini]

When we say in our paper that we apply a "weak augmentation" then this means (among other things): "flip, with 50% probability, the image across the vertical axis". When we say "strong augmentation" then this means (for the case of RA): "apply the RA function 100% of the time". I'm not sure what any other repository is doing but the 50% number in our paper means only that flips are done 50% of the time, RA is treated as a black box that can do whatever it wants to augment. Similarly, when say we use CTA, this means we use the CTA function to augment 100% of the time and let it do whatever it wants.

[psellcam]

Thank you for this fast response,

I believed it was 100% but had to double check

[JoyHuYY1412]

Hi, @carlini, I want to get the weak and strong augmented images of labeled images as the unlabeled in each iteration, but after I change

fixmatch/cta/lib/train.py

Line 49 in d4985a1

batch['probe'] = True

to False, I meet the AssertionError. Could you give me some advice? Is the

fixmatch/fixmatch.py

Line 88 in d4985a1

x_in = tf.placeholder(tf.float32, [None] + hwc, 'x') # Eval images

the strong augmented images of labeled images?

[carlini]

Sorry I don't understand quite what you want. Could you expand?

[JoyHuYY1412]

Sorry I don't understand quite what you want. Could you expand?

Hi @carlini, I mean how to get the strong augmented labeled images? are they 'x_in'?

[carlini]

y_in here gets set with the weakly and strongly augmented images.

fixmatch/fixmatch.py

Line 89 in d4985a1

y_in = tf.placeholder(tf.float32, [batch * uratio, 2] + hwc, 'y') # Training unlabeled (weak, strong)

What do you want to do with the augmentations?

[JoyHuYY1412]

y_in here gets set with the weakly and strongly augmented images.

fixmatch/fixmatch.py

Line 89 in d4985a1

y_in = tf.placeholder(tf.float32, [batch * uratio, 2] + hwc, 'y') # Training unlabeled (weak, strong)

What do you want to do with the augmentations?

Hi @carlini, thank you for your reply.
I want the weakly and strongly augmented images of only the labeled images. Are they 'xt_in' and 'x_in'?

fixmatch/fixmatch.py

Lines 87 to 88 in d4985a1

	xt_in = tf.placeholder(tf.float32, [batch] + hwc, 'xt') # Training labeled
	x_in = tf.placeholder(tf.float32, [None] + hwc, 'x') # Eval images

[carlini]

The labeled images will get passed as part of the unlabeled set as well, and that's how the labeled images are strongly augmented.

I'm still confused what your overall objective is, though. If you just want to weakly and strongly augment a particular set of images then it would be easier to call directly into the augmentation libraries. Do you ant something else?

[JoyHuYY1412]

The labeled images will get passed as part of the unlabeled set as well, and that's how the labeled images are strongly augmented.

I'm still confused what your overall objective is, though. If you just want to weakly and strongly augment a particular set of images then it would be easier to call directly into the augmentation libraries. Do you ant something else?

Hi @carlini, I just want the strong augmented images for only the labeled images in a batch. For example, in each batch, I have 64 labeled images L and 64*7*2 unlabeled images U, and I want the corresponding 64 strong augmented images of L.

I know that labeled images will get passed as part of the unlabeled set, but I just want the strong augmented labeled images in each batch. I want to know what is ''x_in'', because I think they are the strong augmented version of "xt_in". Am I right?

If not, what augmentation libraries should I call to get the strong augmented images of L in each iteration?

Thank you so much.

[carlini]

FixMatch does not strongly augment each of the labeled images in the batch on every iteration. It only weakly augments those images. The only time the labeled images become strongly augmented is when they are passed as unlabeled images. (See Algorithm 1, line 2.)

So if you want strongly augmented labeled images, you will need to get collect them from the unlabeled dataset when they happen to appear there.

[JoyHuYY1412]

FixMatch does not strongly augment each of the labeled images in the batch on every iteration. It only weakly augments those images. The only time the labeled images become strongly augmented is when they are passed as unlabeled images. (See Algorithm 1, line 2.)

So if you want strongly augmented labeled images, you will need to get collect them from the unlabeled dataset when they happen to appear there.

Hi @carlini, thank you for your reply.
I have just one last question: what is the 'x_in' here in the following code? I am a little bit confused. I think they are not the validation data.

fixmatch/fixmatch.py

Lines 87 to 88 in d4985a1

	xt_in = tf.placeholder(tf.float32, [batch] + hwc, 'xt') # Training labeled
	x_in = tf.placeholder(tf.float32, [None] + hwc, 'x') # Eval images

And I found some explanation in #4 (comment), but I still don't understand how to generate x_in. Thank you so much.

[carlini]

xt_in has the training examples, and x_in has the evaluation images. You can see that x_in is only used in the classification ops:

fixmatch/fixmatch.py

Lines 137 to 138 in d4985a1

	classify_raw=tf.nn.softmax(classifier(x_in, training=False)), # No EMA, for debugging.
	classify_op=tf.nn.softmax(classifier(x_in, getter=ema_getter, training=False)))

[JoyHuYY1412]

xt_in has the training examples, and x_in has the evaluation images. You can see that x_in is only used in the classification ops:

fixmatch/fixmatch.py

Lines 137 to 138 in d4985a1

	classify_raw=tf.nn.softmax(classifier(x_in, training=False)), # No EMA, for debugging.
	classify_op=tf.nn.softmax(classifier(x_in, getter=ema_getter, training=False)))

Hi @carlini, the code in https://github.com/google-research/fixmatch/blob/master/fixmatch.py#L38 seems to give augmentation to training labeled images and generate x_in.

The code is a little complicated for me, and I hope you don't mind my bother.

[carlini]

Yeah, the code is a bit convoluted. It evolved from three projects. If you'd like to find a simpler implementation David wrote one here
https://github.com/google/objax/tree/master/examples/fixmatch

However, for your question, no those augmentations aren't going into x_in. I'm not quite sure why you think they are. If you look at the code then xin

fixmatch/fixmatch.py

Line 88 in d4985a1

x_in = tf.placeholder(tf.float32, [None] + hwc, 'x') # Eval images

is only ever used here, and exported as x

fixmatch/fixmatch.py

Lines 136 to 138 in d4985a1

	xt=xt_in, x=x_in, y=y_in, label=l_in, train_op=train_op,
	classify_raw=tf.nn.softmax(classifier(x_in, training=False)), # No EMA, for debugging.
	classify_op=tf.nn.softmax(classifier(x_in, getter=ema_getter, training=False)))

and the x gets fed here as the prediction op

fixmatch/libml/train.py

Lines 228 to 233 in d4985a1

	for x in range(0, images.shape[0], batch):
	p = self.session.run(
	classify_op,
	feed_dict={
	self.ops.x: images[x:x + batch],
	**(feed_extra or {})

[JoyHuYY1412]

Thank you so much and thank you for the simplified version @carlini .
I find the code here feeds x_in as x['probe']:

fixmatch/cta/lib/train.py

Lines 63 to 69 in d4985a1

	def train_step(self, train_session, gen_labeled, gen_unlabeled):
	x, y = gen_labeled(), gen_unlabeled()
	v = train_session.run([self.ops.classify_op, self.ops.train_op, self.ops.update_step],
	feed_dict={self.ops.y: y['image'],
	self.ops.x: x['probe'],
	self.ops.xt: x['image'],
	self.ops.label: x['label']})

Are x['probe'] here the evaluation images, or the training labeled images? I am still not very clear about that... sorry

[carlini]

Sorry for the late reply. If you haven't figured it out already, probe here is used to check how accurate the model is on these probe images as a way to tune the CTAugment scheme. If you're using randaugment then you don't need probes.