How to use AttentionEntropyLoss?

Had · April 14, 2020, 2:05pm

I have found it here

mozilla/TTS/blob/master/layers/losses.py#L86


                x * mask, target * mask, reduction='none')
            loss = loss.mul(out_weights.to(loss.device)).sum()
        else:
            mask = mask.expand_as(x)
            loss = functional.mse_loss(
                x * mask, target * mask, reduction='sum')
            loss = loss / mask.sum()
        return loss




class AttentionEntropyLoss(nn.Module):
    # pylint: disable=R0201
    def forward(self, align):
        """
        Forces attention to be more decisive by penalizing
        soft attention weights


        TODO: arguments
        TODO: unit_test
        """
        entropy = torch.distributions.Categorical(probs=align).entropy()

What is the expected tensor size?

erogol · April 15, 2020, 9:41am

I’ve used it before but it is not active in the current code base. If you like you need to run it over the attention alignment vector.

Had · April 15, 2020, 5:49pm

I want to make alignments for fastspeech train.

So every text token should consist with mel spectrogram frame.
My alignment looks like this with multi head attention.

It tend to skip whitespaces and other punctuation.
Default tacotron is not perfect too.

So I want to try this function to make the strict token-to-token alignment.

Is it fine for [batch, width, height]?
Or I need to run it for every sample like [loss(i) for I in batch]?
Maybe I need something else?

Thanks for attention.

erogol · April 16, 2020, 10:06am

entropy loss would not do that. It works more like a guided attention loss which forces the alignment to be more diagonal.

Alignment looks quite noisy. Pls share the config.json so I can guess better.