Skip to content

SigmoidFocalLoss

Binary / multi-label focal loss operating on raw logits with a sigmoid activation. Drop-in replacement for BCEWithLogitsLoss for imbalanced problems.

Multi-label vs. multiclass

SigmoidFocalLoss applies a sigmoid independently to each logit, so every output is a separate binary prediction. Use it when a sample can belong to multiple classes at once (multi-label), or for a single yes/no decision (binary). If your classes are mutually exclusive (each sample has exactly one correct class), use SoftmaxFocalLoss instead.

imbalanced_losses.focal_loss.SigmoidFocalLoss

Bases: _SampleWeightMixin, Module

Sigmoid Focal Loss as used in RetinaNet.

Binary / multi-label variant operating on raw logits with sigmoid activation. Supports optional DDP all-gather so that the global batch is seen when computing mean/sum reductions.

.. note:: Multi-label vs. multiclass: This loss treats every output logit as an independent binary prediction (sigmoid per element). Use it when a sample can belong to multiple classes simultaneously (multi-label), or for a single yes/no prediction (binary). If your classes are mutually exclusive (each sample belongs to exactly one class), use :class:SoftmaxFocalLoss instead, which couples the outputs via softmax.

Parameters:

Name Type Description Default
alpha float

Weighting factor in [0, 1] to balance positives vs negatives, or -1 to ignore. Default: 0.25.

 0.25
gamma float

Exponent of the modulating factor (1 - p_t). Default: 2.

 2.0
reduction str

'none' | 'mean' | 'sum'. Default: 'mean'. 'mean_positive' is softmax-only: there is no background class in the sigmoid formulation to normalise against.

 'mean'
gather_distributed bool or None

Whether to all-gather inputs and targets across DDP workers before computing the loss. None (default) auto-detects: gathers when torch.distributed is initialized with world_size > 1. Set to False to opt out.

 None
Source code in src/imbalanced_losses/focal_loss.py 
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
class SigmoidFocalLoss(_SampleWeightMixin, nn.Module):
    """
    Sigmoid Focal Loss as used in RetinaNet.

    Binary / multi-label variant operating on raw logits with sigmoid activation.
    Supports optional DDP all-gather so that the global batch is seen when
    computing mean/sum reductions.

    .. note::
        **Multi-label vs. multiclass:** This loss treats every output logit as an
        *independent* binary prediction (sigmoid per element).  Use it when a
        sample can belong to *multiple* classes simultaneously (multi-label), or
        for a single yes/no prediction (binary).  If your classes are
        *mutually exclusive* (each sample belongs to exactly one class), use
        :class:`SoftmaxFocalLoss` instead, which couples the outputs via softmax.

    Parameters
    ----------
    alpha : float
        Weighting factor in [0, 1] to balance positives vs negatives, or -1 to
        ignore. Default: 0.25.
    gamma : float
        Exponent of the modulating factor (1 - p_t). Default: 2.
    reduction : str
        'none' | 'mean' | 'sum'. Default: 'mean'.  ``'mean_positive'`` is
        softmax-only: there is no background class in the sigmoid
        formulation to normalise against.
    gather_distributed : bool or None, optional
        Whether to all-gather inputs and targets across DDP workers before
        computing the loss.  ``None`` (default) auto-detects: gathers when
        ``torch.distributed`` is initialized with world_size > 1.  Set to
        ``False`` to opt out.
    """

    SUPPORTED_REDUCTIONS: tuple[str, ...] = ("none", "mean", "sum")

    def __init__(
        self,
        alpha: float = 0.25,
        gamma: float = 2.0,
        reduction: str = "mean",
        gather_distributed: bool | None = None,
    ):
        super().__init__()
        if not (0.0 <= alpha <= 1.0) and alpha != -1:
            raise ValueError(
                f"Invalid alpha value: {alpha}. alpha must be in [0, 1] or -1 for ignore."
            )
        self.alpha = alpha
        self.gamma = gamma
        self.reduction = reduction
        self.gather_distributed = gather_distributed
        self._gather_resolved: bool | None = None

    def _should_gather(self) -> bool:
        if self._gather_resolved is None:
            self._gather_resolved = _resolve_gather(self.gather_distributed)
        return self._gather_resolved

    def forward(
        self,
        inputs: torch.Tensor,
        targets: torch.Tensor,
        *,
        sample_weight: torch.Tensor | None = None,
    ) -> torch.Tensor:
        """
        Parameters
        ----------
        inputs : Tensor
            Raw logits, arbitrary shape.
        targets : Tensor
            Same shape, float 0/1 labels.
        sample_weight : Tensor, optional
            Per-element weight. Dim 0 must equal ``inputs.size(0)``;
            trailing dims may broadcast (e.g. ``[N, 1, H, W]`` against
            ``[N, C, H, W]``). ``None`` (default) leaves the objective
            unweighted -- when no weight has ever been supplied to this
            instance, the loss and its gradient are bitwise identical to
            the pre-``sample_weight`` release. When provided, must be
            non-negative and finite; it is detached on entry, cast to the
            loss dtype, and never carries gradient. The full dim-0 extent
            is required because ``gather_distributed`` concatenates the
            weight along dim 0: a weight that broadcast over dim 0 would
            validate on one rank and then misalign against the gathered
            batch. Under DDP, all ranks must agree on whether
            ``sample_weight`` is supplied on a given step -- a mismatch
            desynchronizes the collectives across ranks.

        Returns
        -------
        Tensor
            Scalar or per-element loss depending on ``reduction``.
        """
        sample_weight = self._check_sample_weight_broadcastable(
            sample_weight, inputs, dtype=inputs.dtype
        )

        if self._should_gather():
            inputs  = all_gather_with_grad(inputs)
            targets = all_gather_no_grad(targets)
            if sample_weight is not None:
                sample_weight = all_gather_no_grad(sample_weight)

        p = torch.sigmoid(inputs)
        ce_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction="none")
        p_t = p * targets + (1 - p) * (1 - targets)
        loss = ce_loss * ((1 - p_t) ** self.gamma)

        if self.alpha >= 0:
            alpha_t = self.alpha * targets + (1 - self.alpha) * (1 - targets)
            loss = alpha_t * loss

        # No validity mask exists for the sigmoid case: every element of
        # `inputs` is a real prediction, so 'mean' averages over all of them.
        if self.reduction == "none":
            return _reduce_none(loss, sample_weight)
        if self.reduction == "mean":
            return _reduce_mean_all(loss, sample_weight)
        if self.reduction == "sum":
            return _reduce_sum(loss, sample_weight)
        raise _invalid_reduction(self.reduction, self.SUPPORTED_REDUCTIONS)

forward(inputs, targets, *, sample_weight=None)

Parameters:

Name Type Description Default
inputs Tensor

Raw logits, arbitrary shape.

 required
targets Tensor

Same shape, float 0/1 labels.

 required
sample_weight Tensor

Per-element weight. Dim 0 must equal inputs.size(0); trailing dims may broadcast (e.g. [N, 1, H, W] against [N, C, H, W]). None (default) leaves the objective unweighted -- when no weight has ever been supplied to this instance, the loss and its gradient are bitwise identical to the pre-sample_weight release. When provided, must be non-negative and finite; it is detached on entry, cast to the loss dtype, and never carries gradient. The full dim-0 extent is required because gather_distributed concatenates the weight along dim 0: a weight that broadcast over dim 0 would validate on one rank and then misalign against the gathered batch. Under DDP, all ranks must agree on whether sample_weight is supplied on a given step -- a mismatch desynchronizes the collectives across ranks.

 None

Returns:

Type Description
Tensor

Scalar or per-element loss depending on reduction.
Source code in src/imbalanced_losses/focal_loss.py 
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
def forward(
    self,
    inputs: torch.Tensor,
    targets: torch.Tensor,
    *,
    sample_weight: torch.Tensor | None = None,
) -> torch.Tensor:
    """
    Parameters
    ----------
    inputs : Tensor
        Raw logits, arbitrary shape.
    targets : Tensor
        Same shape, float 0/1 labels.
    sample_weight : Tensor, optional
        Per-element weight. Dim 0 must equal ``inputs.size(0)``;
        trailing dims may broadcast (e.g. ``[N, 1, H, W]`` against
        ``[N, C, H, W]``). ``None`` (default) leaves the objective
        unweighted -- when no weight has ever been supplied to this
        instance, the loss and its gradient are bitwise identical to
        the pre-``sample_weight`` release. When provided, must be
        non-negative and finite; it is detached on entry, cast to the
        loss dtype, and never carries gradient. The full dim-0 extent
        is required because ``gather_distributed`` concatenates the
        weight along dim 0: a weight that broadcast over dim 0 would
        validate on one rank and then misalign against the gathered
        batch. Under DDP, all ranks must agree on whether
        ``sample_weight`` is supplied on a given step -- a mismatch
        desynchronizes the collectives across ranks.

    Returns
    -------
    Tensor
        Scalar or per-element loss depending on ``reduction``.
    """
    sample_weight = self._check_sample_weight_broadcastable(
        sample_weight, inputs, dtype=inputs.dtype
    )

    if self._should_gather():
        inputs  = all_gather_with_grad(inputs)
        targets = all_gather_no_grad(targets)
        if sample_weight is not None:
            sample_weight = all_gather_no_grad(sample_weight)

    p = torch.sigmoid(inputs)
    ce_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction="none")
    p_t = p * targets + (1 - p) * (1 - targets)
    loss = ce_loss * ((1 - p_t) ** self.gamma)

    if self.alpha >= 0:
        alpha_t = self.alpha * targets + (1 - self.alpha) * (1 - targets)
        loss = alpha_t * loss

    # No validity mask exists for the sigmoid case: every element of
    # `inputs` is a real prediction, so 'mean' averages over all of them.
    if self.reduction == "none":
        return _reduce_none(loss, sample_weight)
    if self.reduction == "mean":
        return _reduce_mean_all(loss, sample_weight)
    if self.reduction == "sum":
        return _reduce_sum(loss, sample_weight)
    raise _invalid_reduction(self.reduction, self.SUPPORTED_REDUCTIONS)

Quick example

from imbalanced_losses import SigmoidFocalLoss
import torch

loss_fn = SigmoidFocalLoss(alpha=0.25, gamma=2.0)
logits  = torch.randn(32, 1)
targets = torch.randint(0, 2, (32, 1)).float()

loss = loss_fn(logits, targets)
loss.backward()

Parameter guidance

Parameter Default Effect
alpha 0.25 Weights positives; set to -1 to disable
gamma 2.0 Higher = more focus on hard examples; 0 with alpha=-1 = vanilla BCE (with the default alpha=0.25 it is alpha-weighted BCE)
reduction "mean" "mean" averages over elements; "sum" for total; "none" returns per-element tensor
gather_distributed None Auto-detects DDP; set False to opt out