Indeed, thank you!

Thos are great suggestions, thank you!

Yet, I'm not sure about the second one. For example for iou == 0 (complete miss) it returns negative metric which is not right. So I corrected it a little to be torch.clamp(20 * (iou - 0.5), 0, 10).ceil() / 10

Sorry, I left out one line of code that needs to come after the iou_metric calculation:

 iou_metric[iou_metric<0] = 0

Even like that, I think it's not working correctly. Say for iou = 0.5509 your code will return 0.1, while the correct answer should be 0.2 (iou > 0.5 and >0.55, so it's 2/10). For 0.551 it would be correct though.

So, great news, you were sligtly underestimating your models all the time! It's great, it means that they are actually better than you thought :)

I think if I change the 0.451 to 0.450009, it will give the precision on this score until the 5th digit. IOU of 0.55001 will still yield the right result of 0.2

Booth outputs and labels are float

That's exactly the reason, both should be byte or int. Just add this to the functuion body: labels = labels.byte()

And outputs in that case is already thresholded predicted mask. If your net outputs probablities just do output > 0.5 (you may want ot optimize this threshold).

Sure thing! It looks tricky because we do few things at once here. Namely, we map the IOU values to a different scale (x => 2*x - 1) and do the rounding.

We need the mapping because IOU=0.5 means 0 on the leaderboard, and IOU=1.0 means 1.0. Thus the mapping we a looking for is x => 2 * x - 1. And that's why I have that 20 * (iou - 0.5). As we later divide by 10 you can think of it as of 20 * (iou - 0.5) / 10 = 2 * (iou - 0.5) = 2 * iou - 1. But that's not enough. If original IOU was from 0 to 1, than 2 * iou - 1 is from -1 to 1. Now to the second part.

Here we don't care about the values itself, only whether it exceeds specific thresholds. Therefore we need to round our maped values. How do you round numbers to 0.1, 0.2…? You multiply by 10, round to integers and devide by 10. That's why we have (10 * x).ceil() / 10. In reality we have 20 * x but you already know why.

And finnaly we have torch.clamp(..., 0, 10) to omit all IOUs below 0.5 (or mapped IOUs below 0).

Thanks for that, I'll fix it.

(outputs & labels).float().sum((1, 2))

Would be the best looking, I guesss.

Hello! Thanks for your comment, but I think it's not the best one, most because try-except is quite a dangerous pattern. Especially if your except is catching all errors (like in your example). For example, image your function will receive None or some string as an input. It shouldn't, but all can happen. Now np.argwhere(thresholds < iou) will fail, but you wouldn't know about it because of except clause. It will just silently return 0.

A gap between a local validation and a public LB is a normal thing because and happens often, yet 0.05-0.06 is maybe to big for this competition. I observe 0.02 gap (not k-fold, just 20% hold out). Maybe you want to stratify your validation by depth and/or salt %. Otherwise, your hold-out can be quite different from the training/public/private data and you can get a big gap.

You can do something like:

for tr in THRESHOLDS:
    predicted_mask = outputs > tr  # This will be a byte tensor
    iou_pytorch(predicted_mask, labels.byte())  # If your labels are BATCH x H x W
    # or
    # iou_pytorch(predicted_mask, labels.squeeze(1).byte())  # If your labels are BATCH x 1 x H x W

Dor some reason markdown is not working :(