Replying to @jxmnop
The reason this mistake breaks deep ResNets is well known. Lots of papers showed why but I think ours is the simplest: arxiv.org/abs/2002.10444 In short, deep ResNets are trainable if the activations on branches are much smaller than the activations on the skip connection.

Batch Normalization Biases Residual Blocks Towards the Identity...

Batch normalization dramatically increases the largest trainable depth of residual networks, and this benefit has been crucial to the empirical success of deep residual networks on a wide range of...

arxiv.org

Nov 21, 2024 · 11:19 AM UTC
9