Great article Cameron! One question: With the triangular or cyclical LR, how to deal with early stopping criteria while training? Early stopping will stop the training if the validation loss is consistently high for x epochs.
The general recommendation from [1] and [2] would be to make sure early stopping occurs at the end of a decay cycle. Your performance always degrades a bit as you increase the learning rate, then reaches a maximum after the learning rate is decreased again and reaches the end of a decay cycle. The performance improves after each successive cycle, but the peaks always occur at the end of the cycle. So, when using cyclical learning rates you always want to stop training at the end of a cycle.
You can combine this with early stopping by only testing the early stopping criteria as you approach the end of a decay cycle. Or, you could simply record validation accuracy after each cycle and stop training once you've performed a sufficient number of cycles for the performance to reach a plateau.
Great article Cameron! One question: With the triangular or cyclical LR, how to deal with early stopping criteria while training? Early stopping will stop the training if the validation loss is consistently high for x epochs.
The general recommendation from [1] and [2] would be to make sure early stopping occurs at the end of a decay cycle. Your performance always degrades a bit as you increase the learning rate, then reaches a maximum after the learning rate is decreased again and reaches the end of a decay cycle. The performance improves after each successive cycle, but the peaks always occur at the end of the cycle. So, when using cyclical learning rates you always want to stop training at the end of a cycle.
You can combine this with early stopping by only testing the early stopping criteria as you approach the end of a decay cycle. Or, you could simply record validation accuracy after each cycle and stop training once you've performed a sufficient number of cycles for the performance to reach a plateau.