This is really cool, thanks for the great writeup!

If believe there's still so much room to better analyse and leverage offline experimentation, thoroughly and diligently. Your work reminded me of a paper we published at RecSys '19, if you're fancy: https://dl.acm.org/doi/10.1145/3298689.3347032