This past week was draining. I planned to quickly wrap up one project and move on to another. The project that I thought would take a day or two to finish ended up consuming my whole week.

I had trained a model that would take a chest x-ray and classify it as normal or a likely case of covid, pneumonia, or other lung opacity. I saw great results on my test data. But when I put the model into a web app and starting feeding it new images, I realized that it did not generalize well. Its performance was terrible on new images. Attempting to figure out why this was happening and how I could fix it eat up all my project work time for the week. In the end, I still wasn’t happy with the results. I felt bummed about having wasted a week. But the other day I came across a little piece of insight on Twitter. It was a response in a thread of advice for early career programmers. It said:

Every failure contains a hidden treasure. Invest some time to discover it. In the long run those treasures will be worth more than all your successes together.

That felt wise and correct. On the Recurse Center scale, it felt like a failure to spend a week on something that I ended up having to set down. But zoomed out enough, surely it wasn’t a waste of time.

I got to pair with some very smart people. I learned a great deal about training neural nets. I became far more familiar with fastai’s framework by spending hours looking for answers in the source code and explaining issues to those I paired with. I came to understand the limitations of working with such a high level framework. Most importantly, I got hands on experience with things that I had previously only understood in theory, such as overfitting, model generalization, and feature transformations.

The most valuable take away for me was an appreciation of how different and difficult it is to debug machine learning programs. When we think of typical bugs, something doesn’t work. When you fix the bug, that thing now works. ML isn’t like this. To paraphrase one of my batch-mates, it takes a certain ‘spidey-sense’ that tells you, ‘I feel something is wrong here.’ There’s rarely a blatant error. Furthermore, the things that could be causing that wrongness are not constrained to your code.

While debugging my model, one thing another Recurser and I considered was this: all covid images are necessarily less than 2 years old. If the other images were not, then the model could have learned to detect that newness and would over assign the probability of covid to any x-ray taken with more recent technology. Just realizing that this could be an issue takes a certain amount of insight. Figuring out if this issue is real or not requires systemic thinking, diving into the dataset metadata, and, ideally, domain expertise. And while it was frustrating in the moment, I found that I actually really enjoy that type of thinking.

So, I kind of lost the week, but I didn’t waste my time.

Not to mention that outside of that project, I spent time studying algorithms and having some great conversations with other Recursers.

Project wise, I’m looking for something new to dive into this week. I don’t care what it is as long as I’m excited about it. Originally I had wanted to do a Kaggle competition, but I need a little break from ML. I have my eyes on hacking around with some web things.

Outside of that, I’ll be pairing with batch-mates, practicing algorithms, and trying not to think too much about the fact that Recurse is almost over.