ML bugs are so much trickier than bugs in traditional software because rather than getting an error, you get degraded performance (and it's not obvious a priori what ideal performance is). So ML debugging works by continual sanity checking, e.g. comparing to various baselines.

May 14, 2022 · 4:23 PM UTC

46
229
39
1,903
Load newest
Replying to @gdb
Degradation in performance is a classic fault mode in many “traditional” sw systems, esp due to faulty instrumentation.
Replying to @gdb
As an engineer, I often tend to avoid state-of-the-art end to end architectures for the same reason, I prefer to have a more complex solution which its modules are trained separately, rather than a whole black-box solution which I can't say why arrived at a particular outcome.
2
Replying to @gdb
Yes the inability to write fast unit tests degrades the development cycle dramatically. It's all manual regression tests.
1
4
Replying to @gdb
And sometimes your performance isn't even *worse*.
Replying to @gdb
I have more sanity checks for my pipeline than for my mental health 😂
1
Replying to @gdb
Looks like a problem waiting to be fixed. Wink wink, cough cough 😉
Replying to @gdb
Even baselining can be difficult. Your problem evolves as does the solution, with time. Whatever you're trying to predict behaved in a certain way at a point than it does now.
Replying to @gdb
This is why ML should have toy systems where you know the "exact" answer for testing. Researchers in scientific computing have been dealing with this kind of hard to find numerical error for many, many years.
1
1
10
Replying to @gdb
I wonder if ML bugs always degrade rather than enhance performance.
3
2
Replying to @gdb
objectivity wasn't stayed on ou couse it should've had alternate resemblence printing, like augmented not translated more so
Load more