Just before midnight one night last February, Nicolas Spiegelberg's engineering team was getting close to finding the bug. It was a nasty one---responsible for about half of the Facebook mobile app crashes on iPhones and iPads---and that meant it had to be squashed immediately.
Originally a single engineer, Slobodan Predolac, had pulled the job of fixing the problem, but by now Spiegelberg's entire team was on the case. It was just too much fun. "Everybody wanted to solve that bug," remembers Spiegelberg. "Everyone was just running around saying, 'What about this subsystem; what about this subsystem...' It was a hunt."
Spiegelberg and Predolac wrote about their bug-hunt today. Their description reads like a geek mystery, but really, it's a story of unintended consequences, showing how programming decisions made decades ago can take down even the most modern of software.
Starting late last year, users began reporting problems with Facebook's iOS app, but this particular bug got on Facebook Engineering's radar shortly before their February bug-finding free-for-all. It was particularly nasty because, while the Apple utility reporting the problem was giving them bug reports, it was also giving the engineers about a half-dozen different error codes for what turned out to be the same problem.
>It's a story of unintended consequences, showing how programming decisions made decades ago can take down even the most modern of software
The bug, it turned, happened because Unix---the operating system that forms the underpinnings of iOS--- was acting in a way that Facebook's developers just didn't expect. And that was causing the iOS subsystem to write data to places that Facebook developers never expected it, including parts of the computer's memory that were being used by Facebook's database.
I wrote earlier this week about how some of our ideas about password security---for example the practice of forcing passwords to expire after 90 days---date back so far that they're no longer helpful.
And clearly that's part of what got the Facebook team so excited. By the beginning of that February evening, they knew that they were facing a Unix problem; they just didn't know where it was. So they took to the internet, scrutinizing pages and pages of Unix POSIX standards, written decades ago and preserved on the web. "You get this arcane document that was written in ASCII text about 20 years ago and you troll though it for the details you want."
That's a pretty good description of what Unix programmers were doing back in the early '90s to write those standards, too. The world has changed a lot since then, but perhaps not as much as we like to think.
