Codility genesis

Tuesday, August 4, 2009

If you point your browser to websites like zhaopin.com or 51job.cn you will see lots of boxes with logos and links to job offers. In May 2005 one of these boxes had a logo of Exoweb (at that time a small Beijing-based outsourcing business), and at the other end of the hyperlink Ken, Bjørn and myself were waiting for candidates. Several weeks earlier Exoweb landed a promising contract with a Norwegian customer and was hungry for programmers. Hungry, nevertheless picky. Quality mattered.

We decided to adopt quality measures outlined in Joel Spolsky's Guerilla Guide to Inverviewing. In particular, we were running independent interviews (up to 3 tech interviews per candidate) and we were diligently checking whether candidates can actually write correct code. The overwhelming majority could not. It quickly occurred that we were spending long hours in interviews, but the gain was less than modest.

Beijing is a great city and there are more exciting things to do than interviewing programmers who cannot write programs. We had strong motivation to figure out a way to improve the process. The inspiration came from Olympiad in Informatics, the ultimate programming competition for high-school students. I had some experience with Polish chapter of the Olympiad and I was aware that this institution has been employing automated tools for assessing solutions of programming problems since 1990s. I decided to adopt similar means to screening job candidates.

Ken and Bjørn were a bit skeptical about the amount of work that had to be invested, but surely interested in freeing up some of their time on Interview Saturdays. I hacked up an automated evaluator called Exobench and we have set up one machine as a workstation for candidates. From now on, every candidate had to sit in front of a computer and deliver a solution of one simple programming problem using a set of standard programming tools (editor + compiler). Our HR person later ran the automated evaluator to determine whether the candidate goes home or stays for tech interviews.

We quickly noticed that only 1 in 10 candidates was passing the programming test. We were reviewing the solutions of the rejected candidates to make sure we don't throw out the baby with the bath water, but we were increasingly positive that persons unable to get few lines of code straight should never tinker with our complex distributed web application.

When I was leaving Exoweb 2.5 years later, I took a final look at stats. Around that time we have had screened 2500, interviewed 250, hired 50 (I rounded a bit, but the actual numbers were close). Given that we usually organized two to three independent tech interviews per candidate, the automation saved something between 4 and 7 thousands hours of Senior Engineer time.

Based on my experience with Exobench, the team at Enpoka.com created Codility, an on-line system for programming skills assessment. In 2009 Exoweb introduced Codility to its hiring process, becoming our first large customer generating 1200+ evaluations per month.