User
mkuz writes in Gamedev.net forum
GDNet Lounge:
My perspective [prospective] employer gave me a Codility test instead of an interview. [...] In description Codility did not mention that they score not only correctness, but also program performance. They give option to verify your code and when I did it the test result was OK. I submitted the test and was surprised about poor results. [...] My opinion is that the current test is deceiving and illegal to use in hiring process. [...]
Well... I wanted to jump to explaining why this is a misguided judgement, but it turned out that
GDNet Lounge discussant already did it for me. Big thanks! Some highlights follow.
Why would you assume that performance is unimportant?
asks
Sneftel. Exactly.
I don't see how this would be illegal. You failed the tests they set you, and you didn't get the job. How is that unfair?
observes
Talroth.
Antheus looks at Codility from a practical perspective:
[...] at least you didn't need to take a day off, drive 2 hours, be grilled for 3 more, only to fail because you couldn't name all 67 parameters to ls.
One of the frequent concerns of our customers is that perhaps we do not make it clear enough that the code submitted to Codility has to be really well written to score 100%. If this shows up as an issue, we advise customers to warn the candidates, but I always make it clear, that this is lowering the bar. Dan Marchant vel
Obscure sums up this point of view perfectly:
Employers are looking for staff who will do their best work all the time, not just when they are explicitly asked.
There is an interesting thread on whether we should explicitly mention that scalability is important in solutions. Some discussants (
rip-off,
speciesUnknown) think we should stress it more. We believe that candidates who need to be reminded may still be acceptable diligent programmers handling all the boundary cases, but should not score 100% (see also
this post). In line with this thinking
TheBuzzSaw observes:
Algorithmic correctness, protection against overflow, and appropriate big-O design are not exactly stretches for a problem this small. [...] If you don't do these things naturally in your coding, frankly, you're not qualified.
The user
Washu concurs:
Its an interview question. You should do the absolute BEST you can on it. Show your skills.
See the
complete discussion.
The inability to test solutions written on the test against some of the common test cases that are involved in scoring is really what annoys me about Codility. This was never addressed in the GDNet thread due to the fact that everyone but the OP was basing their assumptions on the example tests, in which you can view the results.
ReplyDeleteIn a real Codility test, you are not given enough feedback about possible test cases to determine what extra code might need to go into your solution to get a good score.
Perhaps I am not the ideal programming robot that employers want or that Codility encourages, because I write programs by testing them and modifying, quickly arriving at optimal solutions, rather than blindly writing some "holy perfect solution" the first time every time (which is not realistic when dealing with problems that don't fit into the nice neat stereotypical-computer-science-problem bubble).
I agree with Ross.
ReplyDeleteI think a more fair test would be to give the feedback that one needs to achieve success, but then rate programmers on the time they take and then number of iterations required.
As an employer I'm also interested in the process they took to arrive at the final result. That more of a concern than if they can predict the judging criteria.
What I have found is that most applicants write their codility tests to arrive at the correct solution, and do not attempt to deliver the most algorithmically optimized solution. I am OK with this though, and for promising applicants I simply give them the test again and do an iteration step and say something like, "Performance testing as shown that this piece of code is a performance bottleneck, can you show me how you would change it based on this information."
ReplyDeleteFrom an interviewer's point of view, I would never discount candidates just because they didn't score 100%. It's obvious that sometimes the test cases test lots of extreme edge cases which aren't immediately obvious from the short description.
ReplyDeleteTake the demo test app, you can go from 94% to 100% simply by replacing an int with a long long (and of course in the dynamic languages like Ruby this isn't even a problem so you'd get 100% with the exact same algorithm, since there is no manifest typing).
As long as these tests are used as a phone screen, anyone who 'passes' would probably be good enough to get an on-site interview. If candidates needed to answer the questions to 100% well I'd expect the potential employer to make that abundantly clear and try to be better about pointing out the need to handle edge cases.
But in a 30 minute time slot with no human interaction? Seems unlikely.
I also completely agree with the comment that this is counter-Agile. Normal write-test-improve cycle is just not possible with Codity.
ReplyDeleteMost of online contest allow you to improve your code as long as you're within the time limit. I don't see why this should be different.
One more aspect to point out is the modern programming rule: premature optimisation is the root of all evil! From this point of view, the applicant should be informed that it is necessary in the case of this test.
ReplyDeleteOne wish-list I have for interviewers is to at least spend 30 min to go over the questions.
ReplyDeleteThere was once when the interviewer had a mass walk in interview, but had the time to ask the candidates why we chose our answers one at a time.
The effect was great. They knew whether the questions are vague and easily misunderstood or the candidate made a wild guess. They can easily find out whether we had the fundamental knowledge. It was a win-win situation whereby every party can learn and grow.
I think it's fair that codility provide us the results so that we can also learn from our mistakes.
Since we took a couple of hours to sweat it out doing the tests, it would be nice if we know where we go wrong, rather than shooing us away once it is done.
I took a Codility Problem very recently and I was given a problem which had Point3D[] A as the argument of the method. Since I had never used this class, was not aware of which methods to use for this class. Moreover I couldn't find this Class in JDK, so as to get some clue about this class.
ReplyDeleteIn addition to this, after writing the code, when I clicked on verify button, it gave me an error "NoneObject found for getElementsByTagName" and didn't give any verification results.
Hence, in this case even though my Logic is correct, my score would be zero as my Program wont compile.
I am not sure if this Test was intended to know about my knowledge of a JAVA API or my logic, but I feel that it's unfair to post problems which involve not so commonly used Java APIs. Moreover the error "NoneObject found for getElementsByTagName" which came while I was trying to verify my solution is a bug with Codility or my Browser(I am using FireFox4).
It would be very kind if I get an answer from someone from codility on my previous comment.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteCodility as many other systems is just pointless in my opinion and I would doubt working with a company which rely on using those kind of systems. Programming and code optimization is a process which requires attention which lacks due to the pressure to complete the task. I understand that using those systems help companies in speeding up the selections, but I think it's very unfair especially because companies should hire people based on their potential(as whatever experience someone can have, a new job is always NEW!). Whether you are a computer/software programming geek or not, those systems are unfair. That's it.
ReplyDeleteWell. one has to really prepared because it's not that easy. You don't have any option but to be the best.
ReplyDeletePeople are writing a lot of crap here about "not possible to write, test, improve". What's stopping you? Your code gets executed when you hit verify. Include some tests. Take them out before you hit submit.
ReplyDeleteThat said, it's the process of solving the problem that is interesting to me, not just the result. It's far more revealing to watch someone approach the problem without a time limit than it is to see what they came up with within the limit.