Developing features for products that programmers use is always a challenge–not only do they notice quickly whether the interface works properly or not, but they usually have a pretty good idea of how it could be done better. That is why here at Codility we put great emphasis on making sure we develop the right things the right way.
There’s also the fact that we’re developers ourselves, and we know that programmers’ time is precious. We know from our own (painful) experience that coding new features and then hastily pushing them to production just to get them out the door will come back to bite us. Doing this inevitably forces us to backtrack to enact a laundry list of product improvements to address the negative feedback end users surface.
And to top it all off, like many other SaaS startups we face certain limitations. We can’t invest too much money on user research and we definitely can’t afford to spend too much time on it either. The quicker we develop things, the quicker we can improve the user experience, which will drive more business.
So what do we do?
We organize Candidate Labs–1 day, intensive prototype testing with end users to check our assumptions at the crucial stages of development.
Let’s talk about how we plan and execute these Candidates Labs.
We don’t have anything as fancy as a viewing facility in our office. Instead, we connect two conference rooms using Google Hangouts. Then we start the interviews–the respondents (in our case, professional developers we label “candidates”) come in one at a time, sit down with a researcher and a developer, and find out what we are working on and how they can help us by solving a Codility task in the meeting. We also tell them what feedback we are fishing for specifically so we can dive in.
Why do we include an internal developer of our own in this conversation? To cultivate engagement and help the candidate feel comfortable. Trust me, it’s an interaction way out of the comfort zone for engineers–it’s something that makes their pulse quicken. But it also helps build a connection with the respondent, which will be important for a later part of the Candidate Lab.
The respondent is then asked to solve a coding task and navigate through a specific area of the interface without us in the room (after all, coding is rarely a social activity). They usually interact with a mock-up, i.e. a very limited and only partially functional development that mimics the final solution. It usually takes few days to a week to build, but it’s worth it.
As the coder traverses the interface, the researcher and the internal developer move to another conference room where they can observe the candidate’s screen and every action they take. For this, we use screen-sharing or sometimes more sophisticated software like Lookback. Just be sure the respondent consents to their work being monitored during the exercise. The researcher and internal developer are often accompanied by other developers and designers involved in the tested project. They all observe, learn, and discuss the findings in real-time. If the mock-up allows it, the user research crack team can even introduce changes live so that the following candidate works out of a freshly improved prototype.
What we gain from the exercise:
Once the coding portion of the interview is finished, the researcher and the internal developer reconvene with the candidate for feedback and discussion. Again, having the internal developer present for this part is also helpful. The researchers are rarely developers themselves, so it’s easy for them to miss a piece of information that could be useful. The developer also knows all the questions that their team might have about the tested product. If they hear anything useful when talking to the candidate, they can redirect the conversation to collect more data for the background team.
Each Candidate Lab we run consists of approximately five interviews – we try to do them one-after-another so that we gather quick feedback and start development the very next day. We discovered that doing this was more time-efficient than doing interview sessions here and there, stopping our work every time and taking pause to frequently discuss results from individual sessions.
We wrap-up every Candidate Lab by analyzing the findings and categorizing them based on their significance. For instance, is it crucial for the feature’s performance? How long would it take to do implement? We’re always aiming for the low-hanging fruits.
One example of this, while testing Codility Angular Tasks, we realized that the ‘Run’ button was confusing Candidates. Some of them believed it initiated running tests and others thought it referred to running and submitting the finished code. Renaming the button to ‘Run Tests’ took an hour of work, but it was a fairly quick win to improve the candidate user experience and overall performance.
We did the same thing with changing task introduction to be more specific in the new interface. It was also a great opportunity to learn that the way we decided to test for Angular skills received extremely positive feedback from front-end engineers. It’s such a boost to morale to hear such great reactions after weeks of work.
Some Candidate Labs reveal crucial errors in the way we think about our own product, while others might surface minor cosmetic improvements. Whatever we learn, however, always translates to a better end-solution and helps us understand how we could do things better next time.
Here is some practical advice that might help to get you started with the Candidate Labs in your organization:
But most of all, enjoy the rare moments when you get an inside look on how your product is perceived and how it can be made better–they are precious!
Click below to check out other insights from the Codility team!
© 2009–2016 Codility Ltd., registered in England and Wales (No. 7048726). VAT ID GB981191408. Registered office: 107 Cheapside, London EC2V 6DN