Research Based Development

The following diagrams show how one of my projects applied User Research to drive the software development process. There were two distinct modes of operation – before we went live with Release 1, and afterwards.

Before Release 1

Research was lab based, and essentially theoretical – we would do the best we could to recruit actual and potential customers and find out what they would do if they were faced with our system, and these test subjects would do the best they could to pretend that they needed our services.

Here’s a simplified workflow:

We would start off with a hypothesis, which we’d refine by thinking about it and working with the business. We would design how the lab would run, what experiments we’d perform, all to maximise the learning that we could get from the lab. We’d build multiple prototypes to test the variations on the theme and we’d conduct several lab sessions. At some point our analysis would show whether or not the concept we tested had legs, possibly requiring a few refinements and retests. If proven, we would then tease out what we needed to do to deliver a product increment that capitalised on what we’d learnt, build it and demonstrate it to our stakeholders.

At this point, we’d hope we’ve done the right thing. We’d only really find out when we deploy into production.

Production Based Research

Even after we’d gone live for the first time with our Release One, our reliance on User Research remained. We’d still come up with hypotheses and design suitable experiments to prove or disprove them. All that fundamentally changed though, was how and where we ran our experiments.Instead of conducting the experiment with theoretical users and pretend demands like before, we’d employ A/B or multivariate testing, build alternative flows through our system and just measure what our actual customers did. To measure demand for a new feature, we’d just put a button on the screen and count the number of times it was clicked (users would be given a “we’re assessing the demand for this feature, your interest has been registered, in the meantime do this” message). The proven hypotheses were demonstrably proven as the outcomes were better (as opposed to before when they were still technically just theoretical). The cost was that disproved hypotheses needed additional work to remove the code from the live system. We thought it was a cost worth paying.