UX and the Engineering Mindset (Evidence Based Design)

First posted: August 30 2013

In the past creating software, websites or products has been treated like a technical challenge, the task of designing then creating something that will work and can be sold through marketing. In many cases it was business and engineers coming together to create something. The process often looked something like this.

Even now in the world of startups we still see the same thing. Concepts like Lean Startup are essentially this process with shorter timescales, more iteration and more marketing style research. It is very easy to feel that by speeding things up you are solving the problems quicker. In reality if the process is not effective at delivering software you are doing more work to try and make it effective rather than fix the process.

So why is an engineering based process so bad? Who is better to tame the technology than those who understand it the most? The answer comes from many situations where systems have worked, are technically sound but are not usable. The engineers solve the problem given to them according to the requirements given to them. It is the right solution given all the information. So the issue lies not with the engineers, it is down to the information provided to the engineers. If they are expected to fill in the gaps of the design then the results can literally be fatal.

In 1992 a system created for the London Ambulance Service lead to 11 hour waiting times and was directly related to 30 deaths. Yet the system did exactly what it was designed to do. As requests came into the system they would disappear off key screens and attempts to retrieve the information proved impossible due to the interface. The problem lay at the lack of the right specification and a lack of any testing under real world situations.

In 1995 Americans Airline Flight 965 crashed into the side of a mountain. After the investigation the cause was put down to pilot error. In reality the problem was caused by how way points where identified. When the pilots attempted to find the next needed waypoint with a beacon that was identified as ‘R’ on their chart. The full name of this beacon was ROZO. The pilots entered ‘R’ into the flight system and instead of getting ROZO they got Romeo, a way point in Spain not the US. The system did not link ROZO with R. It gave no feedback that this might not be the right waypoint and just diligently reset the course with disastrous results. The system was logically correct but fatally incorrect in terms of it’s user experience. This example comes from Alan Cooper’s book The Inmates are Running the Asylum:

Dangers of The Engineering Mindset

Why am I blaming the engineering mindset for this? Actually I see the engineering mindset vital for solving engineering problems. The issue is that when designing for a user all the good tricks and skills that are useful for getting a computer or device to do something are potentially dangerous when applied to users. Where ever I say Engineers I mean someone who is using an engineering mindset. A person can use different mindsets during a project but it is very rare that someone can switch between them rapidly without a high cognitive load.

This is a summary of the engineering mindset.

  • Engineers Love complex problems, and the challenge of solving them..
  • Engineers love elegant technically sophisticated solutions.
  • Engineers are very knowledgeable about technical language.
  • Engineers think literally and logically.
  • Engineers approach problems in a sequential way.
  • Engineers like a challenge to be set out for them. Engineers will design for themselves and their peers.

Given these traits which are vital for solving technical issues it is no wonder that specifications are literally translated into something that does what it should do. In worse situations the specifications will be very badly written and lead engineers to attempt to design solutions. This leads to solutions that will be technically orientated. For example engineers will use terms like ‘submit’ and ‘field’ on a form as those are the terms that they use for form entry, but it is not what the users use. An engineer would never put two search boxes on the same screen that do the same thing because that would make no logical sense, even when the user is focused on the content of the screen and the search box is tucked in the top right.

Given this it is vital that the right information is provided to the engineers when they perform their tasks. It is not a case of throwing ultra-detailed specifications over a virtual fence but ensuring the engineering part of a project is acknowledged, listened to and provided with what is needed. Engineers are people too.

So how is the right information provided?

The Solution = User Research.

The only way to really know if what you are doing is right for end users and that is by talking to them. Many companies will believe they already do that by asking users what they want and listening to their feedback. Here’s a slide I created that is deliberately antagonistic to that way of thinking.

Let me explain each of these points and why they are important.

‘Don’t listen to users’ is to encourage everyone to instead of listening to what users have to say in focus groups to instead watch the behaviour of a user use something. There is a big difference between what users say they do and what they actually do.

‘Don’t ask why they need’ is to prevent people asking for feature wishlists. By working out what users do and what they need to do it will give you much better answers than a set of things that user think they need. For example the BBC asked user what they needed and many said a personalsed page, so they built one. A few years later on they replaced it with a page that could not be personalised as less than 1% of all the users did anything more than change the location of the weather forecast.

Don’t ask for feedback’ as it often users will tell you what you want to hear and will be unable to tell you really what the issue is if they do have an issue. For example have you ever had a small problem with the food you’re eating at a restaurant but when asked ‘is everything okay’ said ‘yes, it’s good’? It’s not uncommon.

This leads onto this slide about bad User Research methods.

There are multiple problems with focus groups in user research. First it is reliant upon people telling you what they need and relying upon anecdotal evidence. The group dynamic also hinders all members being able to communicate in a free way. If you are trying to gauge reactions in social settings, focus groups work but outside of that they can be deeply misleading. Also they suffer from ‘the vividness’ effect where the best story tellers and communicators sell their story to those looking much better regardless of the validity of the story. Avoid.

Surveys often also lead to distorted results and are best used to determine rough numbers of types of users.

Eye Tracking is a very expensive way of companies telling you things they should already know. A good designer will know how a user looks at a page or scans a mobile phone screen as eye tracking has been used effectively in academia to determine overall user behaviour patterns. Repeating them for a website or mobile phone app add more noise to the proceedings. What users are looking at is not the same as what they are seeing.

Lastly feature wish lists are great for making users feel they are being listened to and can be mined for low level design solutions (often there are some good ideas put forward in terms of implementation) but they should be avoided for user research.

Instead here are a few good techniques.

Above all one to one contextual / ethnographic interviews remain the best way to understand the end user. Unlike user testing you can conduct 40 interviews and still be learning about the user. How best to do these interviews is a separate article but in a nutshell get to know the user, who they are, what they do and get them to talk about the task before getting them to show you. This may go counter to some of what I have said before but it’s about letting the user get themselves back in to their context of use.

Diary Studies / Experience Sampling are a way to take the contextual approach further and see how a product fits into their life.

User Testing is a much more low level method of finding out if something is working. If done in a lab it is a great way to get stakeholder buy in, although it will be an unnatural environment for many users. I’ve also run user testing on competitor and comparable websites and got vital information. You don’t need more than six users for any test and is something that can be done as soon as there is something to show. Avoid myths like sketches are better than final designs because users find it easier to comment on incomplete work. You want the experience to be as close to end result as possible to get the best feedback. Don’t user test wireframes for example.

Lastly A/B / multivariate testing is great for fine tuning. You’ll can’t design using A/B testing but you can resolve detail. It’s also great to understand reaction to content used and wording. There are many new multivariate solutions becoming available that allow solutions to be evolved over time.

So put all this together and you get a process something like this.

The user is listened to throughout the process and beyond. For those who may be wondering ‘what about Agile’, just apply these stages with an iterative loop. Agile is a way to focus engineers and provide a heartbeat to a project, it’s not a process in itself and is dependent upon the techniques used and the people on the team.

What this means for businesses.

Letting engineers define functionality and chase features will result in bigger software with more things that can be sold to the end user. It also appears to be a way of dealing with feedback, but as we seen, feedback can be misleading. What happens time after time is that a device where user experience is seen as an afterthought is leap frogged by products with less features but much more consideration for the end user. Doing one thing well is more attractive than doing 10 things okay. Chefs do no use Swiss Army knifes when they prepare food, they often have a limited set of specialist knives, usually a core set of one with one knife being used the majority of the time.

Example 1: Altavista vs Google.

Here are two examples of where the engineers solution has been leapfrogged by a solution that is built up on User Experience. First up is Google. Here is a side by side comparison of the then top search engine vs the beta version of Google.

AltaVista back in 1999 was the king of search engines. It is what we all used. Yet on July 9th 2013 it was killed off by Yahoo. These side by screens may explain why. AltaVista had got the portal bug that has killed off so many search engines and sites. Yahoo itself was the place to go to find a site prior to AltaVista and Webcrawler, but then it got portal fever and has survived only through diversifying and throwing money at problems.

Meanwhile Google is, despite it’s solid engineering credentials, always had the user at the heart of what it does. Google was and is about User Experience first as it declares this in its ‘Ten things we know to be true’.

Focus on the user and all else will follow.
Since the beginning, we’ve focused on providing the best user experience possible. Whether we’re designing a new Internet browser or a new tweak to the look of the homepage, we take great care to ensure that they will ultimately serve you, rather than our own internal goal or bottom line.”

Example 2: Nokia N95 vs iPhone 1

I owned a Nokia N95 for a week. It’s terrible battery life and awful overall usability could swamped the potential good things it could do. Here’s a quick side by side comparison.

On paper the N95 should have walked it. The camera was much better, it sounded better and it had real buttons. If you asked users if they wanted real buttons they would give a resounding yes! Another example of where feedback leads to giving users what they ask for, but not what they need. When users used both of these phones in real life it was clear that playing with the touch screen somehow blinded the user to all the technical goodness that Nokia had squeezed into it’s device. In many ways the N95 marked the low point of Nokia, although it has definitely come out fighting and with it’s N9. In name it was only a 5 away, but it in usability it was the best phone no one bought (a contradiction created by Nokia essentially killing it off at birth).

Meanwhile the iPhone was created by a company that invented the term User Experience, or rather Don Norman who was working for Apple coined the term in the mid 1990s. Users who were new to it love flicking between photos and were amazed they could zoom by pinching. It did a few things and it did them well. Despite being behind on the numbers of features and the quality of the camera Apple release something that users loved and found easy and quick to use. Other companies remained perplexed on how a phone without a physical keyboard could survive, especially those weaned on blackberries. It was not just because of Apple's marketing that they succeeded.

The moral:

If you want your company to be great then you put the user above the technology. The engineering team is there to solve epic challenges set in the task of providing the best possible user experience and that user experience has to be based upon.

The Engineering Mindset and User Experience Professionals...

Having User Experience in your job title does not mean you are immune from the Engineer Mindset. It is often asked should we code? The issue that if we do then, if we are not careful we may fall into an Engineering mindset and start putting technical solutions above user solutions. We may fall into the trap of building and then asking for feedback rather than researching and testing our assumptions as we go. Conversely we need to know enough to be able to have meaningful conversations with the vital people who do have an engineering mindset. We have to stay realistic and also push for an outcome that will make a true difference. Knowing which things to push and understanding the engineering issues so we can best negotiate helps.

Ronnie Corbett, one of the UK’s masters of light entertainment, defined a gentleman as - "someone who knows how to play the bagpipes but doesn't." The same should be true for a good User Experience person and coding. A good UX person knows how to code but doesn’t. It is only possible if a person can separate that part of a project form the other.

...And Designers.

The word design currently has a problem. It has two distinct meanings that clash. The first is design as in the process of design thinking, of creating a solution. The second is design as in visual design. These two are not the same.

There is a current trend (as of August 2013) to lump User Experience Design under the same overall job description as Interaction Design and Visual (Graphic) Design. Many jobs treat User Experience as something that is done along side other tasks. My view is that Graphic Design, in my view, overlaps in many ways with the work of an engineering team. The work is often low level and about creating the final output. Many designers fall into a different form of engineering mindset where the final surface of a product swamps the needs of the users. Form crowds out function. This may be acceptable if designing a marketing campaign but a creative director who is used to creating end results without user involvement or through marketing type research often produces user experiences with as many issues as engineers. The same is also true of interaction designers how get stick in trends of interfaces whilst ignoring the needs of the user.

What is even worse is that this misunderstanding is common in many organisations. This leads to not only a focus on technology but also on a focus on the surface of the user experience, that is the final User Interface. I will write more about this soon but UI is a small part of UX.

How the original talk was received.

This article is based upon a talk I gave recently at the first London Skepticamp event, a Skeptics (that is a rationalist evidence based thinking) event in London. It was a non UX crowd and included several self identifying engineers. Two that spoke to me after identified that the main problem they had with their past jobs had been the lack of clear requirements that came through from the marketing department. Giving a clear problem to a skilled engineer solves that problem. It is is vital to ensure that a product is based upon real world evidence. I also added that, in theory, anyone can do good quality user research and recommended the book Rocket Surgery Made Easy: The Do-it-yourself Guide to Finding and Fixing Usability Problems

See Also:

More UX Stew Posts