Usability Testing of Inline Form Validation: 40% Don’t Have It, 20% Get It Wrong

During our just-released checkout usability study, we time and again observed that there’s a direct correlation between how and when users are presented with form validation errors and the pace and effort with which they are able resolve those errors.

While the logic and wording of error messages are very important aspects of the user’s validation error recovery experience, our latest eye-tracking and usability testing shows that in order to perfect the user’s recovery experience, live inline validation must be used.

Definition: “Live inline validation” is where the validity of the user’s inputs are checked live as the user progresses through the form, as opposed to checking the inputs in a lump sum when the user submits the form.

Now, our checkout benchmarks reveal that live inline validation has seen an explosive growth in adoption: in 2012, just 13% of checkouts had live inline validation, whereas in 2016, 60% had implemented it. This makes good sense since the general importance of live inline validation is well documented. In fact, it is surprising that 40% of major e-commerce sites still don’t have inline validation.

Another surprising matter is that the potential implementation pitfalls of inline validation haven’t been highlighted much. During our end-user testing we consistently observe two pitfalls that can end up making live inline validation more frustrating than helpful to the user:

1. Premature Inline Validation – the user’s input is marked as invalid before they have finished typing, essentially reprimanding the user for mistakes they haven’t made and prompting them to look for errors that aren’t there.
2. Outdated Error Messages – error messages aren’t updated and removed “live”, as the user edits their input, resulting in outdated error messages being displayed even though the user has actually fixed the problem.

Despite these two implementation details being critical to a positive form validation experience, of the checkouts that have inline validation, 20% fail to address at least one of the above pitfalls.

In this article we’ll cover: 5 core UX benefits observed for live inline validation (particularly relevant to the 40% of sites that still lack it), the 2 pitfalls of live inline validation (especially relevant to the 20% of site implementations that fail to address them), along with a 3rd consideration for those seeking to perfect their inline validation experience.

5 Benefits of Inline Validation Observed During Testing

During our large-scale usability testing we observed 5 distinct benefits to using inline validation:

1) Locating errors is easier. Users are told about the error immediately, meaning that they won’t to the same degree have to come to a full stop on a dedicated error page, to try to figure out where the error is. With live inline validation, users are alerted to errors immediately, and the process of locating the error is thus significantly simplified.

2) Field context is fresh in mind. Because users are alerted to the input being invalid right after typing it, the amount of time needed to correct it decreases significantly, as the input and its context are still fresh in the user’s mind. On a normal after-the-fact error page, the user will often have to recall the field context and re-read their input since it can be several fields and minutes ago the user read the field label and entered the invalid field value.

3) Positive inline validation increases confidence. If positive inline validation is used, then even users without any invalid input will benefit. Positive inline validation removes some cognitive load from users since they don’t have to review and validate the form for errors before submitting it. Instead they can confidently proceed to the next field, resting assured that their input is valid. Positive inline validation can furthermore instill a sense of rapid progression and accomplishment as the user “successfully completes” each field and the site provides them with encouraging visual feedback.

4) Less likely to unintentionally skip required fields. In addition to making sure that inputs are valid, inline validation is also efficient for highlighting omitted required fields as users tab through the form (which we found more than 50% of users generally do during checkout). With inline validation in place, users are notified immediately if users tab through a required field without entering a value – vastly reducing the odds of a user accidentally skipping a required field.

5) Persisting credit card data despite lower-level PCI compliance. One of the ways that e-commerce sites can get away with lower-level PCI compliance is by never letting the user’s credit card data touch their server but instead having the user send their credit card information directly to the payment processor. However, one downside of this is that any validation errors will tend to clear the form input because the e-commerce site has naturally has no way of persisting data they never touch or receive. Inline validation can prevent invalid inputs (e.g. a card number that’s too short, too long, or fails Luhn validation) from being submitted in the first place, and thus avoid the page reload that clears out the user’s entire input.

Despite these rather significant benefits, 40% of sites still lack inline validation. However, inline validation isn’t great by default – there are implementation pitfalls that must be avoided or the inline validation may end up doing more harm than good. And apparently these are pitfalls that even major sites may succumb to – upon benchmarking 50 major e-commerce sites, we found 20% of the sites with inline validation to stumble on at least one of the two following pitfalls.

Pitfall #1: Premature Inline Validation

“Why are you telling me my e-mail address is wrong,” a subject complained, “I haven’t had a chance to fill it all out yet!” – the site’s live inline validation had shown an error message immediately when the subject began typing his e-mail address.

While the point of live inline validation is to alert users of invalid inputs early on, overly aggressive premature validation can result in users being distracted from their typing, feeling unfairly reprimanded, and wasting time looking for errors that aren’t there – hardly a great checkout experience!

Most people don’t like being told they are wrong – especially when they aren’t. Users therefore naturally find it very frustrating and (quite understandably) feel unfairly reprimanded when a site claims they’ve made a mistake before they’ve had a chance to enter a valid input.

Moreover, during testing, the sudden appearance of an error message disrupted many of the subjects in their typing of a perfectly valid input, as they stopped to read and interpret the error. Even worse, some subjects misinterpreted this message to mean that what they had entered so far contained some error and thus began looking for mistakes that weren’t there.

It’s therefore key to fine-tune the implementation of inline validations to avoid them becoming a distraction and annoyance to the vast majority of users. In essence, premature validations must be avoided. More specifically, when users enter a new field – without any existing errors – and begin their typing, the validity of the input should not be checked until the user has had a chance to fully type a valid input. Despite benchmarking the checkout flows of major e-commerce sites, 10% of the sites fail to get this right.

Depending on the type of input, avoiding premature validation means that the validity of each field input should be checked when the user leaves the field or reaches the intended input length.

For most field types, the validity may be checked once the user leaves the field (by binding to the onblur event of the field). Why not sooner? Because the site has no way to tell if or when the user has finished typing and considers their input “done” or “fully entered”. It’s therefore only safe for the site to conclude that the user considers their input “done” when they leave the field / proceed to the next element in the form.

That said, for certain inputs error messages may be shown while the user is typing. These are inputs with fixed or predictable character lengths and/or formats (such as postal codes, phone numbers, credit card numbers, coupon codes, etc), where the validity can be checked once the user reaches this intended length. This logic can, however, be complicated and tends to be highly input-specific, as it all depends on the possible lengths and formats of the input. There’s thus no general rule that can be applied to all of these inputs, although it is often possible to show immediate warnings for inputs that can have: invalid characters (e.g. alphanumeric characters in a numeric input), impossible character combinations (e.g. Luhn validation), and too long inputs (e.g. a 15-digit phone number), can typically be warned against immediately (i.e. before the user leaves the field).

If resources are constrained, it’s best to “play it safe” and check input validity when the user leaves the field instead of half-seriously dabbling with advanced early warnings. After all, if the logic for these early warnings isn’t flawless, your users can end up with premature validations, and despite all the hard work your live inline validations may end up doing more harm than good.

Pitfall #2: Outdated Error Messages

When an error has occurred and a live inline error message is shown to users, they will try to correct it immediately. However, 10% of sites that have inline validation, do not remove the error message live, as the user resolves the issue.

During testing, the subjects often didn’t click out of a field – instead they would look intensely at the error message as they made corrections (i.e. on a keystroke level), expecting the error to be removed as soon as the field contained the valid input. Not removing error messages live, as the error is corrected, can thus cause grave issues as users are likely to misinterpret their newly corrected (and now valid) input to still contain errors. This is particularly true for fields where the validity is interdependent, such as email and password confirmation fields.

It’s therefore key that when an error is invoked using live inline validation, the logic for re-checking the validity of the invalid field doesn’t just happen when the user leaves the field. Error messages must instead live-update on a keystroke level (i.e. not just onblur), and disappear the moment the user reaches a potentially valid input.

The danger here is in removing the error message too late – effectively the opposite of premature inline validation where the danger is in showing the error message too early!

To avoid the two pitfalls of inline validations it is thus important to give users a fair chance to enter a valid input before showing an error message (a conclusion we can often only draw once the user leaves the field), yet that same error message must be updated and removed instantly upon the user fixing their input (i.e. at each and every keystroke).

Consider Positive Inline Validation Too

Lastly, during testing we observed positive inline validations to perform well. Instead of only telling users what they’ve done wrong, a site may also tell them when they’re doing something right. During testing, sites that used positive inline validation, by adding a checkmark next to valid input values, added a sense of delight and progression to the whole typing experience.

Positive inline validation is not only observed to add a sense of progression to an otherwise dull typing experience. It can also provide a better typing experience for risk-averse users trying to preempt validation errors, as well as users who otherwise tend to meticulously review the entire form before submitting it.

Our benchmark reveals that only 20% of major e-commerce sites with inline validation also use positive inline validation.

Inline Validation Requires an Advanced Live Logic

During the past 7 years of checkout testing we have consistently found inline validation – when implemented correctly – greatly improves the user’s error recovery experience. Luckily, this hasn’t gone unnoticed by the e-commerce industry at large, as we have observed a 4-fold increase in implementations during the last 4 years.

Yet, a high performing inline validation implementation requires close attention to details, due to the opposing logic for invoking and removing the live error messages:

1. On one hand, avoiding premature validation typically means that the validity of each field input can only be checked when the user leaves the field (something 10% of site implementations fail to do).
2. Yet, once an error has occurred, the displayed error message must live-update on a keystroke level and disappear the moment the user reaches a potentially valid input. Especially for interdependent fields, like e-mail and password confirmation fields. (Another 10% of site implementations fail to do this.)

Lastly, if looking for inspiration, our checkout benchmark shows that Etsy, Crutchfield, and B&H Photo, have particularly great inline validation error implementations.

This article presents the research findings from just 1 of the 580+ UX guidelines in Baymard Premium – get full access to learn how to create a “State of the Art” cart and checkout user experience.