A/B testing cannot substitute for taste in creating appealing products. Relying solely on A/B testing is abdicating your responsibility to the users.
Book: Creative Selection
Douglas Bowman, a designer formerly at Google, justified why he left the company:
"Without a person at (or near) the helm who thoroughly understands the principles and elements of Design, a company eventually runs out of reasons for design decisions … Without conviction, doubt creeps in. Instincts fail … When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. Data in your favor? Ok, launch it. Data shows negative effects? Back to the drawing board. And that data eventually becomes a crutch for every decision … Yes, it's true that a team at Google couldn't decide between two blues, so they're testing 41 shades between each blue to see which one performs better."
Google used A/B tests to make color choices. This is an algorithm approach where the best kind of blue is defined as the one people click the most. In contrast, Apple never considered the notion of an algorithmically correct color. They used demos and taste to pick colors.
While A/B tests can help determine colors that will get people to click something more often, they can't produce cohesive and pleasing products. They lack refined responses and fail to recognize the importance of balancing choices. Compared to Google, Apple values taste more in design and never staged A/B tests for iPhone software. Instead, they put faith in their sense of taste. They use heuristics (subjective judgment) over algorithms (objective measurement) when making design decisions like picking colors.
Brian Chesky, CEO of Airbnb, shared a similar perspective on A/B testing at the Config Conference. He stated that A/B testing is like giving up your accountability to the users. At Airbnb, they only use A/B testing if they have a clear hypothesis and know why one option might be superior to another.