Communicating our life experiences is about sharing memories. Our memory is largely dependent upon how impactful an event is, our concentration levels, or repetition, and what we recall can be changed by many types of memory biases. Chances are you can’t remember much about a mundane task from a week ago, or even a day ago, but will be able to remember a significant event from ages ago such as your first kiss or competing in a championship game. Is it easy to remember how audio gear sounds? When you hear a new piece of audio gear, do know if it sounds better or worse than another piece of gear you haven’t heard in a while? How does this play into our perception of gear, and what are the implications for audio reviews?
Researchers at the University of Iowa performed a study to compare three types of memory: visual, tactile, and auditory. The study used pure tones through headphones, color variations, and vibrations with a baseline and a test sample separated by one to 32 seconds. The study found that memory declined for all types of events, but the decline for sound was much greater, starting around four to eight seconds.
A second study was performed by playing back familiar sounds, watching silent video, and touching common objects that couldn’t be seen, tracking recall between an hour and a week later. The results showed that visual and tactile memory recall was similar, but the test subjects were worse at remembering the sounds they heard.
The takeaway is that visual and tactile memory is recorded in a different way and superior to auditory memory for recall reliability. It is difficult to remember sounds, let along differences in nuances of a sound both in the short term and long term unless we practice, concentrate, or have a significant event accompany the sound.
AUDITORY MEMORY & REVIEWING
How does memory, or the fact that human audio memory is not the greatest, play into the audio hobby? How do we determine just how good a piece of gear really is? Measurements are an indicator of performance and can relay certain information, but they don’t tell the whole story and require significant know-how and consistency to both test and interpret correctly. The method we prefer to use at The Headphone List is extensive A/B comparisons. A/B comparisons help overcome the poor auditory memory performance of humans, revealing and pinpointing technical differences in gear and helping determine the true sound signatures of different pieces audio equipment. The ultimate goal is not to see how some gear graphs, but how we hear the sound and how it compares with what else is available. ljokerl and I both share technique for A/Bing, and my technique is described in detail below.
MY REVIEW PROCESS
For me, reviewing is a serious business. Sure, it started as a hobby, but now I value accuracy and communicating what I hear in a way the audio enthusiast can understand and relate to over an enjoyable listening experience (and rarely have the time to just listen these days). Over time, my review process has evolved, with better technique and changes to the order and way things are done to get the best results. Before I start reviewing, I burn in all products and after burn in, before testing, they get some ear time.
I typically give a listen upon receipt to make sure everything is working right and to get a feel for the sound. For custom IEMs I also perform a quick fit check. I will then start a new document for that product with brief comments as well as a new entry in my score table. From time to time, I have A/Bed the new product with a familiar and similar product to gain a reference and try to determine if there are any changes during the burn in process. At various intervals I perform the same A/B comparison with the same song and source to determine if there are any changes.
Burn in is a controversial topic, and while I didn’t want to believe it, my experiences have told set me straight. I am not going to go into detail about burn in here, and there is both brain burn in and product burn in, but as an engineer with materials knowledge, I do believe in both occurring to varying extents. I will leave this topic for another time.
Once burn in is complete, I then listen to the product for a while to get a feel for the sound. While this doesn’t give me much perspective, I do get a feel for the sound signature. After listening for at least several hours, I am ready to start reviewing.
All the testing gear is laid out for quick switching between two pieces of gear. It depends on the gear, but I shoot for a 4-6 second switch. Joker and I have discussed ways to improve the A/B testing, and we came close to having some switchers custom built for us, but they never materialized. If I am A/Bing headphones and using a source with dual output like the DX100, I will make sure both headphones sound the same when simultaneously plugged in. If not, I have to manually switch plugs.
Once everything is setup, I listen to one of my test tracks to start my A/Bing. I know my test tracks quite well, and choose only a small segment of a song, almost always at the beginning so I can easily restart the track after switching gear. I focus on one aspect of sound such as bass response, vocal tonality, treble reverb, spatial queues, etc. Some examples include Billy Idol – Prodigal Blues for bass depth and Balmorhea – Context for spatial qualities. I listen for the main musical notes as well as the nuances.
This isn’t done once, but typically 5-20 times per track depending on many factors such as how apparent the differences are and my state of mind. Yes, my state of mind, which includes how long it has been since my last A/B session, how tired I am at the moment, and even my mood affect my A/B sessions. There are times where I realize I can’t A/B at the time for whatever reason, so I will work on something else and A/B later. Once I get in a rhythm, I can A/B quite effectively and relatively quickly. I may be able to A/B 2-3 headphones in one session for example, which require complete concentration and is usually done at night.
During my A/B sessions, I take notes and formulate my scores for my extensive sound signature and technical scoring table. The table allows me to overcome memory limitations and “remember” the performance level and sound signature at detailed levels. In order for me to get a meaningful score, a minimum of 3 comparisons with similarly performing products need to be performed, and at least another 2 to get a much more solid result. The chart allows me to recall headphone performance with certainty. Overall scores as they currently stand are listed in my chart.
In addition to quick A/Bing, I also switch between different pieces of gear after listening to each for longer periods of time (at least 5 minutes). This allows me to better assess overall differences and tonality. The overall differences I found during A/B testing should be there with this type of testing, and knowing what they are, I should be able to easily pick them out. For me, it is much easier to hear a lack of something, such as bass control, treble extension, or spaciousness than it is to hear the improvement. Tonality is also assessed this way as the brain adjusts to what you are listening to as described in Perfect pitch is not infallible, but one will sound more “right” than the other when switching back and forth. I find this to be a good sanity check for my A/Bing.
As a believer that the human interface is extremely important to the audio experience and a very complex, non-linear system that has uniqueness for each individual. I try to take all of the factors I can into account when reviewing to present a consistent and as bias-free viewpoint as possible. Of course, the best way to read a review and judge a reviewer is to hear the gear they are reviewing and see if you can hear the gear as described.
REVIEW PROCESS AND ABILITY
How does auditory memory play into reviewing, why is it important for audio reviewing, and why did I explain my review technique above? Over time, my auditory memory has improved. I believe that various people have different audio abilities based on many factors of the human interface including ear function, brain capability, and mental performance. Most people can improve their memory and ability to hear nuances, spaciousness, and improve auditory memory. It would seem to make sense that like with athletic ability and natural sight, various people have differing levels of natural performance and ultimate ability.
An interesting article that I think somewhat relate are Some people really just don’t like music, study says. To summarize this article, some people don’t have reward systems that respond to music, and therefore they just don’t like music. The article links to a music reward questioner if you want to find out where you fall on the music reward spectrum.
Ultimately, people that are interested in audio will pay attention to the details necessary to hear the differences between gear, and ultimately will enjoy the benefits of better audio gear, and with training, auditory performance can improve. This can explain why non-audiophiles don’t really care about the high end audio gear, as they don’t put in the concentration or time to recognize the differences, even if some can readily hear it. This may also explain why some people can hear differences in gear others can’t, although there too many other factors that also play a role to discuss here.
Have you ever wondered why technique people use for audio reviews? My review technique is listed above, which is largely about the process, practice, consistency, and understanding myself, but I often wonder what techniques others use, especially some of the mainstream reviewers. Reading some reviews of audio gear, whether they are from a mainstream site, small YouTube channel, or even some found on audio enthusiasts sites make me wonder if they listened to a product for more than 5 minutes. How can their review offer so little information, and what they offer is so different from what I am hearing? Do they account for things such as mood and environment that affect the perception of sound?
Is it possible for anyone to truly “memorize” how one audio product sounds and be able to discern the differences from a different product from memory? I have seen audio enthusiast forum members make countless recommendations and statements based purely off memory, and in environments that could affect perception. While it is their prerogative to do so, others make buying decisions based off these statements that can go dramatically against my experiences. Auditory memory is fallible and having two pieces of gear for direct comparison will give much better results. Have you listened to a piece of gear after a long break and realized it sounded different than you remember?
In my opinion, too many reviews are too positive without any direct comparisons or really explaining why. Could it be that they are relying on the fallible human auditory memory to make their judgment?
We want to hear your thoughts on the current state of audio reviews in the mainstream media, on online retailer sites, and audio enthusiast sites. Please let us know what techniques you use for evaluating audio gear and share any tips, tricks, or gear that will help with the review process.