Communicating our life experiences is about sharing memories. Our memory is largely dependent upon how impactful an event is, our concentration levels, or repetition, and what we recall can be changed by many types of memory biases. Chances are you can’t remember much about a mundane task from a week ago, or even a day ago, but will be able to remember a significant event from ages ago such as your first kiss or competing in a championship game. Is it easy to remember how audio gear sounds? When you hear a new piece of audio gear, do know if it sounds better or worse than another piece of gear you haven’t heard in a while? How does this play into our perception of gear, and what are the implications for audio reviews?
Researchers at the University of Iowa performed a study to compare three types of memory: visual, tactile, and auditory. The study used pure tones through headphones, color variations, and vibrations with a baseline and a test sample separated by one to 32 seconds. The study found that memory declined for all types of events, but the decline for sound was much greater, starting around four to eight seconds.
A second study was performed by playing back familiar sounds, watching silent video, and touching common objects that couldn’t be seen, tracking recall between an hour and a week later. The results showed that visual and tactile memory recall was similar, but the test subjects were worse at remembering the sounds they heard.
Another study from the National Academy of Science in 2008 showed that visual memory is remembered much easier, but what I found interesting about this study was the methodology of one of the tests. Participants listened to 64 sound clips of common sounds such as birds chirping, a motorcycle, etc. and were tested immediately after. The test had 32 of the sounds the participants had heard, and 32 new sounds. Correct identification happened 78% of the time, with 20% false positives (remembering something they didn’t hear). In contrast, visual tests of the same type had a 98% rate of correct identification for 600 pictures, vs. 64 sound clips. There were other tests done, and the part of the conclusion was “auditory recognition memory performance is markedly inferior to visual memory performance…” Although, we obviously can remember sounds long term.
The takeaway is that visual and tactile memory is recorded in a different way and superior to auditory memory for recall reliability. It is difficult to remember sounds, let along differences in nuances of a sound both in the short term and long term unless we practice, concentrate, or have a significant event accompany the sound.
AUDITORY MEMORY & REVIEWING
How does memory, or the fact that human audio memory is not the greatest, play into the audio hobby? How do we determine just how good a piece of gear really is? Measurements are an indicator of performance and can relay certain information, but they don’t tell the whole story and require significant know-how and consistency to both test and interpret correctly. The method we prefer to use at The Headphone List is extensive A/B comparisons. A/B comparisons help overcome the poor auditory memory performance of humans, revealing and pinpointing technical differences in gear and helping determine the true sound signatures of different pieces audio equipment. The ultimate goal is not to see how some gear graphs, but how we hear the sound and how it compares with what else is available. ljokerl and I both share technique for A/Bing, and my technique is described in detail below.
MY REVIEW PROCESS
For me, reviewing is a serious business. Sure, it started as a hobby, but now I value accuracy and communicating what I hear in a way the audio enthusiast can understand and relate to over an enjoyable listening experience (and rarely have the time to just listen these days). Over time, my review process has evolved, with better technique and changes to the order and way things are done to get the best results. Before I start reviewing, I burn in all products and after burn in, before testing, they get some ear time.
I typically give a listen upon receipt to make sure everything is working right and to get a feel for the sound. For custom IEMs I also perform a quick fit check. I will then start a new document for that product with brief comments as well as a new entry in my score table. From time to time, I have A/Bed the new product with a familiar and similar product to gain a reference and try to determine if there are any changes during the burn in process. At various intervals I perform the same A/B comparison with the same song and source to determine if there are any changes.
Burn in is a controversial topic, and while I didn’t want to believe it, my experiences have told set me straight. I am not going to go into detail about burn in here, and there is both brain burn in and product burn in, but as an engineer with materials knowledge, I do believe in both occurring to varying extents. I will leave this topic for another time.
Once burn in is complete, I then listen to the product for a while to get a feel for the sound. While this doesn’t give me much perspective, I do get a feel for the sound signature. After listening for at least several hours, I am ready to start reviewing.
All the testing gear is laid out for quick switching between two pieces of gear. It depends on the gear, but I shoot for a 4-6 second switch. Joker and I have discussed ways to improve the A/B testing, and we came close to having some switchers custom built for us, but they never materialized. If I am A/Bing headphones and using a source with dual output like the DX100, I will make sure both headphones sound the same when simultaneously plugged in. If not, I have to manually switch plugs.
Once everything is setup, I listen to one of my test tracks to start my A/Bing. I know my test tracks quite well, and choose only a small segment of a song, almost always at the beginning so I can easily restart the track after switching gear. I focus on one aspect of sound such as bass response, vocal tonality, treble reverb, spatial queues, etc. Some examples include Billy Idol – Prodigal Blues for bass depth and Balmorhea – Context for spatial qualities. I listen for the main musical notes as well as the nuances.
This isn’t done once, but typically 5-20 times per track depending on many factors such as how apparent the differences are and my state of mind. Yes, my state of mind, which includes how long it has been since my last A/B session, how tired I am at the moment, and even my mood affect my A/B sessions. There are times where I realize I can’t A/B at the time for whatever reason, so I will work on something else and A/B later. Once I get in a rhythm, I can A/B quite effectively and relatively quickly. I may be able to A/B 2-3 headphones in one session for example, which require complete concentration and is usually done at night.
During my A/B sessions, I take notes and formulate my scores for my extensive sound signature and technical scoring table. The table allows me to overcome memory limitations and “remember” the performance level and sound signature at detailed levels. In order for me to get a meaningful score, a minimum of 3 comparisons with similarly performing products need to be performed, and at least another 2 to get a much more solid result. The chart allows me to recall headphone performance with certainty. Overall scores as they currently stand are listed in my chart.
In addition to quick A/Bing, I also switch between different pieces of gear after listening to each for longer periods of time (at least 5 minutes). This allows me to better assess overall differences and tonality. The overall differences I found during A/B testing should be there with this type of testing, and knowing what they are, I should be able to easily pick them out. For me, it is much easier to hear a lack of something, such as bass control, treble extension, or spaciousness than it is to hear the improvement. Tonality is also assessed this way as the brain adjusts to what you are listening to as described in Perfect pitch is not infallible, but one will sound more “right” than the other when switching back and forth. I find this to be a good sanity check for my A/Bing.
As a believer that the human interface is extremely important to the audio experience and a very complex, non-linear system that has uniqueness for each individual. I try to take all of the factors I can into account when reviewing to present a consistent and as bias-free viewpoint as possible. Of course, the best way to read a review and judge a reviewer is to hear the gear they are reviewing and see if you can hear the gear as described.
REVIEW PROCESS AND ABILITY
How does auditory memory play into reviewing, why is it important for audio reviewing, and why did I explain my review technique above? Over time, my auditory memory has improved. I believe that various people have different audio abilities based on many factors of the human interface including ear function, brain capability, and mental performance. Most people can improve their memory and ability to hear nuances, spaciousness, and improve auditory memory. It would seem to make sense that like with athletic ability and natural sight, various people have differing levels of natural performance and ultimate ability.
An interesting article that I think somewhat relate are Some people really just don’t like music, study says. To summarize this article, some people don’t have reward systems that respond to music, and therefore they just don’t like music. The article links to a music reward questioner if you want to find out where you fall on the music reward spectrum.
Another interesting article is ‘Seeing’ bodies with sound (no sight required) discussing how in about 70 hours blind people can be taught to ‘see’ human shapes through sound, and even know what posture the person was using. The study was used for the blind, but the article Humans Can Lear to Echolocate showed that people with normal sight can learn to echolocate, although mentions that blind people are better at it.
Ultimately, people that are interested in audio will pay attention to the details necessary to hear the differences between gear, and ultimately will enjoy the benefits of better audio gear, and with training, auditory performance can improve. This can explain why non-audiophiles don’t really care about the high end audio gear, as they don’t put in the concentration or time to recognize the differences, even if some can readily hear it. This may also explain why some people can hear differences in gear others can’t, although there too many other factors that also play a role to discuss here.
Have you ever wondered why technique people use for audio reviews? My review technique is listed above, which is largely about the process, practice, consistency, and understanding myself, but I often wonder what techniques others use, especially some of the mainstream reviewers. Reading some reviews of audio gear, whether they are from a mainstream site, small YouTube channel, or even some found on audio enthusiasts sites make me wonder if they listened to a product for more than 5 minutes. How can their review offer so little information, and what they offer is so different from what I am hearing? Do they account for things such as mood and environment that affect the perception of sound?
Is it possible for anyone to truly “memorize” how one audio product sounds and be able to discern the differences from a different product from memory? I have seen audio enthusiast forum members make countless recommendations and statements based purely off memory, and in environments that could affect perception. While it is their prerogative to do so, others make buying decisions based off these statements that can go dramatically against my experiences. Auditory memory is fallible and having two pieces of gear for direct comparison will give much better results. Have you listened to a piece of gear after a long break and realized it sounded different than you remember?
In my opinion, too many reviews are too positive without any direct comparisons or really explaining why. Could it be that they are relying on the fallible human auditory memory to make their judgment?
We want to hear your thoughts on the current state of audio reviews in the mainstream media, on online retailer sites, and audio enthusiast sites. Please let us know what techniques you use for evaluating audio gear and share any tips, tricks, or gear that will help with the review process.
Thank you Joe!
I have a relatively long list of tracks I use for A/Bing, and I have listened to the tracks so many times I have the nuances of each (good or bad, as some have noticeable issues), allowing me to focus more on the recreation differences. Some additional tracks are as follows: Perry O’Neil – Kubik, Focal – The Boy Who Stole The Blues, Aerosmith – Dream On, Alicia Keys – Fallin, Apocalyptica – On The Rooftop With Quasimodo, Chemical Brothers at Coachella 2011 (24-bit 48kHz), Danilo Perez – Think of One , Electric Light Orchestra – Believe Me Now, Everything But the Girl – Two Star, James Horner – You don’t dream in cryo, Morphine – Lets Take A Trip Together, Nelly Furtado – All Good Things, Nightwish – The Poet And The Pendulum, Sevendust – Too Close to Hate, Symphonie Nr. 9 D-moll op. 125 1. Allego ma non troppo, un poco maestoso, etc.
The tracks all have something a bit different that I can focus on, such as Too Close to Hate, with aggressive cymbals that brings out top-end harshness, Fallin that lets me hear the tonality of a female vocal (despite the track not having the best master), etc. Pick some tracks you know well that can help you isolate, or list to various tracks to hear any parts that stand out, and use those. If you are going to listen to the same tracks over and over, you want to also make sure you like the track.
A/Bing with eyes close does enable better focus on the sound.
Do you A/B with your eyes, open or closed? 🙂 I have found by shutting my eyes I have more focus to spare towards observing sound
Can you please list some tracks which you use to assess specific focus of different sound characteristics during A/B ing? I really liked prodigal blues you mentioned to test bass depth.
Hi henkie, sounds like you do a good job of auditioning, which I am sure pays off!
Yep, we like plenty of comparisons and perspective. Many things without perspective can seem good. Plus, there is so much that is still only theoretical about the instrument we all use to listen: our brains.
On burn in, it does make perfect sense to me that there can be physical changes to surfaces as well as below the surface over time. One of my engineering professors told us about electron channels in silicone due to the doping, not unlike rivers…
I should probably add that I do indeed use only small segments of songs to actually audition stuff. A bit of one song for bass control, another bit for treble detail, another for bass extension, that kind of thing. For auditioning, I find this enough, though it would be better if I could hear more different headphones, but it’s pretty difficult to find headphones (or indeed IEMs) to audition around here.
For reviews, I can totally understand that you’d need to compare much more rigirously in order to find all the minutiae. For that, I would agree that you’d need to A/B the headphones. Either that or own them for a long time and regularly mix them up with different headphones so that you’d eventually just know the differences. Subconsciously if not consciously, in the same way that you can tell who someone is only by the sound of someone’s voice.
For reviewing, it’s great that you look into these things the more theoretical side to help improve your reviewing technique. It’s interesting to read about, too.
As for burn-in, perhaps it’s my mechanical engineering background, but I always imagine it to be something like running in an engine. Surface roughness on the driver that gets smoothed out as it gets used. Similar to the way the surface of a cylinder gets smoothed in the first few minutes/hours by the piston moving. I might be completely wrong, though.
Hi henkie, thanks for reading and commenting. It sounds like the way you audition is a good one and should be effective. I do believe that while there are commonalities, there also can be individualized differences in auditory memory and experience. The 4-6 second rule applies IMO because while a piece of audio gear may sound good in many ways, I don’t think it is truly possible to know the ultimate performance until there is a direct A/B.
Sure, I can tell if something is good or bad from first listen and get a general ballpark, but the ultimate level of resolution is difficult to determine without a direct reference for me. For example, I can listen to a song I know, and if there is detail missing, I can tell from memory, but just how spacious and how much of the background information is recreated is difficult to truly know without comparison. However, I am talking about higher-end products that have a very high performance ceiling and lower cost products are easier to make the determination for me (one reason my reviews take so long). For example, when I first heard Beats, I knew they were really bad without any direct comparison!
joker has earned a lot of respect because he is consistent and we both try to offer what we term the “audio facts” as interpreted by the brain. I am glad you and he have similar ideas of neutral, which helps tremendously!
We are working on fixing the Ã characters, which occurred during an update.
Yes, burn-in is controversial with many possible reasons. While I don’t have research to back this up, my engineering background and physics knowledge lead me to believe it comes down to the fact that sound, when in the electrical form, is electrons flowing through a material, and material changes can change the flow as well as the physical materials of a driver can change, even if ever-so-slightly, affecting the sound. Some examples that could affect sound: a voice coil alignment in the magnetic field ,surround resistance, capacitance changes in wire, temperature, etc.
Nice read. I personally do find your and ljokerl’s reviews most useful because they are consistent and are compared to several other headphones so that one can place the remarks in context. And I have found that ljokerl’s idea of neutral corresponds fairly closely to my idea of neutral, which is in itself pretty unusual, I’ve discovered.
Personally, when I audition, I listen to songs I know well, which usually allows me to quickly determine if I like a certain sound signature (neutrality, for the most part), bass control, bass extension, and how much detail is presented. I’m not sure how this rates in the auditory memory part, but I suspect this rates a bit differently than the 4-6 second auditory memory that was found in the studies.
I do find some reviews rather useless. There are too many that just throw in the superlatives, without comparison to other equipment, that leave me no wiser than I was before. It might also be my limited experience with different hifi gear in general, but I understand only things like detail, sound signature, control, extension. These are things that I can hear, but I have little idea what is meant with ‘a liquid sound’, or ‘lush mids’, or similar terminology.
In all, I like this site a lot, because it has a lot of really useful information. Indeed enough information that I can glean context simply by reading more reviews due to the number of comparisons you guys put in the reviews.
P.S. I’m seeing a lot of Â characters in the text of this article (in multiple browsers and on multiple PCs); seems something is a bit off on the formatting.
P.P.S. I would love to read about the product burn-in, controversial though that might be. I’ve noticed it with some in-ears and even speakers that they would sound like what I interpret as an occasionally sticking driver. Yet other speakers and in-ears do not have this and it seems to pass generally within a few hours. I would be interested to know what could actually physically be causing this initial run-in.
Thanks for reading and the comment!
Listening to a segment on a loop is a good way to go, and that is another way to get it done. I am quite familiar with the tracks I use as I have focused on the specific parts of the songs for years, and I chose to use the beginning of tracks so it is easy to use any source with a simple restart of the track. The important thing is figuring out what works best for you.
As far as objectivity goes, we both focus on the technical aspects, irrespective of the sound signature. Some sound signatures have various strengths vs. others, so my rating system attempts to balance it out. I have seen many subjective reviews, and many without much discussion about the sound other than something like “it’s amazing,” and don’t think they have anything to offer the reader, just the particular manufacturer.
Absolute gem of an article Joe!
Quick A/Bing hasn’t worked very well for me. I inherently second guess my impressions. Quick A/Bing only gives me a broad “I-like-this-one-better” and not the specifics. Also, I never get a good seal in all the hurry, and then it’s too late.
I listen to a segment on a loop (10-20 times) until it’s burnt into my memory and then switch with a pen and pad at the ready. The familiarity with the previous sound sig(for the specific segment) allows me to pick out nuances in the new sound-sig.
But, the most difficult thing…by far… is to keep the review as objective as possible.
This is one of the main reasons(I feel), reviewers like ljokerl and yourself are well respected and are completely clear from the ‘opinion’ fights that are so prevalent on Head-fi. Does ljokerl rank iems above others? Yes, of course he does. But, unlike others, he does it objectively.
PS: That said, I’d like to see a completely subjective and biased review from you (and ljokerl too)! What say? 😀
Maybe on a discontinued old iem(to be safe 😉 )