How We Test True Wireless Headphones (TWS)

To deliver a fair and insightful review of true wireless headphones, we adopt a testing methodology, dubbed the Sparrow System, that blends objective testing with subjective analyses of our expert reviewers.

In total, we have 43 evaluation criteria categorized into 3 nested levels, with each level and category assigned specific weightage. This enables us to calculate two distinct scores — a General Score and an Audiophile Score.

System Update:

• 30th November 2023: Update criteria for charging case battery

Two Overall Score

Our review process culminates in two definitive scores:

• General Score: Calculated from categories that are important for general usage.
• Audiophile Score: Calculated from categories that are important for an audiophile

Score Category And Criteria

From the top, we have 3 primary category to evaluate:

1. Charging case: The case that the earbuds are stored and charged in
2. Earbuds: The true wireless earbuds itself
3. Software: The accompanying software for the TWS

Within each primary level, we have split into different sub-categories. For example, you see that “Charging case” is evaluated based on 4 sub-categories like “Battery“, “UX“, “Design“, and “Portability“.

Here are the primary categories and their relevant sub-categories:

Except for Software, each secondary category contains a list of scoring criteria. The sub-categories of Software are considered the scoring criteria.

We will explore the scoring criteria within the sub-category level later.

Score Weightage

We recognize that not all categories carry equal significance. Hence, we have assigned specific weightage to each scoring category. 

As we mentioned earlier, the two overall scores – General Usage and Audiophile score- have different weights for each scoring category.

For General Usage:

For Audiophile:

Grading Scale

We are using a 5-point grading scale for each score criteria, where a score of 5 indicating outstanding performance, and 1 signifying very poor performance.

• 5 – Outstanding
• 4 – Good
• 3 – Average
• 2 – Substandard
• 1 – Very Poor

Charging Case Battery

Our “Charging Case Battery” score will be evaluateed on 3 key aspects:

1. Battery Life
2. Connector Type
3. Wireless Charging Capability

Battery life is the most significant factor as it directly impacts user convenience. The connector type evaluates the charging case’s compatibility and convenience. Lastly, Wireless charging is included to assess modern features that enhance user experience.

Battery Life

• Weightage: 85%
• V1.1 Grading Criteria:
  • Outstanding: >= 30hrs
  • Good: 26 – 30hrs
  • Average: 21 – 25hrs
  • Substandard: 15 -20hrs
  • Very Poor: <15hrs
• V1.0 Grading Criteria:
  • Outstanding: >= 10hrs
  • Good: >6 – <10hrs
  • Average: >3 – 6hrs
  • Substandard: 2 -3hrs
  • Very Poor: <2hrs

Connector (10%)

• Weightage: 10%
• Grading Criteria:
  • Outstanding: USB A and USB C
  • Good: USB C
  • Average: Micro USB

Wireless Charging

• Weightage: 5%
• Grading Criteria:
  • Outstanding: Yes
  • Very Poor: No

Charging Case UX

Our “Charging Case UX” evaluation focuses on several critical factors:

1. Accessibility
2. Magnetic Hold
3. Light Indicator
4. Ease of Use

These elements collectively define the user experience with the charging case, reflecting its practicality and design sophistication.

Accessibility (Opening)

Accessibility examines how easily the case can be opened and the earbuds removed.

• Weightage: 25%
• Grading Criteria:
  • Outstanding (5 Points): Can open with one hand
  • Good (4 Points): Need multiple tries with one hand
  • Average (3 Points): Need to open with two hands
  • Substandard (2 Points): [Criteria not specified]
  • Very Poor (1 Point): Accessibility requires multiple tries to open with two hands

Accessibility (Taking out the Earbuds)

• Weightage: 25%
• Grading Criteria:
  • Outstanding: One hand
  • Good: Need multiple tries with one hand
  • Average: Need two hands
  • Substandard: [Criteria not specified]
  • Very Poor: Need multiple tries with two hands

Magnetic Hold

Magnetic Hold assesses the security of the earbuds within the case.

• Weightage: 15%
• Grading Criteria:
  • Outstanding: Hard shakes or never drop out
  • Good: A few regular shakes
  • Average: A few light shakes
  • Substandard: Normal shake
  • Very Poor: Light shake

Light Indicator

The Light Indicator’s presence and functionality offer essential visual feedback.

• Weightage: 15%
• Grading Criteria:
  • Outstanding: Shows both case and earbuds battery life left and shows charging indicator
  • Good: Shows estimated battery life left and shows charging indicator
  • Average: Shows charging state and shows battery
  • Substandard: Shows estimated battery life left or shows charging indicator
  • Very Poor: No indication

Ease of Use

Ease of Use considers the overall user interaction with the case, ensuring it is intuitive and user-friendly.

• Weightage: 20%
• Grading Criteria:
  • Outstanding: Idiot-proof
  • Good: 1-2 minor UX problem
  • Average: Major UX problem
  • Substandard: >2 major UX problem
  • Very Poor: Cannot function without reading the manual

Charging Case Design

The “Charging Case Design” category subjectively evaluates 2 criteria:

1. Build Quality
2. Aesthetics

Build Quality

Build Quality focuses on the durability and robustness of the charging case, ensuring it can withstand regular use.

• Weightage: 50%
• Grading Criteria:
  • Outstanding: Premium feel, tough
  • Good: Above average
  • Average: Plastic average feel
  • Substandard: Plastic average feel, substandard
  • Very Poor: Fragile, cheap feeling

Aesthetics

Aesthetics assess the visual appeal of the case, considering its style and how it complements the overall design of the earbuds.

• Weightage: 50%
• Grading Criteria:
  • Outstanding: Fresh, unique design
  • Good: Above average, visually appealing
  • Average: Normal, run-of-the-mill design
  • Substandard: Below average but still usable
  • Very Poor: Ugly, unappealing design

Charging Case Portability

This section addresses the portability of the charging case, an essential factor for users who frequently carry their True Wireless Headphones. The criteria are based on both weight and volume, ensuring a comprehensive evaluation of the case’s portability.

Weight

• Weightage: 50%
• Grading Criteria:
  • Outstanding: ≤ 40 grams
  • Good: > 40 – ≤ 70 grams
  • Average: > 70 – ≤ 100 grams
  • Substandard: > 100 – ≤ 150 grams
  • Very Poor: > 150 grams

Volume

• Weightage: 50%
• Grading Criteria:
  • Outstanding: ≤ 50 cm³
  • Good: > 50 – ≤ 75 cm³
  • Average: > 75 – ≤ 120 cm³
  • Substandard: > 120 – ≤ 150 cm³
  • Very Poor: > 150 cm³

Earbuds Battery

This section evaluates the battery performance of the earbuds, an essential aspect for users. The criteria focus on overall battery life and the efficiency of quick charging, offering a comprehensive view of the earbuds’ battery capabilities.

Battery Life

• Weightage: 60%
• Grading Criteria:
  • Outstanding: ≥ 10 hours
  • Good: > 6 – < 10 hours
  • Average: > 3 – ≤ 6 hours
  • Substandard: > 2 – ≤ 3 hours
  • Very Poor: ≤ 2 hours

Playback Time – 15 Minute Charge

• Weightage: 40%
• Grading Criteria:
  • Outstanding: > 85 – 100 minutes
  • Good: > 70 – ≤ 85 minutes
  • Average: > 55 – ≤ 70 minutes
  • Substandard: > 40 – ≤ 55 minutes
  • Very Poor: ≤ 40 minutes

Earbuds UX

The Earbuds UX considers 4 evaluation criteria:

1. Number of Controls
2. Mono Use Capability
3. Auto-Pause Feature
4. Ease of Use

This category evaluates the earbuds’ functionality, including control versatility, the ability for mono use, and automatic pausing when removed from ears. Ease of Use assesses the overall user interaction with the earbuds, ensuring a seamless and intuitive experience.

Number of Controls

• Weightage: 40%
• Grading Criteria:
  • Outstanding: ≥ 5 controls
  • Good: 4 controls
  • Average: 3 controls
  • Substandard: 2 controls
  • Very Poor: 1 control

Mono Use

• Weightage: 10%
• Grading Criteria:
  • Outstanding: Both buds can work mono
  • Good: Only one bud can work mono
  • Average: [Criteria not specified]
  • Substandard: [Criteria not specified]
  • Very Poor: No mono use

Pause When Take Out

• Weightage: 10%
• Grading Criteria:
  • Outstanding: Yes, and resumes when put back
  • Good: Yes
  • Average: Sometimes
  • Substandard: Rarely
  • Very Poor: No

Number of Actions Needed

• Weightage: 20%
• Grading Criteria:
  • Outstanding: ≤ 10 actions
  • Good: > 10 – ≤ 15 actions
  • Average: > 15 – ≤ 20 actions
  • Substandard: > 20 – ≤ 25 actions
  • Very Poor: > 25 actions

Ease of Use

• Weightage: 20%
• Grading Criteria:
  • Outstanding: Idiot-proof, seamless
  • Good: < 3 small quirks
  • Average: Need time to memorise
  • Substandard: >= 3 small quirks
  • Very Poor: Frustrating to use

Earbuds Mic

This section evaluates the microphone quality of the earbuds, which is a vital feature for calls and voice commands

Noise Pickup

• Weightage: 50%
• Grading Criteria:
  • Outstanding: No noise, sounds like speaking directly to the phone
  • Good: Soft noise, clear and no breakup
  • Average: Audible noise, muffled
  • Substandard: Loud noise, some breakup
  • Very Poor: Much too noisy, can barely hear

Receiving Voice Quality

• Weightage: 50%
• Grading Criteria:
  • Outstanding: Crystal clear, natural voice quality
  • Good: Clear with minimal distortion
  • Average: Some distortion but voice is intelligible
  • Substandard: Noticeable distortion, hard to understand
  • Very Poor: Poor quality, voice hard to recognize

Earbuds Sound

The sound of the earbuds is judged subjectively based on:

1. Noise Leakage
2. Noise Isolation
3. Sound Quality (Detail, Bass, Mids, Treble)

This category examines sound leakage, isolation, and overall sound quality across different frequency ranges. It’s vital for an immersive listening experience, whether for music, calls, or other audio content.

Noise Leakage

• Weightage: 2.5%
• Grading Criteria:
  • Outstanding: Cannot hear a thing outside
  • Good: Can hear something but cannot make out the lyrics
  • Average: Can make out some parts of the lyrics
  • Substandard: Can make out most of the lyrics
  • Very Poor: Can hear everything clearly

Noise Isolation

• Weightage: 2.5%
• Grading Criteria:
  • Outstanding: ≤ 10% to drown ambient sound
  • Good: ≤ 20% to drown ambient sound
  • Average: ≤ 30% to drown ambient sound
  • Substandard: ≤ 40% to drown ambient sound
  • Very Poor: ≤ 50% to drown ambient sound

Detail

• Weightage: 20%
• Grading Criteria:
  • Outstanding: Exceptional clarity and detail in all frequency ranges
  • Good: Good detail, slight lack in certain ranges
  • Average: Average detail, noticeable lack in some ranges
  • Substandard: Poor detail, lacks clarity
  • Very Poor: Very poor detail, muddled sound

Bass

• Weightage: 25%
• Grading Criteria:
  • Outstanding: Deep, impactful, and tight bass
  • Good: Good bass, some impact and tightness
  • Average: Average bass, lacks some depth and tightness
  • Substandard: Weak bass, lacks depth and tightness
  • Very Poor: Very weak bass, no depth or tightness

Mids

• Weightage: 25%
• Grading Criteria:
  • Outstanding: Clear and well-balanced mids
  • Good: Mostly clear mids, slightly off-balance
  • Average: Average mids, some muddiness or imbalance
  • Substandard: Muddy mids, significant imbalance
  • Very Poor: Very muddy mids, extremely imbalanced

Treble

• Weightage: 25%
• Grading Criteria:
  • Outstanding: Crisp, clear treble without being harsh
  • Good: Mostly clear treble, slightly harsh at high volumes
  • Average: Average treble, some harshness
  • Substandard: Harsh treble, uncomfortable at high volumes
  • Very Poor: Very harsh treble, consistently uncomfortable

Earbuds Connectivity

This section assesses the connectivity features of the earbuds, an essential aspect for seamless and hassle-free use.

Supported Codec

• Weightage: 25%
• Grading Criteria:
  • Outstanding: Supports advanced codecs (e.g., aptX, LDAC)
  • Good: Supports AAC
  • Average: Only supports SBC

Ease of Pairing

• Weightage: 25%
• Grading Criteria:
  • Outstanding: No configuration needed
  • Good: Stable, occasional manual reconnection required
  • Average: Slightly unstable, manual reconnection often needed
  • Substandard: Frequently unstable, frequent manual reconnections
  • Very Poor: Frequently unstable, pairing often fails

Stability

• Weightage: 25%
• Grading Criteria:
  • Outstanding: Connection breaks up 0 – 3 times per hour
  • Good: Connection breaks up 3 – 5 times per hour
  • Average: Connection breaks up 5 – 8 times per hour
  • Substandard: Connection breaks up 8 – 12 times per hour
  • Very Poor: Connection breaks up more than 12 times per hour

Multi-point Pairing

• Weightage: 10%
• Grading Criteria:
  • Outstanding: Yes, seamless switching
  • Good: Yes, but requires manual switching
  • Average: Limited support
  • Very Poor: No multi-point pairing

Range

• Weightage: 15%
• Grading Criteria:
  • Outstanding: > 33 feet
  • Good: ≥ 10 – ≤ 33 feet
  • Average: < 10 feet

Earbuds Waterproof

This section evaluates the waterproof capabilities of the earbuds, which is a crucial feature for users who use them in various environments, including those where they might be exposed to moisture or water. The grading is solely based on the IPX rating of the earbuds, providing a clear and standardized measure of their water resistance.

IPX Rating

• Weightage: 100%
• Grading Criteria:
  • Outstanding: IPX6 rating or higher
  • Good: IPX4 or IPX5 rating
  • Average: IPX3 rating
  • Substandard: IPX2 rating
  • Very Poor: IPX1 rating or not rated

Earbuds Design

This section thoroughly evaluates the design aspects of the earbuds. These criteria are critical for ensuring the earbuds perform well and offer a satisfactory physical experience and appearance to the user.

Comfort

• Weightage: 40%
• Grading Criteria:
  • Outstanding: Can wear more than 4 hours comfortably
  • Good: Comfortable for up to 3-4 hours
  • Average: Comfortable for up to 2 hours
  • Substandard: Comfortable for less than 1 hour
  • Very Poor: Uncomfortable to wear even for short periods

Fit

• Weightage: 25%
• Grading Criteria:
  • Outstanding: Secure and tight fit during rigorous activities
  • Good: Snug fit for normal daily activities and light exercise
  • Average: Fits well indoors but may slip during movement
  • Substandard: Slips easily during indoor activities or light movements
  • Very Poor: Does not stay in place even during minimal activity

Build Quality

• Weightage: 20%
• Grading Criteria:
  • Outstanding: Premium feel, highly durable
  • Good: Above average quality, durable for everyday use
  • Average: Average quality, may show wear over time
  • Substandard: Below average quality, prone to damage
  • Very Poor: Fragile, poor quality materials

Aesthetic

• Weightage: 15%
• Grading Criteria:
  • Outstanding: Fresh, unique, and appealing design
  • Good: Attractive and contemporary design
  • Average: Standard design, nothing standout
  • Substandard: Outdated or unappealing design
  • Very Poor: Ugly or very unattractive design

Software Support

The “Software Support” category evaluates 5 evaluation criteria:

1. EQ Customization
2. Ambient Level Adjustment
3. Battery Level Display
4. Find My Earbuds Feature
5. Voice Assistant Integration
6. Configurability

This category examines the software features that enhance the user experience, including audio customization, ambient sound control, practical utilities like battery level display and earbud location, and the integration and configurability of voice assistants.

If there’s no software support, the score will be 0.

Suggestions and Feedbacks

If you spot any errors with our testing methodology or have any good feedback on how we can further improve the test, please feel free to reach out to us.

Other Testing Methodologies

• Falcon System: How We Test Wireless Headphones
• Medusa System: How We Test Wired Headphones and IEMs
• Oracle Scoring: How We Test General Audio Gear