Ever since markerless motion capture systems became widely available in biomechanics research there have been comparison studies between markerless and marker-based systems. The game-changing potential of markerless systems is a clear motivation for these studies, but how should we interpret these results? Even though some consider marker-based tracking to be a gold standard, it comes with inter- and intra-tester/session reliability issues. The root causes of these issues include the challenge of consistent marker placement, soft tissue artifact, and the “lab effect.”
Given that motion capture lacks a true gold standard, how should we assess new technologies? Previous works have focused on measurement reliability as a proxy for validity, reasoning that a valid measurement system should reliably produce the same measurements for repeated measures across contexts. Das, de Paula Oliveira, and Newell take a similar approach using functional limits of agreement to get at the heart of the question
“How can we effectively evaluate new motion capture technologies and decide that we should prefer it?”
A functional approach to agreement
This paper in particular considered marker-based and markerless tracking. The marker-based tracking used 8 infra-red cameras through Qualisys QTM while the markerless tracking used an 8-camera DARI system. The authors used “functional limits of agreement” (fLoA) to estimate the interval within which a proportion of differences between measures lie. You can think of fLoAs as generalizing Bland-Altman plots from 0D measures to 1D curves. Both methods assess the agreement between measures across a range of possible values. And both methods help us understand whether the systems are broadly measuring the same thing.
A Linear Mixed-Effects Model (LMM), capturing linear relationships where observations are grouped categorically, underlies this technique. These relationships are the result of fixed effects (at the population level), random effects (at the individual level), and, of course, error. This experiment had many independent and confounding variables that could cause differences in measurements between motion capture systems. The authors accounted for these variables by using LMMs across 9 different scenarios representing different joints and components.
| (Joint x Component) | Flexion | Abduction | Rotation |
| Spine | Scenario 1 | Scenario 2 | Scenario 3 |
| Right Hip | Scenario 4 | Scenario 5 | Scenario 6 |
| Right Knee | Scenario 7 | Scenario 8 | Scenario 9 |
The authors assessed a 3-level hierarchy of effects within each scenario accounting for: participant, session, and trial. Their goal with this design was to isolate only those measurement differences attributable to the different motion capture systems.
The Marker-based versus Markerless Comparison
The authors concluded from their comparative analysis that the marker-based system was more reliable, with different levels of agreement according to the scenario. We advocate caution, though, as the study’s experimental design favoured the marker-based system:
- The experiment considered data collected by a single lab rather than testing agreement across labs.
- A single technician placed all the markers, avoiding the inter-tester reality of scaling motion capture collection.
- The data set included only 9 subjects, instead of the 100s that can be captured using markerless technologies.
An additional consideration to keep in mind is that this study considered a single markerless system. Different markerless systems will likely show different characteristics and performance levels, meaning that these results are unlikely to generalize directly to other available systems.
We believe that researchers could adopt this framework beyond comparison studies. For example, it could be used to assess changes in pose estimations produced by successive versions of markerless systems.
Whole-curve assessment for motion capture
Stepping back, we wholeheartedly agree with Das, de Oilveira, and Newell about the functional approach in biomechanical analysis. Researchers collect kinematic and kinetic waveforms; these are time-varying signals and we should assess them as such. In fact, given that these waveforms are measured in 3-dimensional space it is already a simplification to analyse 1D signals that are sensitive to our choices of coordinate system.
The fLoA method used in this paper generalizes Bland-Altman plots from 0D measures to 1D curves. This is similar to how 1D Statistical Parametric Mapping (SPM) generalizes from statistical tests for 0D measures to statistical tests for 1D curves. The authors also commented that researchers can use Dynamic Time Warping to compare time series data, noting that the output is not easy to interpret. We see similar comments about Principal Component Analysis compared to SPM in characterizing groups of traces as well. In this we agree with the authors: interpretable analysis techniques are key to further our understanding of human movement.