Equestic validation

Objective-Oriented Design

The study was structured to validate the precision of the Equestic SaddleClip in capturing and analyzing key metrics such as gait recognition, rhythm distribution, impulsion levels, and symmetry in trot.

A cross-validation design was employed, using both the data from the SaddleClip and high-speed video recordings.

Data Collection Methods

Device and Tools

• Equestic SaddleClip: Equipped with an AEMTS Sensor Tag, capturing data at a 50Hz sampling rate.
• High-Speed Video Cameras: Used in parallel to record the training sessions for manual annotation and cross-validation.
  Fixed-position video Cameras, with markers on the horse, were used to measure vertical movement.
• Pressure plate measurements provided a relative indication of the force, helping to further validate the acceleration-based impulsion analysis.
• Equestic Users Feedback Database: The accuracy of the Equestic SaddleClip’s Symmetry analysis has been validated through real-world user feedback and interaction over a period of three years (2020-2023).

Training Session Protocol

A total of 74 training sessions each by 3 minutes at least were conducted, involving a variety of movements like halt, walk, trot, canter, and jumps where applicable.

Study Participants

Selection Criteria for Horses

The study included a diverse range of horses, varying in size (from 120cm to 186cm in height), training levels (from level 1 to Grand Prix), and breeds (including Warmbloods, Hanoverian, Andalusian, Friesians, and Welsh Pony).

Gaited horses were intentionally excluded to standardize the analysis across common gaits.

Selection Criteria for Equestic Users Feedback

Over the three years from 2020 till 2023, the Equestic has issued notifications to users regarding changes in symmetry.

All user feedback received during this time were collected and classified by language and subject. All 230 independent feedback forms related to symmetry notifications in English were selected for evaluation.

All feedback forms related to technical issues or clarification requests were excluded from the evaluation.

Asymmetry warnings, alerting users about changes exceeding 8% in symmetry metrics, indicating significant deviations were included in the study.

Statistical Analysis Methods

Cross-Validation with Video Analysis

Elan Tool. Used for manual annotation of high-speed videos to create a reliable dataset against which the SaddleClip data was compared.

Comparative Analysis. Involved aligning timestamps from the SaddleClip data with manually described events from video footage.

Cross-Validation with Users Feedback

Sentiment Analysis. Statistical benchmarking of user feedback on Equestic Symmetry Warnings, categorized by “Positive”, “Validated”, or “Not confirmed”.

Accuracy Assessment

The main statistical method was comparative accuracy assessment, determining the congruence between the SaddleClip data, the manual annotations of the video and user’s confirmations.

The percentage of total user feedback that is Positive and Validated is used to determine the accuracy and establish the reliability of Equestic symmetry notifications under various equestrian conditions.

This methodology ensures robust and thorough validation of the Equestic SaddleClip, employing a mix of technological and traditional analysis methods to comprehensively assess its performance and accuracy in equestrian training contexts.

Validation Process

The validation of the Equestic SaddleClip involved a detailed, step-by-step process focusing on four key metrics: gait recognition, rhythm, impulsion and symmetry.

This process combined empirical data collection with sophisticated analysis and cross-validation by experts and factual user feedback to ensure the accuracy and reliability of the SaddleClip in a range of equestrian settings.

Gaits recognition

Steps and Tests

1. Data Collection: During the 74 training sessions, the SaddleClip captured gait data alongside high-speed video recordings.
   
2. Algorithm Application: The SaddleClip’s algorithms analyzed the data to classify the gaits (walk, trot, canter).
3. Cross-Validation: The algorithmic gait classification was cross-referenced with manual annotations from the high-speed video analysis using the Elan tool.

Accuracy Criteria

The congruence between the SaddleClip’s gait classifications and the expert manual annotations was the primary criterion. A high level of agreement indicated accurate gait recognition.

Rhythm

Steps and Tests

1. Rhythm Data Collection: Rhythm data for walk, trot, and canter were recorded by SaddleClip.
2. Algorithm Application: The SaddleClip’s algorithms were used to calculate the rhythm in walk, trot and canter.
3. Video Comparison: The rhythm data from the SaddleClip were compared to the rhythm patterns observed by experts in the high-speed video recordings.

Accuracy Criteria

Accuracy was assessed based on how closely the SaddleClip’s rhythm measurements matched the observed rhythm patterns in the video analysis. A high level of agreement indicated accurate Rhythm recognition.

Impulsion

Steps and Tests

1. Impulsion Data Recording: The SaddleClip measured the acceleration data during each training session.
2. Cross-Validation with Video and Pressure Plates: The impulsion data were cross-validated with vertical movement measurements from fixed-position video recordings and relative force indications from pressure plates.
3. Comparative Analysis: The impulsion measurements from the SaddleClip were compared against these external validation sources to assess accuracy.

Accuracy Criteria

The high degree of correlation between the SaddleClip’s impulsion data and the external measurements determined the accuracy of impulsion analysis.

Symmetry

Steps and Tests

1. Initial Data Collection: During riding sessions, the Equestic SaddleClip collected detailed data on the trot gait, focusing on step duration, push-off, and landing forces for each diagonal pair of legs.
2. Algorithmic Analysis: The Equestic Intelligence Platform processed this data to calculate the symmetry metrics, comparing the aforementioned parameters between the left and right diagonals.
3. User Feedback semantic classification and statistical  Analysis: The Equestic Intelligence Platform sent out notifications where high asymmetry changes were identified. User responses were categorized as ‘Positive’, Validated, and ‘Not confirmed’ to reflect the three types of possible feedback respectively: general agreement, confirmation of the notification or disagreement with the notification.

• Positive: includes general positive feedback confirming the Equestic notification.
  
• Validated: includes clear confirmation statement from user confirming the Equestic observation also validated by professional health expert (vet, therapist, farrier).
  
• Not confirmed: includes user statement about the absence of warning from Equestic or contradicting Equestic notification.

Statistical Analysis of the types and frequencies of user responses to symmetry notifications, proving accuracy and insight into the practical effectiveness of the SaddleClip’s symmetry analysis.

Accuracy Criteria

The rate of user confirmation, especially in cases of high asymmetry warnings, served as a practical measure of the algorithm’s effectiveness.

A significant percentage of user “Positive” and “Validated” confirmations would reinforce the accuracy of the SaddleClip’s symmetry measurements.

Validation Results

The validation study of the Equestic SaddleClip yielded substantial data, supporting the device’s high accuracy in capturing and analyzing equestrian training data. The key findings from this comprehensive research are detailed below.

Gait Recognition Accuracy

Findings: The SaddleClip demonstrated a high level of precision in identifying and categorizing different horse gaits (walk, trot, canter).

Data: Cross-validation with manual annotations from high-speed video analysis showed a congruence rate of approximately 99% in gait recognition (p ≤ 0.05).

Statistical Analysis: The agreement between the SaddleClip data and expert video annotations was statistically significant, underscoring the reliability of the device in gait analysis.

Rhythm and Impulsion Analysis

Rhythm Analysis Findings: The SaddleClip accurately analyzed the rhythm within each gait, aligning closely with the patterns observed in the video analysis (p ≤ 0.05).

Impulsion Analysis Findings: The device effectively used acceleration data as a proxy for impulsion, with cross-validation through video and pressure plate data indicating high accuracy.

Symmetry in Trot

Findings: The symmetry analysis for trot gait showed that the SaddleClip could effectively measure and compare step duration, push-off, and landing forces between diagonals.

User Feedback Correlation: Over three years, user responses to symmetry notifications, including a high rate of confirmation for asymmetry warnings, supported the accuracy of the SaddleClip’s symmetry measurements.

Statistical Confirmation: 97% of users confirmed asymmetry notifications from the Equestic’s assessments, while only 3% did not confirm.

Limitations of the Study

While the validation study of the Equestic SaddleClip provides significant insights, it’s important to acknowledge certain limitations that could impact the generalizability and applicability of the findings.

Specificity in Horse Size and Type

Size Limitation: The study did not include horses smaller than 120 cm. Consequently, the accuracy and applicability of the SaddleClip data for ponies or miniature breeds remain unverified.

Exclusion of Gaited Horses: Gaited horses, known for their unique movement patterns, were not part of the study. Therefore, the results may not be applicable to these breeds, and the device’s performance in analyzing their specialized gaits is not established.

Misinterpretation of Complex Movements

Misclassification of Advanced Movements: According to the user manual, certain complex movements like piaffe or flying changes might be incorrectly identified as jumps by the SaddleClip.

This limitation highlights a potential area for algorithm refinement, especially in advanced dressage training contexts.

Additional Considerations

Environmental and Equipment Variables
The study did not extensively account for variations in external factors like different saddle types, rider influence, or environmental conditions, which could affect the data accuracy.