(en)The present disclosure relates to methods and systems of determining swimming metrics of a user during a swimming session. The method can include receiving, by a processor circuit of a user device, motion information from one or more motion sensors of the user device; determining, by the processor circuit using the motion information, a first set of rotational data of the user device, wherein the first set of rotational data is expressed in a first frame of reference; converting, by the processor circuit, the first set of rotational data into a second set of rotational data, wherein the second set of rotational data is expressed in a second frame of reference; determining, by the processor circuit, one or more swimming metrics of the user; and outputting the one or more swimming metrics.

1.ApplicationNumber: US-201715691245-A
1.PublishNumber: US-2018056128-A1
2.Date Publish: 20180301
3.Inventor: NARASIMHA RAO BHARATH
MERMEL CRAIG H.
RAGHURAM KARTHIK JAYARAMAN
PHAM HUNG A.
HOWELL ADAM S.
HINDIYEH Rami Y.
OCHS JAMES P.
MAJJIGI VINAY R.
SINGH ALVARADO ALEXANDER
CHOW SUNNY K.
SRINIVAS UMAMAHESH
TAN XING
HUANG RONALD K.
ARNOLD EDITH MERLE
GUERS Robin T.
DERVISOGLU GUNES
ULLAL ADEETI

4.Inventor Harmonized: NARASIMHA RAO BHARATH(US)
MERMEL CRAIG H(US)
RAGHURAM KARTHIK JAYARAMAN(US)
PHAM HUNG A(US)
HOWELL ADAM S(US)
HINDIYEH RAMI Y(US)
OCHS JAMES P(US)
MAJJIGI VINAY R(US)
SINGH ALVARADO ALEXANDER(US)
CHOW SUNNY K(US)
SRINIVAS UMAMAHESH(US)
TAN XING(US)
HUANG RONALD K(US)
ARNOLD EDITH MERLE(US)
GUERS ROBIN T(US)
DERVISOGLU GUNES(US)
ULLAL ADEETI(US)

5.Country: US
6.Claims:
(en)The present disclosure relates to methods and systems of determining swimming metrics of a user during a swimming session. The method can include receiving, by a processor circuit of a user device, motion information from one or more motion sensors of the user device; determining, by the processor circuit using the motion information, a first set of rotational data of the user device, wherein the first set of rotational data is expressed in a first frame of reference; converting, by the processor circuit, the first set of rotational data into a second set of rotational data, wherein the second set of rotational data is expressed in a second frame of reference; determining, by the processor circuit, one or more swimming metrics of the user; and outputting the one or more swimming metrics.

7.Description:
(en)PRIORITY CLAIM
This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62/381,640, titled “Systems and Methods for Detecting Turns,” which is filed on Aug. 31, 2016 and is incorporated by reference herein in its entirety.
This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62/381,641, titled “Systems and Methods for Detecting Breaths While Swimming,” which is filed on Aug. 31, 2016 and is incorporated by reference herein in its entirety.
This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62/381,988, titled “Systems and Methods for Counting Laps,” which is filed on Aug. 31, 2016 and is incorporated by reference herein in its entirety.
This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62/381,989, titled “Systems and Methods for Determining Swimming Pool Length,” which is filed on Aug. 31, 2016 and is incorporated by reference herein in its entirety.
This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62/381,846, titled “Systems and Methods of Counting Swim Strokes,” which is filed on Aug. 31, 2016 and is incorporated by reference herein in its entirety.
This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62/381,843, titled “Systems and Methods for Detecting Swim Activity Using Inertial Sensors”, which is filed on Aug. 31, 2016 and is incorporated by reference herein in its entirety.


CROSS-REFERENCE TO RELATED APPLICATIONS
This application relates to U.S. patent application Ser. No. ______ (Attorney Docket No.: 337722-630001), titled “Systems and Methods of Swimming Analysis,” which is filed on ______ and is incorporated by reference herein in its entirety.
This application relates to U.S. patent application Ser. No. ______ (Attorney Docket No.: 337722-636001), titled “Systems and Methods of Swimming calorimetry,” which is filed on ______ and is incorporated by reference herein in its entirety.
FIELD
The present disclosure relates generally to detecting swim activity using inertial sensors.
BACKGROUND
When a user is doing exercise or making movements, it is oftentimes useful to detect when the user makes a change in direction. Keeping track of the user's turns or changes in direction can be useful in many applications. As an example, when the user is swimming, detecting a turn made by the user may imply the user completes a lap. As another example, when the user is walking and/or running, knowing the user makes a turn or changes in direction can be useful in tracking the user's location. It is sometimes, however, not easy or practical for the user to keep track of the changes in direction made by him or her. For example, when the user is swimming, he or she may not want to mentally keep track of the number of turns made by him or her. Accordingly, it is desirable to provide methods and systems of detecting turns while swimming.
When a user is swimming, there is often a need to detect when and how frequently the user is taking a breath. For example, this information can be used to detect exertion, level of effort, general fitness, and/or swimming ability. It is generally, however, not practical for the user to keep track of the breaths taken by him or her. Accordingly, it is desirable to provide methods and systems of detecting user's breaths while swimming.
When a user is doing activities that includes multiple types of motions, there is often a need to classify the types of motions. As an example, when a user is swimming laps, the user can switch between two types of motions: swimming and turning. As another example, when a user is running, the user can switch between running and walking. Knowing which type of motions a user is doing is useful in many applications including estimating energy expenditure of the user. Accordingly, it is desirable to provide methods and systems of determining a user's types of motions, including how many strokes the user has taken while swimming.
When a user is swimming, there is often a need to determine the number of laps a user swims during a swimming session. Keeping track of the number of laps the user swims can be useful in many applications, such as to calculate the total distance a user swims and/or the energy expenditure associated with a swimming session. Accordingly, it is desirable to provide methods and systems of determining the number of laps a user swims during a swimming session.
When a user is performing a swimming session, the user may transition from periods of swimming to periods of rest. While it is reasonable to expect that a user tracking his/her swim metrics (e.g., lap count, lap speed, strokes per lap, time splits, distance, calories, etc.) via a wearable device will indicate the start and end of the workout through interaction with the device, it is not always practical to do so. In a typical swim workout, periods of continuous swimming are interspersed with varying durations of rest. Accordingly, it is desirable to detect periods of lap swimming for the purpose of accurate swim metric evaluation.
When a user is swimming in a pool, there is often a need to know the length of the swimming pool. Information of the length of a swimming pool can be used to calculate the total distance a user swims and the energy expenditure associated with a swimming session. The pool length information, however, is not always readily available to users. Additionally, users may not be able to accurately estimate the pool length. Accordingly, it is desirable to provide methods and systems of determining a length of a swimming pool.
SUMMARY
The present disclosure relates to a method for improving an accuracy of a wearable device while determining swimming metrics of a user during a swimming session. In some embodiments, the method can include: receiving, by a processor circuit of the wearable device, motion data from one or more motion sensors of the wearable device; determining, by the processor circuit using the motion data, a first set of rotational data of the wearable device, wherein the first set of rotational data is expressed in a first frame of reference; converting, by the processor circuit, the first set of rotational data into a second set of rotational data, wherein the second set of rotational data is expressed in a second frame of reference; determining, by the processor circuit, one or more swimming metrics of the user based on the second set of rotational data, wherein the one or more swimming metrics comprise at least one of turns, breaths, laps, swimming styles, and swimming strokes; and outputting, by the processor circuit, the one or more swimming metrics of the user.
In some embodiments, the first frame of reference can include a body-fixed frame of reference with respect to the wearable device. In some embodiments, the second frame of reference can include an inertial frame of reference.
In some embodiments, the method can include: determining, by the processor circuit, yaw rotational data from the second set of rotational data; determining, by the processor circuit, one or more turns of the user based on the yaw rotational data. In some embodiments, the method can include: determining unfiltered yaw rotational data, wherein the unfiltered yaw rotational data are part of the second set of rotational data; and filtering the unfiltered yaw rotational data. In some embodiments, the method can include: determining a time constant proportional to a period which the user needs to complete a stroke; and filtering the unfiltered yaw rotational data based on the time constant.
In some embodiments, the method can include: determining a pitch angle from the second set of rotational data; comparing the pitch angle with a threshold angle; and determining one or more breaths of the user based upon comparing the pitch angle with the threshold angle. In some embodiments, the threshold angle can be associated with a swimming style of the user, wherein the swimming style is at least one of freestyle, butterfly, or breast stroke. In some embodiments, the threshold can be associated with a swimming skill level of the user.
In some embodiments, the method can include: converting the second set of rotational data to a set of two-dimensional rotational data; adding one or more constraints to the set of two-dimensional rotational data; and counting, by the processor circuit, one or more swimming strokes from the constrained two-dimensional rotational data. In some embodiments, the method can include: determining a primary axis of rotation based on the second set of rotational data; projecting the second set of three-dimensional rotational data to a two-dimensional space based on the primary axis of rotation; and determining the set of two-dimensional rotational data based on the projection. In some embodiments, the one or more constraints comprises at least one of accelerometer energy, moment arm calculations, or rotational direction. In some embodiments, the method can include counting revolutions of circles in the constrained two-dimensional rotational data. In some embodiments, the method can include counting revolutions of semi-circles in the constrained two-dimensional rotational data.
In some embodiments, the method can include: detecting a number of turns and a number of strokes of the user during the swimming session based on the received motion data; determining a stroke range per lap based on the number of turns and the number of strokes; determining whether a turn is not detected; inserting a missing turn in response to a determination that a turn is not detected; determining a variance of strokes per lap; adjusting the detected number of turns to reduce the variance of strokes per lap; and determining a lap count of the user based on the adjusted number of turns.
In some embodiment, the method can include: determining whether the number of strokes of the user converges in the swimming session; or determining whether the number of strokes made by the user converges in a historical swimming session. In some embodiments, the method can include: determining a standard deviation of the number of strokes among consecutive turns; and comparing the standard deviation with a threshold. In some embodiments, threshold can be 1.5 strokes.
In some embodiments, the method can include: comparing the number of strokes between two consecutive turns with the determined stroke range per lap. In some embodiments, the method can include: comparing the number of strokes between two consecutive turns with a threshold, wherein the threshold is determined by multiplying a mean value of the number of strokes per lap with a ratio.
In some embodiments, the method can include: determining a stroke rate of the user; classifying a stroke style for the user based on the motion data; determining a confidence value based on the stroke rate and the stroke style; determining a motion signature of the user, wherein the motion signature is swimming; and determining the user is swimming based on the confidence value and the motion signature. In some embodiments, the stroke style comprises at least one of freestyle, backstroke, breaststroke, or butterfly.
In some embodiments, the method can include: receiving an input from the user whether to calibrate a length of the swimming pool; if the input indicates that the user selects to calibrate the length of the swimming pool: prompting the user to perform an activity along an edge of the swimming pool, wherein the edge is parallel with a direction the user swims, receiving distance data associated with the activity, calculating the length of the swimming pool based on the distance data, and determining the one or more swimming metrics based on the calculated length of the swimming pool; and if the input indicates that the user does not select to calibrate the length of the swimming pool: counting a number of swimming strokes per lap, and calculating the length of the swimming pool based on the number of strokes per lap and a default stroke length, and determining the one or more swimming metrics based on the calculated length of the swimming pool.
In some embodiments, the method can include: receiving, by the processor circuit, a number of steps associated with the activity from a pedometer of the wearable device, wherein the activity comprises at least one of walking or running. In some embodiments, the method can include: receiving, by the processor circuit, location data from a GPS sensor of the wearable device.
The present disclosure also relates to a system for improving an accuracy of a wearable device while determining one or more swimming metrics of a user during a swimming session. In some embodiments, the system can include: one or more motion sensors configured to collect motion data; and a processor circuit coupled to the one or more motion sensors and configured to execute instructions causing the processor to: determine a first set of rotational data, wherein the first set of rotational data is expressed in a first frame of reference; convert the first set of rotational data into a second set of rotational data, wherein the second set of rotational data is expressed in a second frame of reference; determine one or more swimming metrics of the user based on the second set of rotational data; and output the one or more swimming metrics of the user.
Other features and advantages will become apparent from the following detailed description and drawings.



BRIEF DESCRIPTION OF THE DRAWINGS
Various objects, features, and advantages of the present disclosure can be more fully appreciated with reference to the following detailed description of the present disclosure when considered in connection with the following drawings, in which like reference numerals identify like elements.
FIG. 1 illustrates a wearable device (or a “user device”) according to some embodiments of the present disclosure.
FIG. 2 illustrates a block diagram of a wearable device according to some embodiments of the present disclosure.
FIG. 3 illustrates a companion device according to some embodiments of the present disclosure.
FIGS. 4A-4D illustrate examples of a body-fixed frame of reference according to some embodiments of the present disclosure.
FIG. 5 illustrates a set of rotational data of a wearable device in a body-fixed frame of reference according to some embodiments of the present disclosure.
FIG. 6 illustrates an example of an inertial frame of reference according to some embodiments of the present disclosure.
FIGS. 7A-7D illustrate examples of an inertial frame of reference according to some embodiments of the present disclosure.
FIG. 8 illustrates a set of rotational data of a wearable device in an inertial frame of reference according to some embodiments of the present disclosure.
FIG. 9 illustrates another set of rotational data of a wearable device in an inertial frame of reference according to some embodiments of the present disclosure.
FIG. 10 illustrates a method of determining a direction of gravity according to some embodiments of the present disclosure.
FIG. 11 illustrates a method of determining a direction of gravity according to some embodiments of the present disclosure.
FIG. 12 illustrates a method of determining one or more turns made by a user during a movement according to some embodiments of the present disclosure.
FIG. 13 illustrates filtered yaw data of a wearable device according to some embodiments of the present disclosure.
FIG. 14 illustrates a method of determining one or more breaths taken by a user while swimming according to some embodiments of the present disclosure.
FIG. 15 illustrates wrist angle of a wearable device according to some embodiments of the present disclosure.
FIG. 16 illustrates a flow chart of a method for counting swim strokes, according to some embodiments of the present disclosure.
FIG. 17 is a series of graphical representations of the data collected with respect to systems and methods described herein.
FIG. 18 depicts a sample constraint, according to some embodiments of the present disclosure.
FIG. 19 depicts sample stroke counts, according to some embodiments of the present disclosure.
FIG. 20 illustrates a case study comparing the number of turns detected to a truth lap count.
FIG. 21 illustrates a probability distribution function of the number of strokes per lap according to a case study.
FIG. 22 illustrates a probability distribution function of the number of strokes per lap according to a case study.
FIG. 23 illustrates a flow chart illustrating a computerized process of determining a number of laps a user swims during a swimming session according to some embodiments of the present disclosure.
FIG. 24 illustrates a flow chart illustrating a computerized process of determining a stroke range per lap according to some embodiments of the present disclosure.
FIG. 25 illustrates a flow chart illustrating a computerized process of determining lap count of the user accordingly to some embodiments of the present disclosure.
FIG. 26 illustrates a probability distribution function of lap count error using techniques according to some embodiments of the present disclosure.
FIG. 27 illustrates lap count error using techniques according to some embodiments of the present disclosure.
FIG. 28 illustrates exemplary components for detecting swim activity, according to some embodiments of the present disclosure.
FIG. 29 is a flowchart of a method for detecting swim activity, according to some embodiments of the present disclosure.
FIG. 30 is a graph illustrating exemplary discriminative information to detect swimming, according to some embodiments of the present disclosure.
FIG. 31 illustrates an example of users' estimation of the length of a swimming pool according to some embodiments of the present disclosure.
FIGS. 32A and 32B illustrate a method of determining a length of a swimming pool according to some embodiments of the present disclosure.
FIG. 33 illustrates a method of determining a length of a swimming pool according to some embodiments of the present disclosure.



DETAILED DESCRIPTION
The present disclosure relates to a method and system for detecting swim activity based on motion sensor signals obtained from a wearable device. Generally, user arm movement when swimming has distinct periodic signatures, unlike periods of rest which are typified by random user behavior.
The wearable device can include one or more motion sensors to collect data about the wearable device's position and orientation in space and to track changes to the wearable device's position and orientation over time.
FIG. 1 shows an example of a wearable device (or a “user device”) 100 according to some embodiments of the present disclosure. In some embodiments, wearable device 100 may be any suitable wearable device, such as a watch and/or a fitness band configured to be worn around an individual's wrist.
FIG. 2 depicts a block diagram of exemplary components that may be found within wearable device 100 according to some embodiments of the present disclosure. In some embodiments, wearable device 100 can include a processor 210 , a memory 220 , one or more motion sensors 240 , a display 270 , an interface 280 , and a heart rate sensor 290 . Wearable device 100 may include additional modules, fewer modules, or any other suitable combination of modules that perform any suitable operation or combination of operations.
In some embodiments, processor 210 can include one or more cores and can accommodate one or more threads to run various applications and modules. Software can run on processor 210 capable of executing computer instructions or computer code. Processor 210 can also be implemented in hardware using an application specific integrated circuit (ASIC), programmable logic array (PLA), field programmable gate array (FPGA), or any other integrated circuit.
Memory 220 can be a non-transitory computer readable medium, flash memory, a magnetic disk drive, an optical drive, a programmable read-only memory (PROM), a read-only memory (ROM), or any other memory or combination of memories. Memory 220 can include one or more modules 230 .
Processor 210 can be configured to run module 230 stored in memory 220 that is configured to cause processor 210 to perform various steps that are discussed throughout the present disclosure, such as, for example, the methods described in connection with FIG. 5 . In some embodiments, wearable device 100 can include one or more motion sensors 240 . For example, motion sensors 240 can include a gyroscope 250 and an accelerometer 260 . In some embodiments, accelerometer 260 may be a three-axes accelerometer that measures linear acceleration in up to three-dimensions (for example, x-axis, y-axis, and z-axis). In some embodiments, gyroscope 250 may be a three-axes gyroscope that measures rotational data, such as rotational movement and/or angular velocity, in up to three-dimension (for example, yaw, pitch, and roll). In some embodiments, accelerometer 260 may be a microelectromechanical system (MEMS) accelerometer, and gyroscope 250 may be an MEMS gyroscope. Processor 210 of wearable device 100 may receive motion information from one or more motion sensors 240 to track acceleration, rotation, position, orientation or gravity information of wearable device 100 in six degrees of freedom through three-dimensional space.
The motion information received from one or more motion sensors 240 may be expressed in a body-fixed frame of reference with respect to wearable device 100 . In some embodiments, the motion information can be converted from the body fixed frame of reference to the inertial frame of reference. Conversion of sensor data to the inertial frame of reference is a necessary process prior to stroke detection, as well as stroke counting, turn detection, stroke phase classification and in some aspects of stroke classification, as described in the respective applications referenced above, and incorporated by reference herein in their entirety.
In some embodiments, wearable device 100 may include other types of sensors in addition to accelerometer 260 and gyroscope 250 . For example, wearable device 100 may include an altimeter or barometer, or other types of location sensors, such as a GPS sensor. Wearable device 100 may also include display 270 . Display 270 may be a screen, such as a crystalline (e.g., sapphire) or glass touchscreen, configured to provide output to the user as well as receive input from the user via touch. For example, display 270 may be configured to display a current heart rate or daily average energy expenditure. Display 270 may receive input from the user to select, for example, which information should be displayed, or whether the user is beginning a physical activity (e.g., starting a session) or ending a physical activity (e.g., ending a session), such as a swimming session, a running session, a weight lifting session, a walking session or a cycling session. In some embodiments, wearable device 100 may present output to the user in other ways, such as by producing sound with a speaker (not shown), and wearable device 100 may receive input from the user in other ways, such as by receiving voice commands via a microphone (not shown).
In some embodiments, wearable device 100 may communicate with external devices via interface 280 , including a configuration to present output to a user or receive input from a user. Interface 280 may be a wireless interface. The wireless interface may be a standard Bluetooth (IEEE 802.15) interface, such as Bluetooth v4.0, also known as “Bluetooth low energy.” In other embodiments, the interface may operate according to a cellphone network protocol such as Long Term Evolution (LTE) or a Wi-Fi (IEEE 802.11) protocol. In other embodiments, interface 280 may include wired interfaces, such as a headphone jack or bus connector (e.g., Lightning, Thunderbolt, USB, etc.).
Wearable device 100 can measure an individual's current heart rate from heart rate sensor 290 . Heart rate sensor 290 may also be configured to determine a confidence level indicating a relative likelihood of an accuracy of a given heart rate measurement. In other embodiments, a traditional heart rate monitor may be used and may communicate with wearable device 100 through a near field communication method (e.g., Bluetooth).
Wearable device 100 may be configured to communicate with a companion device 300 ( FIG. 3 ), such as a smartphone, as described in more detail herein. In some embodiments, wearable device 100 may be configured to communicate with other external devices, such as a notebook or desktop computer, tablet, headphones, Bluetooth headset, etc.
The modules described above are examples, and embodiments of wearable device 100 may include other modules not shown. For example, some embodiments of wearable device 100 may include a rechargeable battery (e.g., a lithium-ion battery), a microphone or a microphone array, one or more cameras, one or more speakers, a watchband, water-resistant casing or coating, etc. In some embodiments, all modules within wearable device 100 can be electrically and/or mechanically coupled together. In some embodiments, processor 210 can coordinate the communication among each module.
FIG. 3 shows an example of a companion device 300 according to some embodiments of the present disclosure. Wearable device 100 may be configured to communicate with companion device 300 via a wired or wireless communication channel (e.g., Bluetooth, Wi-Fi, etc.). In some embodiments, companion device 300 may be a smartphone, tablet computer, or similar portable computing device. Companion device 300 may be carried by the user, stored in the user's pocket, strapped to the user's arm with an armband or similar device, placed in a mounting device, or otherwise positioned within communicable range of wearable device 100 . In some embodiments, companion device 300 may include a variety of sensors, such as location and motion sensors (not shown). When companion device 300 is available for communication with wearable device 100 , wearable device 100 may receive additional data from companion device 300 to improve or supplement its calibration or calorimetry processes. For example, in some embodiments, wearable device 100 may not include a GPS sensor as opposed to an alternative embodiment in which wearable device 100 may include a GPS sensor. In the case where wearable device 100 may not include a GPS sensor, a GPS sensor of companion device 300 may collect GPS location information, and wearable device 100 may receive the GPS location information via interface 280 ( FIG. 2 ) from companion device 300 .
In another example, wearable device 100 may not include an altimeter or barometer, as opposed to an alternative embodiment in which wearable device 100 may include an altimeter or barometer. In the case where wearable device 100 may not include an altimeter or barometer, an altimeter or barometer of companion device 300 may collect altitude or relative altitude information, and wearable device 100 may receive the altitude or relative altitude information via interface 280 ( FIG. 2 ) from the companion device 300 .
In another example, wearable device 100 may receive motion information from companion device 300 . Wearable device 100 may compare the motion information from companion device 300 with motion information from one or more motion sensors 240 of wearable device 100 . Motion information such as data from accelerometer 260 and/or gyroscope 250 may be filtered (e.g. by a high-pass, low-pass, band-pass, or band-stop filter) in order to improve the quality of motion information. For example, a low-pass filter may be used to remove some ambient noise.
Wearable device 100 may use sensed and collected motion information to predict a user's activity. Examples of activities may include, but are not limited to, swimming, walking, running, cycling, weight lifting etc. Wearable device 100 may also be able to predict or otherwise detect when a user is sedentary (e.g., sleeping, sitting, standing still, driving or otherwise controlling a vehicle, etc.) Wearable device 100 may use a variety of motion information, including, in some embodiments, motion information from a companion device. In some embodiments, information from one or more of accelerometers, gyroscopes, global positioning (GPS) devices, and heart rate sensors can be used to determine whether a user is engaging in swimming.
FIG. 4A illustrates an example of a body-fixed frame of reference 500 according to some embodiments of the present disclosure. In FIG. 5A , the rotational axes of body-fixed frame of reference 400 are with respect to wearable device 100 . For example, the z-axis is perpendicular to the display surface 160 of wearable device 100 . The x-axis and the y-axis can be chosen relatively arbitrarily as long as the three axes are perpendicular to each other. In FIG. 4A , the x-axis is parallel with the direction pointed by crown 120 of wearable device 100 , and the y-axis is parallel with the direction of band 140 of wearable device 100 (assuming the direction pointed by crown 120 of wearable device 100 is perpendicular to the direction of band 140 of wearable device 100 ).
FIG. 4B-4D illustrates examples to express one or more orientations in body-fixed frame of reference 500 according to some embodiments of the present disclosure. In FIG. 4B , an orientation/direction 410 has an angle (φ) 402 with respect to the positive x-axis, an angle (θ) 404 with respect to the positive y-axis, and an angle (ψ) 406 with respect to the positive z-axis. The direction 410 can be expressed in body-fixed frame of reference 400 as [cos(φ), cos(θ), cos(ψ)], which is a non-limiting example/format of the first set of rotational data. For example, direction 420 in FIG. 4B is parallel with and pointing toward the positive x-axis, so the angle (φ) between direction 420 and the positive x-axis is 0-degree; the angle (θ) between direction 420 and the positive y-axis is 90-degree; and the angle (ψ) between direction 420 and the positive z-axis is 90-degree. Therefore, direction 420 can be expressed as [cos(0), cos(90), cos(90)], which is [1, 0, 0]. As another example, direction 430 in FIG. 4B is parallel with and pointing toward the positive z-axis, so the angle (φ) between direction 430 and the positive x-axis is 90-degree; the angle (θ) between direction 430 and the positive y-axis is 90-degree; and the angle (ψ) between direction 430 and the positive z-axis is 0-degree. Therefore, direction 430 can be expressed as [cos(90), cos(90), cos(0)], which is [0, 0, 1]. As yet another example, direction 440 represents direction of gravity in FIG. 4B and is parallel with and pointing toward the negative y-axis, so the angle (φ) between direction 440 and the positive x-axis is 90-degree; the angle (θ) between direction 440 and the positive y-axis is 180-degree; and the angle (ψ) between direction 440 and the positive z-axis is 90-degree. Therefore, direction 440 can be expressed as [cos(90), cos(180), cos(90)], which is [0, −1, 0].
In FIG. 4C , wearable device 100 is held vertically. As discussed earlier, the x-axis is parallel with direction pointed by crown 120 , the y-axis is parallel with band 140 , and the z-axis is perpendicular to display surface 160 . Direction 450 in FIG. 4C represents the direction pointed by crown 120 , so the angle (φ) between direction 450 and the positive x-axis is 0-degree; the angle (θ) between direction 450 and the positive y-axis is 90-degree; and the angle (ψ) between direction 450 and the positive z-axis is 90-degree. Therefore, direction 450 can be expressed as [cos(0), cos(90), cos(90)], which is [1, 0, 0]. As another example, direction 440 represents direction of gravity in FIG. 4C and is parallel with and pointing toward the negative y-axis, so the angle (φ) between direction 440 and the positive x-axis is 90-degree; the angle (θ) between direction 440 and the positive y-axis is 180-degree; and the angle (ψ) between direction 440 and the positive z-axis is 90-degree. Therefore, direction 440 in FIG. 4C can be expressed as [cos(90), cos(180), cos(90)], which is [0, −1, 0].
In FIG. 4D , wearable device 100 is rotated 45-degree clockwise compared with FIG. 4C . As discussed earlier, the x-axis is parallel with direction pointed by crown 120 , the y-axis is parallel with band 140 , and the z-axis is perpendicular to display surface 160 . Direction 450 in FIG. 4D represents the direction pointed by crown 120 , so the angle (φ) between direction 450 and the positive x-axis is 0-degree; the angle (θ) between direction 450 and the positive y-axis is 90-degree; and the angle (ψ) between direction 450 and the positive z-axis is 90-degree. Therefore, direction 450 can be expressed as [cos(0), cos(90), cos(90)], which is [1, 0, 0]. As another example, direction 440 represents direction of gravity in FIG. 4D . The angle (φ) between direction 440 and the positive x-axis is 45-degree; the angle (θ) between direction 440 and the positive y-axis is 135-degree; and the angle (ψ) between direction 440 and the positive z-axis is 90-degree. Therefore, direction 440 in FIG. 4D can be expressed as [cos(45), cos(135), cos(0)], which is [0.707, −0.707, 0].
It is noted that the expression of direction 450 is the same in FIG. 4C and FIG. 4D even though wearable device 100 has rotated. This is because the body-fixed frame of reference 400 is always fixed with respect to wearable device 100 . As a result, when position of wearable device 100 changes, the three axes in body-fixed frame of reference 400 and direction 450 change too, and relative position between direction 450 and the three axes remain the same. On the other hand, although direction of gravity 440 does not change in an “absolute” sense, it does not rotate together with wearable device 100 . Therefore, the expression of gravity direction 440 can be changed in the body-fixed frame of reference 500 when wearable device changes position.
FIG. 5 illustrates a first set of rotational data of wearable device 100 according to some embodiments of the present disclosure. Specifically, FIG. 5 illustrates estimation of the gravity in the body-fixed frame of reference 400 . The x-axis shows cos(φ) where φ is the angle between gravity and the positive x-axis in the body-fixed frame of reference 400 . The y-axis shows cos(θ), where θ is the angle between gravity and the positive y-axis in the body-fixed frame of reference 400 . The z-axis shows cos(ψ), where ψ is the angle between gravity and the positive z-axis in the body-fixed frame of reference 400 . For example, if at a moment wearable device 100 is facing up toward the sky, and display surface is parallel with the ground, then the gravity direction can be expressed as [0, 0, −1]. As another example, if crown is pointed towards the ground, then the gravity direction can be expressed as [1, 0, 0]. Gravity estimation in body-fixed frame of reference can help indicate when wearable device 100 is making a pitch and/or roll movement. For example, as discussed above, when a user's wrist was in a position such that crown is pointed towards the ground, the gravity direction is [1, 0, 0]. If the user then is rolling his or her wrist up for 90-degree, then display surface of wearable device 100 is facing up toward the sky, and display surface is parallel with the ground, then the gravity direction is expressed as [0, 0, −1]. If the user then is pitching his or her wrist up for 90-degree, then crown of wearable device 100 is facing up toward the sky, and the gravity direction is expressed as [−1, 0, 0]. These examples illustrate that gravity direction in the body-fixed frame of reference 400 can change in response to pitch and/or roll movement. In some embodiments, the gravity estimation in body-fixed frame of reference 400 can be used together with accelerometer 260 to estimate gravity. However, the gravity direction in the body-fixed frame of reference 400 does not change in response to yaw movement. For example, if wearable device 100 is facing up toward the sky, and display surface is parallel with the ground, then the gravity direction is expressed as [0, 0, −1]; then if the user making yaw movement along the horizon plane, the gravity direction remains as [0, 0, −1]. Also, as discussed above, because wearable device 100 is rotating the same as the body-fixed frame of reference 400 , the directions of wearable device 100 and components thereof are fixed. For example, no matter whether crown is pointing up, straight, or down, the crown direction is always expressed in body-fixed frame of reference 400 as [1, 0, 0]. Therefore, in some embodiments, it is more suitable to express the positions of wearable device 100 in a frame of reference that is not body-fixed in order to more readily indicate the movements of wearable device 100 with respect to external references.
FIG. 6 illustrates an inertial frame of reference 600 according to some embodiments of the present disclosure. In FIG. 6 , the z-axis (or the yaw axis) is based on the direction of gravity. The x-axis (or the roll axis) and the y-axis (or the pitch axis) can be chosen relatively arbitrarily as long as the three axes are perpendicular to each other.
FIGS. 7A-7D illustrate an example of an inertial frame of reference 700 according to some embodiments of the present disclosure. FIG. 7A depicts inertial frame of reference 700 in a context where a user is swimming. In FIG. 7A , the user wears wearable device 100 . But the z-axis (or the yaw axis) in the inertial frame of reference is based on the direction of gravity rather than the wearable device itself. Additionally, assuming the user is swimming laps, the x-axis (or the roll axis) is substantially parallel to the direction of the laps, and the y-axis (or the pitch axis) is perpendicular to the other two axes. In some embodiments, the x-axis (or the roll axis) and the y-axis (or the pitch axis) can be chosen relatively arbitrarily as long as the three axes are perpendicular to each other. In FIG. 7A , the z-axis is also referred to as yaw axis because any yaw movement rotates around the z-axis. Similarly, the x-axis is also referred to as roll axis because any roll movement rotates around the x-axis. And the y-axis is also referred to as pitch axis because any pitch movement rotates around the y-axis. By knowing the difference between the three-axis in the fixed-body frame of reference 400 and the three-axis in the inertial frame of reference 700 , the rotational data expressed in the fixed-body frame of reference 400 can be converted into the rotational data expressed in the inertial frame of reference 700 using techniques appreciated by people skilled in the art such as the one discussed in Sabatini.
FIG. 7B illustrates that wearable device 100 can make rotational movement with respect to inertial frame of reference 700 . In FIG. 7B , an orientation/direction 710 has an angle (φ) 702 with respect to the positive x-axis, an angle (θ) 704 with respect to the positive y-axis, and an angle (ψ) 706 with respect to the positive z-axis. The direction 710 can be expressed in body-fixed frame of reference 700 as [cos(φ), cos(θ), cos(ψ)], which is a non-limiting example/format of the second set of rotational data.
FIGS. 7C and 7D illustrate how same orientations in FIGS. 4C and 4D can be expressed differently in inertial frame of reference 700 . In FIG. 7C , wearable device 100 is held vertically, which is the same as FIG. 4C . As discussed earlier, the z-axis is based on the gravity in inertial frame of reference 700 . In FIG. 7C , the positive z-axis is chosen as the direct opposite position of gravity, the x-axis is perpendicular to the z-axis and pointing right horizontally, and the y-axis is perpendicular to both x-axis and y-axis and pointing “out” of FIG. 7C . Direction 450 in FIG. 7C represents the direction pointed by crown 120 , so the angle (φ) between direction 450 and the positive x-axis is 0-degree; the angle (θ) between direction 450 and the positive y-axis is 90-degree; and the angle (ψ) between direction 450 and the positive z-axis is 90-degree. Therefore, direction 450 can be expressed as [cos(0), cos(90), cos(90)], which is [1, 0, 0]. As another example, direction 440 represents direction of gravity in FIG. 7C and is parallel with and pointing toward the negative z-axis, so the angle (φ) between direction 440 and the positive x-axis is 90-degree; the angle (θ) between direction 440 and the positive y-axis is 90-degree; and the angle (ψ) between direction 440 and the positive z-axis is 180-degree. Therefore, direction 440 in FIG. 7C can be expressed as [cos(90), cos(90), cos(180)], which is [0, 0, −1].
In FIG. 7D , wearable device 100 is rotated 45-degree clockwise compared with FIG. 7C . Because the three axes are based on gravity, they can remain the same as FIG. 7C . Direction 450 in FIG. 7D represents the direction pointed by crown 120 , and the angle (φ) between direction 450 and the positive x-axis is 45-degree; the angle (θ) between direction 450 and the positive y-axis is 90-degree; and the angle (ψ) between direction 450 and the positive z-axis is 135-degree. Therefore, direction 450 can be expressed as [cos(45), cos(90), cos(135)], which is [0.707, 0, −0.707]. As another example, direction 440 represents direction of gravity in FIG. 7D . The angle (φ) between direction 440 and the positive x-axis is 90-degree; the angle (θ) between direction 440 and the positive y-axis is 90-degree; and the angle (ψ) between direction 440 and the positive z-axis is 180-degree. Therefore, direction 440 in FIG. 7D can be expressed as [cos(90), cos(90), cos(180)], which is [0, 0, −1].
It is noted that the expression of gravity direction 440 is the same in FIG. 7C and FIG. 7D even though wearable device 100 has rotated. This is because the inertial frame of reference 700 is always fixed with respect to gravity. As a result, when position of wearable device 100 changes, the three axes in inertial frame of reference 700 do not move along. On the other hand, the direction 450 does move with respect to the three axes, so the expression of direction 450 can be changed in the inertial frame of reference 400 even though it is fixed in body-fixed frame of reference 400 .
FIG. 8 illustrates a first set of rotational data of wearable device 100 according to some embodiments of the present disclosure. Specifically, FIG. 8 illustrates estimation of crown direction in the inertial frame of reference 700 while a user is swimming laps. The x-axis shows cos*, where (I) is the angle between crown direction and the positive x-axis in the inertial frame of reference 700 . The y-axis shows cos(θ), where θ is the angle between crown direction and the positive y-axis in the inertial frame of reference 700 . The z-axis shows cos(ψ), where ψ is the angle between crown direction and the positive z-axis in the inertial frame of reference 700 . For example, if at a moment wearable device 100 is facing up toward the sky, display surface is parallel with the ground, and crown is toward the positive x-axis, then the crown direction can be expressed as [1, 0, 0]; if wearable device 100 is making a yaw movements, and crown is toward the negative x-axis, then the crown direction can be expressed as [−1, 0, 0]. As another example, if crown is pointed towards the ground, then the crown direction can be expressed as [0, 0, 1]. The rotational data in FIG. 8 are largely divided into two clusters, 802 and 804 , because every time the user makes a turn, the angle φ between crown direction and the positive x-axis in the inertial frame of reference 700 changes substantially around 180-degree. Therefore, rotational data expressed in FIG. 8 can indicate wearable device 100 undergoes a steady-state change in heading when the data are switching from cluster 802 to cluster 804 , or vice versa.
FIG. 9 shows rotational data of wearable device 100 during a swimming session, where the rotational data are expressed in an inertial frame of reference. The rotational data in FIG. 9 is largely distributed within two clusters—clusters 902 and 904 in FIG. 9 ; when the traces of the rotational data switch from one cluster to the other cluster, the switch indicates that the yaw rotational data are changing significantly, which implies the user is making a turn. In some embodiments, to more clearly indicate the user is making a turn from the rotational data, the rotational data can be filtered to remove the “disturbance” caused by the user's regular strokes. In some embodiments, the rotational data can be low-pass filtered based on a time constant that is proportional to a period that the user needs to complete a stroke. In some embodiments, the time constant can be set by a user. In some embodiments, wearable device 100 can dynamically set the time constant based on average duration between strokes detected by wearable device 100 . For example, if it takes a user three seconds to finish a stroke, then the time constant can be set around or above three seconds so any rotational changes that are frequent (e.g., happen once and more between two consecutive strokes) can be filtered out or attenuated. On the other hand, a user generally makes a turn much less frequent than making a stroke (e.g., if it takes a user 30 seconds to swim a lap, which is a single length of a pool, then the user makes a turn around every 30 seconds), and the less frequent rotational changes will be passed through a low-pass filter. As a result, the filtered rotational data show the more steady-state change of the user's movements.
FIG. 10 shows a method 1000 for determining a direction of gravity according to some embodiments of the present disclosure. Knowing the direction of gravity is important to determine a frame of reference for motion information, such as rotational data, of wearable device 100 . In some embodiments, method 1000 can be modified by, for example, having steps combined, divided, rearranged, changed, added, and/or removed. Gravity determination method 1000 may begin at step 1010
At step 1010 , motion information may be received from the one or more motion sensors 240 on a wearable device (e.g., wearable device 100 ) of a user. In some embodiments, motion information may include three-dimensional rotational information from one or more sensors 240 such as gyroscope 250 and three-dimensional acceleration information from one or more sensors 240 such as accelerometer 260 . In some embodiments, motion information may be filtered such as by a low-pass filter to remove unwanted noise from the ambient.
At step 1020 , the angular velocity of wearable device 100 may be determined with respect to a frame of reference such as a body-fixed frame of reference or an inertial frame of reference.
At step 1030 , the gravity determination method 1000 may determine whether the angular velocity of wearable device 100 determined at step 1020 is below a threshold. For example, the threshold may be approximately 0.05 radians per second, 0.2 radians per second, or 0.5 radians per second, etc. If the angular velocity exceeds the threshold (e.g., when the user is doing exercise), the gravity determination method 1000 may return to step 1010 . In some embodiments, the gravity determination method 1000 may pause or wait for a period of time (e.g., 1 second, 5 seconds, 1 minute, etc.) before proceeding at step 1010 .
If the angular velocity is below the threshold (e.g., when the user is relatively still), the gravity determination method 1000 may proceed to step 1040 . In some embodiments, at step 1030 wearable device 100 also determines if the magnitude of forces acting on wearable device 100 are approximately equal to the normal force of gravity (1 g) before proceeding to step 1040 . If the magnitude is not approximately the normal magnitude, the gravity determination method 1000 may also return to block 1010 . Estimating direction of gravity when the angular velocity is below the threshold (e.g., when the user is relatively still) is important because in that way wearable device 100 will not be interfered or confused by acceleration due to other movements. Hypothetically, if wearable device 100 is having a 1 g acceleration along x-axis, then wearable device 100 may be mistaken the direction of gravity.
At step 1040 , the direction of gravity relative to wearable device 100 may be estimated. For example, in some embodiments, when wearable device 100 is held relatively still, accelerometer 260 within wearable device 100 may provide data about the direction of forces acting on wearable device 100 , which may be attributable primarily to gravity. In some embodiments, gravity determination method 1000 may also determine whether the user wearing wearable device 100 is accelerating (e.g., speeding up or slowing down) or traveling at an approximately constant velocity so as to further improve the estimate of the direction of gravity.
In some embodiments, gravity determination method 1000 may end after outputting the estimated direction of gravity. In other embodiments, the gravity determination method 1000 may return to step 1010 to refine or otherwise repeat the method of estimating the direction of gravity relative to the wearable device.
FIG. 11 shows a method 1100 for determining a direction of gravity according to some embodiments of the present disclosure. In some embodiments, the method 1100 can be modified by, for example, having steps combined, divided, rearranged, changed, added, and/or removed. Gravity determination method 1100 can be used when the user has companion device 300 and may begin at step 1110 .
At step 1110 , gravity determination method 1100 may periodically or continuously check for the presence of a companion device (e.g., companion device 300 ). For example, in some embodiments, wearable device 100 may determine whether a connection (e.g., Bluetooth, IEEE 802.11 Wi-Fi, or other wireless or wired communication channel) has been established or may be established with companion device 300 . If the companion device 300 is present, gravity determination method 1100 may proceed to step 1120 .
At step 1120 , the direction of gravity relative to companion device 300 may be estimated. In some embodiments, in contrast to the gravity determination method 1100 , it may not be necessary to check whether the angular velocity of companion device 300 is below a threshold because most or all of rotation of the angular velocity of companion device 300 may be orthogonal to the direction of gravity.
At step 1130 , the direction of gravity relative to companion device 300 may be outputted. In some embodiments, the direction of gravity relative to companion device 300 may be combined or otherwise compared with the direction of gravity relative to wearable device 100 . In some embodiments, companion device 300 may further determine a rotation rate around the direction of gravity relative to the companion device and output the rotation rate instead of or in addition to the direction of gravity relative to companion device 300 .
In some embodiments, gravity determination method 1100 may end after outputting the estimated direction of gravity. In other embodiments, gravity determination method 1100 may return to step 1110 to refine or otherwise repeat the method of estimating the direction of gravity relative to the wearable device.
Detecting Turns
FIG. 12 shows a flow chart illustrating a process 1200 of determining one or more turns made by a user during movements according to some embodiments of the present disclosure. In some embodiments, the process 1200 can be modified by, for example, having steps combined, divided, rearranged, changed, added, and/or removed. As described in more details below, in some embodiments, the process 1200 can include four steps. At step 1210 , wearable device 100 receives motion information from one or more motion sensors 240 . At step 1220 , wearable device 100 determines a first set of rotational data of wearable device 100 . At step 1230 , wearable device 100 converts the first set of rational data into a second set of rotational data. At step 1240 , wearable device 100 determines the user is making a turn based on the second set of rotational data.
At step 1210 , motion information may be received from one or more motion sensors 240 on wearable device 100 . In some embodiments, motion information may include three-dimensional rotational data of wearable device 100 from gyroscope 250 . In some embodiments, motion information may include three-dimensional accelerations of wearable device 100 from accelerometer 260 .
At step 1220 , wearable device 100 determines a first set of rotational data of wearable device 100 based on the motion information received from one or more motion sensors 240 . In some embodiments, the rotational data of wearable device 100 include how wearable device 100 rotates, such as angular velocities of wearable device 100 , with respect to a frame of reference. In some embodiments, the first set of rotational data is received from gyroscope 250 and is expressed in a body-fixed frame of reference with respect to wearable device 100 .
At step 1230 , wearable device 100 converts the first set of rotational data into a second set of rotational data. As described above, rotational data in the body-fixed frame of reference cannot readily indicate whether or not wearable device 100 undergoes movements with respect to external references. To address this issue, wearable device 100 converts the rotational data in the body-fixed frame of reference into rotational data in an inertial frame of reference using techniques appreciated by people skilled in the art such as the one discussed in “Kalman-filter-based orientation determination using inertial/magnetic sensors: observability analysis and performance evaluation,” Angelo Maria Sabatini, published Sep. 27, 2011, Sensors 2011, 11, 9182-9206.
At step 1240 , wearable device 100 determines that the user wearing wearable device 100 is making a turn based on the set of rotational data expressed in the inertial frame of reference.
FIG. 13 illustrates filtered yaw data of a wearable device according to some embodiments of the present disclosure. In some embodiments, the yaw data shown in FIG. 13 are obtained by first projecting the rotational data shown in FIG. 9 into a 2D vector and then filtering the 2D vector. For example, the data in FIG. 9 is a 3D vector that moves in time and can be represented as i(t)=(x(t), y(t), z(t)). Then, in some embodiments, i(t) can be projected onto the x-y plane using the gravity vector, and the resulting 2D vector can be represented as j(t)=(x(t), y(t)). The x-component and y-component of j(t) are each individually filtered by a low-pass filter as described above. Then the angle plotted in FIG. 13 can be the angle between j(t) at adjacent times to show how j(t) is progressing in time. For example, suppose at t=0, (x=1, y=0), and then at t=1, (x=0, y=1), then the angle change would be 90 degrees. FIG. 13 shows the yaw data of wearable device 100 worn by a user who completes 4 laps in breaststroke. As described earlier, in an inertial frame of reference, the x-axis and y-axis can be chosen relatively arbitrarily as long as the three axes are perpendicular to each other. Therefore, the filtered yaw data of wearable device 100 in one direction can be around a relatively arbitrary value. For example, FIG. 13 shows the filtered raw data oscillates between two steady-state values, which are roughly 130 degree and −50 degree. The absolute values of the two steady-state yaw data (e.g., 130 degree and −50 degree) are not important; what is more important is that the two steady-state yaw data differ by approximately 180 degree, which implies the user is making a turn. In FIG. 13 , the filtered raw data changes abruptly at 1302 , 1304 , 1306 , and 1308 (for example, from 130 degree to −50 degree and/or from −50 degree to 130 degree) when the user is making a turn, and wearable device 100 can detect this abrupt change and determine that the user is making a turn. In some embodiments, the abrupt change can be measured as having at least a threshold change within a threshold period. For example, in some embodiments, if the filtered raw data changes more than 150 degrees per eight seconds, then wearable device 100 can determine that the user is making a turn. In some embodiments, other suitable threshold changes and/or threshold periods can be used.
Detecting Breaths
FIG. 14 shows a flow chart illustrating a process 1400 of determining one or more breaths taken by a user while swimming according to some embodiments of the present disclosure. In some embodiments, the process 1400 can be modified by, for example, having steps combined, divided, rearranged, changed, added, and/or removed. As described in more details below, in some embodiments, the process 1400 can include four steps. At step 1410 , wearable device 100 receives motion information from one or more motion sensors 240 . At step 1420 , wearable device 100 determines a first set of rotational data of wearable device 100 . At step 1430 , wearable device 100 converts the first set of rotational data into a second set of rotational data, where the second set of rotational data include pitch rotational data. At step 1440 , wearable device 100 determines the user is taking a breath based on the second set of rotational data by monitoring the pitch rotational data exceeding a threshold.
At step 1410 , motion information may be received from one or more motion sensors 240 on wearable device 100 . In some embodiments, motion information may include three-dimensional rotational data of wearable device 100 from gyroscope 250 . In some embodiments, motion information may include three-dimensional accelerations of wearable device 100 from accelerometer 260 .
At step 1420 , wearable device 100 determines a first set of rotational data of wearable device 100 based on the motion information received from one or more motion sensors 240 . In some embodiments, the rotational data of wearable device 100 include how wearable device 100 rotates, such as angular position, angular velocity, and/or angular acceleration of wearable device 100 , with respect to a frame of reference. In some embodiments, if the rotational data of wearable device 100 is angular acceleration, then angular velocity and/or angular position can be obtained by integrating the angular acceleration over time. Likewise, if the rotational data of wearable device 100 is angular velocity, then angular position can be obtained by integrating the angular velocity over time. In some embodiments, the first set of rotational data is received from gyroscope 250 and is expressed in a body-fixed frame of reference with respect to wearable device 100 .
At step 1430 , wearable device 100 converts the first set of rotational data into a second set of rotational data. As described above, rotational data in the body-fixed frame of reference cannot readily indicate whether or not wearable device 100 undergoes movements with respect to external references. To address this issue, wearable device 100 converts the rotational data in the body-fixed frame of reference into rotational data in an inertial frame of reference using techniques appreciated by people skilled in the art such as the one discussed in “Kalman-filter-based orientation determination using inertial/magnetic sensors: observability analysis and performance evaluation,” Angelo Maria Sabatini, published Sep. 27, 2011, Sensors 2011, 11, 9182-9206.
At step 1440 , wearable device 100 determines that the user wearing wearable device 100 is taking a breath based on the second set of rotational data by monitoring the pitch rotational data exceeding a threshold. When the user is swimming in freestyle, breast stroke, or butterfly, the user's breaths are often associated with upward movements of arm/wrist. Accordingly, wearable device 100 can determine that the user takes a breath when the user's arm/wrist is making moving upward.
FIG. 15 illustrates wrist angle of a wearable device according to some embodiments of the present disclosure. In FIG. 15 , the wrist angle is an angle between a user's wrist (the wrist wearing wearable device 100 ) and horizon while the user is swimming in freestyle. For example, in FIG. 15 , the wrist angle at 1502 has about −80 degrees, and this corresponds to the user's arm (the arm with the wrist wearing wearable device 100 ) in a freestyle motion with the fingers pointing towards the ground during the pull motion in the water. From 1502 to 1504 , the wrist angle gradually increases from −80 degrees to 0 degrees, and this corresponds to the user's arm gradually pulling toward the swimmer's hip and eventually leaving the water. From 1504 to 1506 , the wrist angle roughly settles between 0 degrees and −20 degrees, and this corresponds to the user's arm gliding. From 1506 to 1508 , the wrist angle gradually decreases from roughly −20 degrees to −80 degrees, and this corresponds to the user's arm is pulling toward the ground. The user is then repeating the same stroke phases from 1508 to 1514 as the user did from 1502 to 1508 . From 1514 to 1516 , the wrist angle gradually increases from −80 degrees to 0 degrees, and this again corresponds to the user's arm is gradually pulling toward the swimmer's hip and eventually leaves the water. Unlike the previous two phases shown in FIG. 15 , from 1516 to 1518 , the wrist angle continues to increase from 0 degrees to about 30 degrees, and this corresponds to the user's arm spikes towards the sky, which indicates the user's nose and/or mouth is above the water to take a breath. From 1518 to 1520 , the user's arm is back to the water level. From 1520 to 1522 , the user's arm is gliding. And from 1522 to 1524 , the user's arm is again pulling toward the ground. Therefore, in some embodiments, by monitoring the wrist angle, one can deduce the user's stroke phase. For example, when the user is taking a breath in styles such as freestyle or butterfly, the user will have his or her head above the water, and during this time, it is likely that the user's wrist is also above the water with a positive wrist angle or less negative wrist angle. When the user is taking a breath in breaststroke, the user may push his or her wrist further down, and the wrist angle can be more negative than stroke periods without taking a breath. Based on the range of the user's wrist angle and the user's swimming style, a threshold can be chosen to indicate when the user is taking a breath. For example, when the user is swimming freestyle or butterfly, and the range of the user's wrist angle is between −80-degree and 20-degree, the threshold can be selected as top 10% because the user's taking breath is associated with a high wrist angle. That means when the wrist angle is entering the region between 10-degree and 20-degree, the user will be estimated to take a breath. In some embodiments, any other suitable percentage can be selected. As another example, when the user is swimming breaststroke, and the range of the user's wrist angle is between −80-degree and 0-degree, the threshold can be selected as bottom 10% because the user's taking breath is associated with a low wrist angle. That means when the wrist angle is entering the region between −80-degree and −72-degree, the user will be estimated to take a breath. In some embodiments, any other suitable percentage can be selected. In some embodiments, the threshold for determining the user is taking a breath can be dynamically set. For example, by forming a range of angles that a stroke takes, and then using statistical methods to decide if a current stroke looks sufficiently aberrant from the typical stroke.
Counting Swim Strokes
FIG. 16 is a flow chart of a method for counting swim strokes 1600 , according to some embodiments of the present disclosure. In some embodiments, the method includes the steps of receiving three dimensional motion information from a motion sensor 1610 ; determining a first set of three dimensional rotational data of a wearable device 1620 ; converting the first set of three dimensional rotational data into a second set of three dimensional rotational data 1630 ; converting the second set of three dimensional rotational data to a set of two dimensional rotational data 1640 ; adding constraints to 2D rotational data to form circles or semi-circles 1645 ; and counting the number of strokes from the set of two dimensional rotational data 1650 .
At step 1610 , wearable device 100 receives three dimensional motion information from a motion sensor 240 .
At step 1620 , the wearable device 100 determines a first set of three dimensional rotational data of the wearable device 100 .
At step 1630 , wearable device 100 converts the first set of three dimensional rotational data into a second set of three dimensional rotational data. As described above, the three dimensional rotational data in the body-fixed frame of reference cannot readily indicate whether or not wearable device 100 undergoes movements with respect to external references. To address this issue, wearable device 100 converts the three dimensional rotational data in the body-fixed frame of reference into three dimensional rotational data in an inertial frame of reference using techniques appreciated by people skilled in the art such as the one discussed in “Kalman-filter-based orientation determination using inertial/magnetic sensors: observability analysis and performance evaluation,” Angelo Maria Sabatini, published Sep. 27, 2011, Sensors 2011, 11, 9182-9206.
When the motion data is transformed to the inertial frame, the data appears as repetitive motion orbits as the user swims. Once it has been determined that the user is swimming, the 3D orbits can be examined in the inertial reference frame using a principal component analysis. For example, the plane that includes the most data points (corresponding to the plane in which the swimmer demonstrates the greatest stroke energy) define the first and second principal component vectors. The third vector that is perpendicular to the first two vectors defines the primary axis of rotation (i.e., the third principal component vector). Once the planes and axis of rotation are determined, the second set of three dimensional rotational data can be projected onto a 2D space (step 1640 ). The equation for the projection can be expressed as: R T (I−ww T ) equation (1), where R=rotation, T=matrix transpose, I is the 3×3 identity matrix and w is the third component vector of the PCA.
At step 1645 , additional constraints can be added to the 2D rotational data to project rotational data with cleaner orbits and facilitate more reliable stroke counting. In some embodiments, these constraints can be accelerometer energy, moment arm calculations, and/or rotational direction (e.g., clockwise and counterclockwise).
At step 1650 , the revolutions of the circles or semi-circles shown in the constrained 2D rotational data can be counted to determine a stroke number.
FIG. 17 are a series of graphical representations of the data collected with respect to systems and methods described herein. For example, inertial frame data 1702 is shown in three dimensions x, y, z for a freestyle swimmer with time domain sampling. Constrained space domain sampling 1704 of the same three dimensional freestyle data is also shown. Space domain sampling refers to constructing a signal from samples that are a minimum distance apart. For 3D spatial sampling, for example, the distance is specified in a norm-2 manner, where each consecutive sample has a minimum norm-2 distance between them. The three dimensional time and space domain data 1702 , 1704 can be converted to a two-dimensional projection 1706 based on a principal component analysis of the 3D data of 1704 . In this example, only x and y are shown in the data, the z axis is used as the axis of rotation to transform the data from a three dimensional projection to a two dimensional projection. By adding additional constraints to two dimensional projection 1706 , a second two dimensional projection 1708 can be generated that projects a cleaner orbit and facilitates more reliable counting. In some embodiments, these constraints can be accelerometer energy, moment arm calculations, and/or rotational direction (e.g., clockwise and counterclockwise).
In some embodiments, data captured for arm motions that do not show sufficient energy (as measured by the accelerometer) or sufficient extension (as measured by the moment arm calculations) to be considered true swim strokes can be eliminated from the 2D projection.
In some embodiments, clockwise and counterclockwise rotational direction data can be used as constraints to eliminate certain data points from the projection. For example, only the clockwise rotational direction data (e.g., for freestyle and butterfly strokes) or the counter clockwise rotational direction data (e.g., for backstroke) can be considered when counting strokes, to eliminate any unintentional gyroscope drift when executing a stroke. For example, FIG. 18 depicts counter clockwise rotational data for a backstroke that includes some drift in the opposite rotational direction between points A and B. At point B, the backstroke starts going off course and exhibits a clockwise motion. The clockwise motion is not considered for stroke counting, and once the stroke resumes its counterclockwise motion, stroke counting picks up again at point B, the last point before the stroke started going off course.
It is from the representation of the data shown in 2D projection 1708 that the number of revolutions can be counted and can be equated with a number of strokes. In order to determine the number of revolutions, a threshold or a number of thresholds can be established along the path of revolution. For example, if the stroke style yields a full rotation (e.g., backstroke, butterfly and freestyle), then a single threshold can be established along the rotational path. Each time the threshold is crossed, another revolution (which represents a stroke) is counted. A threshold is crossed when there is a data point before and after the threshold line.
In some embodiments, when a stroke style only exhibits a partial revolution, instead of a full revolution (e.g., breaststroke), multiple thresholds can be established along a rotational path (e.g., at 45° intervals) to capture the stroke somewhere along its semi-rotational path. When the motion that exhibits a partial rotation crosses one of the established thresholds, the partial rotation can be counted.
FIG. 19 depicts sample stroke counts, according to some embodiments of the present disclosure. For example, stroke count 1902 represents three full circles, which represents three strokes of a stroke such as backstroke. Stroke count 1904 , represents three semi-circles, which could equate to three strokes of breast stroke. Stroke count 1906 can represent three strokes or semi-circles, which can correspond to breaststroke with either a lap change (showing the change in direction) or the change could also be associated with gyroscope drift.
In some embodiments, the system and method can include a spurious stroke check module to eliminate arm motions that are not true strokes. For example, the system and method of the present disclosure can include a voting mechanism module that only counts strokes when they are not too far spread out in time, before committing these strokes as real strokes.
Counting Laps
The present disclosure describes ways to determine the number of laps a swimmer swims in a swimming session. Generally, when a swimmer reaches an end of a swimming lap, he or she will turn to continue to swim. Therefore, finishing a lap is typically associated with a turn made by the swimmer.
In reality, however, a swimmer may be detected to make a turn sometimes even if he or she has not finished a lap. For example, the swimmer may take a goggle break, take an out-of-pool break, make a turn in the middle of the lap, and/or any other activities that may cause the swimmer to intentionally or unintentionally make a turn without reaching the end of a lap. Therefore, a swimmer may be detected to make more than one turn per lap.
For example, FIG. 20 illustrates a case study associated with 188 swimming sessions. In FIG. 20 , the x-axis indicates the number of laps, and the y-axis indicates the number of turns detected. Each point in FIG. 20 represents a swimming session with the number of truth lap count indicated by the corresponding x-axis and the number of turns detected indicated by the corresponding y-axis. And the number next to each point indicates the number of the swimming sessions that has the same associated truth lap count and number of turns detected. For example, 2010 indicates that there is one swimming session that the swimmer swims for 10 laps and is detected to make 14 turns. As another example, 2020 indicates that there is one swimming session that the swimmer swims for four laps and is detected to make three turns. As yet another example, 2030 indicates that there are five swimming sessions that the swimmer swims for eight laps and is detected to make eight turns. The dotted line 2040 has a slope of one in FIG. 20 , and it means that any point on the dotted line 2040 has the same number of lap count and turns detected. Because generally each lap should be associated with one turn, the data points above the dotted line 2040 generally indicate false positive of lap count inferred by the number of turns detected, i.e., there are one or more turns that are not associated with turns at the end of a lap. On the other hand, the data points below the dotted line 2040 generally indicate false negative of lap count inferred by the number of turns detected, i.e., there are one or more laps that are not associated with a turn detected. This can be the case when the swimmer continues to another lap without making a turn—for example, the swimmer can first swim in freestyle, and when the swimmer reaches the end of the swimming pool, the swimmer can switch to backstroke without making a turn.
When the swimmer makes a turn without finishing a lap, the turn can be referred to as a false or premature turn in that the turn does not correspond to the finish of a lap. One way to determine the false turns is to look at the number of strokes between two consecutive turns. For example, if it generally takes a swimmer 15 to 20 strokes to finish a lap, and if there is only eight strokes between two turns, then at least one turn is a false turn. For example, FIG. 21 shows the probability distribution function of half of the number of strokes per lap according to a case study. In FIG. 21 , the x-axis indicates half of the number of strokes per lap in a 25 yards pool, and the y-axis indicates the probability distribution function. FIG. 21 shows half of the number of strokes per lap because FIG. 21 will be compared to FIG. 22 , which shows the probability distribution function of the number of detected strokes per lap. Because wearable device 100 can observe only those strokes by the arm wearing wearable device 100 , FIG. 21 only shows half of the number of true strokes counted per true lap in order to be consistent. FIG. 21 shows that to finish a lap, the half of the number of strokes can be from five to 25, and it is more likely to be around nine to 15. In FIG. 21 , the half of the number of strokes per lap is recorded by a proctor during the swimming session. FIG. 22 also shows the probability distribution function of the number of detected strokes per lap according to the same case study. Unlike FIG. 21 , in FIG. 22 , a lap is indicated by a turn detected. Comparing to FIG. 21 , FIG. 22 shows that sometimes there are less than 5 detected strokes between two consecutive turns, as indicated by 2210 . As discussed earlier, these turns are considered false turns in the sense that they do not indicate a true lap because it is generally not very likely for a not highly skilled swimmer to finish a 25-yard lap with less than five detected strokes.
FIG. 23 illustrates a flow chart illustrating a computerized process 2300 of determining a number of laps a user swims during a swimming session according to certain embodiments of the present disclosure. In some embodiments, the computerized process 2300 can be modified by, for example, having steps combined, divided, rearranged, changed, added, and/or removed.
At step 2302 , motion information may be received from one or more motion sensors 240 on wearable device 100 . In some embodiments, motion information may include three-dimensional rotational data of wearable device 100 from gyroscope 250 . In some embodiments, motion information may include three-dimensional accelerations of wearable device 100 from accelerometer 260 .
At step 2304 , wearable device 100 determines rotational data of wearable device 100 expressed in an inertial frame of reference as described above. In some embodiments, wearable device 100 can, additionally or alternatively, include rotational data expressed in a body-fixed frame of reference.
At step 2306 , wearable device 100 detects each time the user makes a turn during the swimming session based on the rotational data. In some embodiments, wearable device 100 can detect the user makes a turn.
At step 2308 , wearable device 100 detects each time the user makes a stroke and the swimming style associated with the stroke. In some embodiments, wearable device 100 can detect whether or not the user makes a stroke. In some embodiments, wearable device 100 can only detect strokes of the arm wearing wearable device 100 . Therefore, in these embodiments, throughout the application, the strokes can be meant strokes detected by wearable device 100 and can be approximately half of the true strokes made by the user. For example, in these embodiments, the number of strokes between turns is the number of strokes detected by wearable device 100 between turns.
At step 2310 , wearable device 100 rejects certain turns that are detected at step 2306 based on one or more criteria. One purpose of this step is to reject the turns made by the user while the user is not swimming. In some embodiments, wearable device 100 evaluates a turn based on the stroke, turn, and swimming style detected at steps 2306 and 2308 . For example, in some embodiments, when wearable device 100 detects a turn, it will reject the turn unless both of the following two criteria are met. One of the criteria is the stroke rate between two consecutive turns needs to be greater than a minimum stroke rate. The stroke rate can be defined as the number of strokes per minute. In some embodiments, the minimum stroke rate can be eight strokes per minute. In some embodiments, other suitable values can be used as the minimum stroke rate. The other criteria is the number of the strokes with a known style between two consecutive turns needs to be greater than a minimum stroke count. In some embodiments, the minimum stroke count can be three strokes. In some embodiments, other suitable values can be used as the minimum stroke count. In some embodiments, wearable device 100 can reject a turn detected based on less, other, or more criteria. In some embodiments, even if a turn is rejected, the detected turn will still be stored for later adjustment.
At step 2312 , wearable device 100 determines a stroke range per lap. In some embodiments, the range is determined based on whether or not the number of the user's strokes converges in the current swimming session or in the historical session. Generally, the number of the user's strokes converges when the variation among the number of strokes per lap is less than a threshold. The details of step 2312 can be further described in connection with FIG. 24 below.
FIG. 24 illustrates a flow chart illustrating a computerized process 2400 of determining a stroke range per lap according to certain embodiments of the present disclosure. In some embodiments, the computerized process 2400 can be modified by, for example, having steps combined, divided, rearranged, changed, added, and/or removed.
At step 2410 , wearable device 100 determines whether the number of the user's strokes converges in the current swimming session. In some embodiments, wearable device 100 checks six consecutive turns and evaluates the number of strokes among the six consecutive turns. For example, the first number of strokes S 1 is the stroke count between the user starts to swim and the user makes a first turn detected by wearable device 100 ; the second number of strokes S 2 is the stroke count between the user makes a first turn detected by wearable device 100 and the user makes a second turn detected by wearable device 100 ; and S 3 -S 6 can be calculated in a similar way. In some embodiments, if the standard deviation of the number of strokes among the six consecutive turns (e.g., S 1 -S 6 ) is less than a threshold, such as 1.5, wearable device 100 can determine that the number of the user's strokes converges in the current swimming session. In some embodiments, other number of consecutive turns and/or other thresholds of standard deviation can be used to determine whether or not the number of the user's strokes converges in the current swimming session. If the number of the user's strokes converges in the current swimming session, the process 2400 proceeds to step 2420 . If the number of the user's strokes does not converge in the current swimming session, the process 2400 proceeds to step 2430 .
At step 2420 , wearable device 100 determines a stroke range given the number of the user's strokes converges in the current swimming session. In some embodiments, the stroke range can be determined based on the mean value of the number of strokes per lap in the current converged swimming session. For example, if the mean value of the number of strokes per lap in the current converged swimming session is 20 strokes per lap, the range can be +/−6 from the range, i.e., from 14 to 26 strokes per lap. In some embodiments, other suitable ranges can be used.
At step 2430 , wearable device 100 determines whether the number of the user's strokes converges in a historical swimming session. In some embodiments, the factor(s) to determine whether or not the number of the user's strokes converges in a historical swimming session can be the same factor(s) used at step 2410 . If the number of the user's strokes converges in the historical swimming session, the process 2400 proceeds to step 2440 . If the number of the user's strokes does not converge in the historical swimming session, the process 2400 proceeds to step 2450 .
At step 2440 , wearable device 100 determines a stroke range given the number of the user's strokes converges in the historical swimming session. In some embodiments, the stroke range can be determined based on the mean value of the number of strokes per lap in the historical converged swimming session. For example, if the mean value of the number of strokes per lap in the historical converged swimming session is 16 strokes per lap, the range can be +/−6 from the range, i.e., from 10 to 22 strokes per lap. In some embodiments, the range can be increased for a larger mean value and/or decreased for a smaller mean value. For example, if the mean value of the number of strokes per lap in the historical converged swimming session is 24 strokes per lap, the range can be +/−8 from the range, i.e., from 16 to 32 strokes per lap. In some embodiments, the lower bound of the range can be set at other numbers to take into account the possibility that the current session and the historical session are not associated with the same pool length. For example, in some embodiments, the lower bound can be set at 3 and the upper bound remains at 6 strokes over the mean value. For example, if the mean value of the number of strokes per lap in the historical converged swimming session is 16 strokes per lap, the range can be from 3 to 22 strokes per lap. In some embodiments, other suitable ranges can be used.
At step 2450 , wearable device 100 determines a stroke range given the number of the user's strokes does not converge in the current swimming session or in the historical swimming session. In some embodiments, the stroke range can be wider than the ranges determined at step 2320 or 2340 because the number of the user's strokes has not converged yet. In addition, the range can be varied based on the user's swimming style. For example, in some embodiments, if the user is detected to swim in breast stroke, then the range can be from 3 to 72 strokes per lap. In some embodiments, if the user is detected to swim in other styles, then the range can be from 3 to 40 strokes per lap.
As mentioned above, the parameters used in process 1600 can be changed to other suitable values. In addition, in some embodiments, if wearable device 100 first determines the number of the user's strokes does not converge and then later determines the number of the user's strokes converges eventually, then wearable device 100 can adjust the stroke range accordingly.
Now referring back to FIG. 23 . At step 2314 , wearable device 100 determines lap count of the user during the current swimming session. The details of step 2314 can be further described below in connection with FIG. 25 .
FIG. 25 illustrates a flow chart illustrating a computerized process 2500 of determining lap count of the user accordingly to certain embodiments of the present disclosure. In some embodiments, the computerized process 2500 can be modified by, for example, having steps combined, divided, rearranged, changed, added, and/or removed.
At step 2510 , wearable device 100 accepts the turns that meet all criteria to become a lap. At step 2510 , wearable device 100 has already rejected certain turns. At step 2510 , wearable device further rejects a turn if the number of the strokes between the current detected turn and the previous detected turn is outside the stroke range determined at step 1512 . In some embodiments, all detected turns, both accepted and rejected, will be kept at a storage for potential adjustment at step 2530 .
At step 2520 , wearable device 100 inserts a turn if it determines that a turn is not detected. For example, sometimes the user makes a turn with a relatively small change in yaw angle, and wearable device 100 missed detecting this turn. Sometimes if the user reaches the end of the pool using a style other than backstroke and then switches to backstroke to continue, then wearable device 100 may not detect a turn because there may not be much change of yaw angle.
In some embodiments, wearable device 100 can determine there is a missed turn if the number of strokes between two consecutive turns is too large. For example, if in previous turns, the average number of strokes between two consecutive turns is 20, and the number of strokes between the current detected turn and previous detected turn is 40, it is likely that a turn is missed. In some embodiments, wearable device 100 can determine there is a missed turn if the number of the strokes between two turns is greater than a threshold. For example, the threshold can be 1.8 times of the average number of strokes between consecutive turns. In some embodiments, wearable device 100 can also require the threshold is greater than the upper bound of the stroke range. For example, if the average stroke count is 15, and the stroke range is between 9 and 21, then if there are 30 strokes between the current detected turn and the previous turn, wearable device 100 will determine there is a missed turn because 30 is both greater than the upper bound of the stroke range (21) and 1.8 times of the mean value (1.8*15=27). In some embodiments, other suitable parameters can be used to determine whether a turn is missed. In some embodiments, step 2520 is limited to only insert one lap. In some embodiments, step 2520 can insert more than one lap.
At step 2530 , wearable device 100 adjusts turns detected to reduce variance of strokes per lap counted. For example, when wearable device 100 rejects a turn, it will consider if accepting the current turn and rejecting the previous accepted turn will reduce variance of strokes per lap counted. Table I shows an example of turns detected by wearable device 100 and the number of strokes associated with two consecutive turns detected. Wearable device 100 would normally reject the fourth turn since it only associated with 3 strokes. In that case, six turns (turns #1, #2, #3, #5, #6, and #7) will be accepted, and the numbers of strokes between two consecutive accepted turns will be 15, 15, 12, 18, 14, and 15. The mean value of the stroke count will be 14.83, and the standard deviation will be 1.94. As discussed above, at step 2530 , when wearable device 100 rejects a turn, it will also compare to the previous accepted turn and determine if accepting the current turn and rejecting the previous accepted turn will reduce variance of strokes per lap counted. For example, if wearable device 100 rejected the previously accepted turn #3 and accepted instead turn #4, then the numbers of strokes between two consecutive accepted turns will be 15, 15, 15, 15, 14, and 15. The mean value of the stroke count will be around 14.83, and the standard deviation will be around 0.41. Since the variance of the number of strokes per lap counted will be smaller after adjustment, wearable device 100 will accept turn #4 but reject turn #3.





TABLE I



Turns Detected and the Number of Strokes Detected

between Consecutive Turns





Turns Detected











1
2
3
4
5
6
7













number of strokes between
15
15
12
3
15
14
15

consecutive turns






























FIG. 26 illustrates a probability distribution function of lap count error using techniques according to certain embodiments of the present disclosure. In FIG. 26 , the x-axis is the difference between the lap count determined by the current invention and the truth lap count. The y-axis is the probability distribution function of different lap count error, where the lap count error can be defined as the laps counted minus the truth lap count. FIG. 26 records results for 1269 session. FIG. 26 illustrates that the techniques described in the present disclosure render a low error between laps counted by wearable device 100 and truth lap count—the mean value of the error is around 0.13633 and standard deviation is around 0.81847.
FIG. 27 illustrates lap count error using techniques according to certain embodiments of the present disclosure. FIG. 27 records results for 1269 sessions with 24079 total laps. The x-axis is the truth lap count. The y-axis the lap count error. FIG. 27 shows the lap count error is well bounded even for large truth lap count. In other words, the lap count error does not scale with the truth lap count using techniques described in the present disclosure.
Detecting Swim Activity
FIG. 28 depicts in further detail the components that may be found within wearable device 100 according to some embodiments of the present disclosure for detecting swim activity. In some embodiments, wearable device 100 can include motion sensors 240 , including an accelerometer 240 , gyroscope 250 , and GPS 410 ; a sensor fusion module 2820 , a speed estimation module 2830 , a heart rate sensor 290 , a sensor processing module 2840 , a signal fusion module 2845 , and a swimming detection output 2850 . Sensor data is collected by the three axis accelerometer 260 and three axis gyroscope 250 in epochs. Each epoch can range from about 1 second to about 10 seconds, an exemplary epoch being 2.56 seconds. The samples are collected in X, Y, and Z axes. The data from motion sensors 240 can be used to determine a stroke rate 2860 , stroke classification 2870 and various motion signatures 2880 characteristic of swimming.
FIG. 29 is a flowchart of a method for detecting swim activity, according to embodiments of the present disclosure. In some embodiments, method 2900 includes the step of receive motion information from a motion sensor 2910 . Based on that information the method makes the following determinations: is the stroke rate regular 2920 , is there a high confidence in the style classification 2930 , does the motion signature indicate swimming 2940 . If the answer to all of these determinations is yes, then it is determined that the user is swimming 2950 . If the answer to any of these questions is no, then it is determined that the user is not swimming 2960 .
For example, stroke rate can be used to detect swimming. While a user is swimming, the user will generally have a regular, periodic stroke rate/count. For example, the stroke period (e.g., the time between strokes) will have a low variance when averaged over a reasonable time window, for example, ten seconds. Conversely, when the user is not swimming, but is instead taking a break, the stroke period will be sporadic. Accordingly, the motion sensors detecting a consistent stroke period over a period of time would indicate swimming. In some embodiments, the default stroke rate for detecting swimming can be eight strokes per minute or above, corresponding to a beginner/very unskilled swimmer. If a user's stroke rate falls below the default stroke rate, then the stroke rate counter will not detect swimming.
In some embodiments, wearable device 100 can receive training data based on a user's observed stroke rate when the user wears wearable device 100 . The default stroke rate to detect swimming can then be personalized to be calculated based on the user's observed stroke rate. For example, the default stroke rate can be set to the inverse of the median stroke rate of a user that was observed for at least three consecutive stroke periods.
In some embodiments, the motion sensor data also can be used for style classification, which can also provide an indication as to whether a user is swimming. Each stroke is expected to be classified as one of the common four styles: freestyle, backstroke, breaststroke, and butterfly. With popular classifiers used in pattern recognition, such as support vector machines and logistic regression, an additional notion of confidence in a classification determination can be determined. Stroke classification uses a two-tiered decision tree classifier, with logistic regression classifier at each level. In some embodiments, a range [0,1] can be assigned to each stroke classification determination, which indicates the confidence level in the classification decision, e.g., a value closer to 0 implies low confidence in the classification output, and high otherwise, as apparent in logistic regression. Alternately, confidence level can be inferred from a correlated metric such as the number of strokes with known style (i.e. not classified as Unknown) in a pre-defined time-window. Low confidence in a classification style can be a useful indicator of non-steady state swim behavior, whereas high confidence in a classification style can be a useful indicator of swimming activity. For example, low confidence can be interpreted as any value in the range [0, 0.2], while the corresponding high confidence range can be [0.2, 1.0].
Stroke rate and style classification can capture differences between steady and non-steady state swimming. However, sometimes, user arm movement while not swimming can trigger a stroke count and/or a valid style classification. For these scenarios, accelerometer energy and gyroscope signal variance can be investigated to correctly determine a motion signature that indicates swim activity.
Discriminative information in accelerometer and gyroscope energy (based on the accelerometer and gyroscope signals from the device sensor fusion) can be mined for swim activity detection. As discussed above, in some embodiments, the accelerometer and gyroscope signals are converted from the body fixed frame of reference to the inertial frame of reference. Referring to FIG. 30 , graph 3000 depicts the crown orientation of wearable device 100 . Specifically, graph 3000 includes a y axis ranging from −1 to 1, which represents upward direction towards the sky (+1) and downward direction towards the earth (−1), and an x axis, which represents time (t) from 260 seconds to 320 seconds. The graph also includes three curved lines: a bold solid line corresponding to a first inertial axis indicating crown orientation, a regular solid line corresponding to a second inertial axis perpendicular to the first inertial axis, and a dotted line corresponding to a third inertial axis perpendicular to the first and second inertial axes. Specifically, the bold solid line in graph 3000 fluctuates between −1 and 1 during t=260 to t=320 seconds (shown in the graph at 3010 ), indicating that the crown is pointing in various directions, as would typically occur during different stages of a user's swims strokes when he/she is swimming. In contrast, during t=350 to t=370 seconds (shown in the graph at 3020 ), the bold solid line is mostly negative, which indicates that the crown is pointing downwards. This consistent downward direction is typical of a user walking, not swimming.
The above considerations can be combined using a decision tree classifier with the appropriate thresholds to decide if a given epoch corresponds to swim activity or not. Decisions on successive epochs can be chained together, for example of ten seconds or more, to improve the accuracy of detection. The specific choice of threshold values is usually tied to the discriminating feature used for classification and the hierarchical order in the decision tree. A typical threshold in logistic regression is 0.5. For example, feature values greater than or equal to 0.5 represent one style, while feature values less than 0.5 represent the others. This value can be adjusted to bias the accuracy in terms of either improving true positive detection or false positive rejection.
In some embodiments, additional sensory input, for example, heart rate measurements using, for example, a PPG sensor, can be used to improve the accuracy of the swimming determination. Further, speed estimation from GPS 410 also can be used to verify the swimming determination.
The swimming determination can be used by other features of wearable device 100 . For example, if wearable device 100 knows that the user is swimming, the wearable device 100 can accurately count laps, strokes, calories, and distance, as discussed in the respective applications referred to above, and incorporated by reference herein in their entirety. If the device knows that the user is not swimming, the device can disregard data from those time periods where the user is not swimming. Accordingly, the final workout data for a particular user will be more accurate, because it will reflect only those instances and periods where the user is actually swimming and will not include spurious strokes or laps where the user was not swimming, but was instead resting, or walking around the pool.
Determining Swimming Pool Length
When a user is swimming in a pool, there is often a need to know the length of the swimming pool. Information of the swimming pool length can be used to calculate the total distance a user swims and the energy expenditure associated with a swimming session. The pool length information, however, is not always readily available to users. Additionally, users may not be able to accurately estimate the pool length. FIG. 31 illustrates an example of users' estimation of a swimming pool length according to some embodiments of the present disclosure. In FIG. 31 , 48 users are asked to estimate the length of a 25-yard swimming pool, and only 17 users, or 35.4% of the 48 users, have the correct estimation. FIG. 31 also shows that there is a large range between the user's responses. Two users estimate as low as 10-meter, and one user estimates as high as 50-meter.
FIGS. 32A and 32B illustrate a process 3200 of determining a length of a swimming pool according to some embodiments of the present disclosure. In some embodiments, the process 3200 can be modified by, for example, having steps combined, divided, rearranged, changed, added, and/or removed.
The process 3200 starts at step 3205 . At step 3205 , a user can use wearable device 100 to indicate that he or she is about to start a swimming workout session at a swimming pool. The process 3200 then proceeds to step 3210 .
At step 3210 , wearable device 100 provides the user with one or more options to indicate the length of the swimming pool. In some embodiments, the user can choose among three options: standard length, custom length, and calibrate. In some embodiments, other suitable options can also be used. If the user chooses the standard length option, wearable device 100 can provide one or more lengths of a standard pool. Non-limiting examples of the standard length can include 25-yard, 25-meter, 33⅓-meter, or 50-meter. Other suitable standard length can also be used. In some embodiments, the user can choose to enter a standard length of the swimming pool. If the user chooses the custom length option, then the user can choose to enter a length of the swimming pool. The length can be based on information the user possesses or based on the user's estimation. If the user chooses the calibration option, then wearable device can prompt more options/instructions as described in following steps. The process 3200 then proceeds to step 3215 .
At step 3215 , wearable device 100 determines whether or not the user chooses the calibration option. If the user chooses the calibration option, the process 3200 proceeds to step 3220 . If the user does not choose the calibration option, the process 3200 proceeds to step 3235 .
At step 3220 , wearable device 100 encourages the user to take a short walk along the edge of the pool, where the edge of the pool is parallel with the swimming direction of the pool. In some embodiments, the user can choose to run, jump, or any other suitable activity along the edge of the pool. The process 3200 then proceeds to step 3225 .
At step 3225 , wearable device 100 can use pedometer 265 and GPS sensor 295 to estimate the length associated with the user's activity at step 3220 , and further estimate the length of the swimming pool. In some embodiments, pedometer 265 can count number of steps the user takes. The total distance of the user's activity can be calculated by multiplying the number of steps with a step length. In some embodiments, the step length can be a default value, a value previous customized for the user, or a value determined by GPS sensor 295 . In some embodiments, the total distance of the user's activity can be directly estimated by GPS sensor 295 . Once the total distance of the user's activity is estimated, the length of the swimming pool can be estimated based on a ratio between the length of the swimming pool and the length associated with the user's activity. For example, if the length associated with the user's activity at step 3220 is estimated to be 10-yard, and GPS sensor 295 further estimates the length of the swimming pool is 2.5 times of the length associated with the user's activity at step 3220 , then the length of the swimming pool can be estimated to be 25-yard. The process 3200 then proceeds to step 3230 .
At step 3230 , once a length of the swimming pool is identified, the user can proceed with the swimming workout session.
If the user does not choose to calibrate the length of the swimming pool, the user can start to swim, and wearable device 100 can still passively estimate the length of the swimming pool. At step 3235 , wearable device 100 can use default stroke length to estimate the length of the swimming pool. When the user is swimming, wearable device 100 can count the number of strokes the user has had within a short period. In some embodiments, wearable device 100 can only detect the strokes made by the arm wearing the wearable device 100 . In these embodiments, for every stroke detected, the user may make two strokes, and the stroke length can sometimes be adjusted accordingly. The total distance of the user's swimming activity within a short period can be calculated by multiplying the number of strokes with a stroke length. In some embodiments, the stroke length can be a default value, a value previous customized for the user, or a value determined by GPS sensor 295 . Once the total distance of the user's swimming activity within the short period is estimated, the length of the swimming pool can be estimated based on a ratio between the length of the swimming pool and the length associated with the user's swimming activity within the short period. For example, if the length associated with the user's swimming activity with the short period is estimated to be 10-yard, and GPS sensor 295 further estimates the length of the swimming pool is 2.5 times of the length associated with the user's swimming activity within the short period, then the length of the swimming pool can be estimated to be 25-yard. The process 3200 then proceeds to step 3240 .
At step 3240 , wearable device 100 determines whether or not the length estimated at step 3235 is close to a standard length of a swimming pool. In some embodiments, wearable device 100 can determine that an estimated length is close to a standard length if the difference is within 10% of the standard length. In some embodiments, other suitable threshold can be used. For example, if the pool length estimated at step 3240 is 20-yard, and the standard pool length is 25-yard, then wearable device 100 can determine whether or not the estimated length is close to the standard length based on the threshold. If the threshold is selected to be within 10% of the standard length, then an estimated length between 22.5-yard and 27.5-yard would be considered to be close enough, and the estimated 20-yard length would not be considered to be close enough. If the threshold is selected to be within 20% of the standard length, then an estimated length between 20-yard and 30-yard would be considered to be close enough, and the estimated 20-yard estimation would be considered to be close enough. If the estimated length is close to the standard length, the process 3200 proceeds to step 3245 . If the estimated length is not close to the standard length, the process 3200 proceeds to step 3250 .
At step 3245 , wearable device 100 suggests that the user uses the standard length identified to be close to the length of the swimming pool. For example, if the estimated length is 24.3-yard, then wearable device 100 may suggest that the pool length is actually 25-yard, which is a length of a standard pool. The process 3200 then proceeds to step 3230 .
At step 3250 , because the estimated length of the swimming pool is not close to a standard swimming pool, wearable device 100 suggests that the user uses calibration to get an estimation of the length of the swimming pool. If the user chooses to calibrate, then the process 3200 proceeds to step 3220 to start the calibration process.
FIG. 33 illustrates a process 3300 of determining a length of a swimming pool according to some embodiments of the present disclosure. In some embodiments, the process 3300 can be modified by, for example, having steps combined, divided, rearranged, changed, added, and/or removed.
The process 3300 starts at step 3305 . At step 3305 , a user can use wearable device 100 to indicate that he or she is about to start a swimming workout session at a swimming pool. The process 3300 then proceeds to step 3310 .
At step 3310 , wearable device 100 determines whether or not there is recent location information related to the swimming pool. In some embodiments, the location information includes a length of the swimming pool identified by other users. In some embodiments, wearable device 100 searches for the recent location information from storage media local at wearable device 100 . In some embodiments, wearable device 100 searches for the recent location information from a remote storage media. If there is recent location information available, the process 3300 proceeds to step 3315 . If there is no recent location information available, the process 3300 proceeds to step 3335 .
At step 3315 , wearable device 100 determines if the location of the swimming pool is known. If the location is known, the process 3300 proceeds to step 3320 .
At step 3320 , wearable device 100 determines whether or not there is sufficient history of the identified location of the swimming pool. In some embodiments, history can be pool lengths identified by other users. The threshold to determine the sufficiency of the history can be any suitable number. For example, if the threshold is set at 5, then if the length of the swimming pool has been identified by 5 or more users, then there would be sufficient history of the identified location of the swimming pool; if the length of the swimming pool has been identified by less than 5 users, then there would not be sufficient history of the identified location of the swimming pool. If there is sufficient history of the identified location of the swimming pool, the process 3300 proceeds to step 3325 . If there is no sufficient history of the identified location of the swimming pool, the process 3300 proceeds to step 3350 .
At step 3350 , wearable device 100 prompts the user to provide an estimation of the swimming pool, and the user's estimation will be added to the pool length table.
At step 3325 , since there is sufficient history of the identified location of the swimming pool, wearable device 100 looks up previously identified pool lengths associated with the swimming pool. The process 3300 then proceeds to step 3330 .
At step 3330 , wearable device 100 prompts the user with one or more choices of the length of the swimming pool. For example, if the swimming pool has been identified by 10 users as 25-yard and 3 users as 50-yard, then wearable device 100 can provide both choices to the user. In some embodiments, wearable device 100 can provide the length option identified by most users. In some embodiments, for each length option, wearable device 100 can also provide number of users identified such a length.
At step 3335 , wearable device 100 attempts to obtain location information of the swimming pool. In some embodiments, wearable device 100 obtains location information through GPS sensor 295 .
At step 3340 , wearable device 100 determines whether or not location information of the swimming pool is available. If the information is available, the process 3300 proceeds to step 3315 . If the information is not available, the process 3300 proceeds to step 3345 .
At step 3345 , wearable device 100 determines whether or not the user's swimming workout session has ended. If the user's swimming session has ended, the process 3300 proceeds to step 3355 . If the user's swimming session has not ended, the process 3300 proceeds to step 3335 to continue to obtain location information.
At step 3355 , since the user has already ended the swimming workout, wearable device 100 can adjust the timeout period and/or GPS frequency to save power.
Although the present disclosure has been described and illustrated in the foregoing exemplary embodiments, it is understood that the present disclosure has been made only by way of example, and that numerous changes in the details of implementation of the present disclosure may be made without departing from the spirit and scope of the present disclosure, which is limited only by the claims which follow.