Feedback

QuickPose provides human readable feedback to correct a user's posture:

Inside Box	Body Position	Joint Visibility	Exercise Rule

1. Inside Box gives guidance to stand in a certain area e.g."Move whole body into the box"
2. Body Position gives high level feedback on the required body pose e.g."Stand facing camera"
3. Joint Visiblity gives joint specific feedback if it's out of view. e.g."Move your left arm into view"
4. Exercise Rule gives guidance during an exercise if a rule is broken. e.g. "Lift your knees"

If feedback is required, to say capture accurate measurements, then QuickPose will stop highlighting the user, so it's recommend that your display the returned feedback to explain what your users' should do next.

Annotations

The annotated lines drawn over a user is a fundamental form of feedback See our Annotations and Styling page for more details.

Basic Implementation

To show results you'll need to modify your view ZStack, which is we assume is setup as described in the Getting Started Guide:

ZStack(alignment: .top) {
    QuickPoseCameraView(useFrontCamera: true, delegate: quickPose)
    QuickPoseOverlayView(overlayImage: $overlayImage)
}

The basic implementation will require displaying some text to the screen, start with declaring this value in your swiftui view.

@State private var feedbackText: String? = nil

And show this feedback text as an overlay to the view in your branding.

ZStack(alignment: .top) {
    QuickPoseCameraView(useFrontCamera: true, delegate: quickPose)
    QuickPoseOverlayView(overlayImage: $overlayImage)
}
.overlay(alignment: .center) {
    if let feedbackText = feedbackText {
        Text(feedbackText)
            .font(.system(size: 26, weight: .semibold)).foregroundColor(.white).multilineTextAlignment(.center)
            .padding(16)
            .background(RoundedRectangle(cornerRadius: 8).foregroundColor(Color("AccentColor").opacity(0.8)))
            .padding(.bottom, 40)
    }
}

Note the above use of alignment in .overlay(alignment: .center), you can modify this to move the overlay around easily to say the bottom: .overlay(alignment: .bottom).

Next set the string value, in the onFrame callback, referencing the feature from the feedback dictionary.

quickPose.start(features: [feature], onFrame: { status, image, features, feedback, landmarks in
    if case .success(_,_) = status {
        ...
        if let feedback = feedback[feature] {
            feedbackText = feedback.displayString
        } else {
            feedbackText = nil
        }
    }

Caution: Feature Feedbacks are not returned when a user not visible

If the user is not in view, the onFrame callback will return a status of .noPersonFound. Implement this functionality in your own app by checking the status:

quickPose.start(features: [feature], onFrame: { status, image, features, feedback, landmarks in
    guard case .noPersonFound = status else { feedbackText = "Stand in view"; return } // user not in view, show branded 'Come back in view' type message.

or, consider a switch statement to capture all the scenarios

quickPose.start(features: [feature], onFrame: { status, image, features, feedback, landmarks in
    switch status {
    case .success(_, _):
        ...
    case .noPersonFound:
        feedbackText = "Stand in view";
    case .sdkValidationError:
        feedbackText = "Be back soon";
    }

Body Position

With the above integration code, your UI will start showing body position feedback for the relevant features such as The Push up Counter

To review or customise the messaging, this is the Body Position definition:

case body(feedback: BodyFeedback, isRequired: Bool)

And the provided strings:

case standFacing = "Stand facing camera"
case standSideOn = "Stand side on"
case standFacingOrSide = "Stand up"
case floorFacing = "Get on your front, facing camera"
case floorSideOnWithBackOffTheGround = "Get on your front, side on"
case floorWithBackOffTheGround = "Get on floor"
case floorSideOnWithBackOnTheGround = "Get on your back, side on"

To customize these messages, extend the PoseFeedback enum as shown:

extension PoseFeedback {
    public var customDisplayString: String {
        switch self {
        case .body(let feedback, _):
            if feedback == .standFacing {
                return "Stand facing kiosk" // customized message just for .standFacing scenario
            } else {
                return displayString // for other cases use default
            }
        default:
            return displayString // for other cases use default
        }
    }
}

Internationalization

Internationalize body position feedback by using our display string as a key to NSLocalizedString

extension PoseFeedback {
    public var customDisplayString: String {
        switch self {
        case .body(_, _):
            return NSLocalizedString(displayString) // look up your translation
        default:
            return displayString // for other cases use default
        }
    }
}

And update your onFrame callback to reference your new custom implementation:

quickPose.start(features: [feature], onFrame: { status, image, features, feedback, landmarks in
    if case .success(_,_) = status {
        overlayImage = image
        if let feedback = feedback[feature] {
            feedbackText = feedback.customDisplayString
        } else {
            feedbackText = nil
        }
    }

Joint Position

With the above integration code, your UI will start showing joint feedback for the relevant features such as The Lunge Counter and The Squats Counter

These two examples demonstrate the different levels of joint granularity:

1. Joint Feedback for a specific area e.g. Move your left arm
2. Joint Group Feedback for a larger area e.g. Lower your hips

info

Whilst we aim to provide side and joint name:

"Move your left arm into view"

There are some cases where we can't provide a joint, this is typically to reduce the jitteryness of messages as each frame can return a less specific message e.g:

"Move your upper body into view"

To make these messages easier we've generalized them into the following format:

Action + your joint + in an optional direction

This allows us to implement exercise rules using the same approach:

These are the swift definitions:

case joint(action: ActionChange, joint: QuickPose.Landmarks.Body, direction: DirectionChange?, isRequired: Bool)
case group(action: ActionChange, group: QuickPose.Landmarks.Group, direction: DirectionChange?, isRequired: Bool)

And the provided strings for action change and direction change:

@frozen public enum ActionChange: String {
    case move = "move"
    case lean = "lean"
    case straighten = "straighten"
    case lower = "lower"
    case place = "place"
    case bend = "bend"
}

@frozen public enum DirectionChange: String {
    case up = "up"
    case down = "down"
    case forward  = "forward"
    case back = "back"
    case apart = "apart"
    case intoView = "into view"
    case onFloor  = "on floor"
}

Our implementation for Joint Positioning requires implementing both .joint and .group cases in order to create a consistent experience.

Our internal implementation groups the different components together like so:

case .joint(let action, let joint, let direction, _):
    let jointName = QuickPose.Landmarks.Body.toHumanReadableString(joint) ?? ""
    let direction = direction?.rawValue ?? ""
    return "\(action.rawValue.capitalized) \(jointName) \(direction)"

case .group(let action, let group, let direction, _):
    let direction = direction?.rawValue ?? ""
    return "\(action.rawValue.capitalized) \(group.rawValue.lowercased()) \(direction)" 

This handles the combination, but also the capitalization, de-capitalization, and the optional direction parameter. These must be handled when implementated your custom versions of the strings:

extension PoseFeedback {
    public var customDisplayString: String {
        switch self {
        case .joint(let action, let joint, let direction, _):
            let jointName = QuickPose.Landmarks.Body.toHumanReadableString(joint) ?? ""
            let direction = direction?.rawValue ?? ""
            var newAction = action.rawValue
            if action == .lean {
                newAction = "Angle"
            }
            return "\(newAction.capitalized) \(jointName) \(direction)"
        
        case .group(let action, let group, let direction, _):
            let direction = direction?.rawValue ?? ""
            var newAction = action.rawValue
            if action == .lean {
                newAction = "Angle"
            }
            return "\(newAction.capitalized) \(group.rawValue.lowercased()) \(direction)"
           
        default:
            return displayString // leave body position to default implementation
        }
    }
}

Internationalization

Internationalize joint position feedback by using our display string as a key to NSLocalizedString Providing messages as components provides more options for Internationalization. As you can choose to translate the components independently, reducing the impact of missed translations in newer versions of the SDK.

extension PoseFeedback {
    public var customDisplayString: String {
        switch self {
        case .joint(let action, let joint, let direction, _):
            let jointName = QuickPose.Landmarks.Body.toHumanReadableString(joint) ?? ""
            let direction = direction?.rawValue ?? ""
            return "\(NSLocalizedString(action.rawValue).capitalized) \(NSLocalizedString(jointName)) \(NSLocalizedString(direction))" 

        case .group(let action, let group, let direction, _):
            let direction = direction?.rawValue ?? ""
            return "\(NSLocalizedString(action.rawValue).capitalized) \(NSLocalizedString(group.rawValue).lowercased()) \(NSLocalizedString(direction))" 

        default:
            return displayString // leave body position to default implementation
        }
    }
}