Scoring Template

The Scoring Template is a JSON file that defines how to score a given dataset. The JSON schema for the Template can be found on the Schema Reference page.

Uploading a Template

1. Open the Performaar UI in a web browser and log in.
2. Navigate to Settings > Templates.
3. Click the + New Template button.
4. Select Scoring from the dropdown menu.
5. Add a name (optional).
6. Click Choose File and select a Scoring Template .json file.
7. Click Upload

Core Concepts

Literals

Literals are used to define actual values, usually as part of a comparison operation. They encode what kind of value it represented (e.g. time, length, angle, etc.) and in what units (e.g. seconds, feet, degrees, etc.). For numerical literals, the underlying type is always a double-precision floating point value.

Examples

• 2,000 feet

  {
    "literal": {
      "feet": 2000.0
    }
  }

• 34.5 seconds

  {
    "literal": {
      "seconds": 34.5
    }
  }

Conditions

Conditions are used to compare a Dimension against a target value. The target can be a Literal or the same Dimension in a Reference Entity, which will be resolved to a Literal when the scoring is calculated. Only values that resolve to same kind of Literal (e.g. time, length, angle, etc.) can be compared. Comparing Literals of different types will result in an error.

Examples

• The pitch Dimension is greater than or equal to 5.0 degrees

  {
    "pitch": {
      "eq": {
        "target": {
          "literal": {
            "degrees": 5.0
          }
        }
      }
    }
  }

• The altitude Dimension is at 2000.0 feet, +/- 100.0 feet (inclusive)

  {
    "altitude": {
      "near": {
        "target": {
          "literal": {
            "feet": 2000.0
          }
        },
        "within": {
          "startDelta": {
            "inclusive": {
              "feet": -100.0
            }
          },
          "endDelta": {
            "inclusive": {
              "feet": 100.0
            }
          }
        }
      }
    }
  }

• The position dimension is less than 150 meters away from the partner reference entity

  {
    "position": {
      "distance": {
        "to": {
          "entity": {
            "reference": "partner"
          }
        },
        "condition": {
          "lte": {
            "target": {
              "literal": {
                "meters": 150.0
              }
            }
          }
        }
      }
    }
  }

Template Sections

Reference Entities

This section can be used to define entities, either internal to the recording being scored or from external files, that will be used in the rest of the template for comparison and scoring. The object keys defined here will be used to reference the entities in the rest of the Template file.

In both the internal and external cases, a filter must be provided to locate the desired entity. The scoring API will return an error if no entities matching the filter are found. If multiple matching entities are found, one will be chosen non-deterministically.

For external entities, the path must be provided relative to Performaar's media storage root directory (i.e. the place where all of Performaar's recordings are stored). Additionally, the file must be in a format Performaar supports.

Examples

• Define an exemplar reference entity from an external DIS file. Specifically, looking for an entity with ID 12345 in the external file.

  {
    "exemplar": {
      "external": {
        "path": "exemplar/data.dis",
        "kind": "Dis",
        "entity": {
          "id": {
            "fuzzyEq": "12345"
          }
        }
      }
    }
  }

• Define an partner internal reference entity, looking for an entity with name/marking 'SIERRA_22' in the recording being scored.

{
  "exemplar": {
    "internal": {
      "entity": {
        "displayName": {
          "fuzzyEq": "SIERRA_22"
        }
      }
    }
  }
}

Dimensions

Dimensions are the core values of an entity that are used tested against the rubrics. They are derived from a core property, with optionally some additional simple operations. Additionally, dimensions can be specific to a particular part (entry, exit or steady-state) of a Segment (see below).

Examples

• Define a verticalSpeed Dimension based on the yaw of the entity.

  {
    "verticalSpeed": {
      "name": "Vertical Speed",
      "property": "entityLinearVelocityVerticalSpeed"
    }
  }

• Define a avgSteadyStateAltitude Dimension based on the average MSL altitude of an entity during the steady-state of a segment

  {
    "avgSteadyStateAltitude": {
      "name": "Altitude (Average at Steady-State)",
      "property": "entityPositionAltitudeMsl",
      "partition": "steadyState",
      "operations": [
        "mean"
      ]
    }
  }

Segments

The Segments section is used to define ways to partition the entire data set into sets of contiguous points containing only the relevant data. For example, if you want to assess an aircraft climbing between two altitudes, you may only want to consider parts of the entire data where the entity's vertical speed is greater than 1 meter per second.

The condition for a Segment can be null, in which case it will contain the entire data.

Segments can also be further partitioned into entry, exit and steady-state sections based on a particular Dimension. Note that the Dimension used to partition a Segment cannot itself be specific to a partition.

Examples

• Define a climb segment when the entity's verticalSpeed Dimension is greater than 1 meter per second.

  {
    "climb": {
      "name": "Climbing",
      "condition": {
        "when": {
          "check": {
            "dimension": "verticalSpeed",
            "operation": {
              "gte": {
                "target": {
                  "literal": {
                    "metersPerSecond": 1.0
                  }
                }
              }
            }
          }
        }
      }
    }
  }

• Define a route segment that encompasses the all the data.

{
  "route": {
    "name": "Route"
  }
}

Assessments

Assessments define how Segments are graded. They contain a list of rubric and potential variations for those rubrics. Assessments also define how grades are aggregated (if at all) for each dimension in a segment, and for a segment as a whole.

Rubrics

Each rubric defines a grade and the condition required for each relevant Dimension to achieve that grade. The list of rubrics within an assessment should be in order of highest grade to lowest. Additionally, the last rubric in this list must be a catch-all for dimension values that don't fall into any of the previous rubrics.

Variations

Variations provide ways to alter the condition for all Dimensions in each rubric in the assessment. They can be used to relax or constrict the requirements for achieving a particular grade based on external condition, like weather, time of day and so on.

Example

A gentleClimb assessment that defines three grades for the climb segment. The grade will be Perfect is when the verticalSpeed is exactly 1.2 meters per second, Acceptable when it's between 1.0 and 1.4 meters per second, and Unacceptable otherwise.

Additionally, a variation is provided for low visibility conditions that increases the range for each grade by 0.25 meters per second. When this variation is applied, the grade will be Perfect if the vertical speed is 1.2 +/- 0.25 meters per second, Acceptable when it's between 0.75 and 1.65 meters per second, and Unacceptable otherwise.

{
  "gentleClimb": {
    "name": "Gentle Climb",
    "segment": "climb",
    "rubric": {
      "base": [
        {
          "name": "Perfect",
          "color": "#0F0",
          "condition": {
            "verticalSpeed": {
              "eq": {
                "target": {
                  "literal": {
                    "metersPerSecond": 1.2
                  }
                }
              }
            }
          }
        },
        {
          "name": "Acceptable",
          "color": "#FF0",
          "condition": {
            "verticalSpeed": {
              "near": {
                "target": {
                  "literal": {
                    "metersPerSecond": 1.2
                  }
                },
                "within": {
                  "startDelta": {
                    "inclusive": {
                      "metersPerSecond": -0.2
                    }
                  },
                  "endDelta": {
                    "inclusive": {
                      "metersPerSecond": 0.2
                    }
                  }
                }
              }
            }
          }
        },
        {
          "name": "Unacceptable",
          "color": "#F00",
          "condition": {
            "verticalSpeed": {
              "not": {
                "near": {
                  "target": {
                    "literal": {
                      "metersPerSecond": 1.2
                    }
                  },
                  "within": {
                    "startDelta": {
                      "inclusive": {
                        "metersPerSecond": -0.2
                      }
                    },
                    "endDelta": {
                      "inclusive": {
                        "metersPerSecond": 0.2
                      }
                    }
                  }
                }
              }
            }
          }
        }
      ],
      "variations": [
        {
          "name": "Low Visibility",
          "dimensions": {
            "verticalSpeed": {
              "delta": {
                "metersPerSecond": 0.25
              }
            }
          }
        }
      ],
      "gradeAggregation": {
        "dimension": "exact",
        "segment": "lowest"
      }
    }
  }
}

Complete Example

Below is the complete example of a Scoring Template file that scores the vertical speed.

Vertical Speed Scoring JSON

{
  "name": "Scoring Template",
  "dimensions": {
  "verticalSpeed": {
    "name": "Vertical Speed",
    "property": "entityLinearVelocityVerticalSpeed"
  }
},
  "segments": {
    "climb": {
      "name": "Climb",
      "condition": {
        "when": {
          "check": {
            "dimension": "verticalSpeed",
            "operation": {
              "gte": {
                "target": {
                  "literal": {
                    "metersPerSecond": 1.0
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "assessments": {
    "gentleClimb": {
      "name": "Gentle Climb",
      "segment": "climb",
      "rubric": {
        "base": [
          {
            "name": "Perfect",
            "color": "#0F0",
            "condition": {
              "verticalSpeed": {
                "eq": {
                  "target": {
                    "literal": {
                      "metersPerSecond": 1.2
                    }
                  }
                }
              }
            }
          },
          {
            "name": "Acceptable",
            "color": "#FF0",
            "condition": {
              "verticalSpeed": {
                "near": {
                  "target": {
                    "literal": {
                      "metersPerSecond": 1.2
                    }
                  },
                  "within": {
                    "startDelta": {
                      "inclusive": {
                        "metersPerSecond": -0.2
                      }
                    },
                    "endDelta": {
                      "inclusive": {
                        "metersPerSecond": 0.2
                      }
                    }
                  }
                }
              }
            }
          },
          {
            "name": "Unacceptable",
            "color": "#F00",
            "condition": {
              "verticalSpeed": {
                "not": {
                  "near": {
                    "target": {
                      "literal": {
                        "metersPerSecond": 1.2
                      }
                    },
                    "within": {
                      "startDelta": {
                        "inclusive": {
                          "metersPerSecond": -0.2
                        }
                      },
                      "endDelta": {
                        "inclusive": {
                          "metersPerSecond": 0.2
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        ],
        "variations": [
          {
            "name": "Low Visibility",
            "dimensions": {
              "verticalSpeed": {
                "delta": {
                  "metersPerSecond": 0.25
                }
              }
            }
          }
        ]
      },
      "gradeAggregation": {
        "dimension": "exact",
        "segment": "lowest"
      }
    }
  }
}

📄️Schema Reference