Bundle Review Checklist

This checklist was written for two main purposes:
    For a more streamlined review process.
    For bundle developers to ensure they have written a bundle following best practices.

Check assessment identifiers/objective assessment identifiers & namespaces
  • Ensure identifiers are following  governance rules. 
  • TLDR:
  • Capture true hierarchy of the assessment using assessment/objective assessment IDs.
  • Assessment ID should be the highest level at which scores are captured.
  • Often times captured at the subject level, sometimes composite, sometimes something totally different (like form level for MAP Fluency).
  • Title + subject (unless single subject) is a decent rule to follow as a starting point.
  • Do not include vendor in the ID.
  • Namespaces should link to the underlying vendor of that assessment.
  • Ensure an assessment family is included in the data model.
Check studentAssessmentIdentifier to ensure it acurately captures a unique student assessment record
  • If the assessment file already contains a unique identifier for a student record, use that as the studentAssessmentIdentifier .
  • Otherwise, ensure the combination of columns used to create one does result in a unique ID.
Check descriptors & namespaces
  • General guidance:
  • For assessment-specific descriptors (this includes but isn't limited to assessmentPeriods, assessmentReportingMethods, performanceLevel): the namespace should link to the underlying vendor of that assessment (e.x. renaissance for STAR, collegeboard for PSAT/SAT, etc). The codeValue should match the original values (whether that be score names, periods, etc) from the vendor file.
  • These descriptors will be sent to the ODS with the studentAssessment/assessment/objectiveAssessment results.
  • For non-assessment-specific descriptors (this includes but isn't limited to gradeLevels, academicSubjects, etc. - essentially any descriptors that are also used by resources other than assessment-related resources): the namespace/code values should almost always be the default Ed-Fi  namespace/codeValues  .
  • The ONLY time when this may not be true is if the assessment is STRICTLY administered in a specific local context, aka is a state-specific assessment
  • Ensure these default values are reasonable.
  • When VALIDATING your results, you may need to use  project composition  to override these with the local namespace/codeValues - see  this doc  for more information on that process.
  • NO descriptors namespaces should be hard-coded in any template because it is then harder to override
  • resultDataTypeDescriptor namespaces should be filled in via a parameter: DESCRIPTOR_NAMESPACE - this parameter should not be used otherwise - see  this as an example  of this param.
  • Make sure things like subjects seem reasonable as defaults
  • Also check namespaces
Check that all possible scores are mapped
  • The general guidance is that we should send as much information from the source file as possible into Ed-Fi given the wide range of analytic use-cases for assessment data.
  • We generally do not send student demographics as there is no spot in the Ed-Fi model for them and sending them as score results is not best practice.
Check for anything that could be considered invalid json
  • Trailing commas, etc.
  • Ensure descriptor values are in proper json format (not just list of strings).
Check to ensure that we are properly handling missing values/not sending empty strings
  • Ed-Fi will not accept an empty string.
  • All score/PLs should be contained in a list and looped over to ensure they are not null.
  • This should happen at the top of the studentAssessment.jsont file so that we can easily update if necessary.
  • ^ true for scores, performance levels, and objective assessments/corresponding scores/PLs.
#this should be done at the top of the studentAssessment.jsont file
"scoreResults": [
    {%- set possible_scores = [
        [scaleScore, "Scale Score", "Integer"],
        [sem, "SEM", "Integer"]
    ] -%}

    {%- set scores = [] -%}
    {%- for score in possible_scores -%}
      {%- if score[0] is not none and score[0] | length -%}
        {%- set _ = scores.append(score) -%}
      {%- endif -%}
    {%- endfor -%}

#this part must be part of the actual template
    {% for score in scores %}
      {
        "assessmentReportingMethodDescriptor": "{{namespace}}/AssessmentReportingMethodDescriptor#{{score[1]}}",
        "resultDatatypeTypeDescriptor": "uri://ed-fi.org/ResultDatatypeTypeDescriptor#{{score[2]}}",
        "result": "{{score[0]}}"
      } {% if not loop.last %},{% endif %}
    {% endfor %}
  ]
Check that the student ID package would be compatible
  • Empty initial transformation node
transformations:
  input:
    source: $sources.input
    operations: []
  • Parameter defaults
  • STUDENT_ID_NAME
  • Should default to 'edFi_studentUniqueID', which is the column added by the apply_xwalk package of student ID xwalking feature.
Check logic to transform the files, especially if they are wide
  • Goal is to ensure we won't unintentionally run into memory issues, performance issues, etc.
Check grade level mapping
  • The general guidance is to go against the definition of whenAssessedGradeLevel by including the tested grade in that property (instead of the enrolled grade, which is what the definition suggests. Tested grade does not necessarily = enrolled grade).
  • Reasoning behind this: tested grade is typically more useful analytically, and enrolled grade can be found in the studentEducationOrganizationAssociation resource anyway.
  • If both tested and enrolled grades exist in the underlying assessment data, send the enrolled grade as a separate scoreResult.
Check for correct usage of the studentAssessmentEducationOrganizationAssociation resource
  • Ensure that the associationType is correct based on the documentation of the assessment.
  • We prefer using this resource instead of the reportedSchoolAssociation property of the studentAssessment resource because this resource includes the associationType property.
Check for xwalk values that could change yearly, ask for the writer to check if static
  • (if they do change and we don't realize, we would have to reload data, which wouldn't be fun).
  • This includes but is not limited to: performance levels based on thresholds, labels, etc. Example  here  of a confirmed static xwalk.
  • Ideally, our bundles do little mappings in general because districts will want to see the original values, and additional mapping for viz can occur downstream.
Check that a sample anonymized file is included in the bundle
  • Easier testing when this exists.
  • Find real file, subset to ~5 rows, and remove ALL PII.
  • district/school/staff/student names & IDs.
  • district/school so we don't know the original source of the file.
  • ENSURE ANONYMIZED.
Check that a single earthmover.yaml and lightbeam.yaml file exist in the bundle
  • If multiple earthmover config files exist for a particular bundle, add a single file called earthmover.yaml that uses composition to combine, with proper parameterization. See here for an example.
Check that the .README is consistent with other bundles and includes an example command that can be successfully run against the sample file