Introduction

What is earthmover?

•  earthmover  is a python-based, CLI tool for transforming collections of tabular source data into a variety of text-based data formats via YAML configuration and Jinja templates.

What is an earthmover bundle?

• Bundles are pre-built data mappings for converting various data formats to Ed-Fi format using earthmover. They consist of a folder with CSV seed data, JSON template files, and a config.yaml with sources, transformations, and destinations.
• EA maintains a  public github repository  of bundles.

What is the structure of the Ed-Fi assessment domain?

• The  assessment domain of Ed-Fi  consists of multiple entities, including (but not limited to): Assessment, ObjectiveAssessment, StudentAssessment

Why is integrating assessment data different than integrating other data into Ed-Fi?

.1The models are flexible, especially in regards to student assessment results
.2There is no single, standard source of assessment data and instead will be populated by various vendors

.1Vendors can make vastly different decisions about how to populate the assessment-related resources
.2Downstream processes need to handle this flexibility

Development steps:

Review assessment data source(s)

.1Ensure that the structure of the assessment data is consistent

• The goal for creating a new bundle is that it can be used by every district across the state/country that administers that particular assessment. The source should be a  consistent file structure  that is ideally directly from the original assessment vendor.

.2Determine if there is a single source of student assessment results

• Sometimes assessment vendors split their student assessment results into multiple files
• E.g. Renaissance splits their Star assessment results into multiple files, so all are included as  sources in the bundle  and then  joined together 

.3Review any available data dictionaries

• It can be very difficult to correctly transform data from the original structure into the Ed-Fi model if the columns of the source data are not defined
• This will help in determining which columns map to specific Ed-Fi properties

Determine the assessment identifier & namespace

• Assessment Title
• Subject

• What should be included in the assessmentTitle?
• There is often no column with this information in the student data and there might not be a clear answer from the Vendor.
• Title can capture a multitude of constructs, and determining that title requires deeply understanding the assessment:
• For DIBELS, the titles of DIBELS Next and DIBELS 8^th capture a different version of the assessment
• For MAP Reading Fluency, the titles capture the various forms of the assessments (Foundational Skills, Adaptive Oral Reading, etc)
• How should we handle assessments that cannot fit into this structure?
• As we mentioned, this structure can likely work sufficiently well for most assessments, but there will be assessments that cannot easily match this structure, and attempting to do so may result in lost information.
• Reaching out to EA or contacting someone with experience using the data are solid next steps.

Establish the hierarchy of the assessment

.1Are there subscores results?

• These could map to objective assessments. There is a separate Ed-Fi entity for  objective assessments  and the  student objective assessment  results are housed within the StudentAssessment entity.

.2Are there item-level results?
.aThere is also a separate entity for  assessment items  and again, the  student assessment item  results are housed within the StudentAssessment entity.
.3Are there any additional nested structures (like subsections within subtests)?

• In the Ed-Fi assessment domain, you can associate certain objective assessments to a 'parent' allowing for a hierarchy within objective assessments.

• Overall assessment results (Star Reading assessment, Star Math Assessment, Star Early Literacy assessments)
•  Domain  results (E.g. Literature; Informational Text; Numbers and Operations; etc.)
• Skill Areas (E.g. Literature: Plot; Literature: Setting; Literature: Character; etc.)
• Standards (Results split by domain, relating to state learning standards)

Governance Review

Copy the assessment bundle template

Investigate student IDs



Graphical depiction of the 'Compute Match Rates' package logic.

Graphical depiction of the 'Compute Match Rates' package logic.



Graphical depiction of the template bundle with 'apply_xwalk' package logic.

Graphical depiction of the template bundle with 'apply_xwalk' package logic.

Determine bootstrap/metadata source(s)

.1 Assessment  (an Ed-Fi entity, like StudentAssessment)
.2 ObjectiveAssessment  (an Ed-Fi entity, like StudentAssessment)
.3 AssessmentReportingMethod  (an Ed-Fi descriptor)
.4 PerformanceLevel  (an Ed-Fi descriptor)

• uri://www.nwea.org/map/AssessmentReportingMethodDescriptor#RIT Scale Score
• uri://dibels.uoregon.edu/assessment/dibels/AssessmentReportingMethodDescriptor#Composite Score

These RIT scales are stable, equal interval scales that use individual item difficulty values to measure student achievement independent of grade level (that is, across grades). "Equal interval" means that the difference between scores is the same regardless of whether a student is at the top, bottom, or middle of the RIT scale. "Stable" means that the scores on the same scale from different students, or from the same students at different times, can be directly compared, even though different sets of test items are administered. A RIT score also has the same meaning regardless of the grade or age of the student.

• NWEA MAP: 'Fall-To-Fall Projected Growth'
• Renaissance STAR: 'Normal Curve Equivalent'

Start populating the earthmover config

.1Adjust the  config  section as needed

• For example, certain additional parameter_defaults may be needed

.2Build  definitions  as needed

• Examples include  year mapping  and id hashing

.3Add  sources , which can include seeds

• Note: sources can be listed as optional

Build out transformations in earthmover config

.1Are there Ed-Fi requirements that will require the reformation of columns?

• The  Ed-Fi API documentation  is an incredibly useful resource in determining requirements, such as data types and character limits. The transformation section is where those requirements can be addressed and handled.

.2Could the files include rows outside of a single district & year?

• If so, filters should be built into the template as a safeguard. The earthmover output should be ready to send to an ODS, and most ODS' are separated by district & year. As an example, NWEA sometimes sends results across districts & years in a single file if the scope of the data agreement for a particular external vendor extends beyond a single district.

.3Do sources need to be joined?

• If information that is relevant for a single entity exists across multiple files, those sources will need to be joined together. This is the case for Renaissance Star assessments, where overall assessment results are contained in one file and objective assessment results are contained in other files. Student objective assessment results are captured within the StudentAssessment entity, so those scores must be  joined together .

.4Does the output grain match the grain of the Ed-Fi entity?

• Grain can be hard to conceptualize, but  dbt's definition  is probably the most straightforward:

Grain is the combination of columns at which records in a table are unique.

.aIn the case of the StudentAssessment entity, the combination of columns at which records are unique are: AssessmentIdentifier, Namespace, StudentAssessmentIdentifier, and StudentUniqueId. This means that the student assessment output of earthmover should be a single row for each administration of an assessment for a particular student. This will not always match the grain of the assessment results source file. Continuing on the above example for Renaissance Star assessments, the join between the overall and objective assessment files would result in a grain at the objective assessment level instead of the required student assessment level. To handle this, EA wrote a  transformation that grouped the objective assessment results by the student assessment identifier  which resulted in the correct grain that included a column of aggregated student objective assessment results that could slot directly into that section of the template.

Set destinations in earthmover config

Fill out templates

.1EA typically sends information that does not match a particular Ed-Fi property as a score result. This can include subjects, test names, etc.

• Note: each will need a corresponding AssessmentReportingMethod descriptor value

.2

.3Two Ed-Fi requirements exist that can complicate this process:
.aEmpty strings cannot be sent as a performance level or score result.
.bAt least 1 performance level or score result must be included in a student assessment/student objective assessment record, or else those properties must be skipped entirely.

"studentObjectiveAssessments": [
 {%- for obj in obj_assess -%}
  {
    "objectiveAssessmentReference": {
    "assessmentIdentifier": "assessment_identifier",
    "identificationCode": "{{obj}}",
    "namespace": "namespace"
  },
 # this could be a loop
 "performanceLevels": [
  {
    "assessmentReportingMethodDescriptor": "namespace/AssessmentReportingMethodDescriptor#perf_level_1",
    "performanceLevelDescriptor": "namespace/PerformanceLevelDescriptor#{{perf_level_1_result}}"
  },
  {
    "assessmentReportingMethodDescriptor": "namespace/AssessmentReportingMethodDescriptor#perf_level_2",
    "performanceLevelDescriptor": "namespace/PerformanceLevelDescriptor#{{perf_level_2_result}}"
  }
  ],
  # this could be a loop
  "scoreResults": [
    {
      "assessmentReportingMethodDescriptor": "namespace/AssessmentReportingMethodDescriptor#scale_score",
      "resultDatatypeTypeDescriptor": "namespace/ResultDatatypeTypeDescriptor#Integer",
      "result": "{{obj_scale_score_result}}"
    },
    {
    "assessmentReportingMethodDescriptor": "namespace/AssessmentReportingMethodDescriptor#sem", 
    "resultDatatypeTypeDescriptor": "namespace/ResultDatatypeTypeDescriptor#Integer", 
    "result": "{{obj_sem_result}}" 
    }
  ] 
  }{%- if not loop.last -%}, 
  {%- else %}{% endif %}{%- endfor -%} 
 ]

.1If all of the PL/score results are null, the entire objective assessment must be skipped
.2If any of the PL/score results are null, that particular scoreResult or performanceLevel must be skipped

{% if perf_level_1_result != '' or perf_level_2_result != '' or obj_scale_score_result != '' or obj_sem_result != '' %}

"performanceLevels": [
 {%- set possible_pls= [
        ["perf_level_1", perf_level_1_result],
        ["perf_level_2", perf_level_2_result]
    ] -%}
 {% set pls_present = [] %}
 {%- for pl in pls_present-%}
    {%- if pl[1] is not none and pl[1] | length -%}
      {%- set _ = pls_present.append(pl) -%}
    {%- endif -%}
  {%- endfor -%}
 
 {%- for pl in pls_present -%}
  {
    "assessmentReportingMethodDescriptor": "namespace/AssessmentReportingMethodDescriptor#{{pl[0]}}",
    "performanceLevelDescriptor": "namespace/PerformanceLevelDescriptor#{{pl[1]}}"
  }
 {%- if not loop.last -%}, 
 {%- else %}{% endif %}{%- endfor -%} 
]

Flexibility across years/data standard versions

{% if ${API_YEAR} >= 2022 %}
# standardize score names so we can easily convert from wide to long
"mathematics_reporting_category_1_scores": mathematics_reporting_category_1_raw_score
{% endif %}
{% if ${API_YEAR} >= 2023 %}
# starting in 2023, STAAR began reporting "reading" as "reading_language_arts", but we'll rename here to keep consistency in the convert wide-long code below
"reading_language_arts_reporting_category_1_scores": reading_reporting_category_1_raw_score
{% elif ${API_YEAR} == 2022 %}
"reading_reporting_category_1_scores": reading_reporting_category_1_raw_score
{% endif %}

Test earthmover code

earthmover run -c path/to/earthmover.yaml

earthmover run -c path/to/earthmover.yaml -p '{"OUTPUT_DIR":"path/to/my/output/dir"}'

Review  lightbeam  config