SC - Historical Assessment Loading

Rosters:

You have been provided roster files split by both district code and year. You will need to compare assessment data also by district and year to ensure you are not mixing and matching local student IDs.


Project Composition:

For assessments that are not state-specific (national interim assessments) we write bundles using Ed-Fi default descriptors. However, certain states and districts across the country use their own custom descriptors and their ODS' do not contain the Ed-Fi defaults. If you were to use your bundles with the Ed-Fi defaults, you will run into numerous validation errors. To resolve this issue, we developed a feature in earthmover called  project composition:  
An earthmover project can import and build upon other earthmover projects by importing them as packages, similar to the concept of dbt packages. When a project uses a package, any elements of the package can be overwritten by the project. This allows you to use majority of the code from a package and specify only the necessary changes in the project.
Utilizing project composition to override default descriptors is simple. First, you will create a new earthmover.yaml file (not to be saved in the bundles repo, because this is for local use) that called the original bundle as a package. Then, you will be able to override anything from the original bundle and earthmover will do the work of stitching your local changes with the original code. This is exactly why we contain most of our descriptors within seed files, because then you can simply override the seed sources with a local version of the seeds:

Original bundle seed file example:
assessmentIdentifier
gradeLevelDescriptor
fake_assessment
 edfi.org/gradeLevelDescriptor#Kindergarten 
fake_assessment
 edfi.org/gradeLevelDescriptor#First  Grade
Local seed file example:
assessmentIdentifier
gradeLevelDescriptor
fake_assessment
 sc.ed.gov/gradeLevelDescriptor#0 
fake_assessment
 sc.ed.gov/gradeLevelDescriptor#1 
Local earthmover.yaml:
version: 2

config:
  parameter_defaults:
    # This is the name of the original assessment bundle
    ASSESSMENT_BUNDLE: MAP_Growth
    ASSESSMENT_BUNDLE_BRANCH: main

packages: # keep this section as-is
  student_id_wrapper: 
    git: https://github.com/edanalytics/earthmover_edfi_bundles.git
    branch: main
    subdirectory: packages/student_id_wrapper/

sources:
  # Override default seeds to use SC-namespaced descriptors
  gradeLevelDescriptors:
    file: ./seeds/gradeLevelDescriptors.csv
    header_rows: 1

The above code will override the gradeLevelDescriptor seed from the bundle with the local version of that seed, but otherwise use the original code from the bundle.

Student ID Xwalk:

In order to utilize our  student ID xwalking feature , you will be using the  student ID wrapper package  in earthmover_edfi_bundles repo. The first step will be to create an earthmover_packages.yaml locally, where you will again use project composition to install the original versions of the assessments you are working on, as well as the student ID package:
version: 2

packages:
  # do not change this
  student_ids:  
    git: https://github.com/edanalytics/earthmover_edfi_bundles.git
    branch: main
    subdirectory: packages/student_ids/
    
  # examples - put your assessments here
  Dibels_8_Benchmark: 
      git: https://github.com/edanalytics/earthmover_edfi_bundles.git
      branch: main
      subdirectory: assessments/Dibels_8_Benchmark
  iStation: 
      git: https://github.com/edanalytics/earthmover_edfi_bundles.git
      branch: main
      subdirectory: assessments/iStation
Once you have all assessments included in this file (more can be added later), you can run earthmover deps to install the 'packages'. (NOTE - this deps step only needs to be run once per environment, or anytime a new assessment is added to the list):

cd ~code/earthmover_edfi_bundles/packages/student_id_wrapper/ 
cp ~code/path_to_the_above_earthmover_packages.yaml . # copy the packages into the wrapper folder
earthmover deps 
rm earthmover_packages.yaml

Then, you will be running the student ID wrapper bundle, pointing toward your bundle:
# do not change directories from the student_id_wrapper folder
earthmover run -p '{
"INPUT_FILE": "path/to/file/split/by/district/year.csv",
"OUTPUT_DIR": "path/to/output/dir/",
"EDFI_ROSTER_FILE": "path/to/student_ed_org/file/split/by/same/district/year.jsonl",
}'


Implementation:

    You will need to split your assessment file by district and year so that you are outputting your jsonl by the same grain.
    You will then loop over each district & year combination and run the second command above to transform the assessment data and use the student ID xwalk functionality.
    When uploading that data to give back to EA, it either needs to be in a folder structure that is split by district_code (integer) and year, or the district codes/years must be consistent in the file name itself.
    For each run (each district & year), a separate file of unmatched students will be output, which are records in the original file that could not be matched to any roster record. You will need to also maintain which assessment, district, & year this file is associated to.