Ed-Fi Assessment Integrations using earthmover & lightbeam Documentation

This initial document provides an overview of Ed-Fi Assessment Integrations using earthmover and lightbeam and includes links to more detailed documentation on various topics.

A corresponding slide deck on these topics can be found  here .

Important Links

Ed-Fi Assessment Integration Process Overview:

When building out an integration for a new assessment (AKA an assessment that is not currently listed in the  bundles repository ), the general process is as follows:


1. Background Information

    Familiar yourself with the  earthmover  &  lightbeam  tools.
    The READMEs of those repos contain all necessary documentation to install/use the tools.
    Multiple earthmover  example projects  exist to help familiarize yourself to its features.
    Review the documentation on the  Assessment domain  of the Ed-Fi data model.
    It is impossible to map data to Ed-Fi without understanding the destination.
    Explore the  template bundle  to get a better sense of the structure/best-practices of an assessment bundle.
    This bundle contains fake data and can be run in any python environment with earthmover installed.

2. Data Governance

    Gather relevant documentation about the assessment that you are working with. In order to properly map an assessment, you must deeply understand the structure and background information of that assessment. This documentation typically includes:
    Sample files
    Data dictionary
    Technical reports from the vendor
    Fill out the  governance artifact template⁠  that will be reviewed by an assessment data governance committee.
    The decisions here will impact how the specific assessment is modeled into the Ed-Fi standard, therefore must be done before any actual coding can occur.

3. Bundle Development

    Create a fork of the  earthmover_edfi_bundles  repository then duplicate the template bundle as a starting point for new assessment bundles.
    Update the template bundle code with necessary transformations, jinja logic, etc. for the given assessment.
    Ensure that data modeling matches the decisions from the governance artifact review.
    Test earthmover run to ensure no errors occur.
    Check that the resulting json files are valid.
    This can be done in multiple ways, including  IDE extensions ,  jq , etc.
    (If possible) Manually run lightbeam  validate  against an ODS.
    Open a PR against the earthmover_edfi_bundles repository and tag Julianna Alvord for review (github user:@jalvord1).

4. Implementation of the Bundle

    Build out process to run earthmover & (depending on the circumstance) lightbeam.
    Run manually or built into a process that will run on a schedule?