Aspects plugin for Tutor
Project description
Aspects Learner Analytics combines several free, open source, tools to add analytics and reporting capabilities to the Open edX platform. This plugin offers easy installation, configuration, and deployment of these tools using Tutor. The tools Aspects uses are:
ClickHouse, a fast, scalable analytics database that can be run anywhere
Apache Superset, a data visualization platform and data API
OpenFUN Ralph, a Learning Record store (and more) that can validate and store xAPI statements in ClickHouse
Vector, a log forwarding tool that can be used to forward tracking log and xAPI data to ClickHouse
event-routing-backends, an Open edX plugin that transforms tracking logs into xAPI and optionally forwards them to one or more Learning Record Stores in near real time
event-sink-clickhouse, an Open edX plugin that exports course structure and high level data to ClickHouse at publish time
dbt, a tool to build data pipelines from SQL queries. The dbt project used by this plugin is aspects-dbt.
See https://github.com/openedx/openedx-aspects for more details about the Aspects architecture and high level documentation.
Aspects is a community developed effort combining the Cairn project by Overhang.io and the OARS project by EduNEXT, OpenCraft, and Axim Collaborative.
Note: Aspects is in heavy development and not yet production ready! Please feel free to experiment with the system and offer feedback about what you’d like to see by adding Issues in this repository.
Compatibility
This plugin is compatible with Tutor 15.0.0 and later and is expected to be compatible with Open edX releases from Nutmeg forward.
Installation
The Aspects project can be installed in a Tutor environment with the following command
pip install tutor-contrib-aspects
Or to install the main branch you can:
- ::
pip install git+https://github.com/openedx/tutor-contrib-aspects
Usage
Enable the plugins:
tutor plugins enable aspects
Save the changes to the environment:
tutor config save
Because we’re installing a new app in LMS (event-routing-backends) you will need to rebuild your openedx image:
tutor images build openedx
Run the initialization scripts in your chosen environment (dev or local):
tutor [dev|local] do init
(Optional) Load test xAPI data into Ralph/Clickhouse/Superset (with --help for usage):
tutor [dev|local] do load-xapi-test-data
(Optional) Sink course data from the LMS to clickhouse (see https://github.com/openedx/openedx-event-sink-clickhouse for more information):
tutor [dev|local] do dump-courses-to-clickhouse --options "--force" # If you already have some courses in your clickhouse sink, its better to drop --options "--force" as it will create duplicates of the pre-existing courses.
(Optional) Sink Historical event data to ClickHouse:
tutor [dev|local] run lms ./manage.py lms transform_tracking_logs --source_provider LOCAL --source_config '{"key": "/openedx/data", "container": "logs", "prefix": "tracking.log"}' --transformer_type xapi # Note that this will work only for default tutor installation. If you store your tracking logs any other way, you need to change the source_config option accordingly. # See https://event-routing-backends.readthedocs.io/en/latest/howto/how_to_bulk_transform.html#sources-and-destinations for details on how to change the source_config option.
Superset Assets
Aspects maintains the Superset assets in this repository, specifically the dashboards, charts, datasets, and databases. That means that any updates made here will be reflected on your Superset instance when you update your deployment.
But it also means that any local changes you make to these assets will be overwritten when you update your deployment. To prevent your local changes from being overwritten, please create new assets and make your changes there instead. You can copy an existing asset by editing the asset in Superset and selecting “Save As” to save it to a new name.
Contributing Charts and Dashboards to Aspects
The Superset assets provided by Aspects can be found in the templated assets.yaml file. For the most part, these files what Superset exports, but with some crucial differences which make these assets usable across all Tutor deployments.
To contribute assets to Aspects:
Export the assets you want to contribute as described in Sharing Charts and Dashboards
Expand the .zip file.
Update any database connection strings to use Tutor configuration template variables instead of hard-coded strings, e.g. replace clickhouse with {{CLICKHOUSE_HOST}}. Passwords can be left as {{CLICKHOUSE_PASSWORD}}, though be aware that if you are adding new databases, you’ll need to update SUPERSET_DB_PASSWORDS in the init scripts. Here is the default connection string for reference:
``clickhousedb+connect://{{CLICKHOUSE_REPORT_URL}}``Remove any metadata.yaml files from the export. We generate these as needed during import.
Merge your exported files into the directories and files in the assets.yaml.
Submit a PR with screenshots of your new chart or dashboards, along with an explanation of what data question they answer.
Virtual datasets in Superset
Superset supports creating virtual datasets, which are datasets defined using a SQL query instead of mapping directly to an underlying database object. Aspects leverages virtual datasets, along with SQL templating, to make better use of table indexes.
To make it easier for developers to manage virtual datasets, there is an extra step that can be done on the output of tutor aspects serialize. The sql section of the dataset yaml can be moved to its own file in the queries directory and included in the yaml like so:
sql: "{% include 'openedx-assets/queries/query.sql' %}"However, please keep in mind that the assets declaration is itself a jinja template. That means that any jinja used in the dataset definition should be escaped. There are examples of how to handle this in the existing queries, such as dim_courses.sql.
License
This software is licensed under the terms of the AGPLv3.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tutor-contrib-aspects-0.23.1.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a5c2caad6b7b6a1ad79babfae1775e4c69823fad2558396a5ff666321a0d6ae2 |
|
| MD5 | 67e52b13f54ba6291d53d687596f8339 |
|
| BLAKE2b-256 | a2a0ae8defaa1df61c465545547749bd1280515a68c196e45b40e97dc9b69d38 |
Hashes for tutor_contrib_aspects-0.23.1-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 0074b248356355ccfb0fa2767df3e7ef549dafda25710ed75287410305d8c930 |
|
| MD5 | 69b500014070aa0ca288760a9ba1b10b |
|
| BLAKE2b-256 | 87a5db4f2eef904cd8dadf6b3e31c6c41ad66c1e77fffb9bef24411fbfefa910 |
