Kafka implementation for Open edX event bus.
Project description
edX Event Bus - Kafka
Kafka implementation for Open edX event bus.
Overview
This package implements an event bus for Open EdX using Kafka.
The event bus acts as a broker between services publishing events and other services that consume these events. Implementing the event bus will allow for asynchronous messaging across services which greatly improves efficiency as we don’t have to wait for scheduled batch synchronizations. Additionally, since the services all speak to the event bus, they are independent of one another and can still function if one service crashes.
This package contains both the publishing code, which processes events into messages to send to the broker, and the consumer code, which polls the broker using a while True loop in order to turn messages back into event data to be emitted. The actual Kafka host will be configurable.
The goal for event-bus-kafka is to eventually have a flexible event bus that can be easily brought into other apps and repositories to produce and consume arbitrary topics. Ideally, the event bus itself will also be an abstraction behind which platform maintainers can use non-Kafka implementations (Redis, Pulsar, etc.). The documentation/ADRs may also be moved to more appropriate places as the process matures.
There are a hefty number of “# TODO (EventBus)” annotations left in to help guide further development. While still under development, this app will be subject to frequent and rapid changes. Outside of testing this app, it is best to leave the EVENT_BUS_KAFKA_CONSUMERS_ENABLED setting off.
The repository works together with the openedx/openedx-events repository to make the fully functional event bus.
For manual testing, see docs/how_tos/manual_testing.rst.
Documentation
OEP-52 documentation: https://open-edx-proposals.readthedocs.io/en/latest/architectural-decisions/oep-0052-arch-event-bus-architecture.html (TODO: Set up documentation)
Development Workflow
One Time Setup
# Clone the repository
git clone git@github.com:openedx/event-bus-kafka.git
cd event-bus-kafka
# Set up a virtualenv using virtualenvwrapper with the same name as the repo and activate it
mkvirtualenv -p python3.8 event-bus-kafkaEvery time you develop something in this repo
# Activate the virtualenv
workon event-bus-kafka
# Grab the latest code
git checkout main
git pull
# Install/update the dev requirements
make requirements
# Run the tests and quality checks (to verify the status before you make any changes)
make validate
# Make a new branch for your changes
git checkout -b <your_github_username>/<short_description>
# Using your favorite editor, edit the code to make your change.
vim …
# Run your new tests
pytest ./path/to/new/tests
# Run all the tests and quality checks
make validate
# Commit all your changes
git commit …
git push
# Open a PR and ask for review.Installation
This library is not intended to be a direct dependency of any service. Instead, it will likely be installed by some private mechanism. Unless the platform gains new processes for installing private dependencies, upgrades will need to be manually managed via a pin.
License
The code in this repository is licensed under the AGPL 3.0 unless otherwise noted.
Please see LICENSE.txt for details.
How To Contribute
Contributions are very welcome. Please read How To Contribute for details. Even though they were written with edx-platform in mind, the guidelines should be followed for all Open edX projects.
The pull request description template should be automatically applied if you are creating a pull request from GitHub. Otherwise you can find it at PULL_REQUEST_TEMPLATE.md.
The issue report template should be automatically applied if you are creating an issue on GitHub as well. Otherwise you can find it at ISSUE_TEMPLATE.md.
Reporting Security Issues
Please do not report security issues in public. Please email security@edx.org.
Getting Help
If you’re having trouble, we have discussion forums at https://discuss.openedx.org where you can connect with others in the community.
Our real-time conversations are on Slack. You can request a Slack invitation, then join our community Slack workspace.
For more information about these options, see the Getting Help page.
Change Log
Unreleased
[0.4.3] - 2022-08-24
Fixed
Never evict producers from cache. There wasn’t a real risk of this, but now we can rely on them being long-lived. Addresses remainder of https://github.com/openedx/event-bus-kafka/issues/16.
[0.4.2] - 2022-08-24
Fixed
Properly load auth settings for producer/consumer. (Auth settings were ignored since 0.3.1.)
[0.4.1] - 2022-08-18
Changed
Remove confluent-kafka as a formal dependency of the repository.
Note: This library will not work without confluent-kafka.
Add an ADR to explain why this work was done.
[0.4.0] - 2022-08-15
Changed
Rename settings to have consistent prefix.
KAFKA_CONSUMERS_ENABLED becomes EVENT_BUS_KAFKA_CONSUMERS_ENABLED
CONSUMER_POLL_TIMEOUT becomes EVENT_BUS_KAFKA_CONSUMER_POLL_TIMEOUT
Updates to documentation and tests for various settings previously renamed
[0.3.1] - 2022-08-11
Changed
Refactored consumer to use common configuration.
[0.3.0] - 2022-08-10
Changed
Moved configuration onto separate file.
Updated configuration settings to have EVENT_BUS_KAFKA prefix.
[0.2.0] - 2022-08-09
Fixed
Cache producers so that they don’t lose data.
[0.1.0] - 2022-06-16
Added
First release on PyPI.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for edx_event_bus_kafka-0.4.3.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 6fed61bdc64dfeaa187172f141d897ca90ec02ab64342679139502fcf81cd60f |
|
| MD5 | be13e7f9cc426c8147ea2ba377f0e8cc |
|
| BLAKE2b-256 | 42dd3eead0898b14bee9b9579cbae940b7976074991f8ffa241214ad6b5bcc6c |
Hashes for edx_event_bus_kafka-0.4.3-py2.py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | cecd79d5e036c49ed9f00eefd07af677a5c29fe46e5aa5c58a75e1c850834ffd |
|
| MD5 | f6b25f9137c61380b77f46b6a5e2b0d9 |
|
| BLAKE2b-256 | 26511b9a67af917e024d42bb3c2d4f240d7e34e0dc1694b605c0c70ba1cbaf85 |
