Collective Knowledge - a lightweight knowledge manager to organize, cross-link, share and reuse artifacts and workflows
Project description
Collective Knowledge SDK
Linux/MacOS: 
Introduction
Developing complex computational systems for emerging workloads (ML, AI, Quantum, IoT) and moving them to production is a very tedious, ad-hoc and time consuming process due to continuously changing software, hardware, models, data sets and research techniques.
The Collective Knowledge project (CK) is our attempt to solve these problems with a common SDK and an open repository of knowledge to decompose complex systems and research projects into reusable, portable, customizable and non-virtualized CK components with the unified automation actions, Python APIs, CLI and JSON meta description. We can then use the DevOps methodology to connect such components together into portable CK solutions that can automatically adapt to evolving models, data sets and bare-metal platforms.
Our goal is to help researchers and practitioners to collaboratively validate innovative techniques, benchmark and optimize novel computational systems, and find the most efficient ones in terms of speed, accuracy, energy, size and different costs using a public repository of knowledge with live SOTA scoreboards and reproducible papers.
We hope that such approach will make it possible to better understand what is happening inside complex and "black box" computational systems, integrate them with production and legacy systems, use them inside Docker and Kubernetes, share them along with published papers, and apply the DevOps methodology in deep tech research and computational science. We also hope that the CK technology will complement existing reproducibility efforts such as PapersWithCode, ACM and artifact evaluation.
Please check our MLPerf automation demo and feel free to get in touch if you have any feedback and suggestions! You can learn more about our project in the CK documentation and the following presentations and white papers: 2019, 2018, 2017, 2009.
Even though the CK technology is used in production for more than 5 years, it is still a proof-of-concept prototype requiring further improvements and standardization. Depending on the available resources, we plan to develop a new, backward-compatible and more user-friendly version - please get in touch if you are interested to know more!
Open knowledge portal
- cKnowledge.io: the open portal with stable CK components, workflows, reproduced papers, and SOTA scoreboards for complex computational systems (AI,ML,quantum,IoT):
- Our reproducibility initiatives for systems and ML conferences
CK use cases
- Real world use cases from our industrial and academic partners
- MLPerf benchmark automation demo with a CK SOTA scoreboard
- Demo of a CK-based live research paper (collaboration with the Raspberry Pi foundation).
Documentation
Older wiki-based documentation (we gradually move it to Sphinx-based docs above):
Installation
Follow this guide to install the CK SDK on your platform.
Check CK-based demo to participate in reproducible MLPerf benchmarking with a public SOTA scoreboard.
CK supports the following platforms:
| As a host platform | As a target platform | |
|---|---|---|
| Generic Linux | ✓ | ✓ |
| Linux (Arm) | ✓ | ✓ |
| Raspberry Pi | ✓ | ✓ |
| MacOS | ✓ | ± |
| Windows | ✓ | ✓ |
| Android | ± | ✓ |
| iOS | TBD | TBD |
Get involved
-
This is an ongoing community project and there is a lot to be improved - don't hesitate to get in touch using our slack, mailing list, twitter and email if you have any feedback or would like to collaborate.
-
Check this outdated guide to add your workflows and components. We are gradually rewriting this guide with our limited resources and developing a CK wrapper (cBrain) to simplify the user experience and provide a CK GUI at the open cKnowledge.io platform.
Acknowledgments
We would like to thank all CK users and partners for fruitful discussions and feedback!
Copyright 2015-2020 Grigori Fursin and the cTuning foundation
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
