The Open Data Platform for your community

Datadex is a fully open-source, serverless, and local-first Data Platform that enhances the way communities collaborate on Open Data. Rather than being a new tool, Datadex is an opinionated bridge connecting existing tools more seamlessly.

Datasets generated by this project are ready to explore and consume at HuggingFace. Check them out. Learn more about the approach in this post or check other real-world production implementations of the Datadex pattern working in the following repositories:

• LUNG-SARG: The Open Data Platform for Sustainable, Accessible Lung Radiogenomics.
• Gitcoin Grants Data Portal: A data hub for Gitcoin Grants data, improving access for data scientists and guiding community-driven analysis.
• Filecoin Data Portal: A data hub for Filecoin data, akin to Dune, but running on your laptop.

Principles

• Open: Code, standards, infrastructure, and data are public and open source.
• Modular and Interoperable: Each component can be replaced, extended, or removed, allowing flexibility in various environments.
• Permissionless: Fork it and improve. You don’t need to ask—just enhance and innovate.
• Data as Code: Creating declarative stateless transformations tracked in git, this encourages reproducibility and accessibility.
• Glue: Act as a bridge between tools and methodologies, using software engineering best practices.

What can you do with Datadex?

• Add new data sources locally!
• Model existing datasets using Python and SQL.
• Explore your data using tools like Jupyter Notebooks, BI Tools, Excel, etc.
• Share your findings online as visually appealing static websites.

Setup

To get started with Datadex, you have two approaches based on the Datadex pattern: using a Python Virtual Environment or Docker Dev Containers. If you encounter any issues, please open an issue!

Python Virtual Environment

To install all dependencies in a Python virtual environment, follow these steps:

make setup

Clone the repository and execute the commands from the root folder:

bash
make setup

Alternatively, you can create a virtual environment using your system’s Python installation:

bash
# Create a virtual environment
python3 -m venv .venv
source .venv/bin/activate
# Install the package and dependencies
pip install -e .[dev]

Now, you should be able to run Dagster UI by executing make dev or dagster dev and access it locally at 127.0.0.1:3000.

Docker Dev Containers

The quickest way to get started is through VSCode Remote Containers, which requires Docker. Open your project in VSCode and initiate the project in a container by clicking in the bottom right corner.

After entering the development environment, simply run make dev to launch the Dagster UI locally. This approach also pre-installs some valuable extensions configured to work seamlessly with your project. The development environment can also run in your browser via GitHub Codespaces!

Motivation

This project originated from brainstorming what an Open Data Protocol could resemble. You can read more about it here.

Troubleshooting

If you run into obstacles, consider these troubleshooting ideas:

• Check your Python version; it should be compatible with the existing dependencies.
• Ensure that Docker is running correctly if you’re using Docker Dev Containers.
• If there are issues with package installations, clear the cache and reinstall the packages.
• Examine the logs for helpful error messages that can guide you in debugging your setup.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.