Feldera is a fast query engine for incremental computation. Feldera has the unique ability to evaluate arbitrary SQL programs incrementally, making it more powerful, expressive, and performant than existing alternatives like batch engines, warehouses, stream processors, or streaming databases.
Incremental Computation Engine
Our approach to incremental computation is simple. A Feldera pipeline is a set of SQL tables and views, which can be deeply nested. Users can start, stop, or pause pipelines to manage and advance a computation. The magic happens when pipelines continuously process changes, which consist of various inserts, updates, or deletes to a set of tables. When changes occur, Feldera incrementally updates all the views, focusing only on the changes rather than the entire dataset, making it incredibly efficient.
Defining Features of Feldera
- Full SQL support and more: Our engine supports extensive SQL syntax and semantics incrementally, including joins, aggregates, subqueries, window functions, and more.
- Fast out-of-the-box performance: Users can implement complex use cases in 30 minutes or less, experiencing millions of events per second performance on a laptop.
- Handling large datasets: Feldera handles datasets larger than available RAM by efficiently spilling to disk, leveraging modern NVMe storage capabilities.
- Strong consistency and freshness: Feldera ensures a strongly consistent state of views, corresponding to what would arise from a batch processing system.
- Connectors for data sources: Feldera integrates seamlessly with various data sources like Kafka, HTTP, S3, and more. If you need additional connectors, you can let us know.
Understanding the Architecture
The architecture of Feldera is key to its functionality, allowing for both batch and real-time analytics, making it suitable for complex scenarios.
Quick Start with Docker
Getting started with Feldera is easy! Just follow these steps:
-
- Ensure you have Docker Compose installed.
- Run the command to download and deploy the Feldera Platform:
curl -L https://github.com/feldera/feldera/releases/latest/download/docker-compose.yml | docker compose -f - --profile demo up- After a brief wait, visit http://localhost:8080 to access the Feldera web console.
- For a guided experience, check out our tutorial.
Running Feldera from Source
To run Feldera from source code:
-
- Install the required dependencies: Rust, Java, Maven, and TypeScript.
- Build the SQL compiler using:
cd sql-to-dbsp-compiler & build.sh-
- Run the pipeline-manager:
cargo run --bin=pipeline-manager --features pg-embed- Finally, visit http://localhost:8080 to view the Feldera web console.
Documentation and Learning Resources
To dive deeper into Feldera, explore the following links:
Benchmarks and Performance
Feldera has been tested against various systems and is generally faster and more memory-efficient. Benchmarks are performed continuously, and for detailed results, visit benchmarks.feldera.io.
Contributing to Feldera
Feldera’s software follows an open-source model. If you’re interested in contributing, please refer to our guidelines.
The Theory Behind Feldera
At the core of Feldera is DBSP, a formal model that addresses how streaming operators and queries transform input streams.
- Semantics: DBSP defines how queries process input streams to produce output.
- Algorithm: It generates incremental dataflow programs that operate efficiently based on input changes.
Troubleshooting
If you encounter issues while using Feldera, consider the following tips:
- Ensure all dependencies are correctly installed.
- Check the correctness of your SQL queries.
- Consult the logs for any error messages.
- Visit GitHub Issues to see if others have reported similar problems.
For further assistance or to collaborate on AI development projects, stay connected with fxis.ai.
At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.