Floneum makes it easy to develop applications that use local pre-trained AI models. There are two main projects in this monorepo:

• Kalosm: A simple interface for pre-trained models in Rust
• Floneum Editor (preview): A graphical editor for local AI workflows

See the user documentation or plugin documentation for more information.

Kalosm: The Simple Interface for AI Models

Kalosm is the engine behind Floneum, facilitating interaction with various pre-trained models including language, audio, and image models.

Model Support

Here is a list of the currently supported models:

Model	Modality	Size	Description	Quantized	CUDA + Metal Accelerated	Example
Llama	Text	1b-70b	General purpose language model	llama 3 chat
Mistral	Text	7-13b	General purpose language model	mistral chat
Phi	Text	2b-4b	Small reasoning focused language model	phi 3 chat
Whisper	Audio	20MB-1GB	Audio transcription model	live whisper transcription
RWuerstchen	Image	5gb	Image generation model	rwuerstchen image generation
TrOcr	Image	3gb	Optical character recognition model	Text Recognition
Segment Anything	Image	50MB-400MB	Image segmentation model	Image Segmentation
Bert	Text	100MB-1GB	Text embedding model	Semantic Search

Utilities

Kalosm offers a variety of utilities for working with pre-trained models:

• Extracting, formatting and retrieving context for LLMs: Extract context from various file types.
• Chunk that context: Manage and organize your data effectively.
• Searching relevant context with vector database integrations.
• Transcribing audio from your microphone or file.
• Crawling and scraping content from web pages.

Performance

Kalosm employs the Candle machine learning library to run models efficiently in pure Rust. It supports quantized and accelerated models, delivering impressive performance:

Mistral 7b Performance Comparison:

Accelerator	Kalosm (ts)	llama.cpp (ts)
Metal (M2)	39	27

Structured Generation

Kalosm allows structured generation with arbitrary parsers through its custom parser engine. You can define any Rust type and add #[derive(Parse, Schema)] to leverage structured generation.

rust
use kalosm::language::*;

#[derive(Parse, Schema, Clone, Debug)]
struct Character {
    #[parse(pattern = [A-Z][a-z]{2,10} [A-Z][a-z]{2,10})]
    name: String,
    #[parse(range = 1..=100)]
    age: u8,
    #[parse(pattern = [A-Za-z ]{40,200})]
    description: String,
}
#[tokio::main]
async fn main() -> Result<(), Box> {
    let model = Llama::phi_3().await?;
    let task = Task::builder_for::<[Character; 10]>()
        .build();
    let mut stream = task.run("Create a list of random characters", model);
    stream.to_std_out().await.unwrap();
    let character = stream.await.unwrap();
    println!("{:?}", character);
}

Using regular expressions invites flexibility in generating structured data, ensuring customized outputs ranging from JSON to HTML and XML.

Kalosm Quickstart!

Follow these steps to set up a simple chatbot using Kalosm:

1. Install Rust.
2. Create a new project:

sh
cargo new kalosm-hello-world
cd kalosm-hello-world

3. Add Kalosm as a dependency:

sh
cargo add kalosm --features language
cargo add tokio --features full

4. Update your main.rs file:

rust, no_run
use kalosm::language::*;

#[tokio::main]
async fn main() -> Result<(), Box> {
    let model = Llama::phi_3().await?;
    let mut chat = Chat::builder(model)
        .with_system_prompt("You are a pirate called Blackbeard")
        .build();
    loop {
        chat.add_message(prompt_input("Enter your message: ")?) 
        .to_std_out()
        .await?;
    }
}

5. Run your application:

sh
cargo run --release

6. Check the chat bot demo.

Troubleshooting

If you encounter any issues while using Kalosm or Floneum, here are a few troubleshooting tips:

• Ensure that you have the latest version of Rust installed.
• Double-check your dependencies in your project’s Cargo.toml.
• Consult the user documentation for additional guidance.
• For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.