In the ever-evolving landscape of data retrieval and processing, ranking passages effectively is a crucial skill for enhancing search results. This blog serves as a user-friendly guide on how to implement and utilize the passage-ranker-v1-XS-en model developed by Sinequa, which is designed to produce relevance scores for query-passage pairs. Let’s dive in!

Understanding the Model

The passage-ranker-v1-XS-en model is like a referee in a sports game, ensuring that the most relevant search results are at the forefront based on the implemented query. Just as a referee assesses players’ performances to determine the winning team, this model evaluates passages relative to a query to determine their relevance using a score.

Supported Languages

• English

Scores Metric

The model produces a relevance score, specifically an NDCG (Normalized Discounted Cumulative Gain) score at 10. Here’s the detail:

Relevance (NDCG@10): 0.438

This score is an average computed over 14 different retrieval datasets, ensuring a well-rounded assessment of passage ranking capabilities.

Inference Times and GPU Memory Usage

When using this model, the inference times will vary slightly based on the hardware setup. Below is a quick breakdown:

GPU          Quantization Type   Batch Size 1   Batch Size 32
NVIDIA A10   FP16               1 ms            2 ms
NVIDIA A10   FP32               1 ms            8 ms
NVIDIA T4    FP16               1 ms            6 ms
NVIDIA T4    FP32               3 ms           23 ms
NVIDIA L4    FP16               1 ms            3 ms
NVIDIA L4    FP32               2 ms            8 ms

When it comes to GPU memory usage, it is essential to note the following:

Quantization Type   Memory
FP16                 150 MiB
FP32                 200 MiB

Requirements

To ensure a smooth operation of the passage ranker, you’ll need to meet the following requirements:

• Minimum Sinequa version: 11.10.0
• Minimum Sinequa version for using FP16 models and GPUs with CUDA compute capability of 8.9+: 11.11.0
• Cuda compute capability: above 5.0 (above 6.0 for FP16 use)

Model Details

To further understand this model, here are some additional details:

• Number of parameters: 11 million
• Base language model: English BERT-Mini
• Insensitive to casing and accents
• Training procedure: MonoBERT

Training Data

The model has been trained on the Probably-Asked Questions dataset, which can be found in the following resources:

• Paper
• Official Page

Evaluation Metrics

The relevance score is derived from averaging results evaluated on datasets referenced in the BEIR benchmark, focusing exclusively on English datasets. Here are some evaluated datasets and their corresponding scores:

Dataset               NDCG@10
Average               0.438
Arguana              0.524
CLIMATE-FEVER        0.150
DBPedia Entity       0.338
FEVER                0.706
FiQA-2018            0.269
HotpotQA             0.630
MS MARCO             0.328
NFCorpus             0.340
NQ                   0.429
Quora                0.722
SCIDOCS              0.141
SciFact              0.627
TREC-COVID           0.628
Webis-Touche-2020    0.306

Troubleshooting

If you encounter issues while implementing the passage-ranker-v1-XS-en model, consider the following troubleshooting steps:

• Ensure that your Sinequa version meets the required minimum.
• Check your CUDA compatibility for the appropriate GPU model.
• Monitor the GPU memory usage to prevent overloading.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.