The AVeriTeC dataset is an invaluable asset for researchers looking to delve into real-world claim verification. This blog post provides a comprehensive guide on how to reproduce the baseline experiments associated with AVeriTeC. If you’re keen on engaging in the upcoming 7th FEVER workshop co-hosted at EMNLP 2024, this article is your roadmap!

What You Need to Get Started

Before diving into the steps, ensure you have the following components on hand:

• A computer with a stable internet connection.
• Basic understanding of programming, especially in Python.
• Tools and libraries as specified in the official README.

Step-by-Step Reproduction Guide

0. Set Up Your Environment

First things first, you need the right environment. Here’s how to set it up:

git lfs install
git clone https://huggingface.co/chenxwh/AVeriTeC
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/chenxwh/AVeriTeC
conda create -n averitec python=3.11
conda activate averitec
pip install -r requirements.txt
python -m spacy download en_core_web_lg
python -m nltk.downloader punkt
python -m nltk.downloader wordnet
conda install pytorch pytorch-cuda=11.8 -c pytorch -c nvidia

Think of this step as setting up a kitchen before cooking a gourmet meal: you need all your ingredients and tools organized!

1. Scrape Text from URLs

Next, you’ll scrape text from URLs obtained through predefined search queries. This step requires execution of the scraping script.

bash scripts/scraper.sh split start_idx end_idx  # e.g., bash scripts/scraper.sh dev 0 500

Imagine you’re a journalist gathering evidence – you gather information from various articles to piece together a story.

2. Rank Sentences Using BM25

This step focuses on ranking the scraped sentences based on their relevance to the claims.

python -m src.reranking.bm25_sentences

Think of this process as getting the best quotes from those articles – only the most relevant will make it to your report.

3. Generate Question-Answer Pairs

With sentences ranked, you’ll use a model to generate question-answer pairs.

python -m src.reranking.question_generation_top_sentences

This resembles turning those best quotes into compelling questions that you want your readers to ponder.

4. Rerank the QA Pairs

The next step involves reranking the question-answer pairs to finalize your evidence.

python -m src.reranking.rerank_questions

Like finalizing quotes from your interviews, this stage culls the gold from the gilt.

5. Veracity Prediction

Lastly, you will predict the veracity of the claims based on the evidence gathered.

python -m src.prediction.veracity_prediction

This final step is akin to writing the conclusion of your piece – you’ll assert the truthfulness of the claims based on your collected evidence.

Formatting Submissions

If you plan to submit your findings, the submission format must be specific:

json[ 
  { 
    claim_id: "sample_id",
    claim: "claim text",
    pred_label: "predicted label",
    evidence: [
      {
        question: "question text",
        answer: "answer text",
        url: "source URL",
        scraped_text: "text scraped from the URL"
      }
    ]
  }
]

It’s like the final polish on your article – ensuring it’s ready to impress your readers!

Troubleshooting

While following these steps, you may encounter a few hiccups along the way. Here are some common solutions:

• Issues with Installation: Make sure you’re using the correct version of Python and that you have all dependencies installed correctly. Consider reinstalling any libraries that are causing issues.
• Scraping Errors: If the scraping process fails, check if the URLs are accessible or if your internet connection is stable. You may want to try running the script again after resolving any connection issues.
• Model Conflicts: Ensure that compatibility between various models is maintained, especially during the reranking and prediction stages.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.