The AVeriTeC dataset is an invaluable asset for researchers looking to delve into real-world claim verification. This blog post provides a comprehensive guide on how to reproduce the baseline experiments associated with AVeriTeC. If you’re keen on engaging in the upcoming 7th FEVER workshop co-hosted at EMNLP 2024, this article is your roadmap!
What You Need to Get Started
Before diving into the steps, ensure you have the following components on hand:
- A computer with a stable internet connection.
- Basic understanding of programming, especially in Python.
- Tools and libraries as specified in the official README.
Step-by-Step Reproduction Guide
0. Set Up Your Environment
First things first, you need the right environment. Here’s how to set it up:
git lfs install
git clone https://huggingface.co/chenxwh/AVeriTeC
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/chenxwh/AVeriTeC
conda create -n averitec python=3.11
conda activate averitec
pip install -r requirements.txt
python -m spacy download en_core_web_lg
python -m nltk.downloader punkt
python -m nltk.downloader wordnet
conda install pytorch pytorch-cuda=11.8 -c pytorch -c nvidia
Think of this step as setting up a kitchen before cooking a gourmet meal: you need all your ingredients and tools organized!
1. Scrape Text from URLs
Next, you’ll scrape text from URLs obtained through predefined search queries. This step requires execution of the scraping script.
bash scripts/scraper.sh split start_idx end_idx # e.g., bash scripts/scraper.sh dev 0 500
Imagine you’re a journalist gathering evidence – you gather information from various articles to piece together a story.
2. Rank Sentences Using BM25
This step focuses on ranking the scraped sentences based on their relevance to the claims.
python -m src.reranking.bm25_sentences
Think of this process as getting the best quotes from those articles – only the most relevant will make it to your report.
3. Generate Question-Answer Pairs
With sentences ranked, you’ll use a model to generate question-answer pairs.
python -m src.reranking.question_generation_top_sentences
This resembles turning those best quotes into compelling questions that you want your readers to ponder.
4. Rerank the QA Pairs
The next step involves reranking the question-answer pairs to finalize your evidence.
python -m src.reranking.rerank_questions
Like finalizing quotes from your interviews, this stage culls the gold from the gilt.
5. Veracity Prediction
Lastly, you will predict the veracity of the claims based on the evidence gathered.
python -m src.prediction.veracity_prediction
This final step is akin to writing the conclusion of your piece – you’ll assert the truthfulness of the claims based on your collected evidence.
Formatting Submissions
If you plan to submit your findings, the submission format must be specific:
json[
{
claim_id: "sample_id",
claim: "claim text",
pred_label: "predicted label",
evidence: [
{
question: "question text",
answer: "answer text",
url: "source URL",
scraped_text: "text scraped from the URL"
}
]
}
]
It’s like the final polish on your article – ensuring it’s ready to impress your readers!
Troubleshooting
While following these steps, you may encounter a few hiccups along the way. Here are some common solutions:
- Issues with Installation: Make sure you’re using the correct version of Python and that you have all dependencies installed correctly. Consider reinstalling any libraries that are causing issues.
- Scraping Errors: If the scraping process fails, check if the URLs are accessible or if your internet connection is stable. You may want to try running the script again after resolving any connection issues.
- Model Conflicts: Ensure that compatibility between various models is maintained, especially during the reranking and prediction stages.
For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.
Conclusion
At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.