Comments (6)
I guess there are some errors related to the Serper key.
Also, I just updated the code to handle different Serper response codes. Please pull the latest version, it should help you identify the bug.
from openfactverification.
ok, so using mistral I have this (not sure why I have some errors)
ERROR]2024-04-18 10:48:02,369 CheckWorthy.py:52: ====== Error: , the LLM response is: {
"MBZUAI is the first university in the world focused on artificial intelligence.": "Maybe (This statement might be verifiable with clear and specific references to support its factuality, such as reliable sources or official records.)"
}
[ERROR]2024-04-18 10:48:02,369 CheckWorthy.py:53: ====== Our input is: [[{'role': 'system', 'content': 'You are a helpful assistant designed to output JSON.'}, {'role': 'user', 'content': 'Your task is to evaluate each provided statement to determine if it presents information whose factuality can be objectively verified by humans, irrespective of the statement\'s current accuracy. Consider the following guidelines:\n1. Opinions versus Facts: Distinguish between opinions, which are subjective and not verifiable, and statements that assert factual information, even if broad or general. Focus on whether there\'s a factual claim that can be investigated.\n2. Clarity and Specificity: Statements must have clear and specific references to be verifiable (e.g., "he is a professor" is not verifiable without knowing who "he" is).\n3. Presence of Factual Information: Consider a statement verifiable if it includes factual elements that can be checked against evidence or reliable sources, even if the overall statement might be broad or incorrect.\nYour response should be in JSON format, with each statement as a key and either "Yes" or "No" as the value, along with a brief rationale for your decision.\n\nFor example, given these statements:\n1. Gary Smith is a distinguished professor of economics.\n2. He is a professor at MBZUAI.\n3. Obama is the president of the UK.\n\nThe expected output is:\n{\n "Gary Smith is a distinguished professor of economics.": "Yes (The statement contains verifiable factual information about Gary Smith\'s professional title and field.)",\n "He is a professor at MBZUAI.": "No (The statement cannot be verified due to the lack of clear reference to who \'he\' is.)",\n "Obama is the president of the UK.": "Yes (This statement contain verifiable information regarding the political leadership of a country.)"\n}\n\nFor these statements:\n1. MBZUAI is the first university in the world focused on artificial intelligence.\n\nThe output should be:\n'}]]
[ERROR]2024-04-18 10:48:21,235 CheckWorthy.py:52: ====== Error: , the LLM response is: {
"MBZUAI is the first university in the world focused on artificial intelligence.": "Maybe (This statement might be verifiable with clear and specific references to support its factuality, such as reliable sources or official records.)"
}
[ERROR]2024-04-18 10:48:21,235 CheckWorthy.py:53: ====== Our input is: [[{'role': 'system', 'content': 'You are a helpful assistant designed to output JSON.'}, {'role': 'user', 'content': 'Your task is to evaluate each provided statement to determine if it presents information whose factuality can be objectively verified by humans, irrespective of the statement\'s current accuracy. Consider the following guidelines:\n1. Opinions versus Facts: Distinguish between opinions, which are subjective and not verifiable, and statements that assert factual information, even if broad or general. Focus on whether there\'s a factual claim that can be investigated.\n2. Clarity and Specificity: Statements must have clear and specific references to be verifiable (e.g., "he is a professor" is not verifiable without knowing who "he" is).\n3. Presence of Factual Information: Consider a statement verifiable if it includes factual elements that can be checked against evidence or reliable sources, even if the overall statement might be broad or incorrect.\nYour response should be in JSON format, with each statement as a key and either "Yes" or "No" as the value, along with a brief rationale for your decision.\n\nFor example, given these statements:\n1. Gary Smith is a distinguished professor of economics.\n2. He is a professor at MBZUAI.\n3. Obama is the president of the UK.\n\nThe expected output is:\n{\n "Gary Smith is a distinguished professor of economics.": "Yes (The statement contains verifiable factual information about Gary Smith\'s professional title and field.)",\n "He is a professor at MBZUAI.": "No (The statement cannot be verified due to the lack of clear reference to who \'he\' is.)",\n "Obama is the president of the UK.": "Yes (This statement contain verifiable information regarding the political leadership of a country.)"\n}\n\nFor these statements:\n1. MBZUAI is the first university in the world focused on artificial intelligence.\n\nThe output should be:\n'}]]
[ERROR]2024-04-18 10:48:39,850 CheckWorthy.py:52: ====== Error: , the LLM response is: {
"MBZUAI is the first university in the world focused on artificial intelligence.": "Maybe (This statement might be verifiable with clear and specific references to support its factuality, such as reliable sources or official records.)"
}
[ERROR]2024-04-18 10:48:39,850 CheckWorthy.py:53: ====== Our input is: [[{'role': 'system', 'content': 'You are a helpful assistant designed to output JSON.'}, {'role': 'user', 'content': 'Your task is to evaluate each provided statement to determine if it presents information whose factuality can be objectively verified by humans, irrespective of the statement\'s current accuracy. Consider the following guidelines:\n1. Opinions versus Facts: Distinguish between opinions, which are subjective and not verifiable, and statements that assert factual information, even if broad or general. Focus on whether there\'s a factual claim that can be investigated.\n2. Clarity and Specificity: Statements must have clear and specific references to be verifiable (e.g., "he is a professor" is not verifiable without knowing who "he" is).\n3. Presence of Factual Information: Consider a statement verifiable if it includes factual elements that can be checked against evidence or reliable sources, even if the overall statement might be broad or incorrect.\nYour response should be in JSON format, with each statement as a key and either "Yes" or "No" as the value, along with a brief rationale for your decision.\n\nFor example, given these statements:\n1. Gary Smith is a distinguished professor of economics.\n2. He is a professor at MBZUAI.\n3. Obama is the president of the UK.\n\nThe expected output is:\n{\n "Gary Smith is a distinguished professor of economics.": "Yes (The statement contains verifiable factual information about Gary Smith\'s professional title and field.)",\n "He is a professor at MBZUAI.": "No (The statement cannot be verified due to the lack of clear reference to who \'he\' is.)",\n "Obama is the president of the UK.": "Yes (This statement contain verifiable information regarding the political leadership of a country.)"\n}\n\nFor these statements:\n1. MBZUAI is the first university in the world focused on artificial intelligence.\n\nThe output should be:\n'}]]
[INFO]2024-04-18 10:48:39,851 __init__.py:111: == State: Done! (Nothing to check.)
{
"0_response": "MBZUAI is the first AI university in the world",
"1_decompose": [
"MBZUAI is the first university in the world focused on artificial intelligence."
],
"2_checkworthy": [],
"2_checkworthy_pairwise": {
"MBZUAI is the first university in the world focused on artificial intelligence.": "Maybe (This statement might be verifiable with clear and specific references to support its factuality, such as reliable sources or official records.)"
},
"3_query_generator": {},
"4_evidence_retrieve": {},
"5_claim_verify": {}
}
with wizardlm2 I have this; not sure I understand why the API key is a problem as it is self hosted:
poetry run python -m factcheck --modal string --input "MBZUAI is the first AI university in the world" --client local_openai --model wizardlm2 --prompt factcheck/config/sample_prompt.yaml
== Init decompose_model with model: wizardlm2
[INFO]2024-04-18 11:00:07,223 __init__.py:53: == Use specified client: local_openai
== Init checkworthy_model with model: wizardlm2
[INFO]2024-04-18 11:00:07,223 __init__.py:53: == Use specified client: local_openai
== Init query_generator_model with model: wizardlm2
[INFO]2024-04-18 11:00:07,223 __init__.py:53: == Use specified client: local_openai
== Init evidence_retrieval_model with model: wizardlm2
[INFO]2024-04-18 11:00:07,223 __init__.py:53: == Use specified client: local_openai
== Init claim_verify_model with model: wizardlm2
[INFO]2024-04-18 11:00:07,223 __init__.py:53: == Use specified client: local_openai
[INFO]2024-04-18 11:00:07,223 __init__.py:67: ===Sub-modules Init Finished===
[INFO]2024-04-18 11:00:07,223 multimodal.py:89: == Processing: Modal: string, Input: MBZUAI is the first AI university in the world
[INFO]2024-04-18 11:00:07,223 multimodal.py:103: == Processed: Modal: string, Input: MBZUAI is the first AI university in the world
[INFO]2024-04-18 11:01:17,452 __init__.py:78: == response claims 0: MBZUAI was established as the first artificial intelligence university globally.
[INFO]2024-04-18 11:01:17,452 __init__.py:78: == response claims 1: The Mohammed Bin Zayed University of Artificial Intelligence (MBZUAI) was founded.
[INFO]2024-04-18 11:03:55,539 __init__.py:86: == Check-worthy claims 0: The Mohammed Bin Zayed University of Artificial Intelligence (MBZUAI) was founded.
[INFO]2024-04-18 11:05:50,913 __init__.py:117: == Claim: The Mohammed Bin Zayed University of Artificial Intelligence (MBZUAI) was founded. --- Queries: ['The Mohammed Bin Zayed University of Artificial Intelligence (MBZUAI) was founded.', 'Who founded the Mohammed Bin Zayed University of Artificial Intelligence?', 'When was the Mohammed Bin Zayed University of Artificial Intelligence established?']
[INFO]2024-04-18 11:05:50,916 SerperEvidenceRetrieve.py:30: Collecting evidences ...
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/__main__.py", line 45, in <module>
check(args)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/__main__.py", line 30, in check
res = factcheck.check_response(content)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/__init__.py", line 122, in check_response
claim_evidence_dict = self.evidence_crawler.retrieve_evidence(claim_query_dict=claim_query_dict)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/core/Retriever/SerperEvidenceRetrieve.py", line 33, in retrieve_evidence
evidence_list = self._retrieve_evidence_4_all_claim(
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/core/Retriever/SerperEvidenceRetrieve.py", line 62, in _retrieve_evidence_4_all_claim
serper_response = self._request_serper_api(query_list)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/core/Retriever/SerperEvidenceRetrieve.py", line 193, in _request_serper_api
raise Exception("Failed to authenticate. Check your API key.")
Exception: Failed to authenticate. Check your API key.
from openfactverification.
ok for serper, I will set a key and try again today
from openfactverification.
I am not sure why my serper API key is not accepted. is there a way to debug it further:
[INFO]2024-04-18 18:41:18,230 __init__.py:117: == Claim: The Mohammed Bin Zayed University of Artificial Intelligence (MBZUAI) was founded. --- Queries: ['The Mohammed Bin Zayed University of Artificial Intelligence (MBZUAI) was founded.', 'Who founded the Mohammed Bin Zayed University of Artificial Intelligence?', 'When was the Mohammed Bin Zayed University of Artificial Intelligence established?']
[INFO]2024-04-18 18:41:18,245 SerperEvidenceRetrieve.py:30: Collecting evidences ...
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/__main__.py", line 45, in <module>
check(args)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/__main__.py", line 30, in check
res = factcheck.check_response(content)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/__init__.py", line 122, in check_response
claim_evidence_dict = self.evidence_crawler.retrieve_evidence(claim_query_dict=claim_query_dict)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/core/Retriever/SerperEvidenceRetrieve.py", line 33, in retrieve_evidence
evidence_list = self._retrieve_evidence_4_all_claim(
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/core/Retriever/SerperEvidenceRetrieve.py", line 62, in _retrieve_evidence_4_all_claim
serper_response = self._request_serper_api(query_list)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/core/Retriever/SerperEvidenceRetrieve.py", line 193, in _request_serper_api
raise Exception("Failed to authenticate. Check your API key.")
Exception: Failed to authenticate. Check your API key.
from openfactverification.
Hi, to assist with debugging Serper, I have added a sample function. Please try this
python -m factcheck.core.Retriever.SerperEvidenceRetrieve --serper_api_key YOUR_SERPER_KEY
If your Serper key is correct, you shall be able to see results. If still getting the same error, please try to get a new key from https://serper.dev/
from openfactverification.
it was my fault, I made a typo :-(
from openfactverification.
Related Issues (6)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openfactverification.