primerai / blanc Goto Github PK

View Code? Open in Web Editor NEW

93.0 93.0 11.0 1.44 MB

Human-free quality estimation of document summaries

License: MIT License

Python 100.00%

blanc's People

Contributors

Stargazers

Watchers

Forkers

dakinggg dreamware-nz evgenykos davebulaval xxxlil morobro romanwerpachowski sxjscience techthiyanes pablogope mikasie

blanc's Issues

CLI error: name 'min_token_length_normal_tune' is not defined

Hey!
I am trying the CLI, but it fails with

Traceback (most recent call last):
  File "/Users/user/.local/bin/blanc", line 8, in <module>
    sys.exit(main())
  File "/Users/user/.local/pipx/venvs/blanc/lib/python3.8/site-packages/blanc/__main__.py", line 283, in main
    min_token_length_normal_tune=min_token_length_normal_tune,
NameError: name 'min_token_length_normal_tune' is not defined

How to reproduce:

install blanc (both pipx and pure venv result in error)

pipx install --python python3.8 blanc==0.3.3
or
pip install blanc==0.3.3

run the command from README.md

blanc help --gap 6 --doc "Jack drove his minivan to the bazaar to purchase milk and honey for his large family." --summary "Jack bought milk and honey."

Help understanding Shannon().go(...)

Hi,
I just read the paper "Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluation". Thanks for making the metric accessible!

Could you please help me understand the Shannon().go(...) function.
What are measure_t, measure_summ used for?
What are the components of the return? Do they include ShannonScore, InformationDifference and BlancShannon?

I tried to inspect the code, but i didn't fully get it yet. My best guess atm is that the return consists of the components used to calculate ShannonScore and InformationDifference. If that's correct: How do I use them to get the final scores?

kind regards.

IndexError: string index out of range

When I run the "Basic usage" example in README, I received the error, "IndexError: string index out of range".

~\anaconda3\lib\site-packages\blanc\blanc.py in run_inference_batch(self, model, batch)
318 predictions = {}
319 for idx in model_input.masked_idxs:
--> 320 predicted_id = model_output[idx].argmax()
321 (predicted_token,) = self.model_tokenizer.convert_ids_to_tokens([predicted_id])
322 predictions[idx] = predicted_token

IndexError: string index out of range

Security Policy violation Branch Protection

This issue was automatically created by Allstar.

Security Policy Violation
PR Approvals below threshold 0 : 1 for branch master

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Can't run CLI

I installed blanc for pypi using the command:
pip3 install blanc
as instructed but when I then try to use the CLI I get the following output

$> blanc
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/bin/blanc", line 5, in <module>
    from blanc.__main__ import main
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/blanc/__main__.py", line 231
    'when True, mask every `gap` tokens ('gap_mask' tokens at once) that are longer than `min_token_length`'
                                                 ^
SyntaxError: invalid syntax

I'm able to get the code to work through python otherwise however...

Estime returns NaN

In some cases estim returns NaN. Is this on purpose or a bug?

If needed I could schedule a rerun and post an example.

import error

when I try to import blanc after pip installing it
from blanc import BlancHelp, BlancTune

I face the error below whether it is in google colab, local, or even in CLI

`FileNotFoundError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_8508/2694449195.py in
----> 1 from blanc import BlancHelp, BlancTune

~\anaconda3\lib\site-packages\blanc_init_.py in
----> 1 from .version import version
2 from .blanc import BlancHelp, BlancTune
3 from .estime import Estime
4 from .shannon import Shannon

~\anaconda3\lib\site-packages\blanc_version_.py in
2 import os
3
----> 4 with open(os.path.join(os.path.dirname(file), "version.json")) as reader:
5 version = json.load(reader)["version"]

FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\elsha\anaconda3\lib\site-packages\blanc\version.json'`

I was running the same notebook a few weeks ago and did not face with such problem

Estime ZeroDivisionError

from blanc import Estime

Estime(output=['alarms', 'soft', 'coherence'], device="cuda").evaluate_claims(
    """23 March 2017 Last updated at 14:55 GMT""",
    ["""northern ireland secretary martin mcguinness says he\'s " shocked " by the outcome of the assembly election."""])

leads to

File ".../python3.11/site-packages/blanc/estime.py", line 312, in _evaluate
    cos_raw_avg /= len(embs_summ)
ZeroDivisionError: division by zero

(The doc and summary pair are taken from the aggrefact benchmark as an unfaithful pair.)

import error

version.json is not packaged with the source code

#37 introduced a problem when building e.g. a wheel package: the new version.json file is not packaged with the rest of the code.

#38 fixes that.

Migration to Transformers version 4.x

Hi,

It would be great if the current blanc implementation could be adjusted to work with Transformer Version 4.x, as it is failing due to some breaking changes!

https://huggingface.co/transformers/migration.html

Thanks!

`pip install -e .` does not work in a new Python environment

When I try to install blanc in a new Python environment using pip install -e ., this fails because the installation process requires nltk and torch to be present during installation.

Batched shannon

Hi,

does Shannon support batched evaluation? If so, how to use it? As far as I see the Shannon.go() function only takes a single summary document pair. I am looking for a batched version in the hope to boost performance with respect to time consumption.

kind regards

AttributeError: 'NoneType' object has no attribute 'tokenize'

Unable to run the BlancTune or BlancHelp due to the following error
To reproduce the error

from blanc import BlancHelp, BlancTune
document = "Jack drove his minivan to the bazaar to purchase milk and honey for his large family."
summary = "Jack bought milk and honey."
blanc_tune = BlancTune(device='cuda', inference_batch_size=24, finetune_mask_evenly=False, finetune_batch_size=24)
blanc_tune.eval_once(document, summary)

Ouput

AttributeError                            Traceback (most recent call last)
<ipython-input-22-a8659d54e616> in <module>()
      1 document = "Jack drove his minivan to the bazaar to purchase milk and honey for his large family."
      2 summary = "Jack bought milk and honey."
----> 3 blanc_tune.eval_once(document, summary)

4 frames
/usr/local/lib/python3.7/dist-packages/blanc/blanc.py in eval_once(self, doc, summary)
    100             score (float): The BLANC score for the input
    101         """
--> 102         (doc_score,) = self.eval_summaries_for_docs([doc], [[summary]])
    103         (score,) = doc_score
    104         return score

/usr/local/lib/python3.7/dist-packages/blanc/blanc.py in eval_summaries_for_docs(self, docs, doc_summaries)
    644 
    645         doc_summaries_use = [[None for s in summs] for summs in doc_summaries]
--> 646         base_outputs, base_answers = self.mask_and_infer(self.base_model, docs, doc_summaries_use)
    647 
    648         finetuned_outputs, finetuned_answers = [], []

/usr/local/lib/python3.7/dist-packages/blanc/blanc.py in mask_and_infer(self, model, docs, doc_summaries, sep)
    171             doc_inputs, doc_answers = [], []
    172             for summary in summaries:
--> 173                 summary_inputs, summary_answers = self.get_inference_inputs(doc, summary, sep)
    174                 doc_inputs.append(summary_inputs)
    175                 doc_answers.append(summary_answers)

/usr/local/lib/python3.7/dist-packages/blanc/blanc.py in get_inference_inputs(self, doc, summary, sep)
    215         doc = clean_text(doc)
    216         doc_sents = sent_tokenize(doc)
--> 217         doc_sent_tokens = [self.model_tokenizer.tokenize(sent) for sent in doc_sents]
    218 
    219         summary_sent_tokens = None

/usr/local/lib/python3.7/dist-packages/blanc/blanc.py in <listcomp>(.0)
    215         doc = clean_text(doc)
    216         doc_sents = sent_tokenize(doc)
--> 217         doc_sent_tokens = [self.model_tokenizer.tokenize(sent) for sent in doc_sents]
    218 
    219         summary_sent_tokens = None

AttributeError: 'NoneType' object has no attribute 'tokenize'

On going through the source code , I guess there is an error on line 90 in blanc.py

        if self.model_name.lower().find('albert') >= 0:
            self.model_tokenizer = AlbertTokenizer.from_pretrained(model_name)
        else:
            self.model_tokenizer = AlbertTokenizer.from_pretrained(model_name) ## possible fix some other name needs to be given here

possible options for model_name are all BERT based , while the tokenizer being used is albert hence it returns None type

Thanks for your help

Can't run Shannon score

I am encountering an error with Shannon score. When I clone the repository and run the example script using python blanc/blanc/shannon.py --simple I get this error:

Traceback (most recent call last):
  File "shannon.py", line 219, in <module>
    results = s.go(doc, summ, measure_t=args.measure_t, measure_summ=args.measure_summ)
  File "shannon.py", line 169, in go
    base_sent_lls, base_sent_success = self.measure(sent_tokens, base_prompt)
  File "shannon.py", line 111, in measure
    probs = F.softmax(logits, dim=-1).view(-1)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 1680, in softmax
    ret = input.softmax(dim)
AttributeError: 'str' object has no attribute 'softmax'

I get the same error if I pip install blanc and try running Shannon score as follows:

from blanc import Shannon
estimator = Shannon()
doc = 'Jack drove his minivan to the bazaar to purchase milk and honey for his large family'
summ = 'Jack bought milk and honey from the bazaar'
estimator.go(doc, summ)

Reproducibility of Shannon score paper results

Dear authors,

Thank you very much for making your implementation available. I am trying to reproduce the results for Shannon score and Information Difference score using the SummEval benchmark. However, the results I am getting are really poor compared to the paper's (all correlations < .3). I am running the 1.6k source texts and their summaries through the class below.

Am I doing anything wrong? Thank you!

from blanc.blanc.shannon import Shannon

class ShannonScorer:
    def __init__(self):
        pass
    
    def score(self, srcs, hyps, verbose=False):
        
        assert len(srcs)==len(hyps)
        
        scorer = Shannon()

        def score_aux(doc, summ):
            ll_base, ll_help, ll_full, _, _, _ = scorer.go(doc, summ)
            shannon = (ll_help - ll_base) / (ll_full - ll_base)
            infodif = ll_help - ll_base
            return [shannon, infodif]
        
        scores = []
        for i in tqdm(range(len(hyps)), disable=not verbose):
            scores.append(score_aux(srcs[i], hyps[i]))
            
        return scores

Security Policy violation SECURITY.md

This issue was automatically created by Allstar.

Security Policy Violation
Security policy not enabled.
A SECURITY.md file can give users information about what constitutes a vulnerability and how to report one securely so that information about a bug is not publicly visible. Examples of secure reporting methods include using an issue tracker with private issue support, or encrypted email with a published key.

To fix this, add a SECURITY.md file that explains how to handle vulnerabilities found in your repository. Go to https://github.com/PrimerAI/blanc/security/policy to enable.

For more information, see https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository.

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Shannon error when doc has too many linebreaks

Hi,

I encountered an error using the Shannon metric. The error seems to occure if there are too many line breaks in the doc.

from blanc import Shannon
import re

doc = """nder
Election 2015: Week Ahead - The unveiling of manifestos
Election 2015: Expenditure on the NHS will be a priority, says David Gauke
Election 2015:  Voting issues for Bristol prop makers
Election 2015 smaller parties: National Health Action (NHA)
Election 2015: Northampton voters offered political mug
Election 2015: Andrew Neil's Friday campaign report
Rail fares debate: Eric Pickles v Jack Dromey
Election 2015: Andrew Neil's Thursday campaign report
Thatcher's armoured bus from Northern Ireland for sale
Election 2015 smaller parties: Liberty GB
Election 2015: Voting issues for Somerset stonemasons
Election 2015: Games and online sites about voting
Election 2015: Ed Balls talking about non-dom status
Election 2015: Andrew Neil's Wednesday campaign report
Election 2015: Voting issues for Cornwall cheese-makers
Election: Christian People's Alliance and Christian Party
Mahmood on Labour bid to abolish non-dom rules
Election 2015: Tory and Lib Dems on coalition taxes
Election 2015 smaller parties: Peace Party policies
Election 2015: Voters at National Aquarium in Plymouth
Election 2015: Opinion polls and role of focus groups
Election 2015: Priti Patel and Chris Leslie on Europe
Election 2015: Priti Patel and Chris Leslie on health
Election 2012: Market affected by hung parliament results
Election 2015 smaller parties: Community Party of Britain
Election 2015: TV viewers asked about leaders' debate
Election 2015: Andrew Neil's Thursday campaign report
Election 2015: How union members could affect vote
Election 2015: Labour or Conservative choice on economy
Election 2015: Andrew Neil's Wednesday campaign report
Election 2015: Trader on Labour's zero hours contract policy
Election 2015: Independence from Europe Party
Election 2015: Rat, hedgehog, James Bond and Joey Essex
Zero hours contract debate: Javid, Cable and Leslie
Election 2015: Andrew Neil's campaign Morning Report
Election 2015: Tax levels in UK and other countries
Election 20105: Voters views on political campaigns
Election 2015: Plaid leader Leanne Wood at party launch
Election 2015: Cannabis is Safer than Alcohol Party
Election 2015: Andrew Neil's campaign Morning Report
Election 2015: What the UK and Scottish polls predict
Election 2015 smaller parties: Mebyon Kernow
Election 2015: Issuing 650 writs to get voting started
Lucy Powell: Labour government would ban exploitative zero hour contracts
Peter Kellner: There is a "real Labour bounce" in latest poll
Alan Duncan: Cameron's third term decision 'not unwise'
How will the general election campaigns pan out?
Was it wrong for Tories to try and oust Commons speaker?
Labour's Lucy Powell clashes with presenter Andrew Neil
Famous faces: MPs retiring and leaving political stage
BBC News Timeliner hosts election archives
How many archive election broadcasts can you remember?
How does Big Ben cope with the change to summer time?
Burnham: NHS is going backwards on this government's watch
What happened to coalition predictions?
La Reine le veult: What is prorogation in Parliament?
MacKenzie: "White poor thickos" claiming the benefits
Would you want to do these jobs?
When should Prince Charles’ letters be published?
Secret ballots for future Speaker elections?
Hancock and Mahmood: Tax and national insurance pledges
Election 2015: Artist Adam Dant drawing the campaign
PMQs highlights 2010-2015: Cameron, Miliband and MPs
London Marathon bid in election run-up by Dan Jarvis MP
PMQs: Cameron on British deaths in A320 Alps air crash
PMQs: Cameron and Miliband on post-election VAT rises
PMQs: Cameron and Miliband on national insurance and taxes
PMQs: Cameron on Connarty 'standing down' at election
PMQS review: Patel and Umunna join Landale and Neil
Election: Speechwriters Collins and FinkelsteinDaily Politics highlights of 2015
Election 2015: DUP's Donaldson on hung parliament talks
What do UKIP and Green councillors think?
Brian May on Common Decency campaign
Chris Leslie on Labour election VAT pledge
Why did Cameron announce f
"""
doc_withoutLinebreaks = re.sub(r'\n', ' ', doc)
doc_firstHalf = doc[len(doc)//2]
summ = """the daily and sunday politics are on-air six days a week for much of the year reporting the political news from westminster and beyond."""

Shannon().go(doc_withoutLinebreaks, summ)
Shannon().go(doc_firstHalf, summ)
Shannon().go(doc, summ)

error message:

Traceback (most recent call last):
  File ".../shannon_error_minimal.py", line 87, in <module>
    Shannon().go(doc, summ)
  File ".../.local/lib/python3.11/site-packages/blanc/shannon.py", line 206, in go
    full_sent_lls, full_sent_success = self.measure(sent_tokens, full_prompt)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../.local/lib/python3.11/site-packages/blanc/shannon.py", line 124, in measure
    past = [t[:, :, :, 1:, :] for t in past]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../.local/lib/python3.11/site-packages/blanc/shannon.py", line 124, in <listcomp>
    past = [t[:, :, :, 1:, :] for t in past]
            ~^^^^^^^^^^^^^^^^
TypeError: tuple indices must be integers or slices, not tuple

The error happens for the Shannon().go(doc, summ) line, indicating that the doc without linebreaks and the shorter doc work fine.
(doc and summ are taken from the aggrefact benchmark. The string in "doc" has weird start and end because I tried to reduce the example in a loop. The whole doc had the same error.)

Is this a bug, or am I supposed to pre-process the text and remove the newlines?

kind regards

requirements.txt is missing scipy

blanc/estime.py imports scipy, but it's not mentioned in requirements.txt.

How can we use the code to give aspect scores?

At the bottom of the repo and in the paper, blanc gives the aspect correlation like fluency and consistency, but how can we compute that kind of correlation? My understanding is that blanc can calculate a quality score, and the fluency correlation and consistency correlation are calculated by the same given by blanc with different aspect scores annotated by humans. Is my understanding correct?

primerai / blanc Goto Github PK

blanc's People

Contributors

Stargazers

Watchers

Forkers

blanc's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs