denispeskoff / fednlp Goto Github PK

In addition to the two transcript requests,if <$200

run all the MINUTES data the same way as the statements
https://github.com/trigaten/FedNLP/blob/main/data/minutes.csv

INPUT ENTIRE minutes -> OUTPUT single -1 to 1 prediction.

@Benjamin Wachspress could you please plot the sentence-level results for us?

In the results folder, it is this file.

line plot
date on the x-axis
hawk/dove score on the y-axis
make sure y-axis label indicates which is hawkish and which is dovish.

Thanks!

For all available statements, predict the entire statement (all sentences together) as -1, -.5, 0, .5, 1.

For each speaker in each meeting, take all of their sentences and give each one a score, then report by sentence scores and averages.

In addition to the by-speaker version of the transcript (#7), If the cost is <$200,

run all the transcript data the same way as the statements
https://github.com/trigaten/FedNLP/blob/main/data/transcripts.csv

INPUT ENTIRE transcript -> OUTPUT single -1 to 1 prediction.

For starters, here is the paper I referenced earlier today where researchers at the Fed tested GPT against other traditional NLP methods. Let's start by recreating this using the statements in our dataset from 1994-2016. Specifically, we should prompt GPT-4 to characterize each sentence exactly as they did in their paper, from -1 to 1 (photo attached). Once we have all of the sentences marked, we can take an unweighted average of the scores for all sentences within a meeting to give us a "score" for that date. Let me know if you have questions, but I think starting here makes sense as it is more straightforward than working with the minutes/transcripts.

@trigaten could you please run each of these sentences to get a score on the hawk dove scale?
figure_1_examples.csv

We need a beautiful figure to put either in the top right of the second column on the first page that summarizes our task.

Or one on the top of the second page (spanning the entire width).

(Or likely both, with one explaining the task, and one presenting the results)

@sanderschulhoff could you please push the results for future exercises as a .csv to the results folder?

Thanks.

For each available transcript, calculate BY SPEAKER a prediction of -1, -.5, 0, .5, 1. Keep this separated across unrelated transcripts.

Example:
Joe: sentence 1. sentence 2. sentence 3
Adam: sentence 1
Joe sentence 1. sentence 2

Should be predicted as:
Joe: .5
Adam: 0
Joe: 1

@sanderschulhoff Next step is to process the transcripts. The main difference from the statements is we need to produce a hawk/dove score for each speaker, for each meeting. So the first new task is, for each speaker, create a list of all of their sentences. We will then iterate and score those speaker sentences.

For each transcript from 1994-2016:
Get all sentences for each speaker:
For each sentence by speaker:
Measure each sentence on scale of -1, -0.5, 0, 0.5, 1 (according to Category definitions from above)
Record average measurement for each speaker
Return average measurement for each transcript

For example output: {19940204: {greenspan:0.46, yellen:-0.25, geitner:0.65, ... , avg:0.26},
{19940204: {greenspan:0.1, yellen:-0.38, geitner:0.45, ... , avg:-0.15}, etc}

@sanderschulhoff in addition to the unweighted average, it would be helpful to also have access to the raw counts for each sentence level classification. For the task you have already completed as well as for the upcoming sentence-level work on transcripts.

For example output: {19940204: {-1: 10, -0.5: 2, 0:25, 0.5:4, 1:10, avg:0.4},
19940322: {-1: 4, -0.5: 0, 0:18, 0.5:3, 1:6, avg:0.3333333333333333}, etc}

1.X
2.Y
3.Z

For all available statements, predict EACH sentence as -1, -.5, 0, .5, 1. Keep separated by statement.

Statements -> each sentence gets a score
-> intest entire statement for score
Minutes -> ingest entire meeting for score
Transcripts -> identify speaker and give each a single score based on ingesting all of their speech at once

We want to create data that lumps everything said by a speaker within a transcript together:

John: sentence 1, sentence 2, sentence 6, sentence 7
Sally: sentence 3, sentence 4
Joe: sentence 5

We will want to be able to 1) evaluate the hawk/dove stance for a speaker 2) the aggregate stance of all speakers in the transcript 3) how the same speaker changes throughout different years of transcripts

Run each one as a separate sentence prediction:

1. “The gradual increase in core inflation over the past year is a concern to me.”
2. “I do not believe that there is a very great risk of an unmanageable outbreak of inflation during the relevant policy horizon.”
3. “I think we should not slow our pace of easing moves at this meeting.”

(this one is a statement but don't think that changes anything)
4.“The Committee continues to believe that against the background of its long-run goals of price stability and sustainable economic growth and of the information currently available, the risks are weighted mainly toward conditions that may generate economic weakness in the foreseeable future.”

For every available meeting minutes, ingest the entire minutes and predict -1, -.5, 0, .5, 1

We need to go through all the statements we are sending to GPT and rank them -1, -.5, 0, .5, 1 the same way Alan did in the email to create a gold label.

Following this guidance

Please optimizing accuracy first then speed second, classify all statements.
If you are unsure then mark is as ? and we can escalate to Alan for grey area decisions.

Also keep an eye peeled for anything especially notable/worth commenting on. (the 1 out of 100 statement)

Alan's thoughts were:
Nov. 1994 is a definite +1, not so much for the wording (though that fits your scale) as for the huge 75 bp move.
Dec. 2008 is way beyond a -1. The Fed threw the kitchen sink at the failing economy then—big rate cut, forward guidance,…

@sanderschulhoff
can you do for the ENTIRE statement? That should be easy if ingestible. Then we can compare that how that looks relative to a sentence average.

We need 3-5 good examples of speaker -> [-1, -.5, 0, 1] prediction.

I did a quick attempt but need somebody to verify my labels are accurate. I'm confident of hawk vs dove but not confident of boundary between .5 and 1

Tarullo_-1_dove.txt
Mester_.5_Hawk.txt
George_1_hawk.txt

Adam will provide 10 examples in json (text:prediction). Re-run all statements (double check with Adam but I think it should be ENTIRE statement, not by sentence).