fednlp's Issues
Minutes: Entire Minutes
For every available meeting minutes, ingest the entire minutes and predict -1, -.5, 0, .5, 1
Task 5: Plot Sentence-Level Results
@Benjamin Wachspress could you please plot the sentence-level results for us?
In the results folder, it is this file.
line plot
date on the x-axis
hawk/dove score on the y-axis
make sure y-axis label indicates which is hawkish and which is dovish.
Thanks!
Run all minutes data
In addition to the two transcript requests,if <$200
run all the MINUTES data the same way as the statements
https://github.com/trigaten/FedNLP/blob/main/data/minutes.csv
INPUT ENTIRE minutes -> OUTPUT single -1 to 1 prediction.
Task 3: Record Descriptive Stats from Sentence Level Processing
@sanderschulhoff in addition to the unweighted average, it would be helpful to also have access to the raw counts for each sentence level classification. For the task you have already completed as well as for the upcoming sentence-level work on transcripts.
For example output: {19940204: {-1: 10, -0.5: 2, 0:25, 0.5:4, 1:10, avg:0.4},
19940322: {-1: 4, -0.5: 0, 0:18, 0.5:3, 1:6, avg:0.3333333333333333}, etc}
Prioritization
1.X
2.Y
3.Z
Task 7: finalizing figure
We need a beautiful figure to put either in the top right of the second column on the first page that summarizes our task.
Or one on the top of the second page (spanning the entire width).
(Or likely both, with one explaining the task, and one presenting the results)
Task 9: Prepare transcript data by speaker for GPT processing
We want to create data that lumps everything said by a speaker within a transcript together:
John: sentence 1, sentence 2, sentence 6, sentence 7
Sally: sentence 3, sentence 4
Joe: sentence 5
We will want to be able to 1) evaluate the hawk/dove stance for a speaker 2) the aggregate stance of all speakers in the transcript 3) how the same speaker changes throughout different years of transcripts
Transcripts: By Speaker
For each available transcript, calculate BY SPEAKER a prediction of -1, -.5, 0, .5, 1. Keep this separated across unrelated transcripts.
Example:
Joe: sentence 1. sentence 2. sentence 3
Adam: sentence 1
Joe sentence 1. sentence 2
Should be predicted as:
Joe: .5
Adam: 0
Joe: 1
Task 8: Manual annotation of ALL statements
We need to go through all the statements we are sending to GPT and rank them -1, -.5, 0, .5, 1 the same way Alan did in the email to create a gold label.
Please optimizing accuracy first then speed second, classify all statements.
If you are unsure then mark is as ? and we can escalate to Alan for grey area decisions.
Also keep an eye peeled for anything especially notable/worth commenting on. (the 1 out of 100 statement)
Alan's thoughts were:
Nov. 1994 is a definite +1, not so much for the wording (though that fits your scale) as for the huge 75 bp move.
Dec. 2008 is way beyond a -1. The Fed threw the kitchen sink at the failing economy then—big rate cut, forward guidance,…
Task 2: hawk/dove for each statement, not by sentence
@sanderschulhoff
can you do for the ENTIRE statement? That should be easy if ingestible. Then we can compare that how that looks relative to a sentence average.
Task 6: Speaker Level Transcript
@sanderschulhoff Next step is to process the transcripts. The main difference from the statements is we need to produce a hawk/dove score for each speaker, for each meeting. So the first new task is, for each speaker, create a list of all of their sentences. We will then iterate and score those speaker sentences.
For each transcript from 1994-2016:
Get all sentences for each speaker:
For each sentence by speaker:
Measure each sentence on scale of -1, -0.5, 0, 0.5, 1 (according to Category definitions from above)
Record average measurement for each speaker
Return average measurement for each transcript
For example output: {19940204: {greenspan:0.46, yellen:-0.25, geitner:0.65, ... , avg:0.26},
{19940204: {greenspan:0.1, yellen:-0.38, geitner:0.45, ... , avg:-0.15}, etc}
Run 4 sentences (used for our figure) through GPT
Run each one as a separate sentence prediction:
- “The gradual increase in core inflation over the past year is a concern to me.”
- “I do not believe that there is a very great risk of an unmanageable outbreak of inflation during the relevant policy horizon.”
- “I think we should not slow our pace of easing moves at this meeting.”
(this one is a statement but don't think that changes anything)
4.“The Committee continues to believe that against the background of its long-run goals of price stability and sustainable economic growth and of the information currently available, the risks are weighted mainly toward conditions that may generate economic weakness in the foreseeable future.”
Statements: Entire Statement
For all available statements, predict the entire statement (all sentences together) as -1, -.5, 0, .5, 1.
Task 1: recreate sentence level classification
For starters, here is the paper I referenced earlier today where researchers at the Fed tested GPT against other traditional NLP methods. Let's start by recreating this using the statements in our dataset from 1994-2016. Specifically, we should prompt GPT-4 to characterize each sentence exactly as they did in their paper, from -1 to 1 (photo attached). Once we have all of the sentences marked, we can take an unweighted average of the scores for all sentences within a meeting to give us a "score" for that date. Let me know if you have questions, but I think starting here makes sense as it is more straightforward than working with the minutes/transcripts.
scoring select sentences
@trigaten could you please run each of these sentences to get a score on the hawk dove scale?
figure_1_examples.csv
Statements: By Sentence
For all available statements, predict EACH sentence as -1, -.5, 0, .5, 1. Keep separated by statement.
Re-Run statements with few-shot
Adam will provide 10 examples in json (text:prediction). Re-run all statements (double check with Adam but I think it should be ENTIRE statement, not by sentence).
Experiments to run
Statements -> each sentence gets a score
-> intest entire statement for score
Minutes -> ingest entire meeting for score
Transcripts -> identify speaker and give each a single score based on ingesting all of their speech at once
Transcript level analysis
For each speaker in each meeting, take all of their sentences and give each one a score, then report by sentence scores and averages.
Task 4: Add results as .csv to a results folder in the git repo
@sanderschulhoff could you please push the results for future exercises as a .csv to the results folder?
Thanks.
Few-shot examples
We need 3-5 good examples of speaker -> [-1, -.5, 0, 1] prediction.
I did a quick attempt but need somebody to verify my labels are accurate. I'm confident of hawk vs dove but not confident of boundary between .5 and 1
Run all transcript data
In addition to the by-speaker version of the transcript (#7), If the cost is <$200,
run all the transcript data the same way as the statements
https://github.com/trigaten/FedNLP/blob/main/data/transcripts.csv
INPUT ENTIRE transcript -> OUTPUT single -1 to 1 prediction.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.