robert1ridley / cross-prompt-trait-scoring

This repository contains the code used to produce the results from the paper Automated Cross-prompt Scoring of Essay Traits published in AAAI 2021.

Cross-prompt Trait Scoring

This repository contains the code used to produce the results from the paper Automated Cross-prompt Scoring of Essay Traits published in AAAI 2021.

System Requirements

This code uses python 3.6 along with the following packages and version numbers:

• tensorflow=2.0.0
• numpy=1.18.1
• nltk=3.4.5
• pandas=1.0.5
• scikit-learn=0.22.1

Ensure that these packages have been added to the virtual environment before running the project code.

Running the Code

Each model is trained through the use of a bash script. Run the models as follows (for Hi att and AES aug, GloVe embeddings are required. Download glove.6B.50d.txt from https://nlp.stanford.edu/projects/glove/ and place in the embeddings directory):

• To run the Hi att model, run ./train_Hi_att.sh
• To run the AES aug model, run ./train_AES_aug.sh
• To run the PAES model, run ./train_PAES.sh
• To run the CTS no att model, run ./train_CTS_no_att.sh
• To run the CTS model, run ./train_CTS.sh

The bash scripts will run each model on each of the prompts and traits a total of 5 times, once for each of the following seed values:
[12, 22, 32, 42, 52].

Note – Handcrafted Features:

Handcrafted features have been pre-calculated and can be found at data/hand_crafted_v3.csv and so have some additional readability features (data/allreadability.pickle). The code used to generate these features can be found in features.py and create_readability_features.py respectively. If you wish to regenerate the features, you will also need to install the 'textstat', 'spacy' (along with the 'en_core_web_sm' model), and 'readability' (use pip install https://github.com/andreasvc/readability/tarball/master) packages.

cross-prompt-trait-scoring's Issues

Readability features used

Hello. Thank you for your work and releasing the code for the paper.

I have a question about the readability features. I noticed the file allreadability.pickle in the repo which stores readability scores of each essay. What readability scores are them respectively? Can you kindly provide their names in the order of what is provided in the allreadability.pickle file? Thank you very much!

some data problem

Hello, I am very interested in your work, I have a small question, as shown in the screenshot below, for the training data (.pk format), how to get the score of the field in the red box?

signatures do not match.

hello~ Thanks for your sharing~ but I got the following problem trying to run the models.

2022-01-03 19:42:53.234962: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2022-01-03 19:42:53.263178: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300010000 Hz
2022-01-03 19:42:53.268868: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x555ceb9f7740 executing computations on platform Host. Devices:
2022-01-03 19:42:53.268918: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2022-01-03 19:42:54.391833: W tensorflow/core/grappler/optimizers/implementation_selector.cc:310] Skipping optimization due to error while loading function libraries: Invalid argument: Functions '__inference_standard_lstm_1183' and '__inference_standard_lstm_1183_specialized_for_model_hz_lstm_StatefulPartitionedCall_at___inference_distributed_function_1545' both implement 'lstm_3fba9309-7aa6-43e8-be19-e72544162d01' but their signatures do not match.
2022-01-03 19:43:07.977277: W tensorflow/core/grappler/optimizers/implementation_selector.cc:310] Skipping optimization due to error while loading function libraries: Invalid argument: Functions '__inference___backward_cudnn_lstm_with_fallback_4023_4205' and '__inference___backward_standard_lstm_4448_4933_specialized_for_StatefulPartitionedCall_at___inference_distributed_function_5297' both implement 'lstm_57c91e11-3e43-4031-b5ad-a9e1e2765877' but their signatures do not match.

could you tell me how to solve it? thank you very much~

How to generate hand_crafted_v3.csv

How to generate the features from hand_crafted_v3.csv?
e.g., what do sent_var, ess_char_len, clause_per_s,.. mean?
what does allreadability.pickle mean?
Many thanks