Jonathan Asanjarani
Department of Data Analysis and Visualization, CUNY Graduate Center
LLM Fall 2023: Large Language Models and ChatGPT
Professor Michelle McSweeney
December 18, 2023
This project aims to employ BERT for classifying employee sentiments toward their company, utilizing natural language processing (NLP) and sentiment analysis. BERT, a language model hosted by Hugging Face, will be applied to a sentiment classification task using TensorFlow for fine-tuning. The dataset, sourced from Kaggle here, comprises job reviews from Glassdoor. The focus is on providing an overview of BERT model setup and tuning for employee sentiment analysis, along with exploring societal and ethical considerations tied to language model use in workplace sentiment monitoring.
This project leverages BERT, a language model developed in 2018, utilizing neural networks for text representation. BERT understands word context by examining surrounding words on both sides, providing a more accurate representation of linguistic relationships. The architecture incorporates an “Attention Mechanism,” embedding weighted values for each word. The “Transformer” model, based on this mechanism, consists of an encoder with two layers. BERT, being an encoder-only model, stands for Bidirectional Encoder Representations from Transformers. It employs bidirectional training using a “masked language model” (MLM) pre-training objective. This allows BERT to develop accurate contextual relationships without rebuilding the entire model, making it versatile for specific tasks.
Sentiment analysis studies opinions, sentiments, and emotions using computational methods in the context of NLP. Large Language Models (LLMs) perform well in simple sentiment analysis tasks, classifying sentiment as positive or negative. However, for more complex tasks requiring a deep understanding of specific sentiments, LLMs may not outperform traditional models trained with specific, in-domain data.
Leslie Hammer states that employees feeling uncared for or untrusted by employers can lead to lower commitment, perceived psychological safety, and higher stress levels. Companies using AI for workforce monitoring report negative mental health effects among employees. There is skepticism and concern among workers regarding potential misuse of collected information. While some anticipate improved workplace security, others express worries about inappropriate behavior and misuse.
Large Language Models and AI tools in workplace surveillance raise ethical concerns. Companies may monitor employees extensively, posing risks of micromanaging and potential AI training on private or unethically sourced data. Cases like Barclays Bank's fine for unauthorized employee monitoring underscore the need for clear regulations balancing surveillance and privacy. The increased demand for workplace monitoring solutions during the pandemic, without adequate regulations, emphasizes the importance of ethical guidelines as AI technologies advance.
The BERT model was fine-tuned using a Kaggle dataset of employee reviews. The dataset, accessible here, consists of job reviews from Glassdoor. TensorFlow, an open-source machine learning platform, was employed for fine-tuning. The BERT Tokenizer was used for text tokenization, and the dataset was converted into a TensorFlow dataset. The model was compiled with an "Adam" optimizer for prediction enhancement and "sparse categorical crossentropy" for evaluation. Training involved two epochs, and the model was tested on a separate set of reviews from CareerBliss.
The code for this project is available on GitHub.
During the evaluation phase, the employee sentiment analyzer accurately predicted sentiments in a sample of 10 employee reviews. The model demonstrated proficiency in discerning and categorizing sentiments, aligning with employees' expressions in all instances.
BERT effectively classified binary sentiment relationships, accurately categorizing basic employee sentiments. However, limitations exist in this binary classification approach, as it excludes nuanced or subtle employee reviews. Future research should incorporate a broader spectrum of sentiments, including neutral or mixed emotions, to provide a more comprehensive understanding of employee feedback and enhance sentiment classification.
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019, May 24). Bert: Pre-training of deep bidirectional Transformers for language understanding.
- Lerner, M. (2023, September 7). Electronically monitoring your employees? it’s impacting their mental health.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2023, August 2). Attention is all you need.
- Weidinger, L., Gabriel, I., Irving, G., Legassick, S., Isaac, W., Hendricks, L. A., Rimell, L., Haas, J., Birhane, A., Biles, C., Stepleton, T., Hawkins, W., Brown, S., Kenton, Z., Kasirzadeh, A., Balle, B., Glaese, C.-M., Myra, Huang, P.-S., … Mellor, J. (2021, December 8). Ethical and social risks of harm from language models - arxiv.org.
- Giuntella, O. & K. (2023, February 2). Artificial Intelligence and Workers’ well-being.