danai-antoniou / topic_models

This section introduces a generative text modelling technique, known as topic modelling, in order to identify natural topic groupings in the property descriptions. The purpose of generative models is to discover hidden structures which are accountable for the observed set of data. In our case the descriptions of the properties, and, specifically, their words, are the observations and we are interested in uncovering their latent structure, which is precisely the topics conveyed. The analysis that follows focuses on the use of the Latent Dirichlet Allocation (hereafter LDA) as the topic model of choice which assumes that documents are mixtures of a multitude of topics and every topic is viewed as a probability distribution over the vocabulary of terms.

We introduce the model’s assumptions before applying it to the property descriptions. The unigram Bag-of-Words assumption and the DTM are adopted as the LDA model is built upon them. LDA postulates the Dirichlet distribution is responsible for determining the features of topics and documents in the corpus. The number of topics, T , used in the model is fixed a-priori and the parameters used to describe the generative process are defined below.

We henceforth assume the prior distributions of θ and φ are symmetric Dirichlet distributions with the hyperparameters α and β being scalars, however, they can be extended to take vector-form. The topic assignments are determined using the method proposed by Griffiths and Steyvers of estimating the posterior distribution of the word-to-topic assignments p(z|w) through Markov Chain Monte Carlo (hereafter MCMC) and, in particular, Gibbs Sampling – the estimates of θ and φ are later retrieved from the posterior distribution.

Τhe number of topics to be fitted is also set a-priori and a reoccurring issue in topic modelling is determining this number. There are numerous approaches adopted in literature to identify the optimal number of topics, but Griffiths and Steyvers introduced the use of the marginal likelihood p(w|T) and the Harmonic Mean Estimator.