Embedded Files
Let's use Natural Language Processing (NLP) to understand employees' attitudes about an organization
Let's use Natural Language Processing (NLP) to understand employees' attitudes about an organization
What's the purpose of this project?
While employee engagement surveys can allow an assessor to quantify attitudes or opinions, this method limits participants from freely speaking. Free responses in surveys are great, but are very time consuming to interpret. Also, to understand the deeper story about a textual dataset, statistics is no longer a handy tool. NLP- a form of supervised machine learning- allows us to analyze textual data.
About this data set: I discovered this dataset on Github here. It contains 149 employees' 3 character variables: employee ID, opinion, and assessment of whether the opinion is positive or negative.
Loading and cleaning data
At first glance, we can see our variable "Assessment" is detected by R as a character. Let's change it to a variable to make it easier to work with.
Tokenize dataset
Before we can tell our machine to analyze our comments and display cool findings, we need to teach our machine's algorithms, well...everything. Right now, it just sees the opinions as a bunch of letters and spaces. We need to tell our machine what is one word, two words, three words and a sentence.
1 word
2 words
3 words
What's interesting here is that <int> indicates how many sentences make up an employee's opinion.
Stop Words
Common words, such as "a", "the", "and" do not provide much information about a text's analysis since they serve a grammatical function rather than help indicate positive or negative sentiment. Articles, prepositions, and adverbs are commonly used words that we should tell our machine's algorithims to ingore.
It looks like the most common words in our opinions would be considered stop words.
In the table, we can see how our tokens are different when the opinions include or exclude the stop words.
Visualizing the opinions
Count
Now that our machine's algorithms are a little smarter, we can visualize what specific words are associated with a positive and negative opinion. Let's count up the most common words again.
However, we see the same words in both charts like "job" and "management".
Frequency
Let's try something else. Frequency ignores the differences between an opinion's length and number of positive or negative opinions. This method helps eliminate some noise within the dataset.
Unfortunately, the number of positive and negative opinions appears to be the same, so we see similar results.
if-idf
Because the same words were appearing in both kinds of opinions, let's add the tf-idf statistic. More information can be found here, but the TL;DR is that it detects if a word is common in 1 opinion and is uncommon in the remainder of the opinions. If an idf= 0, it means the commonly appears in both positive and negative opinions, so it's not a good token to be differentiated.
Now our machine has learned more about the dataset. We can now see the words that are important and ordered by frequency. So far, these are the best predictors of whether an opinion is positive or negative.
Adding a Lexicon
Lucky for us, smarter people developed groups of words to use for NLP. Lexicons are dictionaries which group words by different sentiments.
Let's use the Lexicon bing (by Bing Liu and collaborators). Bing assigns words a binary variable (-1 for negative emotion, +1 for positive emotion). We can see examples of words it classifies as positive and negative from the opinions.
For a final visualization, let's see how Bing classifies the opinions. Positive opinions are turquoise and negative opinions are orange. We can see that there is a greater difference in positive words than negative words in positive opinions, and a lesser difference in positive words than negative words in a negative opinions.
To summarize, NLP can analyze textual data to detect underlying phenomena. One could further advance their machine's algorithm by using logistic regression to create a predictive model for future datasets. There are numerous other lexicons and use cases for NLP, but I chose to conduct a sentiment analysis since I wanted to work with a hypothetical HR dataset.
Page updated
Google Sites
Report abuse