R cheasheet

Item HOWTO
CheatSheet https://www.rstudio.com/wp-content/uploads/2016/10/r-cheat-sheet-3.pdf
Vector create manually c(1,2,3) # => c is combine
function term.frequency <- function(row) { row / sum(row) }
apply(someMatrix, 1, term.frequency) # 1: rows, 2: cols
new column with booleans usedcars$conservative <- usedcars$color %in% c("Black", "Gray", "Silver", "White")
Matrix  
Remove column matrix[, -c[2,4] # => remove columns 2,4
Filter get indexes indexes <- matrix(somedata$Label == spam`) # => all indexes of spam.
Inmem Data  
Create matrix dataframe
Parsing Text  
Read csv from url
   
Read text file mylog <- readLines("~./someserver.log")
Remove two first lines mylog <- mylog[3:length(mylog)] // keep from line 3 to length
number of rows nrow(mylog)
grep grep(“INFO”, my log, value = TRUE)
search replace mylog_nofirst_space = sub(“^\s”, “”, mylog) # first space —> nothing
remove first line all_data <- all_data[-1] # remove element number 1
lines not in indexes all_data[-rep_date_entries] # lines not in indexes in array rep_date_entries
text to table mytable <- read.table(text = non_date_lines)
add headers to my table names(mytable) <- c(“action_id”, “question_id”, “rep_change”)
extract substring with(data, str_extract(rep_change, “[0-9]+”)) # data.rep_change column extract numers
str to number as.numeric(some_column)
update column inplace dat$rep_change <- with(dat, as.numeric(str_extract(rep_change, “[0-9]+”)))
how many lines between lines * actions_per_day <- c(rep_date_entries[1], diff(rep_date_entries)) - 1
* ie. If some of the lines have dates and some not, then here we extract the indexes of all lines having date
* -1 at the end because we remove 1 from each result
Filter vector actionsPerDay[actionsPerDay >= 0] // only values larger than zero.
Add date to dateless lines * dat$rep_date = rep(all_data[rep_date_entries], times = actions_per_day)
* we know for each date how many times to repeat -> actions_per_day
rep means repeat, so repeat for dateless lines dat$rep_date
* new column added take the lines from all_data[rep_date_entries and for each such add]
string to date dat$rep_date = strptime(dat$rep_date, “%Y-%m-%d”)
loop - apply apply function over rows or cols of matrix `apply(mymatrix, 1, myfunc) # 1: do it on th erows of the my matrix
Plot http://r-statistics.co/Top50-Ggplot2-Visualizations-MasterList-R-Code.html
 
scatter plot Explore relationship between 2 variables: plot(x = carsData$year, y = carsData$price)
Resoures  
NLP with R https://www.youtube.com/watch?v=4vuw0AsHeGw
nlp package Quanteda (tokenization, filtering, ..)

Read log file and print line chart

df = read.table("~/tmp/heapmem.log", header = FALSE)
matplot(df$V4, type="l")

in column 4 we have the heapmem it will produce the linechart.

Getting help and examples

help(read.table)
example(read.table)
vignettes() # PDF documentation on a topic.
vignettes(package = .packages(all.available = TRUE)) # include more non standard topics.

Plotting

diamond data source

  carat       cut color clarity depth table price     x     y
  <dbl>     <ord> <ord>   <ord> <dbl> <dbl> <int> <dbl> <dbl>
1  0.23     Ideal     E     SI2  61.5    55   326  3.95  3.98
2  0.21   Premium     E     SI1  59.8    61   326  3.89  3.84
3  0.23      Good     E     VS1  56.9    65   327  4.05  4.07

plot the proportion of cut (without ..prop.. would simply count it)

ggplot(data = diamonds) + geom_bar(mapping = aes(x = cut, y = ..prop.., group = 1))

Log file analysis

install.packages(“devtools”) library(devtools install_github(“kevinushey/rex”) library(rex)

http://www.mjdenny.com/Text_Processing_In_R.html

diy <- grep("getAllForType.multithreaded", readLines("~/Downloads/diy-play-fail-log.html"), value = TRUE)

When text data is in a nice CSV format, read.csv is enough to parse it into a useable format. But if this is not the case, getting the data into a useable format is not so straightforward. In this post I particularly illustrate the use of regular expressions for complex and flexible text processing, and the power of vectorization in R. Vectorization means that we operate on vectors as a whole, not operate on individual elements of a vector.

Take for example a snippet of this data which I downloaded from StackOverflow:

2  15466134 (10)
 2  15466134 (10)
 1  15462529 (15)
 2  13265177 (10)
 2  15475139 (10)
 2  15486973 (10)
-- 2013-03-18 rep +65   = 15552     
 2  14376993 (10)
 2  14376993 (10)
 2  14376993 (10)
 1  15493353 (15)
 2  12598625 (10)
 2  14376993 (10)
-- 2013-03-19 rep +65   = 15617     
 2  15520314 (10)
 2  15520314 (10)
-- 2013-03-20 rep +20   = 15637     
 1  15541210 (15)
 2  15541210 (10)
 2  15541210 (10)
 2  15541210 (10)

The entire data file can be downloaded here.

In this post I’ll be stepping through the R code needed to get this text data into a useable format. First, we want to read the data into a character vector:

all_data = readLines("rep.dat")
head(all_data)
[1] "total votes: 2325" " 2   8150378 (10)" " 2   8167111 (10)"
[4] " 2   8167111 (10)" " 2   8167111 (10)" " 2   8167461 (10)"

where each element of the vector is a line in the the text file. Already we see that the first line is some header information which we want to skip:

all_data = all_data[-1]

note the use of negative indexing to remove an element. Next we want to find all the elements in the vector that relate to the date for which the data is representative, we do that by using a regular expression which looks for lines that start with -:

rep_date_entries = grep("^-", all_data)

and find the amount of actions, upvotes or downvotes etc, that have taken place on each day, i.e. the index of a certain day minus the index of the day before that:

actions_per_day = c(rep_date_entries[1], diff(rep_date_entries)) - 1

note that we add rep_date_entries[1] because diff cuts off the first element. Now that we know which elements relate to the date, we can read all other lines into a nice data.frame:

dat = read.table(text = all_data[-rep_date_entries])  
names(dat) = c("action_id", "question_id", "rep_change")

The reputation column has a somewhat strange format ((10)), we need to get rid of the brackets. A nice way of doing that is using a regular expression, and the str_extract function from the stringr package:

require(stringr)
dat$rep_change = with(dat, as.numeric(str_extract(rep_change, "-*[0-9]+")))

The regular expression [0-9]+ matches one or more numbers, and str_extract gets the number out of the string. Now we have the data, we need to add a column which says for each row to which date it belongs. We know which lines in the data belong to a date (rep_date_entries) and we know how much data entries there are per day (actions_per_day). We can now simply repeat each element in rep_date_entries as many times as there are actions:

dat$rep_date = rep(all_data[rep_date_entries], times = actions_per_day)
head(dat)
  action_id question_id rep_change                             rep_date
1         2     8150378         10 -- 2011-11-17 rep +95   = 96
2         2     8167111         10 -- 2011-11-17 rep +95   = 96
3         2     8167111         10 -- 2011-11-17 rep +95   = 96
4         2     8167111         10 -- 2011-11-17 rep +95   = 96

You can see that the date is not yet in a nice format, we need to get rid of all the text, except the date itself. Again, we can use a regular expression, combined with str_extract for this:

dat$rep_date = str_extract(dat$rep_date, "[0-9]{4}-[0-9]{2}-[0-9]{2}")

The regular expression "[0-9]{4}-[0-9]{2}-[0-9]{2}" matches any occurence of 4 numbers-2 numbers-2 numbers. Finally, we transform the date from a string to a real date object using strptime:

dat$rep_date = strptime(dat$rep_date, "%Y-%m-%d")

The end result is the following data.frame:

head(dat)
  action_id question_id rep_change   rep_date
1         2     8150378         10 2011-11-17
2         2     8167111         10 2011-11-17
3         2     8167111         10 2011-11-17
4         2     8167111         10 2011-11-17
5         2     8167461         10 2011-11-17
6         1     8167461         15 2011-11-17
summary(dat)
   action_id       question_id         rep_change
 Min.   : 1.000   Min.   :  489821   Min.   :  0.000
 1st Qu.: 2.000   1st Qu.: 9521651   1st Qu.: 10.000
 Median : 2.000   Median :11738823   Median : 10.000
 Mean   : 2.336   Mean   :11475310   Mean   :  9.873
 3rd Qu.: 2.000   3rd Qu.:13175326   3rd Qu.: 10.000
 Max.   :16.000   Max.   :15541210   Max.   :100.000
    rep_date
 Min.   :2011-11-17 00:00:00
 1st Qu.:2012-04-08 00:00:00
 Median :2012-08-30 00:00:00
 Mean   :2012-08-02 19:55:59
 3rd Qu.:2012-11-16 00:00:00
 Max.   :2013-03-23 00:00:00

All this code together leads to the following function:

parse_so_rep_page = function(rep_file) {
   require(stringr)
   all_data = readLines(rep_file)
   all_data = all_data[-1]
   
   rep_date_entries = grep("^-", all_data)
   actions_per_day = c(rep_date_entries[1], diff(rep_date_entries)) - 1 
   
   dat = read.table(text = all_data[-rep_date_entries])
   names(dat) = c("action_id", "question_id", "rep_change")
   dat$rep_change = with(dat, as.numeric(str_extract(rep_change, "-*[0-9]+")))
   
   dat$rep_date = rep(all_data[rep_date_entries], times = actions_per_day)
   dat$rep_date = str_extract(dat$rep_date, "[0-9]{4}-[0-9]{2}-[0-9]{2}")
   dat$rep_date = strptime(dat$rep_date, "%Y-%m-%d")
   return(dat)
}
res = parse_so_rep_page("data/rep.dat")

I think this nicely illustrates the power of both vectorization, very short and to-the-point for-loop-less syntax, and regular expressions in editing strings.

comments powered by Disqus