Install is install_github within Hadley's devtools package.
install.packages("devtools")
require(devtools)
install_github("citeulike", "ropensci")
require(citeulike)
Some documentation here:
I don't know of any official CiteULike API documentation, but if you do email us the location on at [email protected].