HDF-5 temp data - is the datetime conversion wrong? about neon-data-skills

[lwasser]

heya! thanks :) please tell me -- did the plot turn out different for you? i tried that fix and the plot still looks exactly the same. this could be my misunderstanding of ggplot as well. thoughts?

[scelmendorf]

Yes. Peak temp ~5pm, which I think is biologically right? I’m trying to paste it here. It probably will not render on git, but I can email it to you.

[cid:[email protected]]

From: Leah Wasser [mailto:[email protected]]
Sent: Tuesday, May 26, 2015 10:30 PM
To: lwasser/NEON-Data-Skills
Cc: Sarah Elmendorf
Subject: Re: [NEON-Data-Skills] HDF-5 temp data - is the datetime conversion wrong? (#35)

heya! thanks :) please tell me -- did the plot turn out different for you? i tried that fix and the plot still looks exactly the same. this could be my misunderstanding of ggplot as well. thoughts?

—
Reply to this email directly or view it on GitHubhttps://github.com/lwasser/NEON-Data-Skills/issues/35#issuecomment-105752877.

[lwasser]

that would be great -- thanks! i might be running it incorrectly. i totally agree - the numbers don't make sense and i never really focused enough on that sadly as i've been so focused on the code. i just cant seem to implement your changes. thanks @scelmendorf

[scelmendorf]

Try re-reading in the hdf5 file, then setting the date as UTC and converting. Through trial and erro (I made the exact same plot twice too ☺) I found that once I set the date wrong the as.POSIX etc didn’t change anything (seemingly you can’t as.posix something that has already been as posixed?)

f <- "NEON_TowerDataD3_D10.hdf5"

View structure of file

h5ls(f)

temp <- h5read(f,"/Domain_03/OSBS/min_1/boom_1/temperature")
temp$date <- as.POSIXct(temp$date ,format = "%Y-%m-%d %H:%M:%S", tz = "GMT")
attributes(temp$date)$tzone <- "America/New_York"

ordwayPlot <- qplot (date,mean,data=temp,geom="line", title="ordwayData",
main="Mean Temperature - Ordway Swisher", xlab="Date",
ylab="Mean Temperature (Degrees C)")

ordwayPlot <- ggplot(temp,aes(x=date,y=mean))+
geom_path()+
ylab("Mean temperature") + xlab("Date")+
theme_bw()+
ggtitle("3 Days of temperature data at Ordway Swisher")+
scale_x_datetime(breaks=pretty_breaks(n=4))

#let's check out the plot
ordwayPlot

From: Leah Wasser [mailto:[email protected]]
Sent: Tuesday, May 26, 2015 10:34 PM
To: lwasser/NEON-Data-Skills
Cc: Sarah Elmendorf
Subject: Re: [NEON-Data-Skills] HDF-5 temp data - is the datetime conversion wrong? (#35)

that would be great -- thanks! i might be running it incorrectly. i totally agree - the numbers don't make sense and i never really focused enough on that sadly as i've been so focused on the code. i just cant seem to implement your changes. thanks @scelmendorfhttps://github.com/scelmendorf

—
Reply to this email directly or view it on GitHubhttps://github.com/lwasser/NEON-Data-Skills/issues/35#issuecomment-105753747.

[lwasser]

ahhh ok i got it :) hey thanks! so do you think it makes sense for max temp to fall in the late afternoon like that? isnt it normally around 3 or something? i suppose florida in an exposed spot might be different?

[scelmendorf]

Google suggests yes this makes sense:
https://www.google.com/search?q=diurnal+temperature+curve&es_sm=93&tbm=isch&tbo=u&source=univ&sa=X&ei=tUplVZjfGMvisAW_0oLgBA&ved=0CDIQsAQ&biw=1455&bih=705#imgrc=QbXX8JV8aTBrCM%253A%3BPiUwWdmJl9pBLM%3Bhttp%253A%252F%252Fweb.gccaz.edu%252F~lnewman%252Fgph111%252Ftopic_units%252Ftemperature1%252Ftemperature%252Fheating-curve.jpg%3Bhttp%253A%252F%252Fweb.gccaz.edu%252F~lnewman%252Fgph111%252Ftopic_units%252Ftemperature1%252Ftemperature%252Ftemperature2.html%3B1088%3B632

From: Leah Wasser [mailto:[email protected]]
Sent: Tuesday, May 26, 2015 10:39 PM
To: lwasser/NEON-Data-Skills
Cc: Sarah Elmendorf
Subject: Re: [NEON-Data-Skills] HDF-5 temp data - is the datetime conversion wrong? (#35)

ahhh ok i got it :) hey thanks! so do you think it makes sense for max temp to fall in the late afternoon like that? isnt it normally around 3 or something? i suppose florida in an exposed spot might be different?

—
Reply to this email directly or view it on GitHubhttps://github.com/lwasser/NEON-Data-Skills/issues/35#issuecomment-105754023.

[lwasser]

yea -- that's it!! thank you :) i'll update tomorrow probably. i have to fix one more thing on the back end. i'm moving to RMD files so i am writing this script that knits and purls and such to files that git can play nicely with. i have to do a quick conversion on this file!

i'll make the changes and post tomorrow - thank you for catching this. what a good catch.

[lwasser]

hahaha look what you have just allowed us to create- a LEARNING MOMENT for our learners. hmmmm...

can i use your google reference?? brilliant.
Take away - be careful with dates. ;)
leah

[lwasser]

oh right and a learning moment for yours truly as well 👍

[scelmendorf]

Also, many years of graduate education in the central valley of California (hot), including some months of very Hot and very Pregnant and Very miserable with no AC at home suggest 5pm is really – always -- the worst time of day. You can learn a lot in grad school.

From: Leah Wasser [mailto:[email protected]]
Sent: Tuesday, May 26, 2015 10:42 PM
To: lwasser/NEON-Data-Skills
Cc: Sarah Elmendorf
Subject: Re: [NEON-Data-Skills] HDF-5 temp data - is the datetime conversion wrong? (#35)

hahaha look what you have just allowed us to create- a LEARNING MOMENT for our learners. hmmmm...

can i use your google reference?? brilliant.
Take away - be careful with dates. ;)
leah

—
Reply to this email directly or view it on GitHubhttps://github.com/lwasser/NEON-Data-Skills/issues/35#issuecomment-105754256.

[lwasser]

oh my!
ok -- i'm signing off as i'm tired from too much manual labor today (house work). i'll dig in again tomorrow and see how far i can get. at the least, i'll fix the code. i'm not sure if folks have already downloaded the data or not. i might have a few questions for you tomorrow if you don't mind!
thanks again!

[scelmendorf]

• more intuitive code suggestion to achieve the same result (since we think in UTC and eastern):

#read in as UTC
temp$date <- as.POSIXct(temp$date ,format = "%Y-%m-%d %H:%M:%S", tz = "UTC")
#convert to local time for pretty plotting
attributes(temp$date)$tzone <- "US/Eastern" 

• Another idea might be in the exercise to create the code to specify the timezone formatting in the hdf5 metadata. So were one to read the metadata you’d know that it’s reported in UTC, but the local time is US/Eastern, giving you all the smartipants tools you’d need to convert it.
• And last - yaxis beautification suggestion for the qplot
  ylab=expression(paste("Temperature (",degree*C, ")")

[lwasser]

ok - @scelmendorf - here's a first stab. i haven't pushed this yet. Can yo uplease read this and let me know if it makes sense. i will be honest - i don't fully understand what UTC is. It's time without daylight savings. it is like a global clock? i want to explain it right!

Also i noticed we have two types of plotting. should i just use GGPLOT for all plots??
thanks for your feedback on this :)

# Let's clean up the plot abovet. We can first add dates to the x axis. 
# In order to list dates, we need to specify the format that the date field is in.
# First read in the time as UTC format
temp$date <- as.POSIXct(temp$date ,format = "%Y-%m-%d %H:%M:%S", tz = "GMT")



ordwayPlot <- qplot (date,mean,data=temp,geom="line", title="ordwayData",
                 main="Mean Temperature - Ordway Swisher", 
                 xlab="Date / time in UTC", 
                 ylab="Mean Temperature (Degrees C)")
ordwayPlot

Let's have a close look at this plot. Notice anything unusual with it?
HINT: when would you expect the temperature to be the warmest during the day?

In many parts of the world, we'd expect the maximum temperature to occur during late afternoon.

In this case, our data are in UTC time zone. The UTC time zone is a standardized time zone
that does not observe daylight savings time. If your data are in UTC, then you
will need to CONVERT them to your local time zone for the dates and times to
make sense when you plot / analyze your data.

For example, find the time zone for Denver, Colorado in this

wikipedia page. How many hours difference is UTC from the local time in
Denver, Colorado?

To adjust for time, we need to tell R that the time zone is eastern standard. We
can use the attributes function to set the time zone.

#convert to local time for pretty plotting
attributes(temp$date)$tzone <- "America/New_York"

ordwayPlot2 <- ggplot(temp,aes(x=date,y=mean))+
  geom_path()+
  ylab("Mean temperature") + xlab("Date in Eastern Standard Time")+
  theme_bw()+
  ggtitle("3 Days of Temperature Data at Ordway Swisher")+
  scale_x_datetime(breaks=pretty_breaks(n=4))

#let's check out the plot
ordwayPlot2

[scelmendorf]

Yeah – more or less, UTC is like the goldstandard for times which allow us to represent publish all our sensor data on the same ’clock’ without worrying about location and/or daylight savings. It’s handy because the utc time is the same everywhere on earth, unlike local time. But for some analyses, it’s easier to work in local time.

I would suggestion you change this line:

temp$date <- as.POSIXct(temp$date ,format = "%Y-%m-%d %H:%M:%S", tz = "GMT")

to

temp$date <- as.POSIXct(temp$date ,format = "%Y-%m-%d %H:%M:%S", tz = "UTC")

Just so people understand the rationale for the digression into UTC

And this to

#convert to local time for pretty plotting

attributes(temp$date)$tzone <- "US/Eastern"

And for what it’s worth I think (confession – I did not look this up, but I would guess since there doesn’t appear to be a daylight variant)that the “US/Eastern” knows based on date whether it’s standard or daylight. Not sure what the dates are in your example, but might want to just say in ‘Eastern Time’ rather than ‘Eastern Standard Time’, it could be during a daylight savings part of the year??

Re gggplot:
I don’t think it matters whether you use ggplot or qplot, my understanding is qplot is a shortcut for some of the simpler ggplot routines.

From: Leah Wasser [mailto:[email protected]]
Sent: Wednesday, May 27, 2015 12:59 PM
To: lwasser/NEON-Data-Skills
Cc: Sarah Elmendorf
Subject: Re: [NEON-Data-Skills] HDF-5 temp data - is the datetime conversion wrong? (#35)

ok - @scelmendorfhttps://github.com/scelmendorf - here's a first stab. i haven't pushed this yet. Can yo uplease read this and let me know if it makes sense. i will be honest - i don't fully understand what UTC is. It's time without daylight savings. it is like a global clock? i want to explain it right!

Also i noticed we have two types of plotting. should i just use GGPLOT for all plots??
thanks for your feedback on this :)

Let's clean up the plot abovet. We can first add dates to the x axis.

In order to list dates, we need to specify the format that the date field is in.

First read in the time as UTC format

temp$date <- as.POSIXct(temp$date ,format = "%Y-%m-%d %H:%M:%S", tz = "GMT")

ordwayPlot <- qplot (date,mean,data=temp,geom="line", title="ordwayData",

             main="Mean Temperature - Ordway Swisher",

             xlab="Date / time in UTC",

             ylab="Mean Temperature (Degrees C)")

ordwayPlot

Let's have a close look at this plot. Notice anything unusual with it?
HINT: when would you expect the temperature to be the warmest during the day?
[https://camo.githubusercontent.com/bf77bbc40a21da12c0f27da8e325b588d3ce6762/687474703a2f2f7765622e676363617a2e6564752f7e6c6e65776d616e2f6770683131312f746f7069635f756e6974732f74656d7065726174757265312f74656d70657261747572652f68656174696e672d63757276652e6a7067]In many parts of the world, we'd expect the maximum temperature to occur during late afternoon.

In this case, our data are in UTC time zone.http://en.wikipedia.org/wiki/Coordinated_Universal_Time The UTC time zone is a standardized time zone
that does not observe daylight savings time. If your data are in UTC, then you
will need to CONVERT them to your local time zone for the dates and times to
make sense when you plot / analyze your data.

For example, find the time zone for Denver, Colorado in this

wikipedia page.http://en.wikipedia.org/wiki/List_of_tz_database_time_zones How many hours difference is UTC from the local time in
Denver, Colorado?

To adjust for time, we need to tell R that the time zone is eastern standard. We
can use the attributes function to set the time zone.

#convert to local time for pretty plotting

attributes(temp$date)$tzone <- "America/New_York"

ordwayPlot2 <- ggplot(temp,aes(x=date,y=mean))+

geom_path()+

ylab("Mean temperature") + xlab("Date in Eastern Standard Time")+

theme_bw()+

ggtitle("3 Days of Temperature Data at Ordway Swisher")+

scale_x_datetime(breaks=pretty_breaks(n=4))

#let's check out the plot

ordwayPlot2

—
Reply to this email directly or view it on GitHubhttps://github.com/lwasser/NEON-Data-Skills/issues/35#issuecomment-106036445.

[lwasser]

Hey thanks -- i have a few more tweaks to the lesson but this is all making sense :) fantastic catch on that! how did we all miss it for the past year?!! :)

One last question. where are you getting the list of options for time zones. I totally get how it works now AND if you use and set an incorrect time zone

attributes(temp$date)$tzone <- "US/Eastern"

it barfs when you call it

attributes(temp$date)$tzone

But my question is -- how do we look up the various time zone options in terms of syntax / spelling, etc?

this is cool to see our current time zone
Sys.timezone()

[lwasser]

i found this too..
https://stat.ethz.ch/R-manual/R-devel/library/base/html/timezones.html
str(OlsonNames())

[scelmendorf]

I found them here:
http://en.wikipedia.org/wiki/List_of_tz_database_time_zones

not sure which ones R knows about, I would have assumed all the wikipedia ones but maybe not?

[chrlaney]

It's been a while since I've looked at this in detail, but https://stat.ethz.ch/R-manual/R-devel/library/base/html/timezones.html has good info.

[lwasser]

Thanks guys! I think my thick brain is finally wrapping itself around this concept and absorbing :)
much appreciated!!! I think it will be interesting to see how folks respond to this time series activity tomorrow. It really builds in complexity. it will be interesting to see how it's received and where folks get hung up!

[lwasser]

ahhh -- i found a few more opportunities to tweak the code, simplify significantly and highlight the programmatic power of HDF5 via embedded metadata! it will involve actually modifying the file structure so i won't do it tomorrow BUT.. would you guys be up for looking at it with me sometime before august and just coming up with a few general conventions for teaching purposes? it doesn't have to be the NEON final standard.

i'd like to add some better machine readable attributes like
tower level: 1 or Level_1
Latitude:
Longitude:
etc etc... you get the idea i think :) woo hoo HDF5.
see you both manana :)

[chrlaney]

Sorry about sending you a url you already posted, leah - didn't see that! :)

[lwasser]

Hey @scelmendorf i wanted to clear this up before moving forward

if you go to the documentation

https://stat.ethz.ch/R-manual/R-devel/library/base/html/as.POSIXlt.html
it says:
tz
A time zone specification to be used for the conversion, if one is required. System-specific (see time zones), but "" is the current time zone, and "GMT" is UTC (Universal Time, Coordinated). Invalid values are most commonly treated as UTC, on some platforms with a warning.

---

so in essence i read that as R uses GMT to represent UTC. but if you type UTC it could luckily default to UTC because that's the default when it's uncertain. i'd rather be precise in the code than risk R doing something weird. But have a look and let me know what you think. then i'll adjust and close this as we see fit.
Thank you!
leah

[neondataskills]

This appears to have been corrected in the lesson: http://neondataskills.org/HDF5/TimeSeries-Data-In-HDF5-Using-R/