lewismc / ipres

iPReS - An Internationalization (i18n) Product Retrieval Service for NASA JPL's PO.DAAC

iPReS

Introduction

The Internationalization (i18n) Product Retrieval Service is a web service and client providing i18n-type access to products and product metadata contained within NASA JPL's Physical Oceanography Distributed Active Archive Center otherwise known as PO.DAAC.

The software implements a RESTful web-service and client to obtain data from the NASA JPL's PO.DAAC Web-Services API. It then translates the product metadata into a target language provided along with the initial call to the service.

Citation

There are a couple of ways to cite iPReS.

IEEE 2015 Paper

L. J. McGibbney, K. D. Whitehall, C. A. Mattmann and P. M. Carter, "Enabling Linguistic Analysis of Scientific Metadata through Internationalizing NASA JPL's PODAAC," 2015 IEEE International Conference on Information Reuse and Integration, San Francisco, CA, 2015, pp. 207-210. doi: 10.1109/IRI.2015.41 - URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7300977&isnumber=7300933

Note, if you cannot access the above paper, please check out the IEEE IRI 2015 Poster

Zenodo

Supported Product Translations

Approaching one hundred... see here for an out-the-box list.

It should be noted that because iPReS relies upon Apache Tika's Translate module, the supported translations differ depending on which translation service implementation you decide to use. By default iPReS uses Tika's YandexTranslator meaning that you need to create a Yandex API Key (click login on top right to create account and API key). Once you have this key, simply edit ./app/resources/org/apache/tika/language/translate/translator.yandex.properties and you are good to go!

The following translation implementations are available

• Google Translate
• Apache Joshua
• Lingo24
• Bing (Microsoft) Translator
• Moses statistical machine translation system
• Yandex

In order to configure one of the above the steps are always the same

• Obtain credentials for using the target translation service
• Edit the relevant properties file in ./app/resources/org/apache/tika/language/translate/
• Replace all instances of YandexTranslator in ./app/src/app/core.clj with the tika-translate implementation you wish to use.

Installation

Please refer to the Application Installation instructions. This also details the prerequisite software installation requirements.

Project Website

The project website can be located at - http://lewismc.github.io/iPReS/

Community, Support and Development

Community Mailing List

Please visit and subscribe to our Google Group for project news and development.

Issue Tracking

Please open a ticket in the issue tracker. Please use labels to classify your issue.

License

iPReS is licensed permissively under the Apache License v2.0. A copy of that license is distributed with this software.

Copyright and Export Classification

Copyright 2016, by the California Institute of Technology. ALL RIGHTS RESERVED.
United States Government Sponsorship acknowledged. Any commercial use must be
negotiated with the Office of Technology Transfer at the California Institute
of Technology.
This software may be subject to U.S. export control laws. By accepting this software,
the user agrees to comply with all applicable U.S. export laws and regulations.
User has the responsibility to obtain export licenses, or other export authority
as may be required before exporting such information to foreign countries or
providing access to foreign persons.

ipres's Issues

NPE when starting Lein server at localhost

When I run lein ring server from within the app directory I get the following NPE

Retrieving ring/ring-servlet/1.2.1/ring-servlet-1.2.1.jar from clojars
Retrieving ring-refresh/ring-refresh/0.1.2/ring-refresh-0.1.2.jar from clojars
Retrieving watchtower/watchtower/0.1.1/watchtower-0.1.1.jar from clojars
2015-04-12 19:01:00.008:INFO:oejs.Server:jetty-7.6.8.v20121106
2015-04-12 19:01:00.049:INFO:oejs.AbstractConnector:Started [email protected]:3000
Started server on port 3000
java.lang.NullPointerException: null
                                 core.clj:113 compojure.core/routing[fn]
                                core.clj:2515 clojure.core/some
                                 core.clj:113 compojure.core/routing
                              RestFn.java:423 clojure.lang.RestFn.invoke
                              handler.clj:166 app.handler/fn
                                 core.clj:177 compojure.core/wrap-context[fn]
                                  core.clj:46 compojure.core/if-route[fn]
                                 core.clj:113 compojure.core/routing[fn]
                                core.clj:2515 clojure.core/some
                                 core.clj:113 compojure.core/routing
                              RestFn.java:139 clojure.lang.RestFn.applyTo
                                 core.clj:626 clojure.core/apply
                                 core.clj:118 compojure.core/routes[fn]
                        keyword_params.clj:35 ring.middleware.keyword-params/wrap-keyword-params[fn]
                         nested_params.clj:84 ring.middleware.nested-params/wrap-nested-params[fn]
                                params.clj:64 ring.middleware.params/wrap-params[fn]
                                  json.clj:65 ring.middleware.json/wrap-json-response[fn]
                                 Var.java:379 clojure.lang.Var.invoke
                                reload.clj:18 ring.middleware.reload/wrap-reload[fn]
                            stacktrace.clj:17 ring.middleware.stacktrace/wrap-stacktrace-log[fn]
                            stacktrace.clj:80 ring.middleware.stacktrace/wrap-stacktrace-web[fn]
                                 jetty.clj:18 ring.adapter.jetty/proxy-handler[fn]
                             (Unknown Source) ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle
                      HandlerWrapper.java:116 org.eclipse.jetty.server.handler.HandlerWrapper.handle
                              Server.java:363 org.eclipse.jetty.server.Server.handle
              AbstractHttpConnection.java:483 org.eclipse.jetty.server.AbstractHttpConnection.handleRequest
              AbstractHttpConnection.java:920 org.eclipse.jetty.server.AbstractHttpConnection.headerComplete
              AbstractHttpConnection.java:982 org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete
                          HttpParser.java:635 org.eclipse.jetty.http.HttpParser.parseNext
                          HttpParser.java:235 org.eclipse.jetty.http.HttpParser.parseAvailable
                  AsyncHttpConnection.java:82 org.eclipse.jetty.server.AsyncHttpConnection.handle
               SelectChannelEndPoint.java:628 org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle
                SelectChannelEndPoint.java:52 org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run
                    QueuedThreadPool.java:608 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
                    QueuedThreadPool.java:543 org.eclipse.jetty.util.thread.QueuedThreadPool$3.run
                              Thread.java:745 java.lang.Thread.run
java.lang.NullPointerException: null
                                 core.clj:113 compojure.core/routing[fn]
                                core.clj:2515 clojure.core/some
                                 core.clj:113 compojure.core/routing
                              RestFn.java:423 clojure.lang.RestFn.invoke
                              handler.clj:166 app.handler/fn
                                 core.clj:177 compojure.core/wrap-context[fn]
                                  core.clj:46 compojure.core/if-route[fn]
                                 core.clj:113 compojure.core/routing[fn]
                                core.clj:2515 clojure.core/some
                                 core.clj:113 compojure.core/routing
                              RestFn.java:139 clojure.lang.RestFn.applyTo
                                 core.clj:626 clojure.core/apply
                                 core.clj:118 compojure.core/routes[fn]
                        keyword_params.clj:35 ring.middleware.keyword-params/wrap-keyword-params[fn]
                         nested_params.clj:84 ring.middleware.nested-params/wrap-nested-params[fn]
                                params.clj:64 ring.middleware.params/wrap-params[fn]
                                  json.clj:65 ring.middleware.json/wrap-json-response[fn]
                                 Var.java:379 clojure.lang.Var.invoke
                                reload.clj:18 ring.middleware.reload/wrap-reload[fn]
                            stacktrace.clj:17 ring.middleware.stacktrace/wrap-stacktrace-log[fn]
                            stacktrace.clj:80 ring.middleware.stacktrace/wrap-stacktrace-web[fn]
                                 jetty.clj:18 ring.adapter.jetty/proxy-handler[fn]
                             (Unknown Source) ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle
                      HandlerWrapper.java:116 org.eclipse.jetty.server.handler.HandlerWrapper.handle
                              Server.java:363 org.eclipse.jetty.server.Server.handle
              AbstractHttpConnection.java:483 org.eclipse.jetty.server.AbstractHttpConnection.handleRequest
              AbstractHttpConnection.java:920 org.eclipse.jetty.server.AbstractHttpConnection.headerComplete
              AbstractHttpConnection.java:982 org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete
                          HttpParser.java:635 org.eclipse.jetty.http.HttpParser.parseNext
                          HttpParser.java:235 org.eclipse.jetty.http.HttpParser.parseAvailable
                  AsyncHttpConnection.java:82 org.eclipse.jetty.server.AsyncHttpConnection.handle
               SelectChannelEndPoint.java:628 org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle
                SelectChannelEndPoint.java:52 org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run
                    QueuedThreadPool.java:608 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
                    QueuedThreadPool.java:543 org.eclipse.jetty.util.thread.QueuedThreadPool$3.run
                              Thread.java:745 java.lang.Thread.run

Add Installation Instructions

This is basic stuff.
We need to add a description about how in gods name people install and use the software as well as what the prerequisites are.

Integration Tests for full service

Add pretty printing as an optional paramater to the JSON Response

Right now we get a JSON response object and there is no option to pretty print it so it is difficult to interpret at a glance.
We should consider implementing something like
https://clojuredocs.org/clojure.pprint
as a mechanism for improving aesthetics and interpretation.

Deploy Clojure Demo

• Install whatever is necessary on CentOS box
• Deploy Clojure demo to CentOS box
• Test

Apache Tika to translate

• Utilizes Tika to translate data and metadata into target language
• Tika uses Joshua Translator
• Unit Tests

Develop layer which talks to clients

• Create a ring/compojure layer, similar to the demo, for all routes
• Unit test with mocking

Create iPReS Project Website

It would be really nice for us to create a project website using the free tier provided to us via Github.
The site should be where we keep all data relating to the project as well as links to all of our meeting correspondence, etc.

Async calls to PO.DAAC and Cache

• Based on request info, makes async calls to the distributed cache and PO.DAAC
• Synchronizes return values
• Unit Tests

Request analyzing layer

• Analyzes a Clojure map, figuring out what data set(s) were requested by a client
• Unit Tests

Load Tests for full service

Augment APP README when New Functionality is Added to the REST API

Previously, the App README had a comment

...rest are TBD until they appear to work correctly.

The comment related to the section on https://github.com/lewismc/iPReS/tree/master/app#example-usage

Translation not working in Master

When I use the Tika Microsoft Translator @cartermp added to master, regardless of which language code I enter in the REST call, I can only obtain responses as English 'en'.
Anyone can try this out to replicate it. I am not sure why this is the case. It seems that no language translation is being done at all.

Add key's to generated response

As oppose to returning an array of values within a query response, we should return a map which details keys. Metadata keys are extremely important as they define a vocabulary for referencing data fields.

Embed Google Group in Project Website

We can use the following code to embed the group in our project website

<iframe id="forum_embed"
  src="javascript:void(0)"
  scrolling="no"
  frameborder="0"
  width="900"
  height="700">
</iframe>
<script type="text/javascript">
  document.getElementById('forum_embed').src =
     'https://groups.google.com/forum/embed/?place=forum/ipres-capstone'
     + '&showsearch=true&showpopout=true&showtabs=false'
     + '&parenturl=' + encodeURIComponent(window.location.href);
</script> 

At NOTICE file to repos and License Headers to al Source Code

Hi Folks,
NOTICE file should be similar to the following
https://github.com/apache/nutch/blob/trunk/NOTICE.txt
License file headers are important.
They should all be Apache v2.0 License Headers

Tests failing in master branch

Hi Folks,
The tests are broken in master branch

lmcgibbn@LMC-032857 /usr/local/iPReS/app(master) $ lein test

lein test app.cache-test

lein test app.core-test
[20120912-MSG02-OSDPD-L2P-MSG02_0200Z-v01.nc
 2012-09-12T02:41:26Z
 ftp://ftp.nodc.noaa.gov/pub/data.nodc/ghrsst/L2P/MSG02/OSDPD/2012/256/20120912-MSG02-OSDPD-L2P-MSG02_0200Z-v01.nc.gz
 20120912-MSG02-OSDPD-L2P-MSG02_0200Z-v01.nc
 2012-09-12T02:45:54.611000Z
 publication
 -75.11000061035156
 75.12300109863281
 -80.5510025024414
 67.36199951171875
 2012-09-12T02:00:00Z
 2012-09-12T02:12:00Z
 OSDPD-L2P-MSG02
 eng
 UTF8
 series
 Eileen Maturi
 OSDPD > NOAA Office of Satellite Data Processing and Distribution
 Technical Contact
 301-763-8102 x172
 301-763-8572
 [email protected]
 Phone/FAX/E-mail
 pointOfContact
 20150413
 ISO 19115-2 Geographic information — Metadata — Part 2: Extensions for imagery and gridded data
 ISO 19115-2:2009-02-15
 eng
 US
 UTF8
 http://www.ngdc.noaa.gov/metadata/published/19115/GHRSST/ISO/CoverageExtensions.xml
 Web Browser
 A description of extensions developed at NGDC to classify coverages.
 information
 GHRSST Level 2P Atlantic Regional Skin Sea Surface Temperature from the Spinning Enhanced Visible and InfraRed Imager (SEVIRI) on the Meteosat Second Generation (MSG-2) satellite
 Sea Surface Temperature, 55W-55E and 55S-55N, at 0.05 degree resolution from MSG-02 Imager Sectors every quarter-hour
 20090727
 creation
 4
 NOAA/NESDIS/OSDPD
 originator
 NOAA/NESDIS
 Camp Springs, MD (USA)
 publisher
 The Meteosat Second Generation (MSG) satellites  are spin stabilized geostationary satellites operated by the European Organization for the Exploitation of Meteorological Satellites (EUMETSAT) to provide accurate weather monitoring data through its primary instrument the Spinning Enhanced Visible and InfraRed Imager (SEVIRI), which has the capacity to observe the Earth in 12 spectral channels. Eight of these channels are in the thermal infrared, providing among other information, observations of the temperatures of clouds, land and sea surfaces at approximately 5 km resolution with a 15 minute duty cycle. This Group for High Resolution Sea Surface Temperature (GHRSST) dataset produced by the US National Oceanographic and Atmospheric Administration (NOAA)  National Environmental Satellite, Data, and Information Service (NESDIS) is derived from the SEVIRI instrument on the second MSG satellite (also known as  Meteosat-9) that was launched on 22 December 2005. Skin sea surface temperature (SST) data are calculated from the infrared channels of SEVIRI at full resolution every 15 minutes. L2P data products with Single Sensor Error Statistics (SSES) are then derived following the GHRSST-PP Data Processing Specification (GDS) version 1.5.
 These data are produced by NOAA/NESDIS funded by NESDIS Office of System Development
 onGoing
 Eileen Maturi
 OSDPD > NOAA Office of Satellite Data Processing and Distribution
 301-763-8102 x172
 301-763-8572
 [email protected]
 pointOfContact
 NETCDF
 3
 GZIP
 Earth Science > Oceans > Ocean Temperature > Sea Surface Temperature > Sea Surface Skin Temperature
 theme
 NASA/GCMD Earth Science Keywords
 Eastern Atlantic
 place
 NASA/GCMD Location Keywords
 None
 None
 grid
 eng
 UTF8
 true
 -81
 81
 -73
 73
 true
 2009-11-22T12:30:00Z
 referenceInformation
 lat
 float
 lon
 float
 time
 int
 PO.DAAC User Services
 NASA/JPL/PODAAC > Physical Oceanography Distributed Active Archive Center, Jet Propulsion Laboratory, NASA
 4800 Oak Grove Drive
 Pasadena
 CA
 91109-8099
 USA
 [email protected]
 http://podaac.jpl.nasa.gov
 distributor
 http://podaac.jpl.nasa.gov/ws/search/granule/?datasetId=PODAAC-GHMG2-2PO01&apidoc
 Web Service (PO.DAAC Labs)
 (Search Granule)
 information
 http://www.ghrsst.org
 none
 GHRSST Project Home Page
 information
 http://ghrsst.jpl.nasa.gov
 none
 Portal to the GHRSST Global Data Assembly Center and data access
 information
 asNeeded
 Translated from GCMD DIF
 SEVIRI > Spinning Enhanced Visible and Infrared Imager
 sensor
 The Spinning Enhanced Visible and InfraRed Imager (SEVIRI)  has the capacity to observe the Earth in 12 spectral channels. Eight of these channels are in the thermal infrared, providing among other information, observations of the temperatures of clouds, land and sea surfaces at approximately 5 km resolution with a 15 minute duty cycle.
 MSG > Meteosat Second Generation
 Meteosat Second Generation (MSG) provides multi-spectral imagery of the Earth's surface and cloud systems from a geostationary orbit.
 NOAA/NESDIS/OSDPD
 NASA/JPL/PODAAC > Physical Oceanography Distributed Active Archive Center, Jet Propulsion Laboratory, NASA
 http://podaac.jpl.nasa.gov
]

lein test :only app.core-test/translate-request-test-korean

ERROR in (translate-request-test-korean) (FutureTask.java:122)
that a call to translate-request correctly goes through all layers.
expected: (= true (and (= "some climate data again" (translate-request "metadata/dataset" {} "kr" "pdf")) (= true (cache/cache-has? :metadatadatasetkr)) (= "some climate data again" (cache/cache-lookup :metadatadatasetkr))))
  actual: java.util.concurrent.ExecutionException: java.lang.ClassCastException: Cannot cast java.lang.Character to java.lang.String
 at java.util.concurrent.FutureTask.report (FutureTask.java:122)
    java.util.concurrent.FutureTask.get (FutureTask.java:188)
    clojure.core$deref_future.invoke (core.clj:2180)
    clojure.core$future_call$reify__6320.deref (core.clj:6420)
    clojure.core$deref.invoke (core.clj:2200)
    clojure.core$pmap$step__6333$fn__6335.invoke (core.clj:6470)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.LazySeq.equals (LazySeq.java:115)
    clojure.lang.LazySeq.equiv (LazySeq.java:100)
    clojure.lang.Util.pcequiv (Util.java:125)
    clojure.lang.Util.equiv (Util.java:32)
    app.core_test$fn__1183$fn__1186.invoke (core_test.clj:65)
    clojure.core$with_redefs_fn.invoke (core.clj:6861)
    app.core_test/fn (core_test.clj:63)
    clojure.test$test_var$fn__7187.invoke (test.clj:704)
    clojure.test$test_var.invoke (test.clj:704)
    clojure.test$test_vars$fn__7209$fn__7214.invoke (test.clj:722)
    clojure.test$default_fixture.invoke (test.clj:674)
    clojure.test$test_vars$fn__7209.invoke (test.clj:722)
    clojure.test$default_fixture.invoke (test.clj:674)
    clojure.test$test_vars.invoke (test.clj:718)
    clojure.test$test_all_vars.invoke (test.clj:728)
    clojure.test$test_ns.invoke (test.clj:747)
    clojure.core$map$fn__4245.invoke (core.clj:2559)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.Cons.next (Cons.java:39)
    clojure.lang.RT.boundedLength (RT.java:1654)
    clojure.lang.RestFn.applyTo (RestFn.java:130)
    clojure.core$apply.invoke (core.clj:626)
    clojure.test$run_tests.doInvoke (test.clj:762)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invoke (core.clj:624)
    user$eval85$fn__140$fn__171.invoke (form-init6878369847205765737.clj:1)
    user$eval85$fn__140$fn__141.invoke (form-init6878369847205765737.clj:1)
    user$eval85$fn__140.invoke (form-init6878369847205765737.clj:1)
    user$eval85.invoke (form-init6878369847205765737.clj:1)
    clojure.lang.Compiler.eval (Compiler.java:6703)
    clojure.lang.Compiler.eval (Compiler.java:6693)
    clojure.lang.Compiler.load (Compiler.java:7130)
    clojure.lang.Compiler.loadFile (Compiler.java:7086)
    clojure.main$load_script.invoke (main.clj:274)
    clojure.main$init_opt.invoke (main.clj:279)
    clojure.main$initialize.invoke (main.clj:307)
    clojure.main$null_opt.invoke (main.clj:342)
    clojure.main$main.doInvoke (main.clj:420)
    clojure.lang.RestFn.invoke (RestFn.java:421)
    clojure.lang.Var.invoke (Var.java:383)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.main.main (main.java:37)
Caused by: java.lang.ClassCastException: Cannot cast java.lang.Character to java.lang.String
 at java.lang.Class.cast (Class.java:3094)
    clojure.lang.Reflector.boxArg (Reflector.java:427)
    clojure.lang.Reflector.boxArgs (Reflector.java:460)
    clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:58)
    clojure.lang.Reflector.invokeInstanceMethod (Reflector.java:28)
    app.core$translate_with_tika$fn__1105.invoke (core.clj:126)
    clojure.core$pmap$fn__6328$fn__6329.invoke (core.clj:6466)
    clojure.core$binding_conveyor_fn$fn__4145.invoke (core.clj:1910)
    clojure.lang.AFn.call (AFn.java:18)
    java.util.concurrent.FutureTask.run (FutureTask.java:262)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)
    java.lang.Thread.run (Thread.java:745)

lein test :only app.core-test/translate-request-test-spanish

ERROR in (translate-request-test-spanish) (FutureTask.java:122)
that a call to translate-request correctly goes through all layers.
expected: (= true (and (= "some climate data" (translate-request "metadata/dataset" {} "es" "pdf")) (= true (cache/cache-has? :metadatadatasetes)) (= "some climate data" (cache/cache-lookup :metadatadatasetes))))
  actual: java.util.concurrent.ExecutionException: java.lang.ClassCastException: Cannot cast java.lang.Character to java.lang.String
 at java.util.concurrent.FutureTask.report (FutureTask.java:122)
    java.util.concurrent.FutureTask.get (FutureTask.java:188)
    clojure.core$deref_future.invoke (core.clj:2180)
    clojure.core$future_call$reify__6320.deref (core.clj:6420)
    clojure.core$deref.invoke (core.clj:2200)
    clojure.core$pmap$step__6333$fn__6335.invoke (core.clj:6470)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.LazySeq.equals (LazySeq.java:115)
    clojure.lang.LazySeq.equiv (LazySeq.java:100)
    clojure.lang.Util.pcequiv (Util.java:125)
    clojure.lang.Util.equiv (Util.java:32)
    app.core_test$fn__1172$fn__1175.invoke (core_test.clj:57)
    clojure.core$with_redefs_fn.invoke (core.clj:6861)
    app.core_test/fn (core_test.clj:55)
    clojure.test$test_var$fn__7187.invoke (test.clj:704)
    clojure.test$test_var.invoke (test.clj:704)
    clojure.test$test_vars$fn__7209$fn__7214.invoke (test.clj:722)
    clojure.test$default_fixture.invoke (test.clj:674)
    clojure.test$test_vars$fn__7209.invoke (test.clj:722)
    clojure.test$default_fixture.invoke (test.clj:674)
    clojure.test$test_vars.invoke (test.clj:718)
    clojure.test$test_all_vars.invoke (test.clj:728)
    clojure.test$test_ns.invoke (test.clj:747)
    clojure.core$map$fn__4245.invoke (core.clj:2559)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.Cons.next (Cons.java:39)
    clojure.lang.RT.boundedLength (RT.java:1654)
    clojure.lang.RestFn.applyTo (RestFn.java:130)
    clojure.core$apply.invoke (core.clj:626)
    clojure.test$run_tests.doInvoke (test.clj:762)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invoke (core.clj:624)
    user$eval85$fn__140$fn__171.invoke (form-init6878369847205765737.clj:1)
    user$eval85$fn__140$fn__141.invoke (form-init6878369847205765737.clj:1)
    user$eval85$fn__140.invoke (form-init6878369847205765737.clj:1)
    user$eval85.invoke (form-init6878369847205765737.clj:1)
    clojure.lang.Compiler.eval (Compiler.java:6703)
    clojure.lang.Compiler.eval (Compiler.java:6693)
    clojure.lang.Compiler.load (Compiler.java:7130)
    clojure.lang.Compiler.loadFile (Compiler.java:7086)
    clojure.main$load_script.invoke (main.clj:274)
    clojure.main$init_opt.invoke (main.clj:279)
    clojure.main$initialize.invoke (main.clj:307)
    clojure.main$null_opt.invoke (main.clj:342)
    clojure.main$main.doInvoke (main.clj:420)
    clojure.lang.RestFn.invoke (RestFn.java:421)
    clojure.lang.Var.invoke (Var.java:383)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.main.main (main.java:37)
Caused by: java.lang.ClassCastException: Cannot cast java.lang.Character to java.lang.String
 at java.lang.Class.cast (Class.java:3094)
    clojure.lang.Reflector.boxArg (Reflector.java:427)
    clojure.lang.Reflector.boxArgs (Reflector.java:460)
    clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:58)
    clojure.lang.Reflector.invokeInstanceMethod (Reflector.java:28)
    app.core$translate_with_tika$fn__1105.invoke (core.clj:126)
    clojure.core$pmap$fn__6328$fn__6329.invoke (core.clj:6466)
    clojure.core$binding_conveyor_fn$fn__4145.invoke (core.clj:1910)
    clojure.lang.AFn.call (AFn.java:18)
    java.util.concurrent.FutureTask.run (FutureTask.java:262)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)
    java.lang.Thread.run (Thread.java:745)
[OSDPD-L2P-MSG02
 eng
 UTF8
 series
 Eileen Maturi
 OSDPD > NOAA Office of Satellite Data Processing and Distribution
 Technical Contact
 301-763-8102 x172
 301-763-8572
 [email protected]
 Phone/FAX/E-mail
 pointOfContact
 20150413
 ISO 19115-2 Geographic information — Metadata — Part 2: Extensions for imagery and gridded data
 ISO 19115-2:2009-02-15
 eng
 US
 UTF8
 http://www.ngdc.noaa.gov/metadata/published/19115/GHRSST/ISO/CoverageExtensions.xml
 Web Browser
 A description of extensions developed at NGDC to classify coverages.
 information
 GHRSST Level 2P Atlantic Regional Skin Sea Surface Temperature from the Spinning Enhanced Visible and InfraRed Imager (SEVIRI) on the Meteosat Second Generation (MSG-2) satellite
 Sea Surface Temperature, 55W-55E and 55S-55N, at 0.05 degree resolution from MSG-02 Imager Sectors every quarter-hour
 20090727
 creation
 4
 NOAA/NESDIS/OSDPD
 originator
 NOAA/NESDIS
 Camp Springs, MD (USA)
 publisher
 The Meteosat Second Generation (MSG) satellites  are spin stabilized geostationary satellites operated by the European Organization for the Exploitation of Meteorological Satellites (EUMETSAT) to provide accurate weather monitoring data through its primary instrument the Spinning Enhanced Visible and InfraRed Imager (SEVIRI), which has the capacity to observe the Earth in 12 spectral channels. Eight of these channels are in the thermal infrared, providing among other information, observations of the temperatures of clouds, land and sea surfaces at approximately 5 km resolution with a 15 minute duty cycle. This Group for High Resolution Sea Surface Temperature (GHRSST) dataset produced by the US National Oceanographic and Atmospheric Administration (NOAA)  National Environmental Satellite, Data, and Information Service (NESDIS) is derived from the SEVIRI instrument on the second MSG satellite (also known as  Meteosat-9) that was launched on 22 December 2005. Skin sea surface temperature (SST) data are calculated from the infrared channels of SEVIRI at full resolution every 15 minutes. L2P data products with Single Sensor Error Statistics (SSES) are then derived following the GHRSST-PP Data Processing Specification (GDS) version 1.5.
 These data are produced by NOAA/NESDIS funded by NESDIS Office of System Development
 onGoing
 Eileen Maturi
 OSDPD > NOAA Office of Satellite Data Processing and Distribution
 301-763-8102 x172
 301-763-8572
 [email protected]
 pointOfContact
 NETCDF
 3
 GZIP
 Earth Science > Oceans > Ocean Temperature > Sea Surface Temperature > Sea Surface Skin Temperature
 theme
 NASA/GCMD Earth Science Keywords
 Eastern Atlantic
 place
 NASA/GCMD Location Keywords
 None
 None
 grid
 eng
 UTF8
 true
 -81
 81
 -73
 73
 true
 2009-11-22T12:30:00Z
 referenceInformation
 lat
 float
 lon
 float
 time
 int
 PO.DAAC User Services
 NASA/JPL/PODAAC > Physical Oceanography Distributed Active Archive Center, Jet Propulsion Laboratory, NASA
 4800 Oak Grove Drive
 Pasadena
 CA
 91109-8099
 USA
 [email protected]
 http://podaac.jpl.nasa.gov
 distributor
 http://podaac.jpl.nasa.gov/ws/search/granule/?datasetId=PODAAC-GHMG2-2PO01&apidoc
 Web Service (PO.DAAC Labs)
 (Search Granule)
 information
 http://www.ghrsst.org
 none
 GHRSST Project Home Page
 information
 http://ghrsst.jpl.nasa.gov
 none
 Portal to the GHRSST Global Data Assembly Center and data access
 information
 asNeeded
 Translated from GCMD DIF
 SEVIRI > Spinning Enhanced Visible and Infrared Imager
 sensor
 The Spinning Enhanced Visible and InfraRed Imager (SEVIRI)  has the capacity to observe the Earth in 12 spectral channels. Eight of these channels are in the thermal infrared, providing among other information, observations of the temperatures of clouds, land and sea surfaces at approximately 5 km resolution with a 15 minute duty cycle.
 MSG > Meteosat Second Generation
 Meteosat Second Generation (MSG) provides multi-spectral imagery of the Earth's surface and cloud systems from a geostationary orbit.
 NOAA/NESDIS/OSDPD
 NASA/JPL/PODAAC > Physical Oceanography Distributed Active Archive Center, Jet Propulsion Laboratory, NASA
 http://podaac.jpl.nasa.gov
]

lein test :only app.core-test/translate-request-test-french

ERROR in (translate-request-test-french) (FutureTask.java:122)
that a call to translate-request correctly goes through all layers.
expected: (= true (and (= "some climate data again, but french" (translate-request "metadata/dataset" {} "fr" "pdf")) (= true (cache/cache-has? :metadatadatasetfr)) (= "some climate data again, but french" (cache/cache-lookup :metadatadatasetfr))))
  actual: java.util.concurrent.ExecutionException: java.lang.ClassCastException: Cannot cast java.lang.Character to java.lang.String
 at java.util.concurrent.FutureTask.report (FutureTask.java:122)
    java.util.concurrent.FutureTask.get (FutureTask.java:188)
    clojure.core$deref_future.invoke (core.clj:2180)
    clojure.core$future_call$reify__6320.deref (core.clj:6420)
    clojure.core$deref.invoke (core.clj:2200)
    clojure.core$pmap$step__6333$fn__6335.invoke (core.clj:6470)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.LazySeq.equals (LazySeq.java:115)
    clojure.lang.LazySeq.equiv (LazySeq.java:100)
    clojure.lang.Util.pcequiv (Util.java:125)
    clojure.lang.Util.equiv (Util.java:32)
    app.core_test$fn__1194$fn__1197.invoke (core_test.clj:73)
    clojure.core$with_redefs_fn.invoke (core.clj:6861)
    app.core_test/fn (core_test.clj:71)
    clojure.test$test_var$fn__7187.invoke (test.clj:704)
    clojure.test$test_var.invoke (test.clj:704)
    clojure.test$test_vars$fn__7209$fn__7214.invoke (test.clj:722)
    clojure.test$default_fixture.invoke (test.clj:674)
    clojure.test$test_vars$fn__7209.invoke (test.clj:722)
    clojure.test$default_fixture.invoke (test.clj:674)
    clojure.test$test_vars.invoke (test.clj:718)
    clojure.test$test_all_vars.invoke (test.clj:728)
    clojure.test$test_ns.invoke (test.clj:747)
    clojure.core$map$fn__4245.invoke (core.clj:2559)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.Cons.next (Cons.java:39)
    clojure.lang.RT.boundedLength (RT.java:1654)
    clojure.lang.RestFn.applyTo (RestFn.java:130)
    clojure.core$apply.invoke (core.clj:626)
    clojure.test$run_tests.doInvoke (test.clj:762)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invoke (core.clj:624)
    user$eval85$fn__140$fn__171.invoke (form-init6878369847205765737.clj:1)
    user$eval85$fn__140$fn__141.invoke (form-init6878369847205765737.clj:1)
    user$eval85$fn__140.invoke (form-init6878369847205765737.clj:1)
    user$eval85.invoke (form-init6878369847205765737.clj:1)
    clojure.lang.Compiler.eval (Compiler.java:6703)
    clojure.lang.Compiler.eval (Compiler.java:6693)
    clojure.lang.Compiler.load (Compiler.java:7130)
    clojure.lang.Compiler.loadFile (Compiler.java:7086)
    clojure.main$load_script.invoke (main.clj:274)
    clojure.main$init_opt.invoke (main.clj:279)
    clojure.main$initialize.invoke (main.clj:307)
    clojure.main$null_opt.invoke (main.clj:342)
    clojure.main$main.doInvoke (main.clj:420)
    clojure.lang.RestFn.invoke (RestFn.java:421)
    clojure.lang.Var.invoke (Var.java:383)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.main.main (main.java:37)
Caused by: java.lang.ClassCastException: Cannot cast java.lang.Character to java.lang.String
 at java.lang.Class.cast (Class.java:3094)
    clojure.lang.Reflector.boxArg (Reflector.java:427)
    clojure.lang.Reflector.boxArgs (Reflector.java:460)
    clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:58)
    clojure.lang.Reflector.invokeInstanceMethod (Reflector.java:28)
    app.core$translate_with_tika$fn__1105.invoke (core.clj:126)
    clojure.core$pmap$fn__6328$fn__6329.invoke (core.clj:6466)
    clojure.core$binding_conveyor_fn$fn__4145.invoke (core.clj:1910)
    clojure.lang.AFn.call (AFn.java:18)
    java.util.concurrent.FutureTask.run (FutureTask.java:262)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)
    java.lang.Thread.run (Thread.java:745)

lein test app.handler-test

Ran 45 tests containing 58 assertions.
0 failures, 3 errors.
Tests failed.

Integration Tests for Service-facing end

Develop layer which handles concurrent clients

• Spawns off a thread for each client, packaging up request info
• Handles callbacks (can have fake "work" to do for now)
• Unit Tests
• Load Test
• Deployed to CentOS box

Lein commands are not working with recent install

When I install Leiningen

brew install leiningen

leiningen: stable 2.5.1, HEAD
I follow the steps at https://github.com/lewismc/iPReS/tree/master/app#how-to-run-locally
I get the following

lmcgibbn@LMC-032857 /usr/local/iPReS(master) $ lein ring server
'ring' is not a task. See 'lein help'.

Did you mean this?
         run

Fix escape characters within response

When I pretty print a JSON response I get the following

You can see here that forward slashed are not escaped back to normal format.

Implement logging in Jetty server

Right now there is no logging in the server at all... this is extremely difficult for debugging server requests and what is actually happening (specifically whether or not Language Translation is actually taking place).
I propose we try the following
https://github.com/pjlegato/ring.middleware.logger

Document all funtions within /app

The generated documentation displays that there are a number of functions which are undocumented.
We should make sure that they get documented and any TODO's e.g. https://github.com/lewismc/iPReS/blob/master/app/src/app/core.clj#L137 are addressed

Create documentation for the Clojure codebase

This is a simple one which is extremely helpful for people trying to understand the API design and logic.
I propose to use codox
https://github.com/weavejester/codox

Client-Facing Integration Test

Tests that it all fits together and works when it's up and running.

Filter/Reduce/Map of data and metadata

• Given data sets from the cache and translated PO.DAAC data sets, filters down info, reduces into a single format, and maps into a Clojure Map data structure.
• Unit Tests

Detail available parameters on wiki page

Right now it is not obvious what the parameters are for interacting with the REST service call via terminal or browser.
We should document them.

Create Github WebHook for shadowing Github activity to Google Group

This one allows us to get email notifications (via the Google Group) whenever a PR comes in the door. It is a very handy way of tracking activity around the iPReS repository.

Data-to-file translation layer

• Converts a Clojure Map of raw information into a requested file format, such as HDF5, HDF5, XML, JSON, etc.
• Unit Tests

Distributed Cache

• Utilizes an appropriate caching strategy to cache translated data based on frequency and perhaps other parameters
• Unit Tests