GithubHelp home page GithubHelp logo

opendap / libdap4 Goto Github PK

View Code? Open in Web Editor NEW
13.0 14.0 12.0 152.33 MB

A new version of libdap that contains both DAP2 and DAP4 support

License: GNU Lesser General Public License v2.1

C++ 65.42% C 22.45% Shell 1.63% HTML 0.02% Perl 0.21% Makefile 1.90% Inno Setup 0.04% Limbo 0.01% Awk 0.01% M4 8.11% Filebench WML 0.01% Roff 0.04% D 0.01% CMake 0.17% Eiffel 0.01%

libdap4's Introduction

Please find the libdap4 API documentation here: https://opendap.github.io/libdap4/html/

  • Merge PR from fork regarding configure.ac use of bash-specific syntax.

Updated for version 3.21.0 DOI

  • Added new Direct I/O support so that modules written using libdap can pass compressed data buffers read directly from disk to output files without expensive decompression and recompression operations.
  • Merged contributed fixes from Bo Anderson Fix handling of libtirpc pkg-config files with -L flags (#228)
  • Merged contributed fixes from Dan Horák <da at @danny.cz> add missing include (#227) With GCC 13 the header isn't included thru other headers any more, thus include it explictly. Otherwise uint8_t or uint32_t type remain undefined in Vector.cc. Fixes: #226 add missing big endian baselines (#196)
  • Fixed a bug where the copy ctor for D4Maps failed to correctly set the parent Array.
  • Merged contributed fix from Orion Poplawski Add missing cstdint include for uint32_t (#219)
  • Removed support for RHEL 7 (CentOS 7)
  • Now require C++-11 to build the code. However, configure will use C++-14 if it finds that and the next release will require that.
  • Moved the functionality of is_dap4_projected() into libdap4 (#213). This had a number of consequences and there are new methods to support the feature.
  • Added support for 64-bit sized arrays. This was done by adding a set of 'size methods' that have the suffix '_ll' (for long long). These should be used in place of the old methods, which are still in the code.

Updated for version 3.20.11

  • Fixed bug in computation of request_size_kb()
  • Fixed type issue in HTTPCache.cc (#192)

Updated for version 3.20.10 DOI

  • Support for RHEL8
  • Fix for bugs in the ce parser around inverted indices.
  • Fix for libdap4 github issue 147: Grid::get_map_iter() was off by one
  • Improvements to DAP4 api.
  • Fixed various memory leaks.
  • Replaced instances of &vector[0] with vector.data() (RHEL8)

Updated for version 3.20.9 DOI

  • Started migrating from (deprecated) auto_ptr to C++11 unique_ptr
  • Migrated use of regex functions from outdfated GNU implementation to C++11 implementation (uses compile time swicth)

Updated for version 3.20.8 DOI

  • Modified Error so that it is more in line with C++11.
  • Added (shallow) unit64_t request size computations and max_request_size state.
  • Changed the internal representation of max response size to uint64_t and confined the overflow to just the (deprecated) functions.

Updated for version 3.20.7 DOI

  • Added code coverage (using gcov) to the CI processes.
  • Dropped support for CentOS-6 (whew!)
  • Improved Int64 support.
  • Corrected byte order issue with DAP4 data transmission.
  • Improved error reporting.

Updated for version 3.20.6 DOI

  • Stopped CE parse errors from returning user supplied strings in error messages.
  • README is now called README.md

Updated for version 3.20.5 DOI

  • Memory leaks. Minor bug fixes. Lots of work on CI.

Updated for version 3.20.4 DOI

  • Updated for version 3.20.4
  • Memory leak fixes and C++11 features

Updated for version 3.20.3 DOI

  • Fixes and Debian packaging via Travis CI

Updated for version 3.20.2 DOI

  • Added libdap::Array::rename_dim()

Updated for version 3.20.1

  • Added Continuous Delivery for CentOS 6 and 7

Updated for version 3.20.0

  • We now have a Debian package for libdap.
  • Bug fixes; See the ChangeLog and NEWS files.

Updated for version 3.19.1

  • Bug fixes; See the ChangeLog and NEWS files.

Updated for version 3.18.3

  • Bug fixes; See the ChangeLog and NEWS files.

Updated for version 3.18.2

  • Added support for DAP4 filter operations.
  • For other information, see NEWS and ChangeLog

Updated for 3.16.0

libdap now supports parallel I/O for certain data write operations, mostly those involving larger amounts of data - Marshaller::put_vector and put_vector_part. The behavior can be 'turned off' by #undef the symbol USE_POSIX_THREADS in the XDSStreamMarshaller classes.

Added to libdap is the ability to send DAP2 responses in parts, so that code that builds large responses from multiple reads can write a part that is complete and then return to work on the next part. This is especially beneficial for aggregations that cross many granules. The new method is Marshaller::put_vector_part().

Added to libdap is a virtual void BaseType::clear_local_data() method that can be used to free the memory used by a BaseType to hold data values. This provides a way for the object to persist in memory without holding onto all of its data (which can be a substantial part of the object's total size).

Updated for 3.14.0

Bison 3 is required to build this code.

Updated for 3.13.3

Bug fix release

Updated for 3.13.2

Bug fix release

Updated for 3.13.1

A test baseline was updated to use both the DAP2 and DAP4 version of the XDAP/X-DAP header. This change was likely over doing things on our part, but it's important to have the source releases pass all their tests.

Updated 3.13.0

Support for clang: Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn) added.

Updated for version 3.12.1

Note that the documented behavior of BaseType::read() is now correctly implemented.

Updated for version 3.12.0

The server functions have been moved out of libdap and into their own BES module. Currently this modules is part of the BES, but that will change in the future. This version of libdap supports building very large arrays made up of constant value (e.g., to be used as masks in server functions you write).

For information on the way to make these functions, see: http://docs.opendap.org/index.php/Expanded_arguments_for_Constraint_Expressions Note that this version of libdap requires bison 2.4. This is a change so the parsers can use C++ I/O streams and we can eventually drop the FILE* interfaces.

Updated for version 3.11.7

Minor tweak for the server function caching code: turn on or off the cache by adding or removing the directory /tmp/dap_functions_cache. If the directory is not present no caching of server function calls is done. All the other behaviors are otherwise identical.

Updated for version 3.11.6

There is a new cache for some kinds of response objects. It's size and location are currently fixed to 20GB bytes and "/tmp/dap_functions_cache/" but these will be made BES parameters in a future release.

Updated for version 3.11.5

A memory leak in XDRStreamMarshaller was fixed.

A bug in DDS::add_var_nocopy() was fixed.

Updated for version 3.11.2

Bug fixes and improvements in the implementations of some methods.

Updated for version 3.11.1

Bug fixes only.

Updated for version 3.11.0

Now constraint expressions can have multiple function calls that return data.

I've bumped up the DAP version from 3.3 to 3.4 to reflect this change.

Updated for Version 3.10.2

BaseType::transfer_attributes() and related methods provide a way for handlers to customize how attributes from a DAS object are merged into a DDS.

In the past we supported a kind of client-side system that could augment the attributes call the 'AIS' (Ancillary Information System). This has been removed - our server now supports the NcML language to do much the same thing but in a way that can be set on the server once for all users. It's also an emerging convention that's gaining wide support within the community.

Updated for Version 3.10.0

DAP 3.3 is now supported; see http://docs.opendap.org/index.php/DAP3/4.

This version of libdap contains many changes that are needed for both DAP 4 and the NcML handler. This version of the library is required for the Hyrax 1.6 handlers.

The 'deflate' program is no longer part of this library package since we are no longer supporting the old data server system (based on WWW's CGI specification).

Updated for version 3.9.2

Now libdap supports DAP 3.2. You can read about the evolving DAP 3.x protocol at http://docs.opendap.org/index.php/DAP3/4. If your client sends the XDAP-Accept header with a value of 3.2 the DDX is different (it includes protocol information and also an xmlbase element).

Behavior change for the DAS: In the past the format handlers added double quotes to the values of string attributes when they added those values to the AttrTable object. This meant that the value of the attribute in the C++ object was actually not correct since it contained quotes not found in the original attribute value. I modified libdap so that if an attribute value in the C++ AttrTable object does not have quotes then those quotes are added when the value is output in a DAS response (but not a DDX since there's no need to quote the value in that response). This ensures that the text in the DAS wire representation will parse whether a handler has added quotes or not (paving the way for fixed handlers). At the same time I fixed all of our handlers so that they no longer add the erroneous quotes. This fixes a problem with the DDX where the quotes were showing up as part of the attribute value. The change to libdap is such that a broken handler will not be any more broken but a fixed handler will work for both DAS and DDX generation.

If you have a handler and it's not adding quotes to the String attribute values - good, don't change that! If your handler does add quotes, please modify it so the DDX will be correct.

Our handler's old, broken, behavior can be resurrected by removing the ATTR_STRING_QUOTE FIX define in the appropriate files.

Updated for version 3.8.2 (23 June 2008)

HTTP Cache and win32 installer fixes (the latter are actually in the 3.8.1 installer for winXP). API change: The functions used to merge ancillary data have been moved to their own class (Ancillary).

Updated for version 3.8.1 (10 June 2008)

The syntax for PROXY_SERVER in the .dodsrc file was relaxed. See the .dodsrc file for more information.

Updated for Version 3.8.0 (29 February 2008)

The libdap classes and code are now inside of the libdap namespace. In order to access any of the classes, for example, you will need to do one of the following. After including the libdap headers you can:

  1. add a using statement for the entire libdap namespace:

using namespace libdap ;

  1. add a using statement for the classes that you will be using:

using libdap::DAS ;

  1. inside your code scope the use of libdap classes.

libdap::DAS *das = code_to_get_das() ;

Added method to HTTPCache to return not only the FILE pointer of a cached response but also the name of the file in the cache, to allow for this file name to be passed to data handlers in the BES to be read.

See NEWS for more information about changes for this version and ChangeLog for the gory details.

Updated for Version 3.7.10 (28 November 2007)

A bug fix release. See NEWS.

Updated for Version 3.7.9 (13 November 2007)

This release is a bug fix and refactoring release. Old classes which were no longer used have been removed, the FILE* output methods are slated to be replaced with ones which will use iostream and will support a chucked transfer 'Marshaller,' and the transfer_data() methods have been made a formal part of the library, implemented for all classes, fixed and renamed to intern_data(). Many bugs in the library were also fixed.

Updated for version 3.7.8 (26 June 2007)

The major fixes in this version are memory errors found and fixed in the Regex class and HTTP header processing software. This version also supports pkg-config on hosts that have that installed.

See NEWS for more information about changes for this version and ChangeLog for the gory details.

Notes for version 3.7.7 (2 May 2007)

The major fix here is to the source build. We've fixed the issue where source builds failed to make the dapserver and dapclient libraries.

Notes for version 3.7.6 (12 March 2007)

Two bug fixes, both minor. Problems in the linear_scale() constraint expression function and a bad/missing #include in GNURegex.h were fixed.

There was an error in the INSTALL file sent out in the previous release. It said this library implemented DAP version 3.2, but in fact it implements version 3.1. The version 3.2 release will be along soon (RSN).

Notes for version 3.7.5 (7 Feb 2007)

This version includes many fixes from the first Server4 beta release plus fixes for the server-side functions. It also includes a smoother Win32 build.

Notes for version 3.7.4 (2 Jan 2007)

Release for the Server4 beta release.

Notes for version 3.7.3 (24 Nov 2006)

This version of libdap contains a beta release of the server-side functions geogrid(), geoarray(), linear_scale() and version(). These can be used to select parts of Grids and Arrays using latitude and longitude values instead of array position indexes. The linear_scale() function can be used to scale variables (including those return by other function) using 'y = mx + b'. The version() function can be used to find out which versions of the functions are installed.

EXAMPLES

To get version information use the 'version()' function. Currently, version() can only be called when asking for data, and you must give the name of a data source, although in the default version of version() the data source is not used. The version function takes one optional argument which may be the strings 'help' or 'xml'. Use 'help' to get help on using the function; use 'xml' to get version information encoded using XML instead of plain text:

[jimg@zoe libdap]$ url=http://test.opendap.org/dap/data/nc/coads_climatology.nc [jimg@zoe libdap]$ ./getdap -D "$url?version()" The data: String version = "Function set: version 1.0, grid 1.0, geogrid 1.0b2, geoarray 0.9b1, linear_scale 1.0b1";

[jimg@zoe libdap]$ ./getdap -D "$url?version(help)" The data: String version = "Usage: version() returns plain text information about ...

[jimg@zoe libdap]$ ./getdap -D "$url?version(xml)" The data: String version = " <function name="version" version="1.0"/> <function name="grid" version="1.0"/> <function name="geogrid" version="1.0"/> <function name="geoarray" version="1.0"/> <function name="linear_scale" version="1.0"/> ";

The geogrid function can only be used with variables that are Grids:

[jimg@zoe libdap]$ getdap -d "$url" Dataset { Float64 COADSX[COADSX = 180]; Float64 COADSY[COADSY = 90]; Float64 TIME[TIME = 12]; Grid { Array: Float32 SST[TIME = 12][COADSY = 90][COADSX = 180]; Maps: Float64 TIME[TIME = 12]; Float64 COADSY[COADSY = 90]; Float64 COADSX[COADSX = 180]; } SST; Grid { . . .

Pass the name of the Grid variable and the upper-left and lower-right corners of the lat/lon rectangle to geogrid. Optionally, pass one or more relational expressions to select parts of dimensions that are not lat/lon.

Note: in libdap 3.7.3 calling geogrid with a constraint on each dimension may return incorrect values that indicate missing data even though data should have been returned.

[jimg@zoe libdap]$ getdap -D "$url?geogrid(SST,30,-60,20,-60,\"TIME=366\")" The data: Grid { Array: Float32 SST[TIME = 1][COADSY = 7][COADSX = 2]; Maps: Float64 TIME[TIME = 1]; Float64 COADSY[COADSY = 7]; Float64 COADSX[COADSX = 2]; } SST = { Array: {{{24.4364, 25.0923},{23.7465, 24.4146},{19.843, 23.6033}, {16.8464, 17.7756},{16.65, 16.818},{-1e+34, 15.3656},{18.7214, 13.1286}}} Maps: {366}, {19, 21, 23, 25, 27, 29, 31}, {-61, -59} };

The geoarray() function works like geogrid() except that it's used to select from an Array variable and not a Grid. In addition to the four lat/lon values for selection rectangle, the caller must supply the data's corner points. A subsequent release of libdap will include a version that reads the data extent from the data source when possible so caller's won't normally have to know the data's extent ahead of time.

The linear_scale() function take either one or three arguments. The first (only) argument is the name of a variable or the return from another function. This variable will be scaled using the 'y = mx + b' equation where 'x' is the value(s) of the input variable and 'm' and 'b' are read from the data source using the values of attributes name 'scale_factor' and 'add_offset.' If these are not present, or to over ride their values, m and b can be supplied using the second and third arguments.

Note that there are still some problems with linear_scale() in this release.

See NEWS and ChangeLog for information about other changes

Notes for version 3.7.2

This version of libdap is required for the 9/15/06 alpha release of Server4. The library now contains software which enables Server4 to build the ASCII data response for all types of variables, including Sequence and nested Sequence variables. These features are additions to the API, so older code will work just fine with the new library. See NEWS for more specific info about bug fixes.

Notes for version 3.7.1

This is a bug fix release (mostly) made for users of the netcdf client library who need a fix for a problem dealing with attributes from the HDF4 server.

NOTES for version 3.7.0

This version includes new features and an implementation change.

This version of libdap now returns the DAP protocol version number, 3.1, in an HTTP response header. Use this to determine which protocol version the library implements. The inclusion of a protocol version number is the sole official new feature of DAP 3.1. Use Connect::get_protocol() to get the version number. Clients can use this to determine the features supported by a server. The Connect::get_version() method can still be used to get our server's implementation version. The distinction is that as more groups provide their own implementations of the DAP, the protocol version will provide a way for clients to determine capabilities independently of implementation.

The libdap library now contains an implementation of the DDX object/response, although this is an alpha implementation and it's actually been part of the library for some time now. The implementation contained in this version of the library is close enough to the version we intend for DAP4 that developers can start to use it. Most of the server handlers will return DDXs when asked.

The DDX combines the information previously held by the DDS and DAS objects, making it much easier to associate attributes to variables. As the name suggests, the DDX uses XML rather than curly-braces. You can drop the DDX into your favorite XML parser and get a DOM tree; no need to use our parsers. However, libdap contains a nice SAX parser that will build the libdap objects directly from the XML DDX object/response. Also included in libdap are methods to build a DDX using a DDS and DAS, so there's an easy migration path for both servers and clients.

Finally, the library contains two structural changes. First, the library named 'libdap' now holds the DAP implementation while two new libraries, 'libdapclient' and 'libdapserver', now hold the client and server helper classes which are not strictly part of the DAP. Secondly, the DDS/DDX object now takes the constraint evaluator as a parameter. The class ConstraintEvaluator holds our default evaluator, but it's now possible to use your own evaluator .

NOTES for version 3.6.1

Version 3.6.1 is bug fix release.

NOTES for version 3.6.0

This version of the library may not work older source code. Many of the deprecated methods have been removed.

Added are headers which send information about the version of the DAP protocol that the library implements (in contrast to the implementation of the library itself). A new header named XOPeNDAP-Server is used to send information about the implementation of servers.

The libtool interface version has been incremented from 3 to 4 (these versions do no track the software's release version since several releases might present compatible binary interfaces).

NOTES for version 3.5.3

This version of libdap++ cannot be used to build the 3.4.x and previous clients and/or servers. However, client and servers built using this code will work with the older clients and servers.

WHAT'S IN THIS DIRECTORY?

This directory contains the OPeNDAP C++ implementation of the Data Access Protocol version 2 (DAP2) with some extensions that will be part of DAP3. Documentation for this software can be found on the OPeNDAP home page at http://www.opendap.org/. The NASA/ESE RFC which describes DAP2, implemented by the library, can be found at http://spg.gsfc.nasa.gov/rfc/004/.

The DAP2 is used to provide a uniform way of accessing a variety of different types of data across the Internet. It was originally part of the DODS and then NVODS projects. The focus of those projects was access to Earth-Science data, so much of the software developed using the DAP2 to date has centered on that discipline. However, the DAP2 data model is very general (and similar to a modern structured programming language) so it can be applied to a wide variety of fields.

The DAP2 is implemented as a set of C++ classes that can be used to build data servers and clients. The classes may be specialized to mimic the behavior of other data access APIs, such as netCDF. In this way, programs originally meant to work with local data in those formats can be re-linked and equipped to work with data stored remotely in many different formats. The classes can also by specialized to build standalone client programs.

The DAP2 is contained in a single library: libdap++.a. Also included in the library are classes and utility functions which simplify building clients and servers.

WHAT ELSE IS THERE?

The file README.dodsrc describes the client-side behavior which can be controlled using the .dodsrc file. This includes client-side caching, proxy servers, et c., and is described in a separate file so it's easy to include in your clients.

The file README.AIS describes the prototype Ancillary Information Service (AIS) included in this version of the library. The AIS is (currently) a client-side capability which provides a way to augment DAP attributes. This is a very useful feature because it can be used to add missing metadata to a data source. The AIS is accessed by using the AISConnect class in place of Connect in your client.

This directory also contains test programs for the DAP2, a sample specialization of the classes, getdap (a useful command-line web client created with DAP2) and dap-config (a utility script to simplify linking with libdap.a). Also included as of version 3.5.2 is libdap.m4, an autoconf macro which developers can use along with autoconf to test for libdap. This macro will be installed in ${prefix}/share/aclocal and can be by any package which uses autoconf for its builds. See the file for more information.

We also have Java and C versions of the DAP2 library which inter-operate with software which uses this library. In other words, client programs built with the Java DAP2 implementation can communicate with servers built with this (C++) implementation of the DAP2. The C DAP2 library, called the Ocapi, only implements the client-side part of the protocol. Clients written using the Ocapi are interoperable with both the Java and C++ DAP2 libraries. Note that the Ocapi is in early beta and available only from CVS at this time (5 May 2005).

THREAD SAFETY

We don't need to do this since the STL is also not thread safe. Users of libdap have to be sure that multiple threads never make simultaneous and/or overlapping calls to a single copy of libdap. If several threads are part of a program and each will make calls to libdap, either those threads must synchronize their calls or arrange to each use their own copy of libdap. Some aspects of the library are thread-safe: the singleton classes are all protected as is the HTTP cache (which uses the local file system).

INSTALLATION INSTRUCTIONS

See the file INSTALL in this directory for information on building the library and the geturl client.

COPYRIGHT INFORMATION

The OPeNDAP DAP library is copyrighted using the GNU Lesser GPL. See the file COPYING or contact the Free Software Foundation, Inc., at 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA. Older versions of the DAP were copyrighted by the University of Rhode Island and Massachusetts Institute of Technology; see the file COPYRIGHT_URI. The file deflate.c is also covered by COPYRIGHT_W3C.

libdap4's People

Contributors

blackone-sudo avatar bo98 avatar captainkirk99 avatar dh-opendap avatar eli-schwartz avatar ethanrd avatar hyoklee avatar jgallagher59701 avatar kneumiller avatar kyang2014 avatar ndp-opendap avatar opoplawski avatar rmorris2342 avatar ryandesign avatar sawer297 avatar sharkcz avatar travis-ci-opendap avatar tsgouros avatar vskorole avatar yuanho avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

libdap4's Issues

Build fails with LTO

I tried building with these flags: -flto=4 -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing

The -Werror=* here is important since these indicate cases where the compiler can try to optimize by assuming UB cannot exist.

I got this error:

/bin/sh ./libtool  --tag=CXX   --mode=link x86_64-pc-linux-gnu-g++ --std=c++14 -Wall -W -Wcast-align -Wno-deprecated-register -Wno-inconsistent-missing-override  -g -O2 -DNDEBUG  -march=native -fstack-protector-all -O2 -pipe -fdiagnostics-color=always -frecord-gcc-switches -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-clash-protection -flto=4 -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing  -Wformat -Werror=format-security -std=c++14 -version-info 29:0:2 -Wl,-O1 -Wl,--as-needed -flto=4 -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing -Wl,--defsym=__gentoo_check_ldflags__=0 -o libdap.la -rpath /usr/lib64 libdap_la-AttrTable.lo libdap_la-DAS.lo libdap_la-DDS.lo libdap_la-DataDDS.lo libdap_la-DDXParserSAX2.lo libdap_la-BaseType.lo libdap_la-Byte.lo libdap_la-Int32.lo libdap_la-Float64.lo libdap_la-Str.lo libdap_la-Url.lo libdap_la-Vector.lo libdap_la-Array.lo libdap_la-Structure.lo libdap_la-Sequence.lo libdap_la-Grid.lo libdap_la-UInt32.lo libdap_la-Int16.lo libdap_la-UInt16.lo libdap_la-Float32.lo libdap_la-Constructor.lo libdap_la-BaseTypeFactory.lo libdap_la-SignalHandler.lo libdap_la-Error.lo libdap_la-InternalErr.lo libdap_la-util.lo libdap_la-xdrutil_ppc.lo libdap_la-parser-util.lo libdap_la-escaping.lo libdap_la-Clause.lo libdap_la-RValue.lo libdap_la-ConstraintEvaluator.lo libdap_la-DapIndent.lo libdap_la-XDRUtils.lo libdap_la-XDRFileMarshaller.lo libdap_la-XDRStreamMarshaller.lo libdap_la-XDRFileUnMarshaller.lo libdap_la-XDRStreamUnMarshaller.lo libdap_la-mime_util.lo libdap_la-Keywords2.lo libdap_la-XMLWriter.lo libdap_la-ServerFunctionsList.lo libdap_la-ServerFunction.lo libdap_la-DapXmlNamespaces.lo libdap_la-MarshallerThread.lo GNU/libdap_la-GetOpt.lo GNU/libdap_la-GNURegex.lo libdap_la-D4StreamMarshaller.lo libdap_la-D4StreamUnMarshaller.lo libdap_la-Int64.lo libdap_la-UInt64.lo libdap_la-Int8.lo libdap_la-D4ParserSax2.lo libdap_la-D4BaseTypeFactory.lo libdap_la-D4Dimensions.lo libdap_la-D4EnumDefs.lo libdap_la-D4Group.lo libdap_la-DMR.lo libdap_la-D4Attributes.lo libdap_la-D4Enum.lo libdap_la-chunked_ostream.lo libdap_la-chunked_istream.lo libdap_la-D4Sequence.lo libdap_la-D4Maps.lo libdap_la-D4Opaque.lo libdap_la-D4AsyncUtil.lo libdap_la-D4RValue.lo libdap_la-D4FilterClause.lo -lxml2 -lpthread gl/libgnu.la d4_ce/libd4_ce_parser.la d4_function/libd4_function_parser.la libparsers.la  -ltirpc 
libtool: link: x86_64-pc-linux-gnu-g++  -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../lib64/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/13/crtbeginS.o  .libs/libdap_la-AttrTable.o .libs/libdap_la-DAS.o .libs/libdap_la-DDS.o .libs/libdap_la-DataDDS.o .libs/libdap_la-DDXParserSAX2.o .libs/libdap_la-BaseType.o .libs/libdap_la-Byte.o .libs/libdap_la-Int32.o .libs/libdap_la-Float64.o .libs/libdap_la-Str.o .libs/libdap_la-Url.o .libs/libdap_la-Vector.o .libs/libdap_la-Array.o .libs/libdap_la-Structure.o .libs/libdap_la-Sequence.o .libs/libdap_la-Grid.o .libs/libdap_la-UInt32.o .libs/libdap_la-Int16.o .libs/libdap_la-UInt16.o .libs/libdap_la-Float32.o .libs/libdap_la-Constructor.o .libs/libdap_la-BaseTypeFactory.o .libs/libdap_la-SignalHandler.o .libs/libdap_la-Error.o .libs/libdap_la-InternalErr.o .libs/libdap_la-util.o .libs/libdap_la-xdrutil_ppc.o .libs/libdap_la-parser-util.o .libs/libdap_la-escaping.o .libs/libdap_la-Clause.o .libs/libdap_la-RValue.o .libs/libdap_la-ConstraintEvaluator.o .libs/libdap_la-DapIndent.o .libs/libdap_la-XDRUtils.o .libs/libdap_la-XDRFileMarshaller.o .libs/libdap_la-XDRStreamMarshaller.o .libs/libdap_la-XDRFileUnMarshaller.o .libs/libdap_la-XDRStreamUnMarshaller.o .libs/libdap_la-mime_util.o .libs/libdap_la-Keywords2.o .libs/libdap_la-XMLWriter.o .libs/libdap_la-ServerFunctionsList.o .libs/libdap_la-ServerFunction.o .libs/libdap_la-DapXmlNamespaces.o .libs/libdap_la-MarshallerThread.o GNU/.libs/libdap_la-GetOpt.o GNU/.libs/libdap_la-GNURegex.o .libs/libdap_la-D4StreamMarshaller.o .libs/libdap_la-D4StreamUnMarshaller.o .libs/libdap_la-Int64.o .libs/libdap_la-UInt64.o .libs/libdap_la-Int8.o .libs/libdap_la-D4ParserSax2.o .libs/libdap_la-D4BaseTypeFactory.o .libs/libdap_la-D4Dimensions.o .libs/libdap_la-D4EnumDefs.o .libs/libdap_la-D4Group.o .libs/libdap_la-DMR.o .libs/libdap_la-D4Attributes.o .libs/libdap_la-D4Enum.o .libs/libdap_la-chunked_ostream.o .libs/libdap_la-chunked_istream.o .libs/libdap_la-D4Sequence.o .libs/libdap_la-D4Maps.o .libs/libdap_la-D4Opaque.o .libs/libdap_la-D4AsyncUtil.o .libs/libdap_la-D4RValue.o .libs/libdap_la-D4FilterClause.o  -Wl,--whole-archive gl/.libs/libgnu.a d4_ce/.libs/libd4_ce_parser.a d4_function/.libs/libd4_function_parser.a ./.libs/libparsers.a -Wl,--no-whole-archive  -Wl,--as-needed -lpthread -lxml2 -ltirpc -L/usr/lib/gcc/x86_64-pc-linux-gnu/13 -L/usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../x86_64-pc-linux-gnu/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/13/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-pc-linux-gnu/13/crtendS.o /usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../lib64/crtn.o  -g -O2 -march=native -fstack-protector-all -O2 -fdiagnostics-color=always -frecord-gcc-switches -flto=4 -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing -Werror=format-security -Wl,-O1 -flto=4 -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing -Wl,--defsym=__gentoo_check_ldflags__=0   -Wl,-soname -Wl,libdap.so.27 -o .libs/libdap.so.27.2.0
d4_ce/lex.d4_ce.cc:271:8: error: type 'struct yy_buffer_state' violates the C++ One Definition Rule [-Werror=odr]
  271 | #define YY_STRUCT_YY_BUFFER_STATE
      |        ^
lex.das.cc:445:8: note: a different type is defined in another translation unit
  445 | #define YY_STRUCT_YY_BUFFER_STATE
      |        ^
d4_ce/lex.d4_ce.cc:277:18: note: the first difference of corresponding definitions is field 'yy_input_file'
  277 | /* %if-c++-only */
      |                  ^
lex.das.cc:447:15: note: a field of same name but different type is defined in another translation unit
  447 |         {
      |               ^
d4_ce/lex.d4_ce.cc:271:8: note: type name 'std::basic_streambuf<char, std::char_traits<char> >' should match type name '_IO_FILE'
  271 | #define YY_STRUCT_YY_BUFFER_STATE
      |        ^
dds.tab.hh:123:7: error: type 'union YYSTYPE' violates the C++ One Definition Rule [-Werror=odr]
  123 | union YYSTYPE
      |       ^
Error.tab.hh:95:7: note: a different type is defined in another translation unit
   95 | union YYSTYPE
      |       ^
dds.yy:135:10: note: the first difference of corresponding definitions is field 'word'
  135 |     char word[ID_MAX];
      |          ^
Error.yy:83:9: note: a field with different name is defined in another translation unit
   83 |     int integer;
      |         ^
dds.tab.cc:412:7: error: type 'union yyalloc' violates the C++ One Definition Rule [-Werror=odr]
  412 | union yyalloc
      |       ^
ce_expr.tab.cc:490:7: note: a different type is defined in another translation unit
  490 | union yyalloc
      |       ^
dds.tab.cc:415:11: note: the first difference of corresponding definitions is field 'yyvs_alloc'
  415 |   YYSTYPE yyvs_alloc;
      |           ^
ce_expr.tab.cc:493:11: note: a field of same name but different type is defined in another translation unit
  493 |   YYSTYPE yyvs_alloc;
      |           ^
dds.tab.cc:412:7: note: type name 'YYSTYPE' should match type name 'CE_EXPRSTYPE'
  412 | union yyalloc
      |       ^
lto1: some warnings being treated as errors
lto-wrapper: fatal error: x86_64-pc-linux-gnu-g++ returned 1 exit status
compilation terminated.
/usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../x86_64-pc-linux-gnu/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

Seems like there are multiple flex generated files with overlapping definitions that aren't being namespaced.

using libtirpc instead of glibc RPC breaks EXPR tests on big endian arches

Looks like using libtirpc instead of glibc RPC breaks EXPR tests on big endian arches, as libtirpc is meant as 1:1 replacement, the problem should be in libtirpc rather.

observed on Fedora 28 ppc64 and s390x (only libtirpc available), confirmed with Fedora 27 s390x where both glibc RPC and libtirpc can be used.

I'm going to check, if http://git.linux-nfs.org/?p=steved/libtirpc.git;a=commit;h=145272c2b6d89a1c3a7de86a2cbef43880f1b61b will fix the problem.

404 with `https://www.opendap.org/pub/source/`

👋 I looks like the /pub/source url is giving 404 (the context is homebrew is using the url for livechecking the sw updates). Let me know if that makes sense. Thanks!

$ curl -I https://www.opendap.org/pub/source/
HTTP/2 404
server: GitHub.com
content-type: text/html; charset=utf-8
access-control-allow-origin: *
etag: "64c00bd1-24a3"
content-security-policy: default-src 'none'; style-src 'unsafe-inline'; img-src data:; connect-src 'self'
x-proxy-cache: MISS
x-github-request-id: 1B46:0256:2999922:35D0BE3:65636B51
accept-ranges: bytes
date: Sun, 26 Nov 2023 15:59:14 GMT
via: 1.1 varnish
age: 0
x-served-by: cache-lga21924-LGA
x-cache: MISS
x-cache-hits: 0
x-timer: S1701014354.273242,VS0,VE16
vary: Accept-Encoding
x-fastly-request-id: ebf0dd0322c9fa7e161ab1dab8c12717b42a33c6
content-length: 9379

getdap4 some options not work

I compiled the latest code with commit f80b36f .

I found getdap4 -i option not print server version and -k option not work ,here is the output:

$ ./getdap4 -i https://remotetest.unidata.ucar.edu/d4ts/
<html>
<head>
<title>DAP4 Test Files</title>
<meta http-equiv="Content-Type" content="text/html">
</meta>
...
<hr>
</html>

$ ./getdap4 -k https://remotetest.unidata.ucar.edu/d4ts/
Usage: /root/libdap4/.libs/lt-getdap4
 [dD vVikmzstM][-c <expr>][-m <num>] <url> [<url> ...] | <file> [<file> ...]
...
Options:
        d: For each URL, get the (DAP4) DMR object. Does not get data.
        D: For each URL, get the DAP4 Data response.

        v: Verbose output.
        V: Version of this client; see 'i' for server version.
        i: For each URL, get the server version.
        m: Request the same URL <num> times.
        z: Ask the server to compress data.
        s: Print Sequences using numbered rows.
        M: Assume data read from a file has no MIME headers; use only with files

        c: <expr> is a constraint expression. Used with -d/D
           NB: You can use a `?' for the CE also.
        S: Used in conjunction with -d and will report the total size of the data referenced in the DMR.
$

I noticed the related codes are commented in main function, any plan to fix these options?

-i option commented code:

libdap4/getdap4.cc

Lines 445 to 451 in f80b36f

#if 0
else if (get_version) {
fprintf(stderr, "DAP version: %s, Server version: %s\n",
url->request_protocol().c_str(),
url->get_version().c_str());
}
#endif

-k option commented code:

libdap4/getdap4.cc

Lines 258 to 262 in f80b36f

#if 0
case 'k':
dods_keep_temps = 1;
break; // keep_temp is in Connect.cc
#endif

heap-buffer-overflow in libdap::XDRFileUnMarshaller::get_vector

Hello.

I found a heap-buffer-overflow bug in libdap4.

Please confirm.

Thanks.

OS: Ubuntu 17.04 64bit
Version: commit 724b8ee
PoC Download: libdap__XDRFileUnMarshaller__get_vector.zip

Steps to reproduce:
1.Download the .POC files.
2.Compile the source code with ASan.
3.Execute the following command
: ./getdap -k $POC

=================================================================
==15996==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60300000e040 at pc 0x7faf8d2d50c3 bp 0x7fff519c0500 sp 0x7fff519bfca8
WRITE of size 2 at 0x60300000e040 thread T0
    #0 0x7faf8d2d50c2  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x6a0c2)
    #1 0x7faf8bcc3792 in xdr_array (/lib/x86_64-linux-gnu/libc.so.6+0x13a792)
    #2 0x7faf8ca06ca4 in libdap::XDRFileUnMarshaller::get_vector(char**, unsigned int&, int, libdap::Vector&) /home/karas/libdap4/XDRFileUnMarshaller.cc:186
    #3 0x7faf8c8c3896 in libdap::Vector::deserialize(libdap::UnMarshaller&, libdap::DDS*, bool) /home/karas/libdap4/Vector.cc:813
    #4 0x7faf8cf985ca in libdap::Connect::process_data(libdap::DataDDS&, libdap::Response*) /home/karas/libdap4/Connect.cc:143
    #5 0x7faf8cf9f1d4 in libdap::Connect::read_data_no_mime(libdap::DataDDS&, libdap::Response*) /home/karas/libdap4/Connect.cc:1071
    #6 0x4061f2 in main /home/karas/libdap4/getdap.cc:276
    #7 0x7faf8bba982f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #8 0x40c278 in _start (/home/karas/libdap4/run/bin/getdap+0x40c278)

0x60300000e040 is located 0 bytes to the right of 32-byte region [0x60300000e020,0x60300000e040)
allocated by thread T0 here:
    #0 0x7faf8d3046b2 in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x996b2)
    #1 0x7faf8c8aabe7 in libdap::Vector::m_create_cardinal_data_buffer_for_type(unsigned int) /home/karas/libdap4/Vector.cc:202

SUMMARY: AddressSanitizer: heap-buffer-overflow ??:0 ??
Shadow bytes around the buggy address:
  0x0c067fff9bb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c067fff9bc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c067fff9bd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c067fff9be0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c067fff9bf0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c067fff9c00: fa fa fa fa 00 00 00 00[fa]fa fd fd fd fd fa fa
  0x0c067fff9c10: 00 00 00 07 fa fa 00 00 00 01 fa fa fd fd fd fd
  0x0c067fff9c20: fa fa fd fd fd fd fa fa 00 00 00 07 fa fa 00 00
  0x0c067fff9c30: 00 00 fa fa 00 00 00 00 fa fa 00 00 00 00 fa fa
  0x0c067fff9c40: 00 00 00 00 fa fa 00 00 00 00 fa fa 00 00 00 00
  0x0c067fff9c50: fa fa 00 00 00 00 fa fa 00 00 00 00 fa fa 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==15996==ABORTING

==========
[Acknowledgement]
This work was supported by ICT R&D program of MSIP/IITP. [R7518-16-1001, Innovation hub for high Performance Computing]

support portablexdr

Trying to build libdap on windows with mingw-w64 but I can't get the configure script to pick up on portablexdr (which is the only xdr that I can find for windows).

Example headers here.

error: 'uint32_t' does not name a type when compiling Vector.cc with gcc 13

I believe gcc 13 is stricter again (or changed its header structure) and a build on Fedora 38 on x86_64 (and others) using gcc-13.1.1-2.fc38.x86_64 now fails with

...
/bin/sh ./libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I.  -I./gl -I./gl -I./GNU -I/usr/include/libxml2  -I/usr/include/tirpc   --std=c++14 -Wall -W -Wcast-align -Wno-deprecated-register -Wno-inconsistent-missing-override  -g -O2 -DNDEBUG   -c -o libdap_la-Vector.lo `test -f 'Vector.cc' || echo './'`Vector.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I./gl -I./gl -I./GNU -I/usr/include/libxml2 -I/usr/include/tirpc --std=c++14 -Wall -W -Wcast-align -Wno-deprecated-register -Wno-inconsistent-missing-override -g -O2 -DNDEBUG -c Vector.cc  -fPIC -DPIC -o .libs/libdap_la-Vector.o
In file included from Vector.cc:46:
crc.h:9:14: error: 'uint32_t' does not name a type
    9 | static const uint32_t kCrc32Table[256] = {
      |              ^~~~~~~~
crc.h:1:1: note: 'uint32_t' is defined in header '<cstdint>'; did you forget to '#include <cstdint>'?
  +++ |+#include <cstdint>
    1 | /*
crc.h:79:13: error: 'uint32_t' does not name a type
   79 |     typedef uint32_t checksum;
      |             ^~~~~~~~
crc.h:79:13: note: 'uint32_t' is defined in header '<cstdint>'; did you forget to '#include <cstdint>'?
crc.h:98:24: error: 'uint8_t' does not name a type
   98 |     void AddData(const uint8_t* pData, const uint32_t length)
      |                        ^~~~~~~
crc.h:98:24: note: 'uint8_t' is defined in header '<cstdint>'; did you forget to '#include <cstdint>'?
crc.h:98:46: error: 'uint32_t' does not name a type
   98 |     void AddData(const uint8_t* pData, const uint32_t length)
      |                                              ^~~~~~~~
crc.h:98:46: note: 'uint32_t' is defined in header '<cstdint>'; did you forget to '#include <cstdint>'?
crc.h:110:5: error: 'checksum' does not name a type
  110 |     checksum GetCrc32() const { return ~_crc; }
      |     ^~~~~~~~
crc.h:113:5: error: 'uint32_t' does not name a type
  113 |     uint32_t _crc;
      |     ^~~~~~~~
crc.h:113:5: note: 'uint32_t' is defined in header '<cstdint>'; did you forget to '#include <cstdint>'?
crc.h: In member function 'void Crc32::Reset()':
crc.h:92:20: error: '_crc' was not declared in this scope
   92 |     void Reset() { _crc = (uint32_t)~0; }
      |                    ^~~~
crc.h:92:28: error: 'uint32_t' was not declared in this scope
   92 |     void Reset() { _crc = (uint32_t)~0; }
      |                            ^~~~~~~~
crc.h:92:28: note: 'uint32_t' is defined in header '<cstdint>'; did you forget to '#include <cstdint>'?
crc.h: In member function 'void Crc32::AddData(const int*, int)':
crc.h:100:9: error: 'uint8_t' was not declared in this scope
  100 |         uint8_t* pCur = (uint8_t*)pData;
      |         ^~~~~~~
crc.h:100:9: note: 'uint8_t' is defined in header '<cstdint>'; did you forget to '#include <cstdint>'?
crc.h:100:18: error: 'pCur' was not declared in this scope
  100 |         uint8_t* pCur = (uint8_t*)pData;
      |                  ^~~~
crc.h:100:34: error: expected primary-expression before ')' token
  100 |         uint8_t* pCur = (uint8_t*)pData;
      |                                  ^
crc.h:101:9: error: 'uint32_t' was not declared in this scope
  101 |         uint32_t remaining = length;
      |         ^~~~~~~~
crc.h:101:9: note: 'uint32_t' is defined in header '<cstdint>'; did you forget to '#include <cstdint>'?
crc.h:102:16: error: 'remaining' was not declared in this scope
  102 |         for (; remaining--; ++pCur)
      |                ^~~~~~~~~
crc.h:103:13: error: '_crc' was not declared in this scope
  103 |             _crc = ( _crc >> 8 ) ^ kCrc32Table[(_crc ^ *pCur) & 0xff];
      |             ^~~~
crc.h:103:36: error: 'kCrc32Table' was not declared in this scope
  103 |             _crc = ( _crc >> 8 ) ^ kCrc32Table[(_crc ^ *pCur) & 0xff];
      |                                    ^~~~~~~~~~~
crc.h:98:33: warning: unused parameter 'pData' [-Wunused-parameter]
   98 |     void AddData(const uint8_t* pData, const uint32_t length)
      |                  ~~~~~~~~~~~~~~~^~~~~
crc.h:98:55: warning: unused parameter 'length' [-Wunused-parameter]
   98 |     void AddData(const uint8_t* pData, const uint32_t length)
      |                                        ~~~~~~~~~~~~~~~^~~~~~
In file included from Vector.cc:53:
D4StreamUnMarshaller.h: At global scope:
D4StreamUnMarshaller.h:107:12: error: 'checksum' in 'class Crc32' does not name a type
  107 |     Crc32::checksum get_checksum();
      |            ^~~~~~~~
Vector.cc: In member function 'virtual void libdap::Vector::compute_checksum(Crc32&)':
Vector.cc:978:34: error: cannot convert 'uint8_t*' {aka 'unsigned char*'} to 'const int*'
  978 |                 checksum.AddData(reinterpret_cast<uint8_t*>(d_buf), length_ll() * d_proto->width_ll());
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                  |
      |                                  uint8_t* {aka unsigned char*}
crc.h:98:33: note:   initializing argument 1 of 'void Crc32::AddData(const int*, int)'
   98 |     void AddData(const uint8_t* pData, const uint32_t length)
      |                  ~~~~~~~~~~~~~~~^~~~~
Vector.cc:984:42: error: cannot convert 'const uint8_t*' {aka 'const unsigned char*'} to 'const int*'
  984 |                         checksum.AddData(reinterpret_cast<const uint8_t*>(d_str[i].data()), d_str[i].size());
      |                                          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                          |
      |                                          const uint8_t* {aka const unsigned char*}
crc.h:98:33: note:   initializing argument 1 of 'void Crc32::AddData(const int*, int)'
   98 |     void AddData(const uint8_t* pData, const uint32_t length)
      |                  ~~~~~~~~~~~~~~~^~~~~
At global scope:
cc1plus: note: unrecognized command-line option '-Wno-inconsistent-missing-override' may have been intended to silence earlier diagnostics
cc1plus: note: unrecognized command-line option '-Wno-deprecated-register' may have been intended to silence earlier diagnostics
make[2]: *** [Makefile:1851: libdap_la-Vector.lo] Error 1

3.19.2 missing?

Previously there was a 3.19.2 of libdap4, but that appears to missing. Any reason for its removal?

Please consider change git tagging convention

github and and many other REST frontends provides autogenerate tar balls from exact tag or commit hash.

Latest autogenerated tar ball is https://github.com/OPENDAP/libdap4/archive/version-3.20.6.tar.gz. In that case because all frontendsa adds to base directory name repo name that tar has base directory libdap4-version-3.20.6/.

Please consider change tagging convention to just <version>without any prefixes like most of the maintainers are doing or at lease add duplicated tag consisting only from version string.

BTW. looks like https://www.opendap.org is down and is not possible to download dist tar balls from https://www.opendap.org/pub/source/

test suite fails on big endian arches after recent commit

The test suite fails on big endian arches after commit 30290c1, because it adds only little-endian "gold" results.

...
243: trans ce vol_1_ce_2.xml /inst2 vol_1_ce_2.xml.1.trans_base FAILED (DMRTest.at:647)
244: trans ce vol_1_ce_2.xml inst2 vol_1_ce_2.xml.1.trans_base FAILED (DMRTest.at:648)
245: trans ce vol_1_ce_2.xml /inst2/Point vol_1_ce_2.xml.2.trans_base FAILED (DMRTest.at:649)
246: trans ce vol_1_ce_13.xml /inst2 vol_1_ce_13.xml.1.trans_base FAILED (DMRTest.at:651)
247: trans ce vol_1_ce_13.xml /inst2/inst3 vol_1_ce_13.xml.2.trans_base FAILED (DMRTest.at:652)
248: trans ce vol_1_ce_13.xml /attr_only_global vol_1_ce_13.xml.3.trans_base FAILED (DMRTest.at:654)
249: trans ce vol_1_ce_13.xml /inst2/attr_only vol_1_ce_13.xml.4.trans_base FAILED (DMRTest.at:655)

Could you remind me, how to generate the missing files? Or is there a deeper issue?

ASAN misaligned address 0x611000006a99 for type 'dods...

BaseType::val2buf(void *, bool) encounters alignment errors when evaluated using ASAN in the BES. For an example error message (from the BES fileout_netcdf handler) see below.

In the fileout_netcdf regression tests this is the underlying cause of 5 test failures (via ASAN):

207: bescmd/vdata_test.hdf.dmrpp.bescmd              FAILED (testsuite.at:320)
210: bescmd/vdata_packed_linked_blocks.hdf.dmrpp.bescmd FAILED (testsuite.at:323)
216: bescmd/PointFile.hdf.dmrpp.bescmd               FAILED (testsuite.at:329)
217: bescmd/vdata_test.hdf.bescmd                    FAILED (testsuite.at:333)
218: bescmd/vdata_packed_linked_blocks.hdf.bescmd    FAILED (testsuite.at:334)

Is this a design issue arising from using void * or is this a usage issue in the way that the fileout_netcdf is allocating memory?

Examine and fix if feasible.

Example ASAN error for memory alignment issue.

207. testsuite.at:320: testing bescmd/vdata_test.hdf.dmrpp.bescmd ...
COMMAND: besstandalone  -c bes.nc4.grp.conf -i bescmd/vdata_test.hdf.dmrpp.bescmd
./testsuite.at:320: besstandalone -c $abs_builddir/$bes_conf -i $input > test.nc
--- /dev/null	2024-05-28 14:43:47
+++ /Users/ndp/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests/testsuite.dir/at-groups/207/stderr	2024-05-28 14:43:47
@@ -0,0 +1,25 @@
+Float32.cc:187:13: runtime error: load of misaligned address 0x611000006a86 for type 'dods_float32' (aka 'float'), which requires 4 byte alignment
+0x611000006a86: note: pointer points here
+ 8e 3f 00 00 7b 14  8e 3f 41 61 30 3f 80 00  00 40 00 00 00 40 0e 14  7b 00 01 40 0e 14 7b 42  62 31
+             ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Float32.cc:187:13 in 
+DmrppStructure.cc:154:26: runtime error: load of misaligned address 0x611000006a99 for type 'dods_uint16' (aka 'unsigned short'), which requires 2 byte alignment
+0x611000006a99: note: pointer points here
+ 7b 14 0e  40 00 01 40 0e 14 7b 42  62 31 3f 80 00 00 40 00  00 00 40 55 1e b8 00 02  40 55 1e b8 43
+              ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior DmrppStructure.cc:154:26 in 
+DmrppStructure.cc:154:26: runtime error: load of misaligned address 0x611000006a99 for type 'dods_uint16' (aka 'unsigned short'), which requires 2 byte alignment
+0x611000006a99: note: pointer points here
+ 7b 14 0e  40 00 01 40 0e 14 7b 42  62 31 3f 80 00 00 40 00  00 00 40 55 1e b8 00 02  40 55 1e b8 43
+              ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior DmrppStructure.cc:154:26 in 
+DmrppStructure.cc:154:17: runtime error: store to misaligned address 0x611000006a99 for type 'dods_uint16' (aka 'unsigned short'), which requires 2 byte alignment
+0x611000006a99: note: pointer points here
+ 7b 14 0e  40 00 01 40 0e 14 7b 42  62 31 3f 80 00 00 40 00  00 00 40 55 1e b8 00 02  40 55 1e b8 43
+              ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior DmrppStructure.cc:154:17 in 
+Int16.cc:180:13: runtime error: load of misaligned address 0x611000006a99 for type 'dods_int16' (aka 'short'), which requires 2 byte alignment
+0x611000006a99: note: pointer points here
+ 7b 14 0e  40 01 00 40 0e 14 7b 42  62 31 3f 80 00 00 40 00  00 00 40 55 1e b8 00 02  40 55 1e b8 43
+              ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Int16.cc:180:13 in 
207. testsuite.at:320:  FAILED (testsuite.at:320)

new test failures after recent change

Seems that with commit 5b419ac DMRTest 139, 140 and 141 started to fail on our s390x system (running Fedora 38). It might be a big endian related issue, but I need to double check on ppc64 too.

...
137: trans ce test_simple_7.xml s.i1 test_simple_7.xml.2.trans_base ok
138: trans ce test_simple_7.xml s{i1} test_simple_7.xml.2.trans_base ok
139: trans ce test_simple_8.xml outer test_simple_8.xml.1.trans_base FAILED (DMRTest.at:465)
140: trans ce test_simple_8.xml outer.s.s test_simple_8.xml.2.trans_base FAILED (DMRTest.at:466)
141: trans ce test_simple_8.xml outer{s{s}} test_simple_8.xml.2.trans_base FAILED (DMRTest.at:467)
142: trans ce test_array_7.xml s test_array_7.xml.1.trans_base ok
143: trans ce test_array_7.xml s{i1;s} test_array_7.xml.1.trans_base ok
...
## ------------- ##
## Test results. ##
## ------------- ##

ERROR: All 249 tests were run,
10 failed (7 expected failures).
## ------------------------ ##
## DMRTest.log was created. ##
## ------------------------ ##

Please send `tests/DMRTest.log' and all information you think might help:

   To: <[email protected]>
   Subject: [libdap 3.21.0] DMRTest: 139 140 141 failed

You may investigate any problem if you feel able to do so, in which
case the test suite provides a good starting point.  Its output may
be found below `tests/DMRTest.dir'.

make[3]: *** [Makefile:1958: check-local] Error 1

Cannot install experimental libdap .deb package from Hyrax 1.11 release (Ubuntu 14.04)

I downloaded libdap_3.14.0-1_x86_64.deb from OPeNDAP's experimental debian package index, but ran into a dependency issue when trying to install it with gdebi. Apparently it requires the package libcurl, which is not available in the default repos for Ubuntu 14.04 (Trusty Tahr).

There is, however, a package named libcurl3 available for Trusty Tahr; maybe the dependency could be changed to this in the published .deb?

Below is my output showing the failed installation:

~$ sudo gdebi libdap_3.14.0-1_x86_64.deb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Building data structures... Done
Building data structures... Done
This package is uninstallable
Dependency is not satisfiable: libcurl

The development package, libdap-devel_3.14.0-1_x86_64.deb, installed without issue.

configure: error: Could not find uuid.h

It seems that libuuid, or another library providing uuid.h, is also a requirement for building libdap?

I got an error message saying that uuid.h was not found during configure.

Building latest release, 3.18.0. This was, however, also the case on 3.12 and I believe 3.15 when I tried that (last year sometime).

Configuring with:
configure
--enable-threads=pth
--with-xml2=$PREFIX
--with-curl=$PREFIX
--prefix=$PREFIX

I'm building the binaries for the Anaconda Cloud and will put it out on conda-forge, hence most of the dependencies are from that repository, autoconf, automake, libtool, flex, bison, libuuid, libxml2 and curl are selected directly, the others are indirect dependencies through the other packages:

autoconf: 2.69
automake: 1.14
bison: 3.0.4
ca-certificates: 2016.2.28
curl: 7.49.1
flex: 2.5.39
icu: 56.1
libiconv: 1.14
libtool: 2.4.6
libuuid: 1.0.3
libxml2: 2.9.3
m4: 1.4.17
openssl: 1.0.2h
perl: 5.20.3.1
xz: 5.0.5
zlib: 1.2.8-3

gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)

Grid::get_map_iter() function never returns an iterator for the first map

Example code:

#include <iostream>
#include <libdap/Connect.h>
#include <libdap/Array.h>
#include <libdap/Grid.h>

using namespace libdap;

int main()
{
 Connect cn("http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_53.X/data/2008");
 BaseTypeFactory btf;
 DataDDS data(&btf);

 cn.request_data(data,"surf_el[145:146]");

 data.print(std::cout);

 auto* elb=data.var("surf_el");
 if(elb==nullptr) return 1;

 std::cout<<elb->type_name()<<" "<<elb->length()<<std::endl;

 auto* el=dynamic_cast<Grid*>(elb);
 auto* arr=el->get_array();
 std::cout<<arr->type_name()<<" "<<arr->length()<<std::endl;
 std::cout<<el->components()<<std::endl;
 for(size_t i=0; i<el->components()-1; ++i)
 {
  auto map=*(el->get_map_iter(i));
  std::cout<<map->type_name()<<" "<<map->length()<<std::endl;
 }

 return 0;
}

Output:

Dataset {
    Grid {
      Array:
        Int16 surf_el[time = 2][lat = 3251][lon = 4500];
      Maps:
        Float64 time[time = 2];
        Float64 lat[lat = 3251];
        Float64 lon[lon = 4500];
    } surf_el;
} GLBv0.08/expt_53.X/data/2008;
Grid 1
Array 29259000
4
Array 3251
Array 4500
Segmentation fault

get_map_iter() skips map "time" and cause segmentation fault then called last time.

libdap 3.20.7.

tests fail on big endian arches with 3.17.0

Hello,

we are getting tests failures on big endian arches with 3.17.0

/bin/sh './DMRTest'

-----------------------------------

libdap 3.17.0 test suite: dmr-test.

-----------------------------------

...
73: trans test_array_9.xml FAILED (DMRTest.at:247)
74: trans test_array_12.xml FAILED (DMRTest.at:247)
75: trans test_array_13.xml FAILED (DMRTest.at:247)
76: trans test_array_14.xml FAILED (DMRTest.at:247)
77: trans test_simple_6.2.xml FAILED (DMRTest.at:247)
78: trans test_simple_6.3.xml FAILED (DMRTest.at:247)
79: intern test_array_9.xml FAILED (DMRTest.at:247)
80: intern test_array_12.xml FAILED (DMRTest.at:247)
81: intern test_array_13.xml FAILED (DMRTest.at:247)
82: intern test_array_14.xml FAILED (DMRTest.at:247)
83: intern test_simple_6.2.xml FAILED (DMRTest.at:247)
84: intern test_simple_6.3.xml FAILED (DMRTest.at:247)
...

avoid mkstemps

The mkstemps() function is non-standard and not portable. You either need to test for it in autoconf and provide a fallback, or better: avoid it by using the POSIX.1-2001 standard mkstemp.

CONFORMING TO top
mkstemp(): 4.3BSD, POSIX.1-2001.
mkstemps(): unstandardized, but appears on several other systems.

g -O2 -MT libdap_la-util.lo -MD -MP -MF .deps/libdap_la-util.Tpo -c -o libdap_la-util.lo `test -f 'util.cc' || echo '../libdap-3.19.0/'`util.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../libdap-3.19.0 -I./gl -I../libdap-3.19.0/gl -I../libdap-3.19.0/GNU -IC:/msys64/mingw64/include/libxml2 -Wall -W -Wcast-align -g -O2 -MT libdap_la-util.lo -MD -MP -MF .deps/libdap_la-util.Tpo -c ../libdap-3.19.0/util.cc  -DDLL_EXPORT -DPIC -o .libs/libdap_la-util.o
../libdap-3.19.0/util.cc: In function 'std::__cxx11::string libdap::open_temp_fstream(std::ofstream&, const string&, const string&)':
../libdap-3.19.0/util.cc:1224:19: error: 'mkstemps' was not declared in this scope
     int tmpfile = mkstemps(&name[0], suffix.length());

SHA256 mismatch for libdap 3.19.1

There's a SHA256 mismatch for libdap. Encountered this while installing gdal via homebrew on Mac OS 10.10.5. Was asked to rule out rule out malicious circumstances and to find out why the file changed.

brew install libdap
==> Downloading https://www.opendap.org/pub/source/libdap-3.19.1.tar.gz
######################################################################## 100.0%
Error: SHA256 mismatch
Expected: 5215434bacf385ba3f7445494ce400a5ade3995533d8d38bb97fcef1478ad33e
Actual: fb7014b6047cf3fa47d05c0faf258636f664c5fc232ff0886a78dfc5aae29f8f

newly added tests are broken on big endian arches

Seem that the freshly merged commit 841f6d5 that adds new tests has them broken on big endian arches.

make  check-TESTS check-local
make[4]: Entering directory '/home/sharkcz/projects/libdap4/unit-tests'
make[5]: Entering directory '/home/sharkcz/projects/libdap4/unit-tests'
PASS: marshT
PASS: arrayT
PASS: attrTableT
PASS: structT
PASS: sequenceT
PASS: ddsT
PASS: dasT
PASS: RegexTest
PASS: ArrayTest
PASS: AttrTableTest
FAIL: ByteTest
PASS: MIMEUtilTest
PASS: ancT
PASS: DASTest
PASS: DDSTest
PASS: DDXParserTest
PASS: generalUtilTest
PASS: HTTPConnectTest
PASS: parserUtilTest
PASS: RCReaderTest
PASS: SequenceTest
PASS: SignalHandlerTest
PASS: MarshallerTest
PASS: HTTPCacheTest
PASS: ServerFunctionsListUnitTest
PASS: Int8Test
FAIL: Int16Test
FAIL: UInt16Test
PASS: Int32Test
PASS: UInt32Test
PASS: Int64Test
PASS: UInt64Test
PASS: Float32Test
PASS: Float64Test
PASS: D4MarshallerTest
PASS: D4UnMarshallerTest
PASS: D4DimensionsTest
PASS: D4EnumDefsTest
PASS: D4GroupTest
PASS: D4ParserSax2Test
PASS: D4AttributesTest
PASS: D4EnumTest
PASS: chunked_iostream_test
PASS: D4AsyncDocTest
PASS: DMRTest
PASS: D4FilterClauseTest
PASS: D4SequenceTest
PASS: DmrRoundTripTest
PASS: DmrToDap2Test
============================================================================
Testsuite summary for libdap 3.19.1
============================================================================
# TOTAL: 49
# PASS:  46
# SKIP:  0
# XFAIL: 0
# FAIL:  3
# XPASS: 0
# ERROR: 0

The ByteTest got fixed in between, but the Int16 and UInt16 tests are broken at current HEAD too. Seet the test-suite.log below.

==============================================
   libdap 3.19.1: unit-tests/test-suite.log
==============================================

# TOTAL: 49
# PASS:  46
# SKIP:  0
# XFAIL: 0
# FAIL:  3
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: ByteTest
==============

.....F......


!!!FAILURES!!!
Test Results:
Run:  11   Failures: 1   Errors: 0


1) test: ByteTest::buf2val_test (F) line: 165 ByteTest.cc
assertion failed
- Expression: tb1->buf2val(&v2) == 1 && *(int *)v2 == 6


FAIL ByteTest (exit status: 1)

FAIL: Int16Test
===============

...F.F........


!!!FAILURES!!!
Test Results:
Run:  12   Failures: 2   Errors: 0


1) test: libdap::Int16Test::val2buf_test (F) line: 132 Int16Test.cc
assertion failed
- Expression: i2->value() == 42


2) test: libdap::Int16Test::buf2val_test (F) line: 142 Int16Test.cc
assertion failed
- Expression: i2->buf2val(&v) == 2 && i == 0


FAIL Int16Test (exit status: 1)

FAIL: UInt16Test
================

...F.F........


!!!FAILURES!!!
Test Results:
Run:  12   Failures: 2   Errors: 0


1) test: libdap::UInt16Test::val2buf_test (F) line: 132 UInt16Test.cc
assertion failed
- Expression: i2->value() == 42


2) test: libdap::UInt16Test::buf2val_test (F) line: 142 UInt16Test.cc
assertion failed
- Expression: i2->buf2val(&v) == 2 && i == 0


FAIL UInt16Test (exit status: 1)

This is function should take an unsigned long long type as it's parameter

void set_size(unsigned long size) { d_size = size; }

Because the internal variable d_size is an unsigned long long (and why don't we just migrate to size_t?) and setting it's value with an unsigned long is, well, it defeats the purpose of making the internal variable an unsigned long long and screws the whole deal for large datasets.

For example if I make a D4Dimension with size 3429693028 I get this gem:

<?xml version="1.0" encoding="ISO-8859-1"?>
<Dataset xmlns="http://xml.opendap.org/ns/DAP/4.0#" xmlns:dmrpp="http://xml.opendap.org/dap/dmrpp/1.0.0#" dapVersion="4.0" dmrVersion="1.0" name="" dmrpp:href="https://harmony-uat-staging.s3.us-west-2.amazonaws.com/public/sds/staged/ATL03_20200714235814_03000802_003_01.h5">
    <Byte name="data">
        <Dim size="-865274268"/>
        <dmrpp:chunks byteOrder="LE">
            <dmrpp:chunkDimensionSizes>342969302</dmrpp:chunkDimensionSizes>
            <dmrpp:chunk offset="0" nBytes="342969302" chunkPositionInArray="[0]"/>
            <dmrpp:chunk offset="342969302" nBytes="342969302" chunkPositionInArray="[342969302]"/>
            <dmrpp:chunk offset="685938604" nBytes="342969302" chunkPositionInArray="[685938604]"/>
            <dmrpp:chunk offset="1028907906" nBytes="342969302" chunkPositionInArray="[1028907906]"/>
            <dmrpp:chunk offset="1371877208" nBytes="342969302" chunkPositionInArray="[1371877208]"/>
            <dmrpp:chunk offset="1714846510" nBytes="342969302" chunkPositionInArray="[1714846510]"/>
            <dmrpp:chunk offset="2057815812" nBytes="342969302" chunkPositionInArray="[2057815812]"/>
            <dmrpp:chunk offset="2400785114" nBytes="342969302" chunkPositionInArray="[2400785114]"/>
            <dmrpp:chunk offset="2743754416" nBytes="342969302" chunkPositionInArray="[2743754416]"/>
            <dmrpp:chunk offset="3086723718" nBytes="342969302" chunkPositionInArray="[3086723718]"/>
            <dmrpp:chunk offset="3429693020" nBytes="8" chunkPositionInArray="[3429693020]"/>
        </dmrpp:chunks>
    </Byte>
</Dataset>

Clients using DAP4?

Hi,

is there a list of clients using DAP4? I am struggling to get e.g. ncdump to use dap4 when setting up hyrax through docker, or on thredds for that matter. Is there any performance benchmarks between dap2 and dap4?

I am working a bit more on my rust-server (https://github.com/gauteh/dars) and was wondering if I should go for DAP4-only.

By the way; I tried out your approach with a threaded HDF5 reader and it performs really well: https://github.com/gauteh/hidefix (I hope to make C/C++/Python bindings). Implementing different readers (e.g. S3, whatever, should be straight-forward as long as it can Read and Seek / read slices).

Regards, Gaute

unit-tests/HTTPCacheTest expects filesystem with 4KB blocks

I've tried to run the full test-suite on a Fedora ppc64le system with XFS system and got failure in HTTPCacheTest

.....................F.


!!!FAILURES!!!
Test Results:
Run:  22   Failures: 1   Errors: 0


1) test: libdap::HTTPCacheTest::cache_gc_test (F) line: 812 HTTPCacheTest.cc
assertion failed
- Expression: pc->d_http_cache_table->d_block_size == 4096


FAIL HTTPCacheTest (exit status: 1)

When I checked the pc->d_http_cache_table->d_block_size value I got 64K, then I looked for other differences and the only one was the filesystem, there is no problem with ppc systems with ext4, but this particular one has XFS.

NULL pointer dereference in D4ParserSax2

Hello.

I found a NULL pointer dereference in libdap4.

Please confirm.

Thanks.

OS: CentOS 7 64bit
Version: commit 9806db1
PoC Download: Null_D4ParserSax2.zip

Steps to reproduce:
1.Download the .POC files.
2.Execute the following command
: ./dmr-test -p $FILE

ASAN:DEADLYSIGNAL
=================================================================
==20816==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x0000004b2c1f bp 0x7ffc42e12560 sp 0x7ffc42e11c90 T0)
==20816==The signal is caused by a READ memory access.
==20816==Hint: address points to the zero page.
    #0 0x4b2c1e in AddressIsPoisoned /home/karas/src/llvm/projects/compiler-rt/lib/asan/asan_mapping.h:322
    #1 0x4b2c1e in QuickCheckForUnpoisonedRegion /home/karas/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:43
    #2 0x4b2c1e in printf_common(void*, char const*, __va_list_tag*) /home/karas/src/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors_format.inc:529
    #3 0x4b5725 in __interceptor_vsnprintf /home/karas/src/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1396
    #4 0x7dda14 in libdap::D4ParserSax2::dmr_error(void*, char const*, ...) /home/karas/libdap4/D4ParserSax2.cc:1223:5
    #5 0x7e150f in libdap::D4ParserSax2::process_dimension_def(char const*, unsigned char const**, int) /home/karas/libdap4/D4ParserSax2.cc:209:9
    #6 0x7f82fb in libdap::D4ParserSax2::dmr_start_element(void*, unsigned char const*, unsigned char const*, unsigned char const*, int, unsigned char const**, int, int, unsigned char const**) /home/karas/libdap4/D4ParserSax2.cc:704:30
    #7 0x7fedad7648cf  (/lib64/libxml2.so.2+0x498cf)
    #8 0x7fedad76ae61  (/lib64/libxml2.so.2+0x4fe61)
    #9 0x7fedad76c61d  (/lib64/libxml2.so.2+0x5161d)
    #10 0x8092e7 in libdap::D4ParserSax2::intern(std::istream&, libdap::DMR*, bool) /home/karas/libdap4/D4ParserSax2.cc:1342:9
    #11 0x523d80 in test_dap4_parser(std::string const&, bool, bool) /home/karas/libdap4/tests/dmr-test.cc:94:20
    #12 0x52d2ef in main /home/karas/libdap4/tests/dmr-test.cc:380:15
    #13 0x7fedac74dc04 in __libc_start_main (/lib64/libc.so.6+0x21c04)
    #14 0x42670b in _start (/home/karas/libdap4/tests/dmr-test+0x42670b)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/karas/src/llvm/projects/compiler-rt/lib/asan/asan_mapping.h:322 in AddressIsPoisoned
==20816==ABORTING

==========
[Acknowledgement]
This work was supported by ICT R&D program of MSIP/IITP. [R7518-16-1001, Innovation hub for high Performance Computing]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.