1. Collaborate
  2. Open Data, Services and Software Policies
  3. Earthdata Developer Portal
  4. Hyrax Data Server Installation and Configuration Guide: Chapter 1—Hyrax New Features

Hyrax Data Server Installation and Configuration Guide: Chapter 1—Hyrax New Features

Hyrax Data Server Installation and Configuration Guide

Preface

This manual describes the features and operation of the Hyrax data server, a data server developed by OPeNDAP, Inc. as a reference server for the Data Access Protocol, versions 2 and 4. The Hyrax server is a modular software with a number of handlers that are loaded into a core framework based on the contents of configuration files. Each of the server’s modules provides a distinct functional capability, such as reading data from a certain kind of file, encoding data, or processing data in different ways.

The text contained here was built up over several years as modules were added to the system. Originally, the documentation was built using a Wiki (because it was a collaborative writing tool for a distributed group of people), where each component had a separate page. Over time, as information was spread across many web pages and the Wiki, this became unmanageable. We hope this new format reads more like a guide for people who want to install and configure the server and less like a design document.

Development Sponsored By

Acknowledgments

The High Altitude Observatory at NCAR contributed the BES framework that is the basis for the server’s data processing engine and modular extensibility.

Keith Seyffarth extracted the Wiki’s text that forms the basis of this manual, and Alexander Porrello and Leonard Porrello edited the text.

1. Hyrax New Features (1.15.2)

1.1. Custom Module Configuration with site.conf

1.1.1. About

Prior to Hyrax 1.15.2, if you wanted to customize a module’s configuration, you had to modify its .conf file. Unfortunately, .conf files do not persist through updates, so you were responsible for restoring your custom configurations post-update.

Hyrax 1.15.2 introduces site.conf, a special configuration file that persists through updates, where you can store custom module configurations. To start using site.conf, see its configuration instruction section.

Beginning with release 1.15.3, Hyrax will include a template file, site.conf.proto, that will include many commonly-modified settings. For instructions on how to use the template, see its configuration instruction section.

1.1.2. Theory of Operation

When you launch your server, the BES loads the module configuration files that reside within /etc/bes/modules. The BES then loads site.conf, which resides in /etc/bes.

As the BES reads the custom-configured parameters that you have copied into site.conf, the BES overrides the default configuration parameters that it loaded from the individual module configuration files. For a detailed configuration example, see the example configuration section.

1.1.3. site.conf Configuration Instructions

The following details how you can customize a module’s configuration with site.conf:

  1. Create site.conf in \etc\bes with the following command:
    sudo touch site.conf
        
  2. Locate the .conf file for the module that you would like to customize. All configuration files reside within /etc/bes/modules.
  3. Copy the configuration parameters that you would like to customize from the module’s configuration file into site.conf. For a detailed configuration example, see the next section.
    Information Icon Configuration parameters are generally a key/value pair; for example, the default server administrator email parameter is email:support@opendap.org, where email is the key and support@opendap.org is the value.
  4. Save your updates to site.conf.
  5. Restart the server.
1.1.4. site.conf Configuration Example

The following steps detail how you can update the BES’s server administrator configuration parameters with your organization’s information:

  1. Locate the existing server administrator configuration in /etc/bes/bes.conf:
    BES.ServerAdministrator=email:support@opendap.org
    BES.ServerAdministrator+=organization:OPeNDAP Inc.
    BES.ServerAdministrator+=street:165 NW Dean Knauss Dr.
    BES.ServerAdministrator+=city:Narragansett
    BES.ServerAdministrator+=region:RI
    BES.ServerAdministrator+=postalCode:02882
    BES.ServerAdministrator+=country:US
    BES.ServerAdministrator+=telephone:+1.401.575.4835
    BES.ServerAdministrator+=website:http://www.opendap.org
        
    Light Bulb Icon When adding parameters to the ServerAdministrator configuration, notice how, following the first line, we use += instead of just to add new key/value pairs. += indicates to the BES that we are adding new configuration parameters, rather than replacing those that were already loaded. Had we used just + in the above example, the only configured parameter would have been website.
  2. Copy the above block of text from its default .conf file to site.conf.
  3. In site.conf, update the block of text with your organization’s information; for example…
    BES.ServerAdministrator=email:smootchy@woof.org
    BES.ServerAdministrator+=organization:Mogogogo Inc.
    BES.ServerAdministrator+=street:165 Buzzknucker Blvd.
    BES.ServerAdministrator+=city: KnockBuzzer
    BES.ServerAdministrator+=region:OW
    BES.ServerAdministrator+=postalCode:00007
    BES.ServerAdministrator+=country:MG
    BES.ServerAdministrator+=telephone:+1.800.555.1212
    BES.ServerAdministrator+=website:http://www.mogogogo.org
        
  4. Save your changes to site.conf.
  5. Restart the server.
1.1.5. site.conf.proto Configuration Instructions

The site.conf.proto template resides in \etc\bes. If you want to take advantage of the template, copy site.conf.proto into site.conf with the following command:

cp site.conf.proto site.conf

Uncomment the configuration parameters that you want to modify and update them. For a site.conf configuration example, see the previous section.

1.1.6. Earthdata Login OAuth2

If you wanted to secure the data on your server with NASA’s OAuth2 Earthdata Login services prior to Hyrax 1.15.2, your only option was mod_auth_urs. While mod_auth_urs is robust and thoroughly tested, it is inconvenient for some in that it must be bound to an instance of Apache webserver, which is not required to run Hyrax.

If you do not need Apache webserver but would still like to take advantage of NASA’s OAuth2 Earthdata login services, also known as the User Registration System (URS), Hyrax 1.15.2 offers a standalone implementation of the Earthdata Login Client that can deployed using only Tomcat or a preferred servlet engine.

If you require a robust security solution that has undergone thorough testing, you should implement mod_auth_urs. You can find instructions for configuring mod_auth_urs in the Apache/URS section of this manual.

exclamation icon To take advantage of this feature, you must create an Earthdata Login Application. If you have not yet done this, see the Nasa’s Earthdata Login - OAuth2 (mod_auth_urs) section of this manual.
Enabling Hyrax’s Standalone Earthdata Login Client Implementation

To enable Hyrax’s standalone implementation of the Earthdata login, you need to first modify a few lines of code in web.xml. If you installed Tomcat using the instructions in this manual (via YUM), you can locate web.xml at the following location: /usr/share/tomcat/webapps/opendap/WEB-INF/.

Information Icon You should generally avoid editing web.xml, since the file is overwritten whenever you upgrade to a new version of Hyrax; however, in the software’s current iteration, the only way to enable Hyrax’s standalone implementation of the Earthdata login client is by modifying web.xml.

After navigating to /usr/share/tomcat/webapps/opendap/WEB-INF/

  1. Open web.xml.
  2. Scroll down until you locate the Identity Provider (IdP) filter and the Policy Enforcement Point (PEP) filter, both of which are commented.
  3. Remove the comments around these filters:
    <!-- Uncomment These two filters to enable access control.
        <filter>
            <filter-name>IdP</filter-name>
            <filter-class>opendap.auth.IdFilter</filter-class>
        </filter>
        <filter-mapping>
            <filter-name>IdP</filter-name>
            <url-pattern>/*</url-pattern>
        </filter-mapping>
        <filter>
            <filter-name>PEP</filter-name>
            <filter-class>opendap.auth.PEPFilter</filter-class>
        </filter>
        <filter-mapping>
            <filter-name>PEP</filter-name>
            <url-pattern>/*</url-pattern>
        </filter-mapping>
        -->
        
Light Bulb Icon The IdP filter is responsible for figuring out who users are, and the PEP filter determines what users can or cannot do.
Configuring the Identification Provider

After uncommenting the IdP and PEP filters in web.xml, you must link your server to your Earthdata Login Application by configuring the identification provider.

You must first modify a few lines of code in user-access.xml. If you installed Tomcat via YUM, user-access.xml can be found in one of the following locations:

After accessing the file, scroll down until you locate the IdProvider element:

<idprovider class="opendap.auth.UrsIdP">
        <authcontext>urs</authcontext>
        <isdefault></isdefault>
        <ursclientid>iPlEjZjMvrdwLUlnbaKxWQ</ursclientid>
        <ursclientauthcode>aVBsRWpaak12cmR3TFVsbmJhS3hXUTpKSEdqa2hmZzY3OA==</ursclientauthcode>
        <ursurl>https://uat.urs.earthdata.nasa.gov</ursurl>
</idprovider>

Configure your identification provider by updating the following child elements:

  • authContex: Determines whether you are using the production (urs) or test (uat) service. Only NASA-authorized applications can use the production service.
  • UrsClientID: The Client ID can be located on your application’s Application Administration page.
  • UrsClientAuthCode: You must generate the authorization code using your Earthdata Login Application’s Client ID and password. See the following section for more information.
  • UrsUrl: Depending on the authContext, should be one of the following:
    • https://uat.urs.earthdata.nasa.gov
    • https://urs.earthdata.nasa.gov
Compute <URSClientAuthCode>

You can compute the URS client authorization code with one of the following methods:

  1. Shell Script:
    echo -n "<cid>:<pw>" | base64
        
  2. Perl Script:
    perl -e 'use MIME::Base64; print encode_base64("<cid>:<pw>");'
        
  3. PHP Script:
    php -r 'echo base64_encode("<cid>:<pw>");'
        

In the above examples, <cid> is the Client ID (found on your application’s Application Administration page), and <pw> is your application’s password.

1.2. MetaData Store (MDS)

A new cache, the MetaData Store (MDS), has been added to the BES for metadata responses. This cache is unlike the other BES caches in that it is intended to be operated as either a "cache" or a "store." In the latter case, added items will never be removed. It is an open-ended place where metadata response objects are kept indefinitely. The contents of the MDS (as a cache or a store) will persist through Hyrax restarts.

Light Bulb Icon The MDS is especially important for scenarios where data is not as close as you need it to be. By using the MDS, you can reduce the time it takes to look through files and make quick decisions based on the metadata that the MDS has saved; however, because the underlying data in the MDS does not update automatically, the metadata may be out of sync with the actual data.
1.2.1. Enable or Disable the MDS

To enable or disable the MDS, access dap.conf in the /etc/bes/modules directory, and remove the comment before the following line of code:

DAP.GlobalMetadataStore.path = /usr/share/mds

See the code block below for the MDS section of dap.conf:

#-----------------------------------------------------------------------#
# Metadata Store parameters                                             #
#-----------------------------------------------------------------------#
# Control the Metadata Response Store. Here, DAP metadata responses
# are stored/cached so that they can be returned by the server w/o
# having to touch the data files/objects. Setting the 'path' to null
# disables uses of the MDS. Setting 'size' to zero makes the MDS
# hold objects forever; setting a positive non-zero size makes the
# MDS behave like a cache, purging responses when the size is exceeded.
#DAP.GlobalMetadataStore.path = /usr/share/mds
DAP.GlobalMetadataStore.prefix = mds
# Size in MB
DAP.GlobalMetadataStore.size = 200
# The MDS writes a ledger of additions and removals. By default the
# ledger is kept in 'mds_ledger.txt' in the directory used to start
# the BES.
DAP.GlobalMetadataStore.ledger = /usr/share/mds_ledger.txt
# This tells the BES Framework's DAP module to use the DMR++
# handler for data requests if it find a DMR++ response in the MDS
# for a given granule.
# DAP.Use.Dmrpp = yes
1.2.2. Configure the MDS

The MDS' parameters should be configured in site.conf. For more information, please see the site.conf section.

To configure the MDS to work as a store, rather than a cache, set the Dap.GlobalMetadataSote.size to 0. To configure the MDS as a cache, set it to your desired size.

1.2.3. Cache Ejection Strategy

The MDS caches complete metadata responses and serves those directly.

The MDS can be configured to not exceed a certain size. When the configured size is met, the MDS ejects the metadata files in the cache that were least recently accessed.

By default the metadata files are stored in usr/share/mds. You can change this location by modifying theDAP.GlobalMetadataStore.path. You can modify this parameter in dap.conf, but you should modify it in site.conf. For more information, please see the site.conf section.

1.3. CoverageJSON

CoverageJSON uses heuristics to determine if a data set is suitable for CoverageJSON expression. You can take advantage of the CoverageJSON feature in the following way:

1.4. JSON-LD

This release Hyrax adds JSON-LD content to every browser-navigable catalog page (i.e. */contents.html") and to every dataset/granule OPeNDAP Data Access Form. Along with the site map generation, this feature can be used to assist search engines to catalog, index, and find the data that you want the world to access.

1.4.1. Theory of Operation

The JSON-LD is dynamically built from the metadata. It uses the MDS metadata if it is there; otherwise, it loads it from the file.

1.4.2. Configuration Instructions

The server ships with this feature enabled. By default the publisher information is set to OPeNDAP:

BES.ServerAdministrator=email:support@opendap.org
BES.ServerAdministrator+=organization:OPeNDAP Inc.
BES.ServerAdministrator+=street:165 NW Dean Knauss Dr.
BES.ServerAdministrator+=city:Narragansett
BES.ServerAdministrator+=region:RI
BES.ServerAdministrator+=postalCode:02882
BES.ServerAdministrator+=country:US
BES.ServerAdministrator+=telephone:+1.401.575.4835
BES.ServerAdministrator+=website:http://www.opendap.org

This information can and should be updated to your organization’s information. You can update the information in /etc/bes/bes.conf; however, you should configure the parameters in site.conf. For more information, see the site.conf section.

1 This material is based upon work supported by the National Science Foundation under Grant No. 0430822. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).

Last Updated: Sep 26, 2019 at 2:04 PM EDT