Open Referral HSDS Validator.

A microservice for validating Open Referral HSDS data packages and resources.

Get started Github Project
Validation of data resources according to the Human Services Data Specifications.
A ready to use Docker image that can be plugged into any infrastructure.
A RESTful API with an Open API definition for integration with other services.

Table of contents

Introduction

The Human Services Data Specification (HSDS) is the technical name for Open Referral’s core data exchange format. HSDS is designed to support the publication of open, interoperable community resource directory data.

HSDS is designed to support bulk exchange of data between different information and > referral systems. It is appropriate to use when you want to export a collection of > information about services,the locations where they can be accessed, and the organizations that deliver them.

The Open Referral HSDS validator is a self-standing microservice component that was designed for validating human service data resources that are to be exchanged between different systems.

The HSDS provides some guidelines on how information should be structured in order to be exported or imported between heterogeneous systems. These guidelines are built on top of the Data Packages and Table Schema specifications which are industry adopted standards.

The Open Referral HSDS specication has its own dedicated repository and a dedicated documentation site.

Github Project HSDS Documentation

Running the service

Getting the Docker image

You need to have Docker service installed on your local machine or any other target host.

Once Docker is installed pull the latest pre-built image from the Docker Hub registry by running:

$ docker pull openreferral/validator

Running the service container

After the Docker image is available you can launch a container by running:

$ docker run -d --network=host --name=openreferral-validator openreferral/validator:latest

You can use any name you want, by replacing the “openreferral-validator” value with one of your choice. The container will bind to localhost:1400 by default. You can change the host and port of the container by setting the HOST and PORT environment variables like so:

$ docker run -d --network=host -e HOST=localhost -e PORT=1400 --name=openreferral-validator openreferral/validator:latest

Once the container is launched, you can stop it and start / restart it on demand like so

$ docker stop openreferral-validator
$ docker restart openreferral-validator

Once the service has been launched you can verify that the API is up by hitting http://localhost:1400/health. If everything is ok you should get a blank page (or a 200 response).

Using the service

The validator service can be used in a range of use cases which are described in the following sections.

Validating CSV data resources

A client system can use the validator for validating standalone HSDS data resource files in CSV format. The input can be either in the form of a physical file at some URL or a binary stream.

Validating a CSV resource

In this validation mode the validator is using the resource schemas as defined in the full HSDS specification.

Validating data packages

In this validation mode the client must provide the location of a valid datapackage.json file that describes an HSDS data resource list that is treated as a bundle.

Validating a data package

The validator will process each defined resource sequentially and check whether the resource conforms to the specified schema. There is an extra option that enables the validator to perform a data integrity check among the related files based on primary vs foreign key checks.

API Definition

The micro-service has an OpenAPI 2.x compliant definition that is automatically generated on startup. You can find the OpenAPI (Swagger) definition here http://localhost:1400/swagger. You can parse the OpenAPI definition with any popular API tool like Swagger, Postman, etc and start interacting with the service.

The following screenshots have been captured using the API Spots Chrome extension which is a tool for visualizing and interacting with Open APIs.

API Surface

API surface

API Operations

API operations

GET /health

Description

Returns a 200 OK response

GET /validate/datapackage

Description

Validate an HSDS data package. The operation requires the URI of valid datapackage.json file to be provided as a query parameter. The service will parse the contents of the data package and try to validate all enlisted resources.

Query parameters

Response

The operation returns a collection of validation results, one per resource as defined within the data package descriptor.

Statuses

Example call

Given we have a sample datapackage.json file at http://example.com/openreferral/datapackage.json that contains a small subset of resources, we can run the following command to validate the contained resources:

$ curl 'http://localhost:1400/validate/datapackage?uri=http://example.com/openreferral/datapackage.json'

If all data resources are valid, the service will return a response like:

[{
  "valid": true,
  "resource": "organization"
}, {
  "valid": true,
  "resource": "program"
}]

In case something is wrong, the response would be something like:

[
    {
        "valid": true,
        "resource": "organization"
    },
    {
        "valid": false,
        "errors": [
            {
                "row": 3,
                "description": "Foreign key \"organization_id\" violation in row 3"
            }
        ],
        "resource": "program"
    }
]

POST /validate/csv

Description

Validate an HSDS data resource file. The operation validates an uploaded CSV data stream using the definition of a specified resource as found in the standard Open Referral data packagespecification. Clients should send a form payload containg a type field with the name of the HSDS logical resource and a file that contains the CSV data stream.

Consumes

Payload

Response

The operation returns an object with the validation results.

Statuses

Example call

$ curl -F 'type=contact' -F 'file=@/home/chris/contacts.csv' 'http://localhost:1400/validate/csv'

A successful validation would return something like:

{
    "valid": true,
    "errors": []
}

Sample data sets

You can test drive the validator using the official Open Referral sample data sets found in the dedicated Github repository. Clone the repository or download as a ZIP archive and extract them locally.