Skip to content

Database Schemas

Ioto uses database schemas to define what are the database data entities and how these will be stored in the database.

A schema specifies the application models (entities), fields, indexes and other data access and replication parameters.

Ioto uses the DynamoDB OneTable schema format, implementing the OneTable Schema Spec.

The same schema is used for the cloud side AWS DynamoDB table and for the device side embedded database.

Schema Definition

Schemas look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
const DeviceSchema = {
    format: 'onetable:1.1.0',
    version: '0.0.1',
    indexes: {
        primary: {hash: 'pk', sort: 'sk'},
    },
    process: {
        /*
            Where the model is enabled (cloud, device, both. Default is both)
            Synchronization direction. Up to the cloud, down or both. Default is none.
        */
        Status: {enable: 'cloud'},
        Fault:   { 
            sync: 'up' 
            metrics: [{
                namespace: 'Embedthis/Device',
                fields: ['temperature'],
                dimensions: [{Device: 'deviceId'}]
            }]
        },
    },
    models: {
        Status: {
            pk: {type: 'string', value: 'status#'},
            sk: {type: 'string', value: 'status#'},
            parameters: {type: 'object'},
            version: {type: 'string'},
            updated: {type: 'date'},
        },
        Fault: {
            pk: {type: 'string', value: 'device#${deviceId}'},
            sk: {type: 'string', value: 'fault#${id}'},
            deviceId: {type: 'string', required: true},
            id: {type: 'string', generate: 'ulid'},
            timestamp: {type: 'date', required: true},
            source: {type: 'string', required: true},
            severity: {
                type: 'string',
                required: true,
                enum: ['info', 'warn', 'error', 'critical', 'fatal'],
            },
            subject: {type: 'string', required: true},
            message: {type: 'string', required: true},
        },
    },
    params: {
        isoDates: true,
        timestamps: true,
    },
}

Schema Purpose

AWS DynamoDB can operate most efficiently at scale when using a single table with multiple application entities stored in the one table. To make this design pattern easy to implement, application data schemas such as the one used by DynamoDB OneTable provide the mapping between application level entities and raw database items.

Ioto uses the OneTable schema in both the DynamoDB table and in the Ioto agent database.

When coupled with a high-level database API such as that offered by OneTable and by the Ioto Agent Database, accessing and managing device data is both performant and straightforward.

DynamoDB Schema

In Device Clouds, Ioto uses the DynamoDB OneTable NodeJS library for all access to the ioto DynamoDB table. This library has an extensive suite of features beyond those implemented by the Ioto device agent database.

Please consult the OneTable library and OneTable Documentation for background.

Schema Properties

The valid properties of the schema object are:

Property Type Description
format string Reserved. Must be set to onetable:1.1.0
indexes object Hash of indexes used by the table.
models object Hash of model entities describing the model keys, indexes and attributes.
params object Hash of properties controlling how data is stored in the table.
process object Hash of model entities describing where the model should be and how it should be replicated.
version string A SemVer compatible version string.

The format property specifies the schema format version and must be set to onetable:1.1.0.

The indexes property specifies the key structure for the primary, local and secondary indexes.

The models property contains one or more models with attribute field descriptions. The models collections define the attribute names, types, mappings, validations and other properties.

The params property defines additional parameters for table data formats.

The process map defines how database model data should be processed, stored, replicated and whether metrics should be calculated from the data stream.

The version property defines a version for your DynamoDB model design. It must be a SemVer compatible version string. The version string is used by tools and consumers of the schema to understand compatibility constraints for the schema and stored data.

Schema Models

The schema defines a model for each application entity. For example, consider a music example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
{
    album: {
        pk:     { type: 'string', value: '${_type}:${name}' },
        sk:     { type: 'string', value: '${_type}:' },
        name:   { type: 'string', required: true },
        songs:  { type: 'number' },
    },
    artist: {
        pk:     { type: 'string', value: '${_type}:${name}' },
        sk:     { type: 'string', value: '${_type}:' },
        name:   { type: 'string', required: true },
        address: {
            type: Object, schema: {
                street: { type: 'string' },
                city: { type: 'string' },
                zip: { type: 'string' },
            },
        },
    }
}

For each model, all the entity fields are defined by specifying the field type, validations and other operational characteristics (uniqueness, IDs and templates).

The valid types are: array, binary, boolean, date, number, object, and string.

The database will automatically add a model type via the _type attribute to each model. This is set to the name of the model. You can modify this via the params.typeField setting.

Value Templates

The value of a field can be computed based on the value of other fields and a formatting "value template". This is useful for decoupling your key structure from your entity fields.

The value template is like a Javascript string literal where the value of fields are expressed in a "${field}" format. At runtime, the field references are expanded with the item's value for the named field.

In the example above, the primary key "pk" is calculated from the entity type _type and the name field by using the value template: ${_type}:${name}.

Nested Schemas

For object attributes, you can define a nested schema for the object properties, as in the example above (repeated below).

A nested schema uses the schema property to define a nested map of attributes. Schemas can be nested to an arbitrary depth.

1
2
3
4
5
6
7
address: {
    type: Object, schema: {
        street: { type: 'string' },
        city: { type: 'string' },
        zip: { type: 'string' }
    }
}

Database Indexes

Schema indexes are defined using the schema.indexes property.

The indexes property can contain one or more indexes and must contain the primary key. Additional indexes will be treated as secondary Indexes.

Note

Ioto currently only supports the primary index.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
const MySchema = {
    indexes: {
        primary: {
            hash: 'pk',         //  Schema property name of the hash key
            sort: 'sk',         //  Schema property name of the sort key
        },
        //  Zero or more global secondary or local secondary indexes
        gs1: {
            hash: 'gs1pk',
            sort: 'gs1sk',
        }
    }
}

Schema Params

The schema.params is a map of properties that control how data is stored in the database. It may contain the following properties:

Property Type Description
createdField string Name of the "created" timestamp attribute. Defaults to "created".
hidden boolean Hides templated (value) attributes in Javascript properties. Default true.
isoDates boolean Set to true to store dates as Javascript ISO strings vs epoch numerics. Default false.
nulls boolean Store nulls in database attributes vs remove attributes set to null. Default false.
timestamps boolean | string Make "created" and "updated" timestamps in items. Set to true to create both. Set to 'create' for only "created" timestamp and set to "update" for only an "updated" timestamp. See also: "updatedField" and "createdField" properties. Default false.
typeField string Name of the "type" attribute. Default "_type".
updatedField string Name of the "updated" timestamp attribute. Default "updated".

For example:

1
2
3
4
5
6
const MySchema = {
    params: {
        isoDates: true,
        timestamps: true,
    }
}

References