7.2 AI/ML dataset management
7.2.1 Overview
The mechanism and the resource structure of AI/ML dataset management is similar to the <locationPolicy>, <container>, <contentInstance> and resource and corresponding handling procedures [1]. When an AE creates the <mlDatasetPolicy> resource with preferred policies for the Hosting CSE, the AI/ML dataset is created and managed by the Hosting CSE on behalf of the AE. Once a dataset is managed, other AEs can use the datasets for different applications.
Figure 7.2.1-1: Resource structure for AI/ML dataset management
When the resources get instantiated it would look like the figure 7.2.1-2. One dataset policy resource can have two dataset resources: one for historical dataset which is used for AI/ML model training, and the other is the live dataset for inference with the trained models. Each dataset resource contains dataset fragments.
The datasetFormat attribute in the <mlDatasetPolicy> resource as well as the <datasetFragment> resource represents the format of the dataset. As it is widely used in AI/ML model training, CSV (i.e. MIME type "text/csv") is suggested as one of the format options.
Figure 7.2.1-2: Resource tree example
7.2.2 Resource types
7.2.2.1 Resource Type mlDatasetPolicy
The <mlDatasetPolicy> resource represents a policy to create a machine learning dataset with the data from other data sharing resources. This is used to create training datasets with historical data for building machine learning models. When dataset preparation is done, the CSE creates <dataset> and <datasetFragment> resources to store the dataset. Then data scientists fetch, process the dataset and train models. After that, for newly created resources, inference input data is created and stored the by CSE with the policies. Newly created inference input data can be retrieved or notified with subscription to the machine leaerning applications which deploys the trained models.
Table 7.2.2.1-1: Child resources of <mlDatasetPolicy> resource
| Child Resources of <mlDatasetPolicy> | Child Resource Type | Multiplicity | Description | <mlDatasetPolicy> Child Resource Types |
|---|---|---|---|---|
| [variable] | <semanticDescriptor> | 0..n | See clause 9.6.30 [1] | <semanticDescriptor>, <semanticDescriptorAnnc> |
| [variable] | <subscription> | 0..n | See clause 9.6.8 [1] | <subscription> |
The <mlDatasetPolicy> resource shall contain the attributes specified in table 7.2.2.1-2.
Table 7.2.2.1-2: Attributes of <mlDatasetPolicy> resource
| Attributes of <mlDatasetPolicy> |
Multiplicity | RW/ RO/ WO |
Description | <mlDatasetPolicyAnnc> Attributes |
|---|---|---|---|---|
| resourceType | 1 | RO | See clause 9.6.1.3 [1] | NA |
| resourceID | 1 | RO | See clause 9.6.1.3 [1] | NA |
| resourceName | 1 | WO | See clause 9.6.1.3 [1] | NA |
| parentID | 1 | RO | See clause 9.6.1.3 [1] | NA |
| expirationTime | 1 | RW | See clause 9.6.1.3 [1] | MA |
| accessControlPolicyIDs | 0..1 (L) | RW | See clause 9.6.1.3 [1] | MA |
| labels | 0..1 (L) | RW | See clause 9.6.1.3 [1] | MA |
| creationTime | 1 | RO | See clause 9.6.1.3 [1] | NA |
| lastModifiedTime | 1 | RO | See clause 9.6.1.3 [1] | NA |
| announceTo | 0..1 (L) | RW | See clause 9.6.1.3 [1] | NA |
| announcedAttribute | 0..1 (L) | RW | See clause 9.6.1.3 [1] | NA |
| announceSyncType | 0..1 | RW | See clause 9.6.1.3 [1] | MA |
| dynamicAuthorizationConsultationIDs | 0..1 (L) | RW | See clause 9.6.1.3 [1] | OA |
| creator | 0..1 | RO | See clause 9.6.1.3 [1] | NA |
| custodian | 0..1 | RW | See clause 9.6.1.3 [1] | NA |
| sourceResourceIDs | 1(L) | WO | The resource IDs which builds a dataset When the target resource is a <container> resource, content attribute of the children <contentInstance> resouces get included in the dataset. When the target resource is a <timeSeries> resource, content attribute of the cchildren <timeSeriesInstance> resouces get included in the dataset. When the target resource is a <flexContainer> resource, custom attributes of the children <flexContainerInstance> resouces get included in the dataset. |
OA |
| datasetStartTime | 0..1 | WO | The timestamp filter as the start time of source data resources. creationTime or dataGenerationTime depending on the source resource type, gets filtered. If datasetStartTime and datasetEndTime both are not provided, then all data instanes of source resources get included in the dataset. |
OA |
| datasetEndTime | 0..1 | WO | The timestamp filter as the end time of source data resources. creationTime or dataGenerationTime depending on the source resource type, gets filtered. If datasetStartTime and datasetEndTime both are not provided, then all data instanes of source resources get included in the dataset. |
OA |
| timeCorrelationStartTime | 0..1 | WO | When more than one data from source resources get batched as one, this timestamp indicates the start time of each recurring windows. | OA |
| timeCorrelationDuration | 0..1 | WO | This indicates duration for each data batch window. | OA |
| nullValuePolicy | 0..1 | WO | This indicates null(empty) value handling policy (e.g. leave as null, fill with last-known values) for the created dataset. | OA |
| datasetFormat | 1 | WO | The serialization format of the dataset. (e.g. CSV, JSON) Note: CSV format is not supported in oneM2M yet. |
OA |
| historicalDatasetID | 0..1 | RO | The ID of the <dataset > resource for a training dataset which gets generated with existing source resources (i.e. historical data) referred by the sourceResourceIDs attribute. | OA |
| numberOfRowsForHistoricalDataset | 0..1 | WO | The number of data fragment of the dataset is stored in a single <datasetFragment > resource. Default is all. |
OA |
| numberOfRowsForLiveDataset | 0..1 | WO | The number of data fragment of the dataset is stored in a single <datasetFragment > resource. Default is 1(one). |
OA |
| liveDatasetID | 0..1 | RO | The ID of the <dataset > resource for a dataset which gets generated with newly created source resources after creation of the <mlDatasetPolicy > resource. When the is numberOfDataForInference set, a separate <dataset > resource gets created for newly created source resources. | OA |
7.2.2.2 Resource Type dataset
The <dataset> resource represents a container for a dataset. This is created by a CSE when <mlDatasetPolicy> resource is created. When it is created, the links are stored in trainingDatasetID and infefenceInputDataID in the <mlDatasetPolicy> resource, and mlDatasetPolicyID attribute in the <dataset> resource.
The <dataset> resource works similar to <container> resource, but has additional attributes to have dataset related aspects. This supports <latest> and <oldest> resources. The dataset is fragmented and stored in each child <datasetFragment> resource.
Table 7.2.2.2-1: Child resources of <dataset> resource
| Child Resources of <dataset> | Child Resource Type | Multiplicity | Description | <dataset> Child Resource Types |
|---|---|---|---|---|
| [variable] | <semanticDescriptor> | 0..n | See clause 9.6.30 [1] | <semanticDescriptor>, <semanticDescriptorAnnc> |
| [variable] | <subscription> | 0..n | See clause 9.6.8 [1] | <subscription> |
| la | <latest> | 1 | This virual resource refers the latest <datasetFragment> resource. | None |
| ol | <oldest> | 1 | This virual resource refers the oldest <datasetFragment> resource. | None |
| [variable] | <datasetFragment> | 0..n | See clause 7.1.2.3 | <datasetFragmentAnnc> |
The <dataset> resource shall contain the attributes specified in table 7.2.2.2-2.
Table 7.2.2.2-2: Attributes of <dataset> resource
| Attributes of <dataset> |
Multiplicity | RW/ RO/ WO |
Description | <datasetAnnc> Attributes |
|---|---|---|---|---|
| resourceType | 1 | RO | See clause 9.6.1.3 [1] | NA |
| resourceID | 1 | RO | See clause 9.6.1.3 [1] | NA |
| resourceName | 1 | RO | See clause 9.6.1.3 [1] | NA |
| parentID | 1 | RO | See clause 9.6.1.3 [1] | NA |
| expirationTime | 1 | RO | See clause 9.6.1.3 [1] | MA |
| accessControlPolicyIDs | 0..1 (L) | RW | See clause 9.6.1.3 [1] | MA |
| labels | 0..1 (L) | RW | See clause 9.6.1.3 [1] | MA |
| creationTime | 1 | RO | See clause 9.6.1.3 [1] | NA |
| lastModifiedTime | 1 | RO | See clause 9.6.1.3 [1] | NA |
| announceTo | 0..1 (L) | RW | See clause 9.6.1.3 [1] | NA |
| announcedAttribute | 0..1 (L) | RW | See clause 9.6.1.3 [1] | NA |
| announceSyncType | 0..1 | RW | See clause 9.6.1.3 [1] | MA |
| dynamicAuthorizationConsultationIDs | 0..1 (L) | RW | See clause 9.6.1.3 [1] | OA |
| creator | 0..1 | RO | See clause 9.6.1.3 [1] | NA |
| custodian | 0..1 | RW | See clause 9.6.1.3 [1] | NA |
| mlDatasetPolicyID | 1 | RO | The ID of the <mlDatasetPolicyID > resource which generated this <dataset > resource. | OA |
| listOfFeatures | 1 | RO | The list of dataset feature names. | OA |
NOTE: management mechanism for location attribute of a dataset which merges more than one data sharing resources is FFS
7.2.2.3 Resource Type datasetFragment
The <datasetInstance> resource represents a fragment of a dataset. A whole dataset is split into the number of data items which is configured in the corresponding <mlDatasetPolicy> resource of the parent <dataset> resource.
The <datasetFragment> resource shall contain the attributes specified in table 7.2.2.3-1.
Table 7.2.2.3-1: Attributes of <datasetFragment> resource
| Attributes of <datasetFragment> | Multiplicity | RW/ RO/ WO |
Description | <datasetFragmentAnnc> Attributes |
|---|---|---|---|---|
| resourceType | 1 | RO | See clause 9.6.1.3 [1] | NA |
| resourceID | 1 | RO | See clause 9.6.1.3 [1] | NA |
| resourceName | 1 | RO | See clause 9.6.1.3 [1] | NA |
| parentID | 1 | RO | See clause 9.6.1.3 [1] | NA |
| labels | 0..1 (L) | WO | See clause 9.6.1.3 [1] | MA |
| creationTime | 1 | RO | See clause 9.6.1.3 [1] | NA |
| expirationTime | 1 | WO | See clause 9.6.1.3 [1] | NA |
| announceTo | 0..1 (L) | WO | See clause 9.6.1.3 [1] | NA |
| announcedAttribute | 0..1 (L) | WO | See clause 9.6.1.3 [1] | NA |
| announceSyncType | 0..1 | RW | See clause 9.6.1.3 [1] | MA |
| lastModifiedTime | 1 | RO | See clause 9.6.1.3 [1] | NA |
| datasetFragmentStartTime | 1 | RO | The oldest timestamp among the data in the dataset attribute. | OA |
| datasetFragmentEndTime | 1 | RO | The latest timestamp among the data in the dataset attribute. | OA |
| numberOfRowsInFragment | 1 | RO | The number of data in the dataset attribute. | OA |
| datasetFragment | 1 | RO | The dataset fragment generated by the <mlDatasetPolicy > resource, referred by mlDatasetPolicyID of the parent <dataset > resource. | OA |
| datasetFormat | 1 | RO | The serialization format of the dataset. (e.g. CSV, JSON) Note: CSV format is not supported in oneM2M yet. |
OA |
7.2.3 Procedures
7.2.3.1 Procedures for <mlDatasetPolicy>
No change from the CRUD procedures in clause 10.1 [1].
7.2.3.2 Procedures for <dataset>
A <dataset> resource is created by the <mlDatasetPolicy> resource hosting CSE, so the Create prodecure is not specified as an API. When the CSE creates <dataset> resource after the creation of corresponding <mlDatasetPolicy> resource, the custodian attribute of <dataset> resource is set as the same as <mlDatasetPolicy> resource, if present. No change from the Retrieve, Update and Delete procedures in clause 10.1 [1].
7.2.3.3 Procedures for <datasetFragment>
A <datasetFragment> resource is created by the <dataset> resource hosting CSE, so the Create prodecure is not specified as an API. Also, this resource is immutable so Update procedure is not specified. No change from the Retrieve and Delete procedures in clause 10.1 [1].
Editor’s Note: How to merge structured data into a single dataset is FFS. (e.g. flattening structured data to merge into one dataset table)