Semantic Engine – System Specification

Semantic Engine

Objectives

We developed and implemented dedicated software components for Semantic Engine System as SEED, that is in charge of managing the semantic metadata, descriptions, queries, discoveries, retrieve, create and map descriptions and manipulate registries, federated queries and components interactions and interfaces. To make the interface and use of functionalities easier we present the external operations via APIs that are more agnostic and easy to use also for non-semantic experts.

Building Blocks

For the Semantic subsystems in charge of dealing with “Semantic Data Management” we can highlight parts like:

• Semantic Meta Data Storage: This component on receiving semantically annotated data push data into registry data store, for example, mongodb.
• Semantic Mapping: This component does semantic mappings and transform data (json to json-ld document) received from API endpoints.
• Vocabulary Management: This environment keeps and manages all of the vocabularies, defined as i3-Market Semantic model, used in different operations of the semantic engine.
• Offering Registration: The Semantic Engine basically REST APIs exposed as endpoints. Semantic engine exposes different endpoints for offering registration. Examples are:
  • register data provider,
  • register data offering of a data provider,
  • update data offerings,
  • deleting a data offering
  • query existing offerings, etc.
• Offering Discovery: The Semantic engine exposes different endpoints for offering discoveries and retrieve. Examples are:
  • Retrieve list of Data Offerings,
  • discover data offerings by Providers,
  • discover data offerings by parameters,
  • search for particular metadata, etc.

The following picture shows a detailed landscape of the current set of microservices (cubes), API’s (little yellow rectangles), components (blue rectangles), and storages (white rectangles) on i3-MARKET.

How the above components interact with each other can be explained with the use of a scenario, for instance, registration of data provider. To complete this task, a data provider fills a template provided by offering registration component. The template for a Data Offering description as a JSON request is Semantically mapped by the Semantic Engine component and is forwarded towards the Semantic Meta Data Storage where this annotated data is pushed into the database. A JSON Data Offering object request is shown in the following example:

After a Semantically annotation the above JSON request is transformed to a Json-ld document for the storage in the metadata registry in MongoDB like:

Technical Requirements

For the Semantic Storage the following high-level capabilities have been defined.

Semantic Meta Data Management

The Semantic engine (SEED) relies on a registry database (MongoDB). All the information for instance, data provider, data offerings, consumer, and querying offering are stored as semantic metadata based on the i3-Market Semantic Models.

Name	Description
Metadata Registry Database	The Registry Database is responsible to store semantic meta data and process the management of metadata components can access to the data.
Spatial and Text Data Storage	To support spatial and full-text search queries, the semantic data manager should be able to index spatial and fulltext data.

Offering registration

The Semantic engine will expose APIs to register, query and update offering. A data provider can register offering, for instance, datasets and the price for data, etc.

Name	Description
Offering Registration	Offering Registration is a component that allows the user to manage their data offering. More specifically, it provides the following functionalities: – Register the data offering – Retrieve all the offerings – Update/Delete offerings Subscribe to an offering

Offering Discovery

The Semantic engine will expose APIs to query the existing offerings in i3-Market backplane. A data consumer can query datasets, prices for any dataset, offering, etc.

Name

Description

Offering Discovery

Offering Discovery is a component that allows the Data Consumers to search the offering data available on the marketplace. The Data Consumer has to specify the characteristics of the data they are looking for. The Offering Discovery module will then process the data request and returns a list of available offering data that meet their requirements. More specifically, the Offering Discovery should provide the following functionalities: – Register the consumer data request – Retrieve all the data request of a consumer – Process a data request – Update/Delete a data request – Subscribe to a data request.

Semantic Vocabulary Management

Name

Description

I3-Market Semantic Models Management

The i3/Market Semantic Models are shared not only with Consortium Partners but also with stakeholders and community. As part of open-source assets, the vocabulary models, documentations, and files used in the i3-Market project are made available. The i3-Market Semantic Model info is the documentation that describes in detail all the taxonomies and vocabularies from needed domains used in i3-Market and that describes and represent all the relationships between them to build the graph representation of the i3-Market semantic model. • The Models Files will be shared in GitHub and GitLab public repositories with releases versions where each section contains the online machine-readable files in OWL and other format for online accessibility, the files are maintained and updated regularly to keep the latest version of the models files up to date. – View, search the concepts of i3-Market – Allow user to propose a new concept – Allow the administrator to add a new concept – Allow the administrator to update/delete an existing concept.

Sequence Diagrams

Data Offering Registration:

Once a provider is successfully authenticated, the provider can see all the APIs exposed by the Semantic Engine – called (SEED) – in the backplane swagger interface

Data Offering Discovery/deletion/update Sequence Diagram:

When a data provider interacts with i3-Market backplane and has successful been authenticated, s/he can perform the following tasks:

• Retrieve offering by providing offering id
• Retrieve list of all offering registered by a data provider using its provider id
• Retrieve list of all offerings filtered by category which are registered not only in local instance of SEED (Semantic Engine) but also other instances of SEEDs running in i3Market cluster.
• Update an offering
• Delete a particular data offering by providing its id
• User can also download the data offering template

Interfaces Description

Define the APIs which expose the functionalities provided by the semantic engine components.

The Semantic engine currently the many functionalities via APIs, which include like: registration, searching, retrieving, updating, and deletion of different Data Offerings and delivery of info to other components.

Register Data Provider

A data provider, when for the first-time interacts with the system, can register its information in the i3-Market.  Below is the endpoint address and the request for registration of data provider in the semantic repository. User must provide in “providerID” field, the ID which was provided to user at the authentication process.

{

  “providerId”: “string”,

  “name”: “string”,

  “description”: “string”,

  “organization”: [

    {

      “organizationId”: “string”,

      “name”: “string”,

      “description”: “string”,

      “address”: “string”,

      “contactPoint”: “string”

    }

  ]

}

Register Data Offering

When a user is registered as data provider, the next step would be to create and register data offerings (semantic descriptions) for the data assets that wants to share/sell. Below is the API pointer, and the request template the user can use to register data offerings. It is important to note that value of “provider” in the offering template should be the “ID” of the data provider

Query a registered Data Offering by Offering Id:

the endpoint to fetch a particular offering registered in triple-store. A data provider must provide the “Offering ID”

Query a list of all registered Data Offerings by Provider Id:

The following endpoint is used to fetch all the offerings registered by a data provider. In this endpoint user must provide the “Provider ID”

Query a list of all registered Data Offerings by Category:

In i3-Market project, we use different nodes in cluster and each node has its own semantic engine instance running on it. Furthermore, each instance of semantic engine will have its own data by categories from different pilots

Update a data Data Offering:

The following endpoint is used to update an already registered data offering.

Delete a Data Offering:

This endpoint can be used to permanently remove an offering from the repository. Users must provide the “Offering ID” of the data offerings they want to delete.

Download Data Offering template:

The following endpoint is used to download the offering template.

Query list of offerings by Active state

The following endpoint is used to search data offerings that are “active” and so made available to be seen and searched by their providers

Query list of offerings by Shared state

The following is the end-point to look for data offerings that are set or not as available to be shared in the network by the data marketplaces

Query list of offerings based on Text/Keyword

The following is the end-point can be used for text searches

Query list of offerings in federated network

The Semantic Engine is able to search, discover and retrieve Data Offerings not only in a single instance of a Marketplace but also throughout the entire nodes of the Marketplace belonging to the i3-Market network via federated queries

SDK(-RI) Semantic Engine Services

Once the functionalities of the Semantic Engine from the internal API interfaces are mapped and reflected in the backplane API gateway they are available to be used via the i3-Market development kits in languages like Java and JavaScript (among others) using directly the SDK-Core and/or the SDK-RI services.

i3-Market Semantic Models

i3-Market Models Specifications

The O-CASUS Semantic Models comprise a collection of ontologies and vocabularies to cover the concepts used in the Backplane to define the

• i3-Market Semantic Core Model
• W3c Data Catalog Vocabulary (DCAT and DCAT-AP)
• W3c Vocabulary of Interlinked Datasets (VoID),
• W3c Verifiable Credentials and DID
• SKOS Simple Knowledge Organization System
• IT Service Ontology
• EU Vocabularies  Frequency Named Authority List
• EU Vocabularies File Type Named Authority List
• EU Vocabularies Languages Named Authority List
• EU Vocabularies Continents Named Authority List
• ADMS license type vocabulary    
• Distribution availability vocabulary          
• Domain Annotations

Free* Open Source Software Tools for SMEs, Developers and Large Industries Building/Enhancing their Data Marketplaces.

For more detail and source code please refer below link.

Go to the beginning of the page

Other resources of interest

Documentation

Full Developers Documentation