The secure Data Access API enables data providers secure registration to access and/or exchange data in a peer-to-peer fashion once the contracts and security mechanisms for identity management have been executed and confirmed. This improves scalability and avoids the need that data providers have to share their data assets with intermediaries (e.g. a marketplace operator). In addition, anonymization can be used to hide the provider’s identity.

Data Access API consists of the following main components:

  1. Authentication and Authorization
  2. Policy Management
  3. Role Management
  4. Secure Data Transfer and Anonymization
Figure 1 – Data Access System component diagram

Components of the system

Authentication and Authorization

Authentication – Verifies the identity of the user against the i3-market Backplane

Authorization – Verifies the permissions the authenticated user has in the i3-market platform allowing to perform authorized actions and granting access to resources

The Authentication & Authorization subsystem has the following sub-components:

Policy management

Policy is a set of rules that defines how to protect the assets to provide trust, security and privacy. Policy Management component oversees enforcing the rule set provided by i3-market Backplane within the Data Access system. The responsibilities of the Policy management module are:

  • Intercept access attempts
  • Check attempt against rule set
  • Grant access to permitted assets

Role management

A Role is a set of Policies attached to an entity in order to define the access that entity has within the i3-market Data Access system. The Role management component is in charge with fetching the list of policies and verifying them against the Data Access system. The responsibilities of the Role management module are:

  • Get the list of policies associated with Role from Backplane
  • Verify Role access by invoking Policy management
  • Allow or Deny functionalities

Secure Data Transfer & Anonymization

Secure Data Transfer & Anonymization subsystem has the following components:

Data Encryption

The responsibilities of the Data encryption module are:

  • Key generation and Exchange
  • Transfer data in an encrypted way between endpoints
  • Decrypt data on the consumer side

Data Proxy

The proxy needs to be used when the identity of the Data Provider needs to be hidden. This feature is optional, therefore no needed to implement it if there is no specific requirement referring to the anonymity of the Data Provider. The responsibilities of the Proxy module are:

  • Activate the proxy
  • Configure the parameters to hide the identity
  • Data transfer goes through the proxy

Data Transfer Transparency

Data Transfer Transparency subsystem has the following components:

Data Transfer Management

This component is responsible with the management of the connection between provider and consumer and implements the following functionalities:

  • Initialize the connection
  • Resume the connection
  • Finalize the connection

Data Transfer Tracking

This component implements the following operation:

  • Measure the amount of transferred data

Data Transfer Monitor

The information about how much data was transferred, when the data transfer was initiated and when it was completed are monitored and the following operations are triggered:

  • Inform the i3-Market Backplane that the data transfer was performed and reports how much data was transferred
  • Invoke the linked smart contract

Data Management

Two methods for data transfer are supported by Data Access API which are supported by the following modules:

Batch Data Transfer Management

One-time data transfer for one chunk of data in a session with the following methods:

  • Request data
  • Transfer data

Data Stream Management

Continuous transfer of data based on a subscription, e.g. publish/subscribe mechanism:

  • Subscribe to an offering
  • Trigger data transfer – on the producer side
  • Get data – on the consumer side
  • Unsubscribe

Technical requirements

For Data Access API the capabilities described below have been defined. They are structured as epics and User Stories.

Authentication & Authorization

NameDescription
Policy Management
Policy is a set of rules that defines how to protect the assets in order to provide trust, security and privacy. Policy Management component is in charge of enforcing the rule set provided by i3-market Backplane inside of the Data Access system.
Role Management
A Role is a set of Policies attached to an entity in order to define the access that entity has within the i3-market Data Access system. The Role management component is in charge with fetching the list of policies and verifying them against the Data Access system.
Table 1 – Authentication & Authorization Epics

NameDescription
Intercept access attempts
As Data Provider I want to intercept the Data Access API access attempts so I can check the policy
Check attempt against rule set
As Data Provider I want to check the access attempt of Data Access API against policy so I be able to grant access
Grant access to permitted assets
As Data Provider I want to grant access to assets so the user can access the data
Get the list of policies associated with Role
As Data Provider I want to access Backplane so I obtain the list of policies associated with the user’s role
Verify role access
As Data Provider I want to invoke Policy management so I will verify the role access of the user
Allow or Deny access
As Data Provider I want to allow or deny access so the data can be accessed according to Policy
Table 2 – Authentication & Authorization User Stories

Data Transfer Transparency

NameDescription
Data Transfer Management
Data Transfer Management is a component which is in charge with the control of the connection between provider and consumer.
Data Transfer Tracking
The Data Transfer Tracking component measures the volume of data transferred between producer and consumer.
Data Transfer Monitor
The Data Transfer Monitor component communicates with the Backplane before and after the data transfer.
Table 3 – Data Transfer Transparency Epics

Name Description
Initialize the connectionAs Data Provider I want to initialize a connection so I will be able to start the transfer
Resume the connectionAs Data Provider I want to resume the connection so I be able to continue the transfer
Finalize the connectionAs Data Provider I want to finalize the connection so the I conclude the transfer
Measure transferred dataAs Data Provider I want to measure the transferred data so I can report the information to the Backplane
Inform i3-market BackplaneAs Data Provider I want inform the Backplane so the system can track the volume of transferred data
Invoke linked smart contractAs Data Provider I want to invoke the smart contract so the data can be transferred according to contractural parameters
Table 4 – Data Transfer Transparency User Stories

Secure Data Transfer & Anonymization

NameDescription
Data Encryption The Data Encryption component is responsible with the end-to-end process of encoding and decoding of data during transfer between producer and consumer.
ProxyThe Proxy component can be used when the Data Producer identity needs to be hidden
Table 5 – Secure Data Transfer & Anonymization Epics

NameDescription
Key generation and exchangeAs Data Provider I want to obtain the encryption key so I will be able to transfer the data securely
Transfer encrypted dataAs Data Provider I want to transfer encrypted data so I be able to enforce the transfer safety and confidentiality
Decrypt dataAs Data Consumer I want to decrypt the transferred data so the I access the transferred data
Activate proxyAs Data Provider I want to activate the proxy so I can can hide my identity
Transfer data through proxyAs Data Provider I to transfer the data through proxy so the my identity remains confidential
Table 6 – Secure Data Transfer & Anonymization User Stories

Data Management

NameDescription
Batch Data Transfer ManagementBatch Data Transfer Management refers to one time data transfer and retrieving one chunk of data in a session
Data Stream ManagementData Stream Management component is responsible with the continuous transfer of data based on a subscription, e.g. publish/subscribe mechanism
Table 7 – Data Management Epics

NameDescription
Request batch dataAs Data Consumer I want to request a batch of data so I will be able to obtain the data from a provider
Transfer batch dataAs Data Provider I want to transfer a batch data so I will send the data to consumer
Subscribe to channelAs Data Consumer I want to subscribe to a channel so the I access the streaming data
Trigger data transferAs Data Provider I want to trigger the data transfer so the data is sent on a stream
Get dataAs Data Consumer I want to get the data so I can save data locally
Unsubscribe from channelAs Data Consumer I want to unsubscribe from a channel so I disconnect from the stream of data
Table 8 – Data Management User Stories

Data Access SDK

NameDescription
Batch Data Transfer ManagementAuthentication and Authorization are required for users who call the Data Access API from Data Access SDK
Data Stream ManagementData Transfer is a component which is responsible with the management of the request data and response
Table 9 – Data Access SDK Epics

NameDescription
Authenticate and Authorize the Data ConsumerAs Software developer I want to authenticate and authorization the consumer so I will be able to obtain the data from a provider
Request dataAs Software developer I want to implement a data request so the I get access to data
Get DataAs Software developer I want to implement the get data so the I can transfer the data locally
Table 10 – Data Access SDK User Stories

Interface description

Data Access API

The Data Access API is the interface via which data consumers gain access to the data offered by a data provider or data space. Since this open interface enables direct interactions among stakeholders of different data spaces/marketplaces, we need not only an open interface specification that can be implemented by all, but also a high-level of security, as the data exchange might involve sensitive data, e.g. personal data or commercial data.

The endpoints documented below were grouped by modules.

Batch Transfer Controller

POST /CHECK_BATCH

(batchTransferController.checkBatch)

Verifies whether the data is available for consumption.

  • Consumes

This API call consumes the following media types via the Content-Type request header:

application/json

  • Request body

body DataRequest(required)

Body Parameter —

  • Return type

AvailabilityResponse

  • Example data

Content-Type: application/json

{

“available” : false,

“message” : “Data is not available due to networking issues at provider side.”

}

  • Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

application/json

  • Responses

200

Returns whether the data requested by the consumer is available at the provider side. AvailabilityResponse

400

Occurs when one of the input fields provided in the body is invalid. BadRequest

401

Occurs when an authentication token was not provided. Unauthorized

403

Occurs when the authentication token does not contain the necessary scopes or the token is expired. Forbidden

500

Unknown exception occured while checkind data availability. InternalServerError

POST /BATCH

(batchTransferController.getBatch)

Requests data from a provider in form of stream.

  • Consumes

This API call consumes the following media types via the Content-Type request header:

application/json

  • Request body

body DataRequest (optional)

Body Parameter —

  • Return type

DataResponse

  • Example data

Content-Type: application/json

{

“additionalInfo” : “Transfer status: started. Progress: 0%”,

“transferId” : “c3e3ae76-39fd-11eb-adc1-0242ac120002”

}

  • Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

application/json

  • Responses

200

Accepted. The data consumer provided a valid token for authentication. The data is returned to the consumer. DataResponse

400

Occurs when one of the input fields provided in the body is invalid. BadRequest

401

Occurs when an authentication token was not provided. Unauthorized

403

Occurs when the authentication token does not contain the necessary scopes or the token is expired. Forbidden

500

Unknown exception occured while requesting data. InternalServerError

POST/PROXY_BATCH

(batchTransferController.proxyBatch)

Requests data from a provider in form of batches. The data should be intercepted by a proxy that hides sensitive data.

  • Consumes

This API call consumes the following media types via the Content-Type request header:

application/json

  • Request body

body DataRequest (optional)

Body Parameter —

  • Return type

DataResponse

  • Example data

Content-Type: application/json

{

“additionalInfo” : “Transfer status: started. Progress: 0%”,

“transferId” : “c3e3ae76-39fd-11eb-adc1-0242ac120002”

}

  • Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

application/json

  • Responses

200

Accepted. The data consumer provided a valid token for authentication. The data is returned to the consumer. DataResponse

400

Occurs when one of the input fields provided in the body is invalid. BadRequest

401

Occurs when an authentication token was not provided Unauthorized

403

Occurs when the authentication token does not contain the necessary scopes or the token is expired Forbidden

500

Unknown exception occured while requesting data. InternalServerError

Stream Transfer Controller

POST /CHECK_STREAM

(streamTransferController.checkStream)

Verifies whether the data is available for consumption.

  • Consumes

This API call consumes the following media types via the Content-Type request header:

application/json

  • Request body

body DataRequest(optional)

Body Parameter —

  • Return type

AvailabilityResponse

  • Example data

Content-Type: application/json

{

“available” : false,

“message” : “Data is not available due to networking issues at provider side.”

}

  • Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

application/json

  • Responses

200

Returns whether the data requested by the consumer is available at the provider side. AvailabilityResponse

400

Occurs when one of the input fields provided in the body is invalid. BadRequest

401

Occurs when an authentication token was not provided. Unauthorized

403

Occurs when the authentication token does not contain the necessary scopes or the token is expired. Forbidden

500

Unknown exception occured while checkind data availability. InternalServerError

POST /PROXY_STREAM

(streamTransferController.proxyStream)

Requests data from a provider in form of stream. The data should be intercepted by a proxy that hides sensitive data.

  • Consumes

This API call consumes the following media types via the Content-Type request header:

application/json

  • Request body

body DataRequest (optional)

Body Parameter —

  • Return type

DataResponse

  • Example data

Content-Type: application/json

{

“additionalInfo” : “Transfer status: started. Progress: 0%”,

“transferId” : “c3e3ae76-39fd-11eb-adc1-0242ac120002”

}

  • Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

application/json

  • Responses

200

Accepted. The data consumer provided a valid token for authentication. The data is returned to the consumer. DataResponse

400

Occurs when one of the input fields provided in the body is invalid. BadRequest

401

Occurs when an authentication token was not provided. Unauthorized

403

Occurs when the authentication token does not contain the necessary scopes or the token is expired. Forbidden

500

Unknown exception occured while requesting data. InternalServerError

POST/STREAM

(streamTransferController.stream)

Requests data from a provider in form of stream.

  • Consumes

This API call consumes the following media types via the Content-Type request header:

application/json

  • Request body

body DataRequest (optional)

Body Parameter —

  • Return type

DataResponse

  • Example data

Content-Type: application/json

{

“additionalInfo” : “Transfer status: started. Progress: 0%”,

“transferId” : “c3e3ae76-39fd-11eb-adc1-0242ac120002”

}

  • Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

application/json

  • Responses

200

Accepted. The data consumer provided a valid token for authentication. The data is returned to the consumer. DataResponse

400

Occurs when one of the input fields provided in the body is invalid. BadRequest

401

Occurs when an authentication token was not provided Unauthorized

403

Occurs when the authentication token does not contain the necessary scopes or the token is expired Forbidden

500

Unknown exception occured while requesting data. InternalServerError

Transfer Monitor Controller

POST /REPORT

(transferMonitorController.report)

The user of the marketplace wants to be informed about the data transferred between provider or data space and consumer.

  • Consumes

This API call consumes the following media types via the Content-Type request header:

application/json

  • Request body

body (optional)

Body Parameter —

  • Return type

StatusResponse

  • Example data

Content-Type: application/json

{

“status” : “resumed”

}

  • Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

application/json

  • Responses

200

Returns the status of a transfer. StatusResponse

400

Occurs when one of the input fields provided in the body is invalid. BadRequest

401

Occurs when an authentication token was not provided. Unauthorized

403

Occurs when the authentication token does not contain the necessary scopes or the token is expired. Forbidden

500

Unknown exception occured while checkind data availability. InternalServerError

Transfer Tracking Controller

POST /STATUS

(transferTrackingController.status)

The transfer of data needs to be tracked by consumer once the subscription was confirmed. The consumer wants to know the status of the process meaning that the transfer was: started, interrupted, stopped, resumed. The user of the marketplace need to know if the transfer was successfully completed or the data was corrupted.

  • Consumes

This API call consumes the following media types via the Content-Type request header:

application/json

  • Request body

body (optional)

Body Parameter —

  • Return type

StatusResponse

  • Example data

Content-Type: application/json

{

“status” : “resumed”

}

  • Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

application/json

  • Responses

200

Returns the status of a transfer. StatusResponse

400

Occurs when one of the input fields provided in the body is invalid. BadRequest

401

Occurs when an authentication token was not provided. Unauthorized

403

Occurs when the authentication token does not contain the necessary scopes or the token is expired. Forbidden

500

Unknown exception occured while requesting data. InternalServerError

POST/TRACK_BATCH

(transferTrackingController.trackBatch)

Checks the volume of data that was consumed in batches.

  • Consumes

This API call consumes the following media types via the Content-Type request header:

application/json

  • Request body

body (optional)

Body Parameter —

  • Return type

BatchTrackingResponse

  • Example data

Content-Type: application/json

{

“percent” : “33%”

}

  • Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

application/json

  • Responses

200

Percent of data that was consumed until now. BatchTrackingResponse

400

Occurs when one of the input fields provided in the body is invalid. BadRequest

401

Occurs when an authentication token was not provided Unauthorized

403

Occurs when the authentication token does not contain the necessary scopes or the token is expired Forbidden

500

Unknown exception occured while requesting data. InternalServerError

POST/TRACK_STREAM

(transferTrackingController.trackStream)

Checks the volume of data that was consumed from a stream.

  • Consumes

This API call consumes the following media types via the Content-Type request header:

application/json

  • Request body

body (optional)

Body Parameter —

  • Return type

StreamTrackingResponse

  • Example data

Content-Type: application/json

{

“volume” : “25MB”

}

  • Produces

This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.

application/json

  • Responses

200

Volume of data that was consumed until now. StreamTrackingResponse

400

Occurs when one of the input fields provided in the body is invalid. BadRequest

401

Occurs when an authentication token was not provided Unauthorized

403

Occurs when the authentication token does not contain the necessary scopes or the token is expired Forbidden

500

Unknown exception occured while requesting data. InternalServerError

State of the Art

Loopback

Loopback is an open source solution developed by StrongLoop, an IBM company. It is a framework that enables you to create dynamic end-to-end APIs (RESTful and GraphQL). It is for Node.js and developed in TypeScript, a typed superset of JavaScript. Due to its modular connectors, it can (indeed does) support any DB as well as custom data integrations like blockchains.

Kong

Kong gateway is a scalable, open source API Layer (also known as an API Gateway, or API Middleware). Kong can  run in front of any RESTful API and may be extended through Plugins, which provide extra functionality and services beyond the core platform.

GraphQL

GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools.

Free* Open Source Software Tools for SMEs, Developers and Large Industries Building/Enhancing their Data Marketplaces.

For more detail and source code please refer below link.

Go to the beginning of the page

Other resources of interest

Home

We are a community of experienced developers who understand that it is all about changing the perception of data economy via marketplaces support.

i3-MARKET Architecture

Take a look at the main building blocks and their hierarchy.

Data Storage

The Decentralised storage shall provide highest available security guarantees…

Identity and Access Management

The SSI & IAM subsystem is in charge of providing both “User-centric Authentication” and…

Smart Contracts, Wallets & Accounting

i3-M Wallet is a set of technologies that facilitate the management of their identity to…

Crypto Token and Monetization System

The Data Monetization subsystem is in charge of providing “Standard Payments”…

Semantic Engine

We developed and implemented dedicated software components for Semantic Engine System as…

Specification Deployment

i3-MARKET architecture specification is based on the 4+1 architectural view model approach. One of…

Developers Quickstart SDK

Once a marketplace is part of i3-MARKET, it can issue credentials to its consumers, providers, and…

Documentation

Full Developers Documentation