The secure Data Access API enables data providers secure registration to access and/or exchange data in a peer-to-peer fashion once the contracts and security mechanisms for identity management have been executed and confirmed. This improves scalability and avoids the need that data providers have to share their data assets with intermediaries (e.g. a marketplace operator). In addition, anonymization can be used to hide the provider’s identity.
Data Access API consists of the following main components:
- Authentication and Authorization
- Policy Management
- Role Management
- Secure Data Transfer and Anonymization
Table of Contents
Components of the system
Authentication and Authorization
Authentication – Verifies the identity of the user against the i3-market Backplane
Authorization – Verifies the permissions the authenticated user has in the i3-market platform allowing to perform authorized actions and granting access to resources
The Authentication & Authorization subsystem has the following sub-components:
Policy management
Policy is a set of rules that defines how to protect the assets to provide trust, security and privacy. Policy Management component oversees enforcing the rule set provided by i3-market Backplane within the Data Access system. The responsibilities of the Policy management module are:
- Intercept access attempts
- Check attempt against rule set
- Grant access to permitted assets
Role management
A Role is a set of Policies attached to an entity in order to define the access that entity has within the i3-market Data Access system. The Role management component is in charge with fetching the list of policies and verifying them against the Data Access system. The responsibilities of the Role management module are:
- Get the list of policies associated with Role from Backplane
- Verify Role access by invoking Policy management
- Allow or Deny functionalities
Secure Data Transfer & Anonymization
Secure Data Transfer & Anonymization subsystem has the following components:
Data Encryption
The responsibilities of the Data encryption module are:
- Key generation and Exchange
- Transfer data in an encrypted way between endpoints
- Decrypt data on the consumer side
Data Proxy
The proxy needs to be used when the identity of the Data Provider needs to be hidden. This feature is optional, therefore no needed to implement it if there is no specific requirement referring to the anonymity of the Data Provider. The responsibilities of the Proxy module are:
- Activate the proxy
- Configure the parameters to hide the identity
- Data transfer goes through the proxy
Data Transfer Transparency
Data Transfer Transparency subsystem has the following components:
Data Transfer Management
This component is responsible with the management of the connection between provider and consumer and implements the following functionalities:
- Initialize the connection
- Resume the connection
- Finalize the connection
Data Transfer Tracking
This component implements the following operation:
- Measure the amount of transferred data
Data Transfer Monitor
The information about how much data was transferred, when the data transfer was initiated and when it was completed are monitored and the following operations are triggered:
- Inform the i3-Market Backplane that the data transfer was performed and reports how much data was transferred
- Invoke the linked smart contract
Data Management
Two methods for data transfer are supported by Data Access API which are supported by the following modules:
Batch Data Transfer Management
One-time data transfer for one chunk of data in a session with the following methods:
- Request data
- Transfer data
Data Stream Management
Continuous transfer of data based on a subscription, e.g. publish/subscribe mechanism:
- Subscribe to an offering
- Trigger data transfer – on the producer side
- Get data – on the consumer side
- Unsubscribe
Technical requirements
For Data Access API the capabilities described below have been defined. They are structured as epics and User Stories.
Authentication & Authorization
| Name | Description |
| Policy Management | Policy is a set of rules that defines how to protect the assets in order to provide trust, security and privacy. Policy Management component is in charge of enforcing the rule set provided by i3-market Backplane inside of the Data Access system. |
| Role Management | A Role is a set of Policies attached to an entity in order to define the access that entity has within the i3-market Data Access system. The Role management component is in charge with fetching the list of policies and verifying them against the Data Access system. |
| Name | Description |
| Intercept access attempts | As Data Provider I want to intercept the Data Access API access attempts so I can check the policy |
| Check attempt against rule set | As Data Provider I want to check the access attempt of Data Access API against policy so I be able to grant access |
| Grant access to permitted assets | As Data Provider I want to grant access to assets so the user can access the data |
| Get the list of policies associated with Role | As Data Provider I want to access Backplane so I obtain the list of policies associated with the user’s role |
| Verify role access | As Data Provider I want to invoke Policy management so I will verify the role access of the user |
| Allow or Deny access | As Data Provider I want to allow or deny access so the data can be accessed according to Policy |
Data Transfer Transparency
| Name | Description |
| Data Transfer Management | Data Transfer Management is a component which is in charge with the control of the connection between provider and consumer. |
| Data Transfer Tracking | The Data Transfer Tracking component measures the volume of data transferred between producer and consumer. |
| Data Transfer Monitor | The Data Transfer Monitor component communicates with the Backplane before and after the data transfer. |
| Name | Description |
| Initialize the connection | As Data Provider I want to initialize a connection so I will be able to start the transfer |
| Resume the connection | As Data Provider I want to resume the connection so I be able to continue the transfer |
| Finalize the connection | As Data Provider I want to finalize the connection so the I conclude the transfer |
| Measure transferred data | As Data Provider I want to measure the transferred data so I can report the information to the Backplane |
| Inform i3-market Backplane | As Data Provider I want inform the Backplane so the system can track the volume of transferred data |
| Invoke linked smart contract | As Data Provider I want to invoke the smart contract so the data can be transferred according to contractural parameters |
Secure Data Transfer & Anonymization
| Name | Description |
| Data Encryption | The Data Encryption component is responsible with the end-to-end process of encoding and decoding of data during transfer between producer and consumer. |
| Proxy | The Proxy component can be used when the Data Producer identity needs to be hidden |
| Name | Description |
| Key generation and exchange | As Data Provider I want to obtain the encryption key so I will be able to transfer the data securely |
| Transfer encrypted data | As Data Provider I want to transfer encrypted data so I be able to enforce the transfer safety and confidentiality |
| Decrypt data | As Data Consumer I want to decrypt the transferred data so the I access the transferred data |
| Activate proxy | As Data Provider I want to activate the proxy so I can can hide my identity |
| Transfer data through proxy | As Data Provider I to transfer the data through proxy so the my identity remains confidential |
Data Management
| Name | Description |
| Batch Data Transfer Management | Batch Data Transfer Management refers to one time data transfer and retrieving one chunk of data in a session |
| Data Stream Management | Data Stream Management component is responsible with the continuous transfer of data based on a subscription, e.g. publish/subscribe mechanism |
| Name | Description |
| Request batch data | As Data Consumer I want to request a batch of data so I will be able to obtain the data from a provider |
| Transfer batch data | As Data Provider I want to transfer a batch data so I will send the data to consumer |
| Subscribe to channel | As Data Consumer I want to subscribe to a channel so the I access the streaming data |
| Trigger data transfer | As Data Provider I want to trigger the data transfer so the data is sent on a stream |
| Get data | As Data Consumer I want to get the data so I can save data locally |
| Unsubscribe from channel | As Data Consumer I want to unsubscribe from a channel so I disconnect from the stream of data |
Data Access SDK
| Name | Description |
| Batch Data Transfer Management | Authentication and Authorization are required for users who call the Data Access API from Data Access SDK |
| Data Stream Management | Data Transfer is a component which is responsible with the management of the request data and response |
| Name | Description |
| Authenticate and Authorize the Data Consumer | As Software developer I want to authenticate and authorization the consumer so I will be able to obtain the data from a provider |
| Request data | As Software developer I want to implement a data request so the I get access to data |
| Get Data | As Software developer I want to implement the get data so the I can transfer the data locally |
Interface description
Data Access API
The Data Access API is the interface via which data consumers gain access to the data offered by a data provider or data space. Since this open interface enables direct interactions among stakeholders of different data spaces/marketplaces, we need not only an open interface specification that can be implemented by all, but also a high-level of security, as the data exchange might involve sensitive data, e.g. personal data or commercial data.
The endpoints documented below were grouped by modules.
Batch Transfer Controller
POST /CHECK_BATCH
(batchTransferController.checkBatch)
Verifies whether the data is available for consumption.
This API call consumes the following media types via the Content-Type request header:
application/json
- Request body
body DataRequest(required)
Body Parameter —
AvailabilityResponse
Content-Type: application/json
{
“available” : false,
“message” : “Data is not available due to networking issues at provider side.”
}
This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.
application/json
200
Returns whether the data requested by the consumer is available at the provider side. AvailabilityResponse
400
Occurs when one of the input fields provided in the body is invalid. BadRequest
401
Occurs when an authentication token was not provided. Unauthorized
403
Occurs when the authentication token does not contain the necessary scopes or the token is expired. Forbidden
500
Unknown exception occured while checkind data availability. InternalServerError
POST /BATCH
(batchTransferController.getBatch)
Requests data from a provider in form of stream.
This API call consumes the following media types via the Content-Type request header:
application/json
body DataRequest (optional)
Body Parameter —
DataResponse
Content-Type: application/json
{
“additionalInfo” : “Transfer status: started. Progress: 0%”,
“transferId” : “c3e3ae76-39fd-11eb-adc1-0242ac120002”
}
This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.
application/json
200
Accepted. The data consumer provided a valid token for authentication. The data is returned to the consumer. DataResponse
400
Occurs when one of the input fields provided in the body is invalid. BadRequest
401
Occurs when an authentication token was not provided. Unauthorized
403
Occurs when the authentication token does not contain the necessary scopes or the token is expired. Forbidden
500
Unknown exception occured while requesting data. InternalServerError
POST/PROXY_BATCH
(batchTransferController.proxyBatch)
Requests data from a provider in form of batches. The data should be intercepted by a proxy that hides sensitive data.
This API call consumes the following media types via the Content-Type request header:
application/json
body DataRequest (optional)
Body Parameter —
DataResponse
Content-Type: application/json
{
“additionalInfo” : “Transfer status: started. Progress: 0%”,
“transferId” : “c3e3ae76-39fd-11eb-adc1-0242ac120002”
}
This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.
application/json
200
Accepted. The data consumer provided a valid token for authentication. The data is returned to the consumer. DataResponse
400
Occurs when one of the input fields provided in the body is invalid. BadRequest
401
Occurs when an authentication token was not provided Unauthorized
403
Occurs when the authentication token does not contain the necessary scopes or the token is expired Forbidden
500
Unknown exception occured while requesting data. InternalServerError
Stream Transfer Controller
POST /CHECK_STREAM
(streamTransferController.checkStream)
Verifies whether the data is available for consumption.
This API call consumes the following media types via the Content-Type request header:
application/json
- Request body
body DataRequest(optional)
Body Parameter —
AvailabilityResponse
Content-Type: application/json
{
“available” : false,
“message” : “Data is not available due to networking issues at provider side.”
}
This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.
application/json
200
Returns whether the data requested by the consumer is available at the provider side. AvailabilityResponse
400
Occurs when one of the input fields provided in the body is invalid. BadRequest
401
Occurs when an authentication token was not provided. Unauthorized
403
Occurs when the authentication token does not contain the necessary scopes or the token is expired. Forbidden
500
Unknown exception occured while checkind data availability. InternalServerError
POST /PROXY_STREAM
(streamTransferController.proxyStream)
Requests data from a provider in form of stream. The data should be intercepted by a proxy that hides sensitive data.
This API call consumes the following media types via the Content-Type request header:
application/json
body DataRequest (optional)
Body Parameter —
DataResponse
Content-Type: application/json
{
“additionalInfo” : “Transfer status: started. Progress: 0%”,
“transferId” : “c3e3ae76-39fd-11eb-adc1-0242ac120002”
}
This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.
application/json
200
Accepted. The data consumer provided a valid token for authentication. The data is returned to the consumer. DataResponse
400
Occurs when one of the input fields provided in the body is invalid. BadRequest
401
Occurs when an authentication token was not provided. Unauthorized
403
Occurs when the authentication token does not contain the necessary scopes or the token is expired. Forbidden
500
Unknown exception occured while requesting data. InternalServerError
POST/STREAM
(streamTransferController.stream)
Requests data from a provider in form of stream.
This API call consumes the following media types via the Content-Type request header:
application/json
body DataRequest (optional)
Body Parameter —
DataResponse
Content-Type: application/json
{
“additionalInfo” : “Transfer status: started. Progress: 0%”,
“transferId” : “c3e3ae76-39fd-11eb-adc1-0242ac120002”
}
This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.
application/json
200
Accepted. The data consumer provided a valid token for authentication. The data is returned to the consumer. DataResponse
400
Occurs when one of the input fields provided in the body is invalid. BadRequest
401
Occurs when an authentication token was not provided Unauthorized
403
Occurs when the authentication token does not contain the necessary scopes or the token is expired Forbidden
500
Unknown exception occured while requesting data. InternalServerError
Transfer Monitor Controller
POST /REPORT
(transferMonitorController.report)
The user of the marketplace wants to be informed about the data transferred between provider or data space and consumer.
This API call consumes the following media types via the Content-Type request header:
application/json
- Request body
body (optional)
Body Parameter —
StatusResponse
Content-Type: application/json
{
“status” : “resumed”
}
This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.
application/json
200
Returns the status of a transfer. StatusResponse
400
Occurs when one of the input fields provided in the body is invalid. BadRequest
401
Occurs when an authentication token was not provided. Unauthorized
403
Occurs when the authentication token does not contain the necessary scopes or the token is expired. Forbidden
500
Unknown exception occured while checkind data availability. InternalServerError
Transfer Tracking Controller
POST /STATUS
(transferTrackingController.status)
The transfer of data needs to be tracked by consumer once the subscription was confirmed. The consumer wants to know the status of the process meaning that the transfer was: started, interrupted, stopped, resumed. The user of the marketplace need to know if the transfer was successfully completed or the data was corrupted.
This API call consumes the following media types via the Content-Type request header:
application/json
body (optional)
Body Parameter —
StatusResponse
Content-Type: application/json
{
“status” : “resumed”
}
This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.
application/json
200
Returns the status of a transfer. StatusResponse
400
Occurs when one of the input fields provided in the body is invalid. BadRequest
401
Occurs when an authentication token was not provided. Unauthorized
403
Occurs when the authentication token does not contain the necessary scopes or the token is expired. Forbidden
500
Unknown exception occured while requesting data. InternalServerError
POST/TRACK_BATCH
(transferTrackingController.trackBatch)
Checks the volume of data that was consumed in batches.
This API call consumes the following media types via the Content-Type request header:
application/json
body (optional)
Body Parameter —
BatchTrackingResponse
Content-Type: application/json
{
“percent” : “33%”
}
This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.
application/json
200
Percent of data that was consumed until now. BatchTrackingResponse
400
Occurs when one of the input fields provided in the body is invalid. BadRequest
401
Occurs when an authentication token was not provided Unauthorized
403
Occurs when the authentication token does not contain the necessary scopes or the token is expired Forbidden
500
Unknown exception occured while requesting data. InternalServerError
POST/TRACK_STREAM
(transferTrackingController.trackStream)
Checks the volume of data that was consumed from a stream.
This API call consumes the following media types via the Content-Type request header:
application/json
body (optional)
Body Parameter —
StreamTrackingResponse
Content-Type: application/json
{
“volume” : “25MB”
}
This API call produces the following media types according to the Accept request header; the media type will be conveyed by the Content-Type response header.
application/json
200
Volume of data that was consumed until now. StreamTrackingResponse
400
Occurs when one of the input fields provided in the body is invalid. BadRequest
401
Occurs when an authentication token was not provided Unauthorized
403
Occurs when the authentication token does not contain the necessary scopes or the token is expired Forbidden
500
Unknown exception occured while requesting data. InternalServerError
State of the Art
Loopback
Loopback is an open source solution developed by StrongLoop, an IBM company. It is a framework that enables you to create dynamic end-to-end APIs (RESTful and GraphQL). It is for Node.js and developed in TypeScript, a typed superset of JavaScript. Due to its modular connectors, it can (indeed does) support any DB as well as custom data integrations like blockchains.
Kong
Kong gateway is a scalable, open source API Layer (also known as an API Gateway, or API Middleware). Kong can run in front of any RESTful API and may be extended through Plugins, which provide extra functionality and services beyond the core platform.
GraphQL
GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools.
Free* Open Source Software Tools for SMEs, Developers and Large Industries Building/Enhancing their Data Marketplaces.
For more detail and source code please refer below link.
Go to the beginning of the page
Other resources of interest
Home
We are a community of experienced developers who understand that it is all about changing the perception of data economy via marketplaces support.
i3-MARKET Architecture
Take a look at the main building blocks and their hierarchy.
Data Storage
The Decentralised storage shall provide highest available security guarantees…
Identity and Access Management
The SSI & IAM subsystem is in charge of providing both “User-centric Authentication” and…
Smart Contracts, Wallets & Accounting
i3-M Wallet is a set of technologies that facilitate the management of their identity to…
Crypto Token and Monetization System
The Data Monetization subsystem is in charge of providing “Standard Payments”…
Semantic Engine
We developed and implemented dedicated software components for Semantic Engine System as…
Specification Deployment
i3-MARKET architecture specification is based on the 4+1 architectural view model approach. One of…
Developers Quickstart SDK
Once a marketplace is part of i3-MARKET, it can issue credentials to its consumers, providers, and…
Documentation
Full Developers Documentation