A data source is a core component of a Hypersync app responsible for retrieving data from an external service. Data sources can connect to external systems through REST, GraphQL, or direct database access.
Each Hypersync app includes one data source component that either implements the IDataSource interface or inherits from a base class that does.
A data source can expose multiple data sets, each identified by name. For example, a Jira data source might provide data sets for issues, labels, and users. These data set names are referenced by other components—such as proof types—to specify which data should be used.
Note: Data sets can and should be re-used across multiple proof types, and can also be used in non-proof scenarios such as collecting user data during the
validateCredentialsprocess.
For services that expose their data through a REST API, developers are recommended to derive a data source from the RestDataSourceBase base class. This base class makes it possible to configure the data sets along with filters, sorts, and transformations in a dataSources.json file that's included in your package. You can configure most of your data retrieval functionality without writing any code.
If your external service uses OAuth for authorization, your app's data source should look like this:
export class MyServiceDataSource extends RestDataSourceBase {
constructor(accessToken: string) {
super(config as IRestDataSourceConfig, Messages, {
Authorization: `Bearer ${accessToken}`,
'Content-Type': 'application/json'
});
}
}You should also have createDataSource method like this in your app's Hypersync class:
public async createDataSource(accessToken: string): Promise<IDataSource> {
return new MyServiceDataSource(accessToken);
}For services that use another method of autherization, use the following pattern:
export class MyServiceDataSource extends RestDataSourceBase {
constructor(credentials: CustomAuthCredentials) {
// TODO: Update the headers below for your REST service.
super(config as IRestDataSourceConfig, Messages, {
Authorization: `Basic ...`,
'Content-Type': 'application/json'
});
}
}In your Hypersync app:
public async createDataSource(credentials: CustomAuthCredentials): Promise<IDataSource> {
return new MyServiceDataSource(credentials);
}Once you've created your app's RestDataSourceBase component and updated the createDataSource method in your HypersyncApp, add the dataSource.json file within the /json directory. The RestDataSourceBase base class automatically loads this configuration file when it is instantiated.
For more information on configuring the data sets in your data source using dataSource.json, refer to the REST data source JSON format article.
Many REST APIs use pagination to return data in manageable chunks. For example, an API might accept pageSize and pageNumber parameters to specify how many items to return and which page to retrieve.
The Hypersync SDK supports four pagination styles: Page-Based, Offset and Limit, Next Token, and GraphQL Connections. By default, pagination parameters are automatically added to the query string of the API URL. If the data source uses the POST HTTP method, the paging parameters are included in the request body instead.
Begin paging at a starting value and increment the page value by one (1) after each iteration (1, 2, 3, etc). Return at most limitValue items per page.
"pagingScheme": {
"type": "pageBased",
"request": {
"pageParameter": "pageNumber",
"pageStartingValue": 1,
"limitParameter": "pageSize",
"limitValue": 100
},
"pageUntil": "noDataLeft"
}The request property in a paging scheme defines how the paginated query string is constructed. For example, the first API call in the scenario above would include the query string: ?pageNumber=1&pageSize=100. Each paging scheme must also define a pageUntil property, which specifies the condition that determines when pagination should stop.
If the reachTotalCount condition is used, the totalCount field must be defined in the response object. This field represents the path to the total number of items available from the external service.
Begin paging at a starting value and increment the offset by the number of elements in a full page (0, 100, 200, 300, etc). Return at most limitValue items per page.
"pagingScheme": {
"type": "offsetAndLimit",
"request": {
"offsetParameter": "offset",
"offsetStartingValue": 0,
"limitParameter": "limit",
"limitValue": 100
},
"response": {
"totalCount": "pagination.total"
},
"pageUntil": "reachTotalCount"
}The request property in a paging scheme defines how the paginated query string is constructed. For example, the first API call in this scheme would include the following query string: ?offset=0&limit=100. Each paging scheme must also include a pageUntil property, which specifies when pagination should stop.
If the reachTotalCount condition is used, the totalCount field must be defined in the response object. This field represents the path to the total number of items available from the external service.
Begin paging and continue until nextToken is no longer provided. Return at most limitValue items per page. Tokens may be a unique string returned from the external service or a url.
"pagingScheme": {
"type": "nextToken",
"request": {
"tokenParameter": "token",
"limitParameter": "size",
"limitValue": 20
},
"response": {
"nextToken": "next.token"
},
"pageUntil": "noNextToken",
"tokenType": "token"
}The request property in a paging scheme defines how the paginated query string is constructed. For example, the first API call in this flow would include: ?size=20. Each subsequent call follows the pattern: ?size=20&token=891b629672384d04.
Every paging scheme must also include a pageUntil property, which specifies when pagination should stop.
When the noNextToken condition is used, a nextToken field must be defined in the response object. This field represents the path to the token value returned by the external service.
Following the GraphQL connections specification, continue paging until hasNextPage is false. Return at most limitValue items per page. Supports forward, non-nested pagination.
"body": {
"query": "query($first: Int, $after: String) { attributes(first: $first, after: $after) { nodes { id name } pageInfo { endCursor hasNextPage } } }",
"variables": {
"first": 500
}
},
"method": "POST",
"property": "data.attributes.nodes",
"pagingScheme": {
"type": "graphqlConnections",
"request": {
"limitParameter": "first",
"limitValue": 500
},
"response": {
"pageInfo": "data.attributes.pageInfo"
},
"pageUntil": "noNextPage"
}The paging scheme dynamically adds the first and after variables to the request body. The after variable is derived from the endCursor value in the previous response.
The pageInfo field must be defined in the paging scheme’s response object. This field specifies the path to the pageInfo object in the data returned by the external service.
Note: If the required values are found in the response headers instead of the body, prefix the path with
header:.
For services that don't expose data as REST, or for services that use certain REST patterns that are incompatible with RestDataSourceBase, the Hypersync SDK makes it possible to create a custom data source.
To begin, define your data source class using one of the patterns below:
OAuth authorization
export class MyServiceDataSource implements IDataSource {
private accessToken: string;
constructor(accessToken: string) {
this.accessToken = accessToken;
}
}Custom authorization
export class MyServiceDataSource implements IDataSource {
private credentials: CustomAuthCredentials;
constructor(credentials: CustomAuthCredentials) {
this.credentials = credentials;
}
}Once you've created the class and a properly formatted constructor, all that's left is to implement the getData method. This is the only method defined in the IDataSource interface.
async getData(
dataSetName: string,
params?: DataValueMap
): Promise<DataSetResult<DataObject | DataObject[]>> {
// TODO: Retrieve the data set by name using the provided parameters.
}