Persistent data storage at the solution instance level

Within Tray, the Data Storage connector is a handy connector you can use to store data for re-use in other connectors. There are different scopes that you can use to determine what workflows have access to this data. With Tray Embedded, it may seem like there isn’t a way to create persistent data storage unique to a specific solution instance. Fear not! I have got you covered. Having data be readily accessible in a group of workflows specific to a certain end-user is easy to implement. In this blog post, I will go over the different scopes of the Data Storage connector and provide you with the JSON you can use to import this persistent data storage implementation into your Tray account.

Data Storage Scopes

In the Data Storage connector, there are three different scopes to choose from: Current Run, Workflow, and Account.

Current Run scope is the most temporary scope of the three. Data stored with this scope will only exist for the current run of the workflow and garbage collected upon the conclusion of the workflow execution.

Workflow scope is the next level of persistence. On repeat runs of a workflow, data stored using this scope will be accessible.

Account scope is the highest level of persistence. Any workflow within the workspace will be able to access the data stored at this level.

For more technical details on these scopes, read the documentation found here.

Implementing Solution Instance Level Data Storage

At first glance, it appears that the existing data storage scopes do not address our problem: sharing data amongst a group of workflows that are unique to a specific end-user. We don’t want to use Account level storage because then all end-users’ workflows will be accessing that data. Workflow level storage is only specific to one workflow. We will be using the Workflow level storage creatively to allow for sharing of data between a set of workflows.

The idea here is that we will be creating a workflow within a project that will essentially act as a data storage API for all the other workflows within the project. This data storage API workflow will be using the Workflow level storage and it will keep track of solution instance level specific data. Whenever a workflow in the project needs specific data, it will call this data storage API workflow to get or set this value.

Note: If the data is always static and never changing, I recommend storing that value as a config property. This data storage API workflow is to be used for data that changes based on workflow executions.

The Data Storage API Workflow

Below is a screenshot of what this workflow will look like. Essentially, we will be implementing our own getter and setter methods within a workflow.

The Callable Trigger will accept 3 parameters:

• Key: The key that will hold the value to be stored (required parameter)

• Value: The persistent value to be retrieved (optional parameter)

• Operation: Will represent what action you want to be taken within the workflow like GET and SET (optional parameter)

To set up these 3 parameters, go to the properties panel for the Callable Trigger and click on edit input schema. Add the 3 parameters in this window.

The Branch connector will then check the value of the operation parameter to determine what actions to perform next. If neither GET or SET were used in the operation parameter, then we will travel down the default path. Think of this Branch connector as a pseudo-switch statement.

The GET Action

In the GET path, the Data Storage will simply retrieve the value held in the specific key passed into the key parameter with the Workflow scope.

Then we will set up how to return the retrieved value in the Callable Response connector. Edit the response schema to include a boolean success property and a data property that will house an object. Since the value was retrieved successfully in this branch, mark the success property as true. Then assign the data property the value retrieved in the previous Data Storage step.

The SET Action

In the SET path, the first step will be to set the passed in value from the Callable Trigger on the passed in key with the Workflow scope.

We will then retrieve the recently set value in a subsequent Data Storage connector step. After that, we will insert a Callable Response connector with similar values to the Callable Response in the GET path: a true success and the data retrieved in the previous step.

The Default Action

The Default path is used when the passed in operation when calling this workflow is not GET or SET. In this path, we will simply create a Callable Response that sends back a false success and no data.

And that is all we need to do to set up persistent data storage that is encapsulated within this single workflow.

Calling the Data Storage API Workflow

Whenever you need to retrieve data found in Data Storage, all you have to do is use the Call Workflow connector to trigger the retrieval. See the below screenshot for an example.

When running this workflow, the Debug logs will show exactly how our data storage API workflow responds.

Conclusion

Making persistent solution instance level data storage is, in fact, really easy to implement! Now for the real reason everyone is here on this blog post…

Drum roll please!

Below is the JSON that provides the data storage API workflow along with a demo workflow that calls it.

{ "tray_export_version": 4, "export_type": "project", "workflows": [ { "id": "d7773039-c33a-4115-a883-603af69c7ad4", "created": "2022-02-03T15:22:26.155767Z", "workspace_id": "74977ddf-8a23-4604-a331-7093c0310adf", "project_id": "e8ed69f6-64c5-4929-97bc-353a780d1b84", "group": "3f800002-d63a-49d6-867d-0a67d4d0c280", "creator": "74977ddf-8a23-4604-a331-7093c0310adf", "version": { "id": "b8cd06b0-667e-4fb1-8260-2a23c3e72ce6", "created": "2022-02-03T17:24:01.328208Z" }, "title": "Persistent Data Storage", "enabled": true, "tags": [], "settings": { "config": {}, "input_schema": { "type": "object", "properties": { "key": { "type": "string", "title": "Key", "description": "Key holding stored-value" }, "value": { "title": "Value", "type": "string", "description": "Stored-value held in the key" }, "operation": { "title": "Operation", "type": "string", "description": "Specified operation to determinate what to do with key/value inputs" } }, "additionalProperties": false, "required": [ "key" ], "advanced": [] }, "output_schema": { "type": "object", "properties": { "success": { "title": "Success", "description": "Mark true if value was retrieved", "type": "boolean", "default": false }, "data": { "title": "Data", "type": "object", "properties": {}, "additionalProperties": false, "required": [], "advanced": [] } }, "additionalProperties": false, "required": [], "advanced": [] } }, "steps_structure": [ { "name": "trigger", "type": "normal", "content": {} }, { "name": "branch-1", "type": "branch", "content": { "__default__": [ { "name": "callable-workflow-response-3", "type": "normal", "content": {} } ], "branch1": [ { "name": "storage-1", "type": "normal", "content": {} }, { "name": "callable-workflow-response-1", "type": "normal", "content": {} } ], "branch2": [ { "name": "storage-2", "type": "normal", "content": {} }, { "name": "storage-3", "type": "normal", "content": {} }, { "name": "callable-workflow-response-2", "type": "normal", "content": {} } ] } } ], "steps": { "branch-1": { "title": "Determine Operation", "connector": { "name": "branch", "version": "1.1" }, "operation": "simple", "output_schema": {}, "error_handling": {}, "properties": { "value": { "type": "jsonpath", "value": "$.steps.trigger.data.operation" }, "branches": { "type": "array", "value": [ { "type": "object", "value": { "value": { "type": "string", "value": "GET" }, "label": { "type": "string", "value": "GET" } } }, { "type": "object", "value": { "value": { "type": "string", "value": "SET" }, "label": { "type": "string", "value": "SET" } } } ] } } }, "storage-2": { "title": "Set value on key", "connector": { "name": "storage", "version": "1.4" }, "operation": "set", "output_schema": {}, "error_handling": {}, "properties": { "scope": { "type": "string", "value": "Workflow" }, "key": { "type": "jsonpath", "value": "$.steps.trigger.data.key" }, "value": { "type": "jsonpath", "value": "$.steps.trigger.data.value" } } }, "callable-workflow-response-1": { "title": "Return value retrieved", "connector": { "name": "callable-workflow-response", "version": "1.0" }, "operation": "response", "output_schema": {}, "error_handling": {}, "properties": { "response": { "value": { "success": { "type": "boolean", "value": true }, "data": { "type": "jsonpath", "value": "$.steps.storage-1.value" } }, "type": "object" } } }, "storage-3": { "title": "Get set value to return", "connector": { "name": "storage", "version": "1.4" }, "operation": "get", "output_schema": {}, "error_handling": {}, "properties": { "scope": { "type": "string", "value": "Workflow" }, "default_value": { "type": "null", "value": null }, "key": { "type": "jsonpath", "value": "$.steps.trigger.data.key" } } }, "callable-workflow-response-2": { "title": "Return set value", "connector": { "name": "callable-workflow-response", "version": "1.0" }, "operation": "response", "output_schema": {}, "error_handling": {}, "properties": { "response": { "value": { "success": { "type": "boolean", "value": true }, "data": { "type": "jsonpath", "value": "$.steps.storage-3.value" } }, "type": "object" } } }, "storage-1": { "title": "Get value from key", "connector": { "name": "storage", "version": "1.4" }, "operation": "get", "output_schema": {}, "error_handling": {}, "properties": { "scope": { "type": "string", "value": "Workflow" }, "default_value": { "type": "null", "value": null }, "key": { "type": "jsonpath", "value": "$.steps.trigger.data.key" } } }, "callable-workflow-response-3": { "title": "Return unsuccessful message", "connector": { "name": "callable-workflow-response", "version": "1.0" }, "operation": "response", "output_schema": {}, "error_handling": {}, "properties": {} }, "trigger": { "title": "Callable Trigger", "connector": { "name": "callable-trigger", "version": "2.0" }, "operation": "trigger_and_respond", "output_schema": {}, "error_handling": {}, "properties": {} } }, "dependencies": [] }, { "id": "295d597a-536b-4205-95a9-93b90efae52f", "created": "2022-02-03T15:58:08.944781Z", "workspace_id": "74977ddf-8a23-4604-a331-7093c0310adf", "project_id": "e8ed69f6-64c5-4929-97bc-353a780d1b84", "group": "1ddbfb2d-1f76-48c1-9dc0-de75d0635647", "creator": "74977ddf-8a23-4604-a331-7093c0310adf", "version": { "id": "57937d26-2ea2-4152-871a-e1a64dfac73a", "created": "2022-02-03T15:58:52.227090Z" }, "title": "Demo Retrieval", "enabled": true, "tags": [], "settings": { "config": {}, "input_schema": {}, "output_schema": {} }, "steps_structure": [ { "name": "trigger", "type": "normal", "content": {} }, { "name": "call-workflow-1", "type": "normal", "content": {} } ], "steps": { "trigger": { "title": "Manual Trigger", "connector": { "name": "noop", "version": "1.1" }, "operation": "trigger", "output_schema": {}, "error_handling": {}, "properties": {} }, "call-workflow-1": { "title": "Call Workflow", "connector": { "name": "call-workflow", "version": "2.0" }, "operation": "fire_and_wait_for_response", "output_schema": {}, "error_handling": {}, "properties": { "workflow_id": { "type": "string", "value": "d7773039-c33a-4115-a883-603af69c7ad4" }, "trigger_input": { "type": "object", "value": { "key": { "type": "string", "value": "demoKey" }, "value": { "type": "string", "value": "demoValue" }, "operation": { "type": "string", "value": "SET" } } } } } }, "dependencies": [ { "id": "d7773039-c33a-4115-a883-603af69c7ad4", "name": "Persistent Data Storage" } ] } ], "projects": [ { "id": "e8ed69f6-64c5-4929-97bc-353a780d1b84", "name": "Data Storage Project", "config": {}, "workflows": [ "d7773039-c33a-4115-a883-603af69c7ad4", "295d597a-536b-4205-95a9-93b90efae52f" ], "dependencies": [] } ] }