Data Sources

Explains how to manage data sources for communications.

Using data in your communications adds a level of personalization that helps your customers feel connected and catered for, increasing the likelihood of them taking positive action on your message. EngageOne Communicate supports multiple data import methods and data file types.

The most common method of data import is by simple file upload. For file uploads, support formats are CSV, JSON, XLS, XLSX or XML.

Note: Data source files must be UTF-8 encoded; if the file is encoded using another encoding method, it will not be successfully parsed and an error message will be displayed (Error adding dataSourceName. File contains incorrect encoding, please edit file and try uploading again).

For a series of short video clips on Data Sources, see Help videos: Data.

Select Data. The following screen is displayed, which shows any existing data sources:

Note: Use the up and down arrows to sort the Data Sources alphanumerically by that column (ascending or descending). The default sort is by Last Modified (ascending). You can also filter the results by entering text in the box Type to filter the list; this means that only Data Sources containing the characters entered will be shown in the list.

To add a new data source, click Add Data Source. The following screen is displayed:

Enter a name for the data source, and then select the data source type, which can be one of the following:

  • Upload File - Specify a filename or drag the file across.
  • Spectrum - Enter your Spectrum OnDemand credentials and then click Connect to view the datasets.
  • Compose - Specify a template that was created in EngageOne Compose.

While a data source is being imported, Communicate returns you to the main Data Sources screen with a progress bar showing the progress of the upload. For example:

While the upload continues, you can do other things with Communicate. For example, you could upload another data source; the two uploads will progress in parallel. Clicking the X to the right of the progress bar allows you to cancel the upload (unless it is very close to completion).

Typical upload times (based on benchmark tests) are as follows:

Note: These values are approximate and may vary.
Number of columns of data File size in MB Upload time in seconds
10 1.1 51
10 11.1 290
20 2.0 55
20 21.0 330
30 3.0 55
30 30.0 339
Note: The Delete and Download buttons only become active when the upload is complete.
Note: If multiple tabs are open looking at the same Data Sources screen, and an upload is cancelled from one tab, the other tab must be refreshed to see the cancellation.

Once the data is imported, click View Data to see the data in detail.

Click Data Schema to edit a field name if required. You can also check the Hidden box against any fields that you want to hide (these fields cannot be used in any communication using this data source).

Upload File

The following types of file are supported: CSV, JSON, XLS, XLSX or XML.

Special characters in column names

If the data source file includes special characters in column names, they are encoded by Communicate using the following format:

__<xxxx>

where xxxx is the 4 digit hexadecimal ASCII code for the desired character. (Note the double underscore.)

Note: Special characters are all characters other than the following:

ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890_

CSV files

If you import data from a CSV file:

  • The first row must be the field names, separated by commas. All spaces are replaced by underscores.
  • Each record must be on a separate line within the file.

If the imported CSV file has blank columns, Communicate will automatically remove them and close up the gaps. For example, the following CSV file has blank columns (denoted by adjacent commas in the first row):

name,surname,,,country,,,testcolumn,,,
Joe,Bloggs,test1,test2,test3,test4
Jane,Smith,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19

This CSV file would be tidied up on input to look like this:

Id,name,surname,country,testcolumn
0d789cef-99xy,Jane,Smith,3,6
0c5678ab-88ab,Joe,Bloggs,test3,none

JSON files

You can import more complex data from JSON (JavaScript Object Notation) files. The file should contain an array of objects (an array should be the top element of the structure).
Consider this example:
[
{
	"firstname": "John",
	"lastname": "Smith",
	"email": "john.smith@mycompany.com",
	"order_number": "5555",
	"delivery_address": [
			"Appartment 234",
			"High Road North",
			"Some Town"
	],	
	"order_items": [
		{
			"description":"Complete Puppy Food",
			"price":"3.49",
			"qty":"2",
			"total":"6.98"
		},
		{
			"description":"Chunky Dog Treats",
			"price":"0.99",
			"qty":"5",
			"total":"4.95"
		}
	],
	"subtotal": "27.55",
	"salestax": "6.88",
	"total": "34.43",
	"est_del_date":"September 12 2019"
}
]
This example has two nested arrays:
  • delivery_address is an example of a string array. Communicate imports the key of the array as an array. This means that in the metadata generated for the JSON data, the key of that array is marked as array.
  • order_items is an example of an object array. Communicate imports the key of the array as an array, For all the fields for the JSON objects inside the array, the keys are marked as either string or array, as appropriate.

For information on how to include array data into repeating table rows, see Repeating data in tables.

XLSX files

The first row must contain the field names.

If the imported XLSX file has blanks in the first row, Communicate will automatically close up the gaps. For example, the following XLSX file has blank field names for columns C and E:

This XLSX file would be tidied up on input to look like this:

Unique Identifier

When a data source is uploaded for the first time, the entire dataset is scanned for unique fields/columns. The first unique field/column is selected as the unique identifier and stored against the data source.

When more data sources are uploaded, the upload process confirms that the unique identifier field/column is present and also contains unique data.

Spectrum

To retrieve up-to-date information from third-party sources for your communications, use EngageOne Communicate’s connection with Spectrum OnDemand. See Spectrum OnDemand for more information.

Note: Using Spectrum OnDemand requires work from Professional Services (contact Support for further information).

Enter your Spectrum OnDemand credentials and then click Connect to view the datasets.

Compose

The EngageOne Compose datasource functionality allows you to use the same data with a Compose and Communicate template.

Before you can use Compose templates in Communicate:

  1. Your Communicate team must be integrated with Compose. This can be enabled by contacting Support. If your team is not integrated, Compose related functions will not be displayed.
  2. You must have published some Compose assets to Communicate. For details of how to do this, see EngageOne Compose/Communicate: Integrated Features User's Guide.

To specify a Compose template as a data source, click Compose, select the appropriate template from the dropdown list and then click Add Data Source.

Using Data Sources for Video communications

For video communications, only Data Sources with a flat data structure (such as CSV or XLS files) are supported. Other types such as JSON, XML or data from EngageOne Compose are supported only if the structure is flat (a single level with no nested elements).

For example, a structure such as this is supported:

{
 var1: 1,
 var2: 2
}

However, the following nested structure is not supported: 

{
 node1: {
 var1: 1,
 var2: 2
 }
}

In addition, you cannot use data from Spectrum OnDemand for video communications.

Downloading Data Sources

There may be occasions when you need to download a data source from Communicate to a file. To do this, on the Data Sources screen, click on the download button () on the right of the screen against the appropriate data source.

Communicate will create and download a file of a type that depends on the original data source (the type of file that was imported). The rules are as follows:

Original file type Downloaded file type
XLS CSV
XLSX CSV
CSV CSV
JSON JSON
XML JSON