Data Sources

Explains how to manage data sources for communications.

Using data in your communications adds a level of personalization that helps your customers feel connected and catered for, increasing the likelihood of them taking positive action on your message. EngageOne™ Communicate supports multiple data import methods and data file types.

The most common method of data import is by simple file upload. For file uploads, support formats are CSV, JSON, XLS, XLSX or XML.

Note: Data source files must be UTF-8 encoded; if the file is encoded using another encoding method, it will not be successfully parsed and an error message will be displayed (Error adding dataSourceName. File contains incorrect encoding, please edit file and try uploading again).

For a series of short video clips on Data Sources, see Help videos: Data.

Select Data. The following screen is displayed, which shows any existing data sources:

Data Sources screen

Note: Use the up and down arrows to sort the Data Sources alphanumerically by that column (ascending or descending). The default sort is by Last Modified (ascending). You can also filter the results by entering text in the box Type to filter the list; this means that only Data Sources containing the characters entered will be shown in the list.

To add a new data source, click Add Data Source. The following screen is displayed:

Add Data Source screen

Select the data source type, which can be one of the following:

  • Local File - Browse for a file or drag the file across.
  • Spectrum - Enter your Spectrum OnDemand credentials and then click Connect to view the datasets.
  • Compose - Specify a template that was created in EngageOne™ Compose.

If you select Local File, then select a file, the following screen is displayed:

Add Data Source screen with Local File selected

The Data Standard dropdown allows you to select a locale. This setting determines how the data is interpreted, specifically the formatting of date/time and numeric values. For more information, see Formatting variables.

While a data source is being imported, EngageOne™ Communicate returns you to the main Data Sources screen with a progress bar showing the progress of the upload. For example:

Data Sources screen with uploading progress bar

While the upload continues, you can do other things with Communicate. For example, you could upload another data source; the two uploads will progress in parallel. Clicking the X to the right of the progress bar allows you to cancel the upload (unless it is very close to completion).

Typical upload times (based on benchmark tests) are as follows:

Note: These values are approximate and may vary.
Number of columns of data File size in MB Upload time in seconds
10 1.1 51
10 11.1 290
20 2.0 55
20 21.0 330
30 3.0 55
30 30.0 339
Note: The Delete and Download buttons only become active when the upload is complete.
Note: If multiple tabs are open looking at the same Data Sources screen, and an upload is cancelled from one tab, the other tab must be refreshed to see the cancellation.

Once the data is imported, click the data source in the list, then View Data to see the data in detail.

Data Schema

Click Data Schema to view the schema. The following screen is displayed:

Data Schema tab

In this tab, you can do the following:

  • Edit a field name.
  • Check the Hide box against any fields that you want to hide. Hidden fields do not appear under View Data and cannot be used in any communication using this data source.
  • Click the Data standard option to see a dropdown that allows you to select a locale. This setting determines how the data is interpreted, specifically the formatting of date/time and numeric values. See Formatting variables for more information.
  • Check the Include in Reporting CSV option against all fields that you want to appear in the Reporting CSV file (see Reporting). Any fields without this box checked will not be included in the Reporting CSV file. Note that you should not check this box for hidden fields. See below for a list of points to consider when including a field for reporting.
Note: The field Communicate ID is a hidden field added by EngageOne™ Communicate and it cannot be edited. The Hide checkbox and Include in Reporting CSV checkbox are both checked for this field.

When selecting a field for inclusion in the Reporting CSV file, note the following:

  1. Nested fields (fields with arrays and other fields inside them) are currently not supported. You will not be able to select such fields for reporting.
  2. Field names must be unique. Having more than one field with the same name included in reporting may cause data to overlap between fields, or not be present at all.
  3. Video reporting (see Statistics for a Video Communication) does not currently support fields with a full stop (.) inside them (such as first.example). Including such fields may cause the fields values to be empty.
  4. Field names are case-insensitive; the field with the name "ID" is treated the same way as the field with name "id".

Local File

The following types of file are supported: CSV, JSON, XLS, XLSX or XML.

CSV files

If you import data from a CSV file:

  • The first row must be the field names, separated by commas. All spaces are replaced by underscores.
  • Each record must be on a separate line within the file.

If the imported CSV file has blank columns, Communicate will automatically remove them and close up the gaps. For example, the following CSV file has blank columns (denoted by adjacent commas in the first row):

name,surname,,,country,,,testcolumn,,,
Joe,Bloggs,test1,test2,test3,test4
Jane,Smith,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19

This CSV file would be tidied up on input to look like this:

Id,name,surname,country,testcolumn
0d789cef-99xy,Jane,Smith,3,6
0c5678ab-88ab,Joe,Bloggs,test3,none

JSON files

You can import more complex data from JSON (JavaScript Object Notation) files. The file should contain an array of objects (an array should be the top element of the structure).
Consider this example:
[
{
	"firstname": "John",
	"lastname": "Smith",
	"email": "john.smith@mycompany.com",
	"order_number": "5555",
	"delivery_address": [
			"Appartment 234",
			"High Road North",
			"Some Town"
	],	
	"order_items": [
		{
			"description":"Complete Puppy Food",
			"price":"3.49",
			"qty":"2",
			"total":"6.98"
		},
		{
			"description":"Chunky Dog Treats",
			"price":"0.99",
			"qty":"5",
			"total":"4.95"
		}
	],
	"subtotal": "27.55",
	"salestax": "6.88",
	"total": "34.43",
	"est_del_date":"September 12 2019"
}
]
This example has two nested arrays:
  • delivery_address is an example of a string array. EngageOne™ Communicate imports the key of the array as an array. This means that in the metadata generated for the JSON data, the key of that array is marked as array.
  • order_items is an example of an object array. EngageOne™ Communicate imports the key of the array as an array, For all the fields for the JSON objects inside the array, the keys are marked as either string or array, as appropriate.

For information about how to include array data into repeating table rows, see Repeating data in tables.

XLSX files

The first row must contain the field names.

If the imported XLSX file has blanks in the first row, EngageOne™ Communicate will automatically close up the gaps. For example, the following XLSX file has blank field names for columns C and E:

Example XLSX file

This XLSX file would be tidied up on input to look like this:

XLSX file tidied up

Unique Identifier

When a data source is uploaded for the first time, the entire dataset is scanned for unique fields or columns. The first unique field or column is selected as the unique identifier and stored against the data source.

When more data sources are uploaded, the upload process confirms that the unique identifier field or column is present and also contains unique data.

Spectrum

To retrieve up-to-date information from third-party sources for your communications, use EngageOne™ Communicate's connection with Spectrum OnDemand. For more information, see Spectrum OnDemand.

Note: Using Spectrum OnDemand requires work from Professional Services (contact Support for further information).

Enter your Spectrum OnDemand credentials, then click Connect to view the datasets.

Compose

The EngageOne™ Compose datasource functionality allows you to use the same data with a Compose and Communicate template.

Before you can use Compose templates in Communicate:

  1. Your Communicate team must be integrated with Compose. This can be enabled by contacting Support. If your team is not integrated, Compose related functions will not be displayed.
  2. You must have published some Compose assets to Communicate. For details of how to do this, see EngageOne™ Compose/Communicate: Integrated Features User's Guide.

To specify a Compose template as a data source, click Compose, select the appropriate template from the dropdown list, then click Add Data Source.

Using Data Sources for Video communications

For video communications, Data Sources can have a flat data structure or a nested data structure.

This example shows a flat data structure:

{
 var1: 1,
 var2: 2
}

This example shows a nested data structure:

{
 node1: {
 var1: 1,
 var2: 2
 }
}
Note: You cannot use data from Spectrum OnDemand for video communications.

Downloading Data Sources

There may be occasions when you need to download a data source from EngageOne™Communicate to a file. To do this, on the Data Sources screen, click the download button (Download button) on the right of the screen against the appropriate data source.

EngageOne™Communicate will create and download a file of a type that depends on the original data source (the type of file that was imported). The rules are as follows:

Original file type Downloaded file type
XLS CSV
XLSX CSV
CSV CSV
JSON JSON
XML JSON